Nabble - Netty User Group

Announcing LittleProxy 0.2 based on Netty

2009-11-07T15:22:39Z

I just put the finishing touches on LittleProxy 0.2, now available at:

http://svn.littleshoot.org/svn/littleproxy/tags/littleproxy-0.2

This version fixes the major issues in 0.2, most notably properly
switching HTTP 1.0 responses to use HTTP 1.1 in cases where Netty adds
Transfer-Encoding: chunked to avoid holding large responses in memory.
This release also properly handling Transfer-Encoding: chunked in
requests, improvement LittleProxy's memory use, and more. You can see
the full list or release notes/issues fixes here:

http://bit.ly/2QjVcM

This release does not add the ability to decompress responses and to
modify them. That's pushed to 0.3 in anticipation of Trustin's latest
decompression changes, I believe planned for Netty 3.2.0.

Thanks again to Stephen Haberman for figuring out the fix for the HTTP
1.0/1.1 Transfer-Encoding issue.

Let us know if you have anything you'd like to see added to
LittleProxy, and we'll do our best to add it.

-Adam

--
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com |
http://twitter.com/adamfisk
_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Section 1.9 shutting down your application

2009-11-06T18:28:20Z

Hi Adam,

Actually, they are same. I added Bootstrap.releaseExternalResources()
for convenience. :)

HTH

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 9:42 AM, Adam Fisk <[hidden email]> wrote:

> In the docs, it looks like it should be the bootstraps calling
> releaseExternalResources(), not the factories.
> -Adam
>
> --
> Adam Fisk
> http://www.littleshoot.org | http://adamfisk.wordpress.com |
> http://twitter.com/adamfisk
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Section 1.9 shutting down your application

2009-11-06T16:42:53Z

In the docs, it looks like it should be the bootstraps calling releaseExternalResources(), not the factories.

-Adam

--
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com | http://twitter.com/adamfisk

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Text Protocol -- Variable length, variable number of lines

2009-11-07T14:54:29Z

Hi,

In the telnet/server case each newline would trigger a MessageEvent. In my case it would be inefficient, I intend to handle MessageEvent after an X number (which is variable) of lines have been available for upstream processing to the StreamDecoder. If I simple call --

public void messageReceived(
ChannelHandlerContext ctx, MessageEvent e) {
// Cast to a String first.
// We know it is a String because we put some codec in TelnetPipelineFactory.
String several_lines_protocol_msg= (String) e.getMessage();
:::::
}

there is no guarantee that all the lines of the protocol packet have been captured in
"several_lines_protocol_msg". Or is it the case that if FrameDecoder is not supplied the message received will be all the bytes available in the stream ?

Thanks a lot.
Venkay

Trustin Lee wrote:

Hi Venky,

DelimiterBasedFrameDecoder is perhaps what you are looking for. Also,
please refer to the telnet client/server example.

HTH

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 7:11 AM, venky10 <vsayyagari@gmail.com> wrote:
>
> Hi,
>
> I am a newbie to Netty.
> I have text protocol to decode. It has variable number of lines, all
> delimited by 'newline'.
> There is no other end-of-frame delimiter. I am assuming it would be most
> efficient to read the entire frame till there are no more lines to be read
> on single MessageEvent.
>
> What is the best approach ?
>
> Thanks,
> Venky
> --
> View this message in context: http://n2.nabble.com/Text-Protocol-Variable-length-variable-number-of-lines-tp3961650p3961650.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> netty-users@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
netty-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Text Protocol -- Variable length, variable number of lines

2009-11-07T14:37:32Z

Trustin Lee wrote:

Hi Venky,

DelimiterBasedFrameDecoder is perhaps what you are looking for. Also,
please refer to the telnet client/server example.

HTH

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 7:11 AM, venky10 <vsayyagari@gmail.com> wrote:
>
> Hi,
>
> I am a newbie to Netty.
> I have text protocol to decode. It has variable number of lines, all
> delimited by 'newline'.
> There is no other end-of-frame delimiter. I am assuming it would be most
> efficient to read the entire frame till there are no more lines to be read
> on single MessageEvent.
>
> What is the best approach ?
>
> Thanks,
> Venky
> --
> View this message in context: http://n2.nabble.com/Text-Protocol-Variable-length-variable-number-of-lines-tp3961650p3961650.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> netty-users@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
netty-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Text Protocol -- Variable length, variable number of lines

2009-11-06T23:36:38Z

Hi Venky,

DelimiterBasedFrameDecoder is perhaps what you are looking for. Also,
please refer to the telnet client/server example.

HTH

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 7:11 AM, venky10 <[hidden email]> wrote:

>
> Hi,
>
> I am a newbie to Netty.
> I have text protocol to decode. It has variable number of lines, all
> delimited by 'newline'.
> There is no other end-of-frame delimiter. I am assuming it would be most
> efficient to read the entire frame till there are no more lines to be read
> on single MessageEvent.
>
> What is the best approach ?
>
> Thanks,
> Venky
> --
> View this message in context: http://n2.nabble.com/Text-Protocol-Variable-length-variable-number-of-lines-tp3961650p3961650.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Text Protocol -- Variable length, variable number of lines

2009-11-06T14:11:21Z

Hi,

I am a newbie to Netty.
I have text protocol to decode. It has variable number of lines, all delimited by 'newline'.
There is no other end-of-frame delimiter. I am assuming it would be most efficient to read the entire frame till there are no more lines to be read on single MessageEvent.

What is the best approach ?

Thanks,
Venky

RE: Streaming to file

2009-11-07T02:07:02Z

Thanks Trustin,

That improved the performance a bit. (c: If you get the chance, please take a look at my wget vs. netty message. I could really use some help...

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Trustin Lee (???)
Sent: Friday, November 06, 2009 11:36 PM
To: Netty Users
Subject: Re: Streaming to file

Hi David,

ChannelBuffer.getBytes(int index, GatheringByteChannel out, int length)
ChannelBuffer.getBytes(int index, OutputStream out, int length)
ChannelBuffer.readBytes(GatheringByteChannel out, int length)
ChannelBuffer.readBytes(OutputStream out, int length)

are what you are looking for. :)

— Trustin Lee, http://*gleamynode.net/

On Sat, Nov 7, 2009 at 1:57 AM, Hoyt, David <[hidden email]> wrote:

> What’s the best way to get a ChannelBuffer into a file? I just want to dump
> the incoming bytes into a file.
>
> I’m using NIO and I have a FileChannel and I’m just calling
> "myFileChannel.write(myChannelBuffer.toByteBuffer());
> myFileChannel.flush();" – is that right? Is there a better way?
>
> Is there a way to have my incoming ChannelBuffers automagically be a
> ByteBuffer without having additional overhead of calling .toByteBuffer() and
> risking byte copying?
>
> Thanks,
> - David Hoyt
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://*lists.jboss.org/mailman/listinfo/netty-users
>
>

_______________________________________________
netty-users mailing list
[hidden email]
https://*lists.jboss.org/mailman/listinfo/netty-users

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: wget vs netty

2009-11-07T13:01:57Z

Hi David,

I've got a similar problem using write to file from Java in my project (FTP using Netty).
In fact, it turns out that the problem has nothing related to Netty but of course to JVM way to write files.

To be sure, I've done a little benchmark, trying several ways to read or write files in Java without any network access.

Here are some results, tested on 2.6.5-7.244-smp 2 cpu 2-core 2.8Ghz Xeon using two JVM: Sun JDK 1.6.0.13 and IBM 1.6 SR5 (5/2009).
Take them with caution since they must be used for what they are: testing purpose for my scenario
Each wile is written, flushed (by the way available) then delete and rewrited from scratch.

Write file:
- arbitrary position when write is starting (or restarting)
Note tha FileOutputStream is not allowed since in my scenario we can restart a write from a previous position and FileOutputStream does not allowed to change the position (except 0 or at the end at construction), except if we used the FileChannel with the append mode and the position method of the FileChannel.
- RandomAccessFile : 10 MB files, 10 writes for each thread:
1 thread = around 40 MByte/s ; 50 threads = 3,32 MByte/s
same but 1 MB files 10 times with 500 threads = 2,96 MByte/s (IBM JDK) 2.89 MByte/s (SUN JDK)

- FileChannel over a RandomAccessFile : 10 MB files, 10 writes for each thread:
1 thread = around 40 MByte/s; 50 threads = 3,33 MByte/s
same but 1 MB files 10 times with 500 threads = 2,84 MByte/s (IBM JDK) 2,85 MByte/s (SUN JDK)

Note that most of the time, direct RandomAccessFile is faster than FileChannel for me when more than 100 threads are concurrently writing files.

- start from 0 or from end (append) (FileOutputStream is therefore allowed)
- FileOutputStream : 10 MB files, 10 writes for each thread:
1 thread = around 250 MByte/s; 50 threads = 154,5 MByte/s
same but 1 MB files 10 times with 500 threads = 25,94 MByte/s (IBM JDK) 27,42 MByte/s (SUN JDK)

- FileChannel over FileOutputStream : 10 MB files, 10 writes for each thread:
1 thread = around 40 MByte/s; 50 threads = 3,15 MByte/s
same but 1 MB files 10 times with 500 threads = 2,82 MByte/s (IBM JDK) 2,83 MByte/s (SUN JDK)

- BufferedOutputStream over FileOutputStream : 10 MB files, 10 writes for each thread:
1 thread = 250 MByte/s; 50 threads = 183,3 MByte/s
same but 1 MB files 10 times with 500 threads = 21,02 MByte/s (IBM JDK) 20,67 MByte/s (SUN JDK)

So FileOutputStream is the faster, Buffered one just under but not FileChannel (same level and even a little bit worst than FileChannel based on RandomAccessFile).
Note that Systems info shows that write speed on disk from system point of view says it was about 3 to 4 MBytes/s whatever the test.

For my project, I need to be sure that I can move from one point to another, so I need RandomAccess capability, so no chance for me to use FileOutputStream directly.

Read File: Please note that as each thread access its own file but several times, the system cache is used so the reason of huge performance in reading.
- best is always : FileChannel over a FileInputStream :10 MB files, 10 reads for each thread:
1 thread = 2 210 MByte/s; 50 threads = 3 329 MByte/s
same but 1 MB files with 500 threads = 2 801 MByte/s (IBM JDK) 3 106 MByte/s (SUN JDK)

I do not show the others, but with many threads, FileChannel over FileInputStream tends to be the best.

Hope this helps others too...

Cheers,
Frederic

Trustin Lee wrote:

Hi David,

Just to make sure disk I/O is not a bottleneck, could you try to
configure wget and your Netty application to discard the received data
to /dev/null?

I also wonder if you are using Netty's built-in HTTP codec or your own one.

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 7:02 PM, Hoyt, David <hoyt6@llnl.gov> wrote:
> Our application does a lot of downloading -- I notice that when I test my netty app against wget for some basic non-encrypted http streams, wget outperforms my netty app. My basic test is I start 2 wget processes and have them download some content from our server. I then have netty download the same content. I give each 20 seconds to download as much as they can and then I cancel it (the file contents are produced on-the-fly at a constant rate of ~500 KB/s). wget consistently does better than my java app (by ~3MB per file with 2 concurrent downloads). It's a bit confusing, b/c my CPU is nowhere near maxed out using either wget or netty (java). The discrepancy gets much higher when I start doing 10+ concurrent downloads. wget will download an 11 MB file while my java app gets ~700-800 KB. Every message (http chunk) I get from netty is immediately flushed to disk and I'm using a FileChannel for what I figured was optimum performance. It should be able to do the full 11 MB eve!
> n if it takes a little while for it to come in. 2 streams of 500 KB/s (so ~ 1 MB/s) should be easily handled by java/netty I would think.
>
> I don't think I'm IO-bound - the disk can easily write ~22 MB (the file is ~11 MB) and wget doesn't seem to have a problem writing to disk. I'm on a gigabit line and 22 MB over 20 seconds shouldn't be saturating the connection.
>
> I'm wondering if there's something I'm missing here. I'm following the http examples pretty closely. I use keepAlive, reuseAddress, and tcpNoDelay for the client bootstrap. I made the threads in my thread pool have MAX_PRIORITY to help rule out scheduler issues. Everything else is using the defaults.
>
> I did a quick old IO (OIO) implementation and it seemed to suffer more than netty. I thought perhaps that something I was doing may have cut off the content before it could be flushed to the disk, but using OIO and input streams should have gotten the full content eventually.
>
> I could really use some help or some pointers on what the community does for netty and/or network optimizations.
>
> I'm doing my testing on a Windows box but we'll be deploying for linux. The behavior appears on both platforms (Windows XP and CentOS).
>
> Thanks,
> - David Hoyt
>
> _______________________________________________
> netty-users mailing list
> netty-users@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
netty-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/netty-users

Hardware/Software Architect

Re: wget vs netty

2009-11-07T06:58:33Z

Hi David,

Just to make sure disk I/O is not a bottleneck, could you try to
configure wget and your Netty application to discard the received data
to /dev/null?

I also wonder if you are using Netty's built-in HTTP codec or your own one.

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 7:02 PM, Hoyt, David <[hidden email]> wrote:

> Our application does a lot of downloading -- I notice that when I test my netty app against wget for some basic non-encrypted http streams, wget outperforms my netty app. My basic test is I start 2 wget processes and have them download some content from our server. I then have netty download the same content. I give each 20 seconds to download as much as they can and then I cancel it (the file contents are produced on-the-fly at a constant rate of ~500 KB/s). wget consistently does better than my java app (by ~3MB per file with 2 concurrent downloads). It's a bit confusing, b/c my CPU is nowhere near maxed out using either wget or netty (java). The discrepancy gets much higher when I start doing 10+ concurrent downloads. wget will download an 11 MB file while my java app gets ~700-800 KB. Every message (http chunk) I get from netty is immediately flushed to disk and I'm using a FileChannel for what I figured was optimum performance. It should be able to do the full 11 MB eve!
> n if it takes a little while for it to come in. 2 streams of 500 KB/s (so ~ 1 MB/s) should be easily handled by java/netty I would think.
>
> I don't think I'm IO-bound - the disk can easily write ~22 MB (the file is ~11 MB) and wget doesn't seem to have a problem writing to disk. I'm on a gigabit line and 22 MB over 20 seconds shouldn't be saturating the connection.
>
> I'm wondering if there's something I'm missing here. I'm following the http examples pretty closely. I use keepAlive, reuseAddress, and tcpNoDelay for the client bootstrap. I made the threads in my thread pool have MAX_PRIORITY to help rule out scheduler issues. Everything else is using the defaults.
>
> I did a quick old IO (OIO) implementation and it seemed to suffer more than netty. I thought perhaps that something I was doing may have cut off the content before it could be flushed to the disk, but using OIO and input streams should have gotten the full content eventually.
>
> I could really use some help or some pointers on what the community does for netty and/or network optimizations.
>
> I'm doing my testing on a Windows box but we'll be deploying for linux. The behavior appears on both platforms (Windows XP and CentOS).
>
> Thanks,
> - David Hoyt
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

wget vs netty

2009-11-07T02:02:28Z

Our application does a lot of downloading -- I notice that when I test my netty app against wget for some basic non-encrypted http streams, wget outperforms my netty app. My basic test is I start 2 wget processes and have them download some content from our server. I then have netty download the same content. I give each 20 seconds to download as much as they can and then I cancel it (the file contents are produced on-the-fly at a constant rate of ~500 KB/s). wget consistently does better than my java app (by ~3MB per file with 2 concurrent downloads). It's a bit confusing, b/c my CPU is nowhere near maxed out using either wget or netty (java). The discrepancy gets much higher when I start doing 10+ concurrent downloads. wget will download an 11 MB file while my java app gets ~700-800 KB. Every message (http chunk) I get from netty is immediately flushed to disk and I'm using a FileChannel for what I figured was optimum performance. It should be able to do the full 11 MB eve!
n if it takes a little while for it to come in. 2 streams of 500 KB/s (so ~ 1 MB/s) should be easily handled by java/netty I would think.

I don't think I'm IO-bound - the disk can easily write ~22 MB (the file is ~11 MB) and wget doesn't seem to have a problem writing to disk. I'm on a gigabit line and 22 MB over 20 seconds shouldn't be saturating the connection.

I'm wondering if there's something I'm missing here. I'm following the http examples pretty closely. I use keepAlive, reuseAddress, and tcpNoDelay for the client bootstrap. I made the threads in my thread pool have MAX_PRIORITY to help rule out scheduler issues. Everything else is using the defaults.

I did a quick old IO (OIO) implementation and it seemed to suffer more than netty. I thought perhaps that something I was doing may have cut off the content before it could be flushed to the disk, but using OIO and input streams should have gotten the full content eventually.

I could really use some help or some pointers on what the community does for netty and/or network optimizations.

I'm doing my testing on a Windows box but we'll be deploying for linux. The behavior appears on both platforms (Windows XP and CentOS).

Thanks,
- David Hoyt

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Streaming to file

2009-11-06T23:35:35Z

Hi David,

ChannelBuffer.getBytes(int index, GatheringByteChannel out, int length)
ChannelBuffer.getBytes(int index, OutputStream out, int length)
ChannelBuffer.readBytes(GatheringByteChannel out, int length)
ChannelBuffer.readBytes(OutputStream out, int length)

are what you are looking for. :)

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 1:57 AM, Hoyt, David <[hidden email]> wrote:

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Re: Streaming to file

2009-11-06T10:10:42Z

This is more than you need but it should help: see at the end of this email. This particular implementation is for decoding a set of files and a management object where each file is sent in chunks. FileChannel is obtained from a FileOutputStream to which the bytes are written, cf. processKarmaFileChunk(...). The file channel is saved until all chunks for a particular file are written. The management object that also was written that has information about where the passed files are is itself passed to an application.

AS you can see the ChannelBuffer contains only a byte array that is processed in convertChannelBuffer. Hope this helps.

Mike

P.S. The Karma application is a mobile object as contrasted with a mobile agent application with interfaces that we hope will assist in providing a new IEEE standard that will complement FIPA. We hope to make it public soon and then to begin the IEEE process. Netty is the default transport.

On Nov 6, 2009, at 8:57 AM, Hoyt, David wrote:

What’s the best way to get a ChannelBuffer into a file? I just want to dump the incoming bytes into a file.

I’m using NIO and I have a FileChannel and I’m just calling "myFileChannel.write(myChannelBuffer.toByteBuffer()); myFileChannel.flush();" – is that right? Is there a better way?

Is there a way to have my incoming ChannelBuffers automagically be a ByteBuffer without having additional overhead of calling .toByteBuffer() and risking byte copying?

Thanks,
- David Hoyt
_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Mike McGrady

Principal Investigator AF081-028 AFRL SBIR

Senior Engineer

Topia Technology, Inc.

1.253.720.3365

[hidden email]

* KARMA is free software: you can redistribute it and/or modify it under the

* terms of the GNU LESSER GENERAL PUBLIC LICENSE as published by the Free

* Software Foundation, either version 3 of the License, or (at your option) any

* later version. KARMA is distributed in the hope that it will be useful, but

* WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or

* FITNESS FOR A PARTICULAR PURPOSE. See the GNU LESSER GENERAL PUBLIC LICENSE

* for more details. You should have received a copy of the GNU LESSER GENERAL

* PUBLIC LICENSE along with KARMA. If not, see <http://www.gnu.org/licenses/>.

package net.sourceforge.karma.spi.transport.netty;

import java.io.File;

import java.io.FileNotFoundException;

import java.io.FileOutputStream;

import java.io.IOException;

import java.nio.ByteBuffer;

import java.nio.channels.FileChannel;

import java.security.cert.Certificate;

import java.util.HashMap;

import java.util.Map;

import java.util.UUID;

import java.util.concurrent.atomic.AtomicInteger;

import javax.net.ssl.SSLException;

import javax.net.ssl.SSLPeerUnverifiedException;

import javax.net.ssl.SSLSession;

import javax.net.ssl.SSLEngineResult.HandshakeStatus;

import net.sourceforge.karma.KarmaConstant;

import net.sourceforge.karma.KarmaManager;

import net.sourceforge.karma.spi.KarmaReceiver;

import net.sourceforge.karma.spi.util.KarmaUtility;

import net.sourceforge.karma.spi.util.NettyKarmaSslHandshake;

import net.sourceforge.karma.spi.util.StackTrace;

import org.jboss.netty.buffer.ChannelBuffer;

import org.jboss.netty.buffer.ChannelBuffers;

import org.jboss.netty.channel.Channel;

import org.jboss.netty.channel.ChannelDownstreamHandler;

import org.jboss.netty.channel.ChannelEvent;

import org.jboss.netty.channel.ChannelFuture;

import org.jboss.netty.channel.ChannelFutureListener;

import org.jboss.netty.channel.ChannelHandlerContext;

import org.jboss.netty.channel.ChannelPipelineCoverage;

import org.jboss.netty.channel.ChannelStateEvent;

import org.jboss.netty.channel.ChannelUpstreamHandler;

import org.jboss.netty.channel.ChildChannelStateEvent;

import org.jboss.netty.channel.ExceptionEvent;

import org.jboss.netty.channel.MessageEvent;

import org.jboss.netty.channel.SimpleChannelDownstreamHandler;

import org.jboss.netty.channel.SimpleChannelUpstreamHandler;

import org.jboss.netty.channel.WriteCompletionEvent;

import org.jboss.netty.channel.group.ChannelGroup;

import org.jboss.netty.logging.InternalLogLevel;

import org.jboss.netty.logging.InternalLogger;

import org.jboss.netty.logging.InternalLoggerFactory;

/**

*

* A composition (rather than inheritance) a la Joshua Bloch design for the use

* of JBoss Netty simple channel upstream and downstream handlers with a KARMA

* decoder.

*

*

* The principal method is <code>messageReceivedUpstream</code>.

*

*

* The <code>KarmaReceiverPipelineFactory</code> utilizes this class along with

* the standard JBoss Netty <code>SslHandler</code> and

* <code>SslBufferPool</code>. Where the event is a

* <code>ChannelStateEvent</code> and the channel state is

* <code>CONNECTED</code>, the <code>handleUpstream</code> method initiates an

* handshake with the client on the other end of the channel. The

* <code>handshake</code> method adds a listener (<code>ConnectionNotice</code>

* which implements <code>ChannelFutureListener</code>) to notify the client

* when the handshake is successful.

*

*

* All <code>OPEN</code> channels are added to the field

* <code>allChannels</code> to make sure that shutdown is complete and

* efficient.

*

* @author Mike McGrady

* @version Karma Alpha v001

* @since August 2009.

@ChannelPipelineCoverage("all")

public final class KarmaDecoder implements ChannelUpstreamHandler,

ChannelDownstreamHandler, Comparable<KarmaDecoder> {

private static int managersCount;

private static int channelCount;

String errors;

private static final AtomicInteger chunkCounter = new AtomicInteger(0);

private static final InternalLogger logger = InternalLoggerFactory

.getInstance(KarmaDecoder.class.getName());

private static final AtomicInteger managerCounter = new AtomicInteger(0);

private static final AtomicInteger readCounter = new AtomicInteger(0);

private final ChannelGroup allChannels;

private final Map<UUID, Integer> fileCount = new HashMap<UUID, Integer>();

private final Map<UUID, FileChannel> fileWriters = new HashMap<UUID, FileChannel>();

private final Map<UUID, KarmaManager> managers = new HashMap<UUID, KarmaManager>();

private final KarmaReceiver receiver;

private final SimpleChannelDownstreamHandler scdh = new SimpleChannelDownstreamHandler();

private final SimpleChannelUpstreamHandler scuh = new SimpleChannelUpstreamHandler();

/**

* This listener follows the functionality for the JBoss Netty chat example

* for the most part. This is just a standard use of the

* <code>ChannelFutureListener</code>.

* @author Mike McGrady

private final class ConnectionNotice implements ChannelFutureListener {

private final KarmaSslHandler sslHandler;

ConnectionNotice(final KarmaSslHandler sslHandler) {

this.sslHandler = sslHandler;

}

public void operationComplete(final ChannelFuture future) {

KarmaDecoder.log(InternalLogLevel.INFO, "SslHandler: " //$NON-NLS-1$

+ this.sslHandler);

final HandshakeStatus hs = this.sslHandler.getEngine()

.getHandshakeStatus();

final SSLSession session = this.sslHandler.getEngine().getSession();

KarmaDecoder.log(InternalLogLevel.INFO, "HandshakeStatus: " + hs); //$NON-NLS-1$

KarmaDecoder.log(InternalLogLevel.INFO, "SSLSession: " + session); //$NON-NLS-1$

final Certificate[] localCerts = session.getLocalCertificates();

Certificate[] peerCerts = null;

try {

peerCerts = session.getPeerCertificates();

} catch (final SSLPeerUnverifiedException e) {

log(

InternalLogLevel.INFO,

(KarmaDecoder.this.errors = ("\n\t\tERROR MESSAGE: " //$NON-NLS-1$

+ e.getMessage() + "\n\t\tERROR STACK TRACE: " + StackTrace //$NON-NLS-1$

.trace(e))));

}

for (final Certificate localCert : localCerts) {

log(InternalLogLevel.INFO, "" + localCert); //$NON-NLS-1$

}

if (peerCerts != null) {

for (int i = 0; i < peerCerts.length; i++) {

log(InternalLogLevel.INFO, "" + localCerts[i]); //$NON-NLS-1$

}

if (future.isSuccess()) {

final byte[] protectedSession = KarmaUtility

.toBytes(new NettyKarmaSslHandshake(true,

KarmaDecoder.this.errors));

final ChannelBuffer protectedSessionBuf = ChannelBuffers

.wrappedBuffer(protectedSession);

future.getChannel().write(protectedSessionBuf);

} else {

final byte[] protectedSession = KarmaUtility

.toBytes(new NettyKarmaSslHandshake(false,

KarmaDecoder.this.errors));

final ChannelBuffer protectedSessionBuf = ChannelBuffers

.wrappedBuffer(protectedSession);

future.getChannel().write(protectedSessionBuf);

}

/**

* Creates a new instance with the current system character set.

public KarmaDecoder(final KarmaReceiver receiver,

final ChannelGroup allChannels) {

this.receiver = receiver;

this.allChannels = allChannels;

}

public void channelOpenUpstream(final ChannelHandlerContext ctx,

final ChannelStateEvent event) {

KarmaDecoder.channelCount++;

// Add all open channels to the global group so that they are able to be

// closed on shutdown.

final Channel channel = event.getChannel();

this.allChannels.add(channel);

ctx.sendUpstream(event);

}

public int compareTo(final KarmaDecoder decoder) {

int result = -1;

if (this == decoder) {

result = -1;

}

if (decoder == null) {

result = -1;

}

if (decoder.hashCode() > this.hashCode()) {

result = +1;

} else if (decoder.hashCode() < this.hashCode()) {

result = -1;

} else if (decoder.equals(this)) {

result = 0;

} else {

// Probably impossible.

result = -1;

}

return result;

}

@Override

public boolean equals(final Object obj) {

if (this == obj) { return true; }

if (obj == null) { return false; }

if (!(obj instanceof KarmaDecoder)) { return false; }

final KarmaDecoder other = (KarmaDecoder) obj;

if (this.allChannels == null) {

if (other.allChannels != null) { return false; }

} else if (!this.allChannels.equals(other.allChannels)) { return false; }

if (this.fileCount == null) {

if (other.fileCount != null) { return false; }

} else if (!this.fileCount.equals(other.fileCount)) { return false; }

if (this.fileWriters == null) {

if (other.fileWriters != null) { return false; }

} else if (!this.fileWriters.equals(other.fileWriters)) { return false; }

if (this.managers == null) {

if (other.managers != null) { return false; }

} else if (!this.managers.equals(other.managers)) { return false; }

if (this.receiver == null) {

if (other.receiver != null) { return false; }

} else if (!this.receiver.equals(other.receiver)) { return false; }

if (this.scdh == null) {

if (other.scdh != null) { return false; }

} else if (!this.scdh.equals(other.scdh)) { return false; }

if (this.scuh == null) {

if (other.scuh != null) { return false; }

} else if (!this.scuh.equals(other.scuh)) { return false; }

return true;

}

/**

* @return the allChannels

public ChannelGroup getAllChannels() {

return this.allChannels;

}

/**

* {@inheritDoc} Down-casts the received downstream event into more

* meaningful sub-type event and calls an appropriate handler method with

* the down-casted event.

public void handleDownstream(final ChannelHandlerContext ctx,

final ChannelEvent channelEvent) {

if (channelEvent instanceof MessageEvent) {

try {

this.scdh.writeRequested(ctx, (MessageEvent) channelEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

} else if (channelEvent instanceof ChannelStateEvent) {

final ChannelStateEvent channelStateEvent = (ChannelStateEvent) channelEvent;

switch (channelStateEvent.getState()) {

case OPEN:

if (!Boolean.TRUE.equals(channelStateEvent.getValue())) {

try {

this.scdh.closeRequested(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

break;

case BOUND:

if (channelStateEvent.getValue() != null) {

try {

this.scdh.bindRequested(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

} else {

try {

this.scdh.unbindRequested(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

break;

case CONNECTED:

if (channelStateEvent.getValue() != null) {

try {

this.scdh.connectRequested(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

} else {

try {

this.scdh.disconnectRequested(ctx,

channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

break;

case INTEREST_OPS:

try {

this.scdh.setInterestOpsRequested(ctx,

channelStateEvent);

} catch (final Exception e) {

KarmaDecoder

.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

break;

default:

ctx.sendDownstream(channelEvent);

}

} else {

ctx.sendDownstream(channelEvent);

}

/**

* Down-casts the received upstream event into more meaningful sub-type

* event and calls an appropriate handler method with the down-casted event.

public void handleUpstream(final ChannelHandlerContext ctx,

final ChannelEvent channelEvent) {

if (channelEvent instanceof MessageEvent) {

this.messageReceivedUpstream((MessageEvent) channelEvent);

} else if (channelEvent instanceof WriteCompletionEvent) {

final WriteCompletionEvent writeCompletionEvent = (WriteCompletionEvent) channelEvent;

try {

this.scuh.writeComplete(ctx, writeCompletionEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

} else if (channelEvent instanceof ChildChannelStateEvent) {

final ChildChannelStateEvent childChannelStateEvent = (ChildChannelStateEvent) channelEvent;

if (childChannelStateEvent.getChildChannel().isOpen()) {

try {

this.scuh.childChannelOpen(ctx, childChannelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

} else {

try {

this.scuh.childChannelClosed(ctx, childChannelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

} else if (channelEvent instanceof ChannelStateEvent) {

final ChannelStateEvent channelStateEvent = (ChannelStateEvent) channelEvent;

switch (channelStateEvent.getState()) {

case OPEN:

if (Boolean.TRUE.equals(channelStateEvent.getValue())) {

try {

// Custom coding to make sure that all channels are

// added to the channel group.

this.channelOpenUpstream(ctx, channelStateEvent);

} catch (final Exception e1) {

KarmaDecoder.log(InternalLogLevel.ERROR, e1

.getMessage());

e1.printStackTrace();

}

} else {

try {

this.scuh.channelClosed(ctx, channelStateEvent);

} catch (final Exception e1) {

KarmaDecoder.log(InternalLogLevel.ERROR, e1

.getMessage());

e1.printStackTrace();

}

break;

case BOUND:

if (channelStateEvent.getValue() != null) {

try {

this.scuh.channelBound(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

} else {

try {

this.scuh.channelUnbound(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

break;

case CONNECTED:

if (channelStateEvent.getValue() != null) {

try {

// Custom coding to make sure that the exchanges are

// appropriate encrypted with PKI.

this.handshake(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

} else {

try {

this.scuh.channelDisconnected(ctx,

channelStateEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e

.getMessage());

e.printStackTrace();

}

break;

case INTEREST_OPS:

try {

this.scuh

.channelInterestChanged(ctx, channelStateEvent);

} catch (final Exception e) {

KarmaDecoder

.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

break;

default:

ctx.sendUpstream(channelEvent);

}

} else if (channelEvent instanceof ExceptionEvent) {

try {

this.scuh.exceptionCaught(ctx, (ExceptionEvent) channelEvent);

} catch (final Exception e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

e.printStackTrace();

}

} else {

ctx.sendUpstream(channelEvent);

}

@Override

public int hashCode() {

final int prime = 31;

int result = 1;

result = prime

* result

+ ((this.allChannels == null) ? 0 : this.allChannels.hashCode());

result = prime * result

+ ((this.fileCount == null) ? 0 : this.fileCount.hashCode());

result = prime

* result

+ ((this.fileWriters == null) ? 0 : this.fileWriters.hashCode());

result = prime * result

+ ((this.managers == null) ? 0 : this.managers.hashCode());

result = prime * result

+ ((this.receiver == null) ? 0 : this.receiver.hashCode());

result = prime * result

+ ((this.scdh == null) ? 0 : this.scdh.hashCode());

result = prime * result

+ ((this.scuh == null) ? 0 : this.scuh.hashCode());

return result;

}

/**

* @param messageBuffer

* @return

private Object convertChannelBuffer(final ChannelBuffer messageBuffer) {

final int readableBytes = messageBuffer.readableBytes();

final byte[] readBytes = new byte[readableBytes];

messageBuffer.readBytes(readBytes);

final Object object = KarmaUtility.toObject(readBytes);

return object;

}

private void decode(final Channel channel, final Object msg) {

final ChannelBuffer msgBuf = (ChannelBuffer) msg;

* Converts the channel buffer bytes to an object.

final Object object = this.convertChannelBuffer(msgBuf);

* Keeps track of how many managers and how many file stream chunks have

* been processed. This envisions future additions to functionality.

if (object instanceof KarmaManager) {

KarmaDecoder.managerCounter.incrementAndGet();

} else if (object instanceof KarmaChunk) {

KarmaDecoder.chunkCounter.incrementAndGet();

}

* Uses different methods to process KarmaChunks versus KarmaManagers.

if (object instanceof KarmaChunk) {

* Chunk processing.

final KarmaChunk chunk = (KarmaChunk) object;

this.processKarmaFileChunk(channel, chunk);

} else if (object instanceof KarmaManager) {

* Manager processing.

final KarmaManager manager = (KarmaManager) object;

this.processKarmaManager(manager);

} else {

* This should never happen. TODO - get a more sophisticated

* response to this unwanted alternative.

KarmaDecoder.log(InternalLogLevel.ERROR,

"Not a legimate object type."); //$NON-NLS-1$

}

/**

* Invoked when a {@link Channel} is open, bound to a local address, and

* connected to a remote address.

private void handshake(final ChannelHandlerContext ctx,

final ChannelStateEvent evnet) {

// Get the SslHandler in the current pipeline.

// We added it in SecureChatPipelineFactory.

final KarmaSslHandler sslHandler = ctx.getPipeline().get(

KarmaSslHandler.class);

// Get notified when SSL handshake is done.

ChannelFuture handshakeFuture = null;

try {

handshakeFuture = sslHandler.handshake(evnet.getChannel());

} catch (final SSLException e) {

KarmaDecoder.log(InternalLogLevel.ERROR, (this.errors = e

.getMessage()));

}

handshakeFuture.addListener(new ConnectionNotice(sslHandler));

}

/**

* Invoked when a message object (e.g: {@link ChannelBuffer}) was received

* from a remote peer.

private void messageReceivedUpstream(final MessageEvent event) {

KarmaDecoder.readCounter.incrementAndGet();

final Object originalMessage = event.getMessage();

this.decode(event.getChannel(), originalMessage);

}

/**

*

* This method processes data stream file chunks. A file output stream is

* created that has the appropriate location in ~/.KARMA/.AUM. A Java NIO

* file channel is obtained fromt he file output stream and is saved under

* the chunk's <code>UUID</code>, which is also the ID of the manager in the

* <code>fileWriters</code> <code>Map</code>. N.B.: this assumes that the

* chunks will arrive in order.

*

*

* Eventually we should not assume that the chunks arrive in order and

* should write them according to a given logic. This will allow us to take

* care of divergent delivery scenarios.

*

* @param channel

* @param chunk

private void processKarmaFileChunk(final Channel channel,

final KarmaChunk chunk) {

final byte[] chunkBytes = chunk.bytes;

* When the first file chunk arrives, create the file channel from the

* file output stream for later use.

if (chunk.id == 1) {

final String baseName = KarmaConstant.AUM + chunk.uuid;

final File chunkFile = new File(baseName);

if (!chunkFile.exists()) {

chunkFile.mkdirs();

chunkFile.isDirectory();

}

final String toFileName = baseName + File.separatorChar

+ chunk.fileName;

FileChannel fileChannel = null;

try {

fileChannel = new FileOutputStream(toFileName).getChannel();

} catch (final FileNotFoundException e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

}

this.fileWriters.put(chunk.uuid, fileChannel);

}

* For all chunks, put the bytes into a pre-allocated byte buffer, flip

* and write.

final ByteBuffer buffer = ByteBuffer.allocate(chunkBytes.length);

buffer.put(chunkBytes);

buffer.flip();

try {

this.fileWriters.get(chunk.uuid).write(buffer);

* When the chunk contains the end of the file, remove the file

* channel after writing and close it. This is sort of an

* independent check.

if (chunk.fileLength == chunk.offset) {

final FileChannel fileChannel = this.fileWriters

.remove(chunk.uuid);

fileChannel.close();

}

} catch (final IOException e) {

KarmaDecoder.log(InternalLogLevel.ERROR, e.getMessage());

}

if (chunk.isLast) {

final Integer fileNumber = this.fileCount.get(chunk.uuid);

* This value is NOT otiose. It should be kept. When the value is

* the first and last it does not get to be incremented.

if (fileNumber == null) {

this.fileCount.put(chunk.uuid, new Integer(1));

}

final Integer numberOfFiles = this.fileCount.get(chunk.uuid);

* Note that is NOT the number of CHUNKS but the number of FILES.

int numberOfFilesReceived = numberOfFiles.intValue();

final KarmaManager manager = this.managers.get(chunk.uuid);

final int numberOfManagerFileNames = manager.getResources().size();

* We have now got all CHUNKS and all FILES.

if ((numberOfManagerFileNames == numberOfFilesReceived)

&& (this.managers.get(chunk.uuid) != null)) {

* If all the chunks have arrived.

this.managers.remove(chunk.uuid);

this.fileCount.remove(chunk.uuid);

this.fileWriters.remove(chunk.uuid);

* This is the end of the road for this transmission. At this

* point all the files and the manager have been received and

* the chunks have been written to the file by the file channel.

* The manager will now be sent to the application with all the

* requisite information to get to the file resources and to do

* whatever the application likes.

this.receiver.update(manager);

} else {

* Keep track of how many FILES have arrived.

numberOfFilesReceived++;

this.fileCount.put(chunk.uuid, new Integer(

numberOfFilesReceived));

}

buffer.clear();

}

/**

*

* This method processes the manager objects.

*

* @param manager

private void processKarmaManager(final KarmaManager manager) {

managersCount++;

final UUID uuid = manager.getId();

this.managers.put(uuid, manager);

}

static void log(InternalLogLevel level, String msg) {

logger.log(level, msg);

}

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Streaming to file

2009-11-06T08:57:47Z

What’s the best way to get a ChannelBuffer into a file? I just want to dump the incoming bytes into a file.

I’m using NIO and I have a FileChannel and I’m just calling "myFileChannel.write(myChannelBuffer.toByteBuffer()); myFileChannel.flush();" – is that right? Is there a better way?

Is there a way to have my incoming ChannelBuffers automagically be a ByteBuffer without having additional overhead of calling .toByteBuffer() and risking byte copying?

Thanks,
- David Hoyt

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Http server Session/Context management when server has Netty NIO backend

2009-11-05T14:08:53Z

Hi,

Appreciate if anybody can guide me in what I what to achieve with Netty Http server.

The http server is more like a proxy but not a real proxy. When a request comes in I need to talk to at least two backend servers and I was planning to use Netty NIO client.

1. My question is how do I maintain session/context management in such a scenario and handle events correctly. The stack with the server pipeline and 2 backend client pipelines end handlers

NIO Server
Server-RequstHandler

(context object)

NIO client NIO Client
Backend1-Handler Backend2-handler

As I need to maintain states/context when doing the backend I was thinking of a single context object per request with various methods like onServerMessage recd, onBackend1MessageRecd and more like this to keep states/contexts to handle. I am also planning to use OMAthreads also in all 3 pipelines.

2. I would need to send response back to client in a async way by using the writeComplete event in the Server-RequestHandler.

How can I achieve this using the various handler models. I saw the proxy example but could not find a session/context management example.

Any help appreciated.

Thanks
Raj

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-07T10:32:09Z

When the timeout occurs, how does it run the timeouthandler? Does it
do it from a thread pool, or does it run on the hashedWheelTimer's
thread?

Another quick question. I am writing a piece of PLC Communications
software based on a request and response cycle, and I have one handler
that needs to act as a codec because it needs to maintain state
between the request and response.

My setup is like this

PLC device => (upstream starts) LengthBasedFramer => TCPFrameCODEC
(checks to see if response.transid = lastRequest.transid) = >
BusinessLogicHandler (downstream starts) => TCPFrameCodec (increment
lastRequest.transid) => PLC device

Is this the best way to do this?

Thanks

Garry Watkins

Managing Director
DynaFOCUS, LLC
1097 Rivershore Rd.
Charleston, SC 29492

mobile: 843-276-2808
[hidden email]

On Nov 7, 2009, at 10:17 AM, Trustin Lee (이희승) wrote:

> A HashedWheelTimer creates only one thread. If you create many
> instances of HashedWheelTimer, you will see many threads, and it's
> improper use. There's no thread leak if used properly.
>
> If you want to make it pull a thread from a pool, you can specify a
> custom ThreadFactory in the constructor. However, I don't think this
> is related with thread leak.
>
> — Trustin Lee, http://gleamynode.net/
>
> On Wed, Nov 4, 2009 at 1:36 AM, Garry Watkins <[hidden email]>
> wrote:
>> Ok I fixed item one.
>>
>> However for item two, shouldn't it just check threads out of the
>> thread pool and return the thread when it is done? We have clients
>> with terrible networks, and it really bugs me that it will leak
>> threads every time that the network disconnects my server.
>>
>> Thanks for your help
>>
>> Garry Watkins
>>
>> Managing Director
>> DynaFOCUS, LLC
>> 1097 Rivershore Rd.
>> Charleston, SC 29492
>>
>> mobile: 843-276-2808
>> [hidden email]
>>
>>
>>
>> On Nov 3, 2009, at 11:29 AM, Virat Gohil wrote:
>>
>>> - Show quoted text -
>>> On Tue, Nov 3, 2009 at 9:51 PM, Garry Watkins <[hidden email]>
>>> wrote:
>>>>
>>>> Here is the pipeline factory code:
>>>>
>>>> public class PassThroughMasterPipelineFactory implements
>>>> ChannelPipelineFactory {
>>>> private short unitId;
>>>> private Properties properties;
>>>> private ClientBootstrap bootStrap;
>>>>
>>>> public PassThroughMasterPipelineFactory(short unitId, Properties
>>>> properties, ClientBootstrap bootStrap) {
>>>> this.unitId = unitId;
>>>> this.properties = properties;
>>>> this.bootStrap = bootStrap;
>>>> }
>>>>
>>>> public ChannelPipeline getPipeline() throws Exception {
>>>> ChannelPipeline pipeline = pipeline();
>>>>
>>>> // pipeline.addLast("logger", new
>>>> LoggingHandler(InternalLogLevel.INFO, false));
>>>>
>>>> Timer timer = new HashedWheelTimer();
>>>> pipeline.addLast("reconnect", new PLCDisconnectHandler
>>>> (bootStrap,
>>>> timer));
>>>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5,
>>>> 10, 0));
>>>>
>>>> UnitronicsLengthFieldBasedDecoder tcpFramer = new
>>>> UnitronicsLengthFieldBasedDecoder();
>>>> pipeline.addLast("tcpFramer", tcpFramer);
>>>>
>>>> pipeline.addLast("tcpCodec", new TCPFrameCodec());
>>>>
>>>> PLCStateMachine stateMachine = new PassThroughStateMachine
>>>> (unitId);
>>>> stateMachine.initializeScheduler(properties);
>>>> pipeline.addLast("handler", new
>>>> PLCStateMachineHandler(stateMachine));
>>>>
>>>> pipeline.addLast("test", new RequestStatisticHandler());
>>>>
>>>> return pipeline;
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> Garry Watkins wrote:
>>>>>
>>>>> I have a reconnect handler that handles closed channels and it
>>>>> will try to
>>>>> connect up after 10 seconds. Every time that it opens the
>>>>> connection back
>>>>> up it creates a new thread. I am not sure if it is the wheel
>>>>> timer is
>>>>> causing the leak or if there is something else. I am willing to
>>>>> try and
>>>>> debug this if I can get a pointer in the right direction.
>>>>>
>>>>> Here is what my code looks like:
>>>>>
>>>>> @ChannelPipelineCoverage("one")
>>>>> public class PLCDisconnectHandler extends
>>>>> SimpleChannelUpstreamHandler
>>>>> {
>>>>> private static final Log LOG =
>>>>> LogFactory.getLog(PLCDisconnectHandler.class);
>>>>>
>>>>> final ClientBootstrap bootstrap;
>>>>> private final Timer timer;
>>>>>
>>>>> public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer
>>>>> timer) {
>>>>> this.bootstrap = bootstrap;
>>>>> this.timer = timer;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void channelOpen(ChannelHandlerContext ctx,
>>>>> ChannelStateEvent
>>>>> e) throws Exception
>>>>> {
>>>>> LOG.error("channelOpen");
>>>>>
>>>>> super.channelOpen(ctx, e);
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void channelConnected(ChannelHandlerContext ctx,
>>>>> ChannelStateEvent e) throws Exception {
>>>>> LOG.error("channelConnected: " +
>>>>> ctx.getChannel().getRemoteAddress().toString());
>>>>>
>>>>> super.channelConnected(ctx, e);
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void channelDisconnected(ChannelHandlerContext ctx,
>>>>> ChannelStateEvent e) throws Exception
>>>>> {
>>>>> LOG.error("Channel disconnected: " +
>>>>> ctx.getChannel().getRemoteAddress().toString());
>>>>>
>>>>> super.channelDisconnected(ctx, e);
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void channelClosed(ChannelHandlerContext ctx,
>>>>> ChannelStateEvent
>>>>> e) {
>>>>> LOG.error("channelClosed");
>>>>>
>>>>> timer.newTimeout(new TimerTask() {
>>>>> public void run(Timeout timeout) throws Exception {
>>>>> timeout.cancel();
>>>>> bootstrap.connect();
>>>>> }
>>>>> }, 10, TimeUnit.SECONDS);
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void exceptionCaught(ChannelHandlerContext ctx,
>>>>> ExceptionEvent
>>>>> e) throws Exception {
>>>>> if (e.getCause() instanceof ConnectException){
>>>>> ctx.getChannel().close();
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> TIA
>>>>> Garry
>>>>>
>>>>
>>>> --
>>>> View this message in context: http://n2.nabble.com/HashedWheelTimer-is-leaking-threads-am-I-doing-something-wrong-tp3939379p3939502.html
>>>> Sent from the Netty User Group mailing list archive at Nabble.com.
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>> Hmm,
>>>
>>> 1. This is the reasons for leaks
>>>
>>>> public ChannelPipeline getPipeline() throws Exception {
>>>> Timer timer = new HashedWheelTimer();
>>>> pipeline.addLast("reconnect", new PLCDisconnectHandler
>>>> (bootStrap,
>>>> timer));
>>>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5,
>>>> 10, 0));
>>>
>>> You dont actually need a new instance for every pipeline, I suggest
>>> that you create a single instance of timer and share it across all
>>> channels as follows:
>>>
>>>
>>> public class PassThroughMasterPipelineFactory implements
>>> ChannelPipelineFactory {
>>> private short unitId;
>>> private Properties properties;
>>> private ClientBootstrap bootStrap;
>>> private Timer timer;
>>> public PassThroughMasterPipelineFactory(short unitId, Properties
>>> properties, ClientBootstrap bootStrap) {
>>> this.unitId = unitId;
>>> this.properties = properties;
>>> this.bootStrap = bootStrap;
>>> this.timer =new HashedWheelTimer();
>>> }
>>> public ChannelPipeline getPipeline() throws Exception {
>>> pipeline.addLast("reconnect", new
>>> PLCDisconnectHandler(bootStrap, timer));
>>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10,
>>> 0));
>>> }
>>>
>>> And second:
>>>
>>>>> timer.newTimeout(new TimerTask() {
>>>>> public void run(Timeout timeout) throws Exception {
>>>>> timeout.cancel();
>>>>> bootstrap.connect();
>>>>> }
>>>>> }, 10, TimeUnit.SECONDS);
>>>
>>> It basically creates a new thread every time a connection is closed.
>>>
>>> Thanks,
>>>
>>> Virat
>>> _______________________________________________
>>> netty-users mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-07T07:17:15Z

A HashedWheelTimer creates only one thread. If you create many
instances of HashedWheelTimer, you will see many threads, and it's
improper use. There's no thread leak if used properly.

If you want to make it pull a thread from a pool, you can specify a
custom ThreadFactory in the constructor. However, I don't think this
is related with thread leak.

— Trustin Lee, http://gleamynode.net/

On Wed, Nov 4, 2009 at 1:36 AM, Garry Watkins <[hidden email]> wrote:

> Ok I fixed item one.
>
> However for item two, shouldn't it just check threads out of the
> thread pool and return the thread when it is done? We have clients
> with terrible networks, and it really bugs me that it will leak
> threads every time that the network disconnects my server.
>
> Thanks for your help
>
> Garry Watkins
>
> Managing Director
> DynaFOCUS, LLC
> 1097 Rivershore Rd.
> Charleston, SC 29492
>
> mobile: 843-276-2808
> [hidden email]
>
>
>
> On Nov 3, 2009, at 11:29 AM, Virat Gohil wrote:
>
>> - Show quoted text -
>> On Tue, Nov 3, 2009 at 9:51 PM, Garry Watkins <[hidden email]>
>> wrote:
>>>
>>> Here is the pipeline factory code:
>>>
>>> public class PassThroughMasterPipelineFactory implements
>>> ChannelPipelineFactory {
>>> private short unitId;
>>> private Properties properties;
>>> private ClientBootstrap bootStrap;
>>>
>>> public PassThroughMasterPipelineFactory(short unitId, Properties
>>> properties, ClientBootstrap bootStrap) {
>>> this.unitId = unitId;
>>> this.properties = properties;
>>> this.bootStrap = bootStrap;
>>> }
>>>
>>> public ChannelPipeline getPipeline() throws Exception {
>>> ChannelPipeline pipeline = pipeline();
>>>
>>> // pipeline.addLast("logger", new
>>> LoggingHandler(InternalLogLevel.INFO, false));
>>>
>>> Timer timer = new HashedWheelTimer();
>>> pipeline.addLast("reconnect", new PLCDisconnectHandler
>>> (bootStrap,
>>> timer));
>>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5,
>>> 10, 0));
>>>
>>> UnitronicsLengthFieldBasedDecoder tcpFramer = new
>>> UnitronicsLengthFieldBasedDecoder();
>>> pipeline.addLast("tcpFramer", tcpFramer);
>>>
>>> pipeline.addLast("tcpCodec", new TCPFrameCodec());
>>>
>>> PLCStateMachine stateMachine = new PassThroughStateMachine
>>> (unitId);
>>> stateMachine.initializeScheduler(properties);
>>> pipeline.addLast("handler", new
>>> PLCStateMachineHandler(stateMachine));
>>>
>>> pipeline.addLast("test", new RequestStatisticHandler());
>>>
>>> return pipeline;
>>> }
>>>
>>> }
>>>
>>>
>>> Garry Watkins wrote:
>>>>
>>>> I have a reconnect handler that handles closed channels and it
>>>> will try to
>>>> connect up after 10 seconds. Every time that it opens the
>>>> connection back
>>>> up it creates a new thread. I am not sure if it is the wheel
>>>> timer is
>>>> causing the leak or if there is something else. I am willing to
>>>> try and
>>>> debug this if I can get a pointer in the right direction.
>>>>
>>>> Here is what my code looks like:
>>>>
>>>> @ChannelPipelineCoverage("one")
>>>> public class PLCDisconnectHandler extends
>>>> SimpleChannelUpstreamHandler
>>>> {
>>>> private static final Log LOG =
>>>> LogFactory.getLog(PLCDisconnectHandler.class);
>>>>
>>>> final ClientBootstrap bootstrap;
>>>> private final Timer timer;
>>>>
>>>> public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer
>>>> timer) {
>>>> this.bootstrap = bootstrap;
>>>> this.timer = timer;
>>>> }
>>>>
>>>> @Override
>>>> public void channelOpen(ChannelHandlerContext ctx,
>>>> ChannelStateEvent
>>>> e) throws Exception
>>>> {
>>>> LOG.error("channelOpen");
>>>>
>>>> super.channelOpen(ctx, e);
>>>> }
>>>>
>>>> @Override
>>>> public void channelConnected(ChannelHandlerContext ctx,
>>>> ChannelStateEvent e) throws Exception {
>>>> LOG.error("channelConnected: " +
>>>> ctx.getChannel().getRemoteAddress().toString());
>>>>
>>>> super.channelConnected(ctx, e);
>>>> }
>>>>
>>>> @Override
>>>> public void channelDisconnected(ChannelHandlerContext ctx,
>>>> ChannelStateEvent e) throws Exception
>>>> {
>>>> LOG.error("Channel disconnected: " +
>>>> ctx.getChannel().getRemoteAddress().toString());
>>>>
>>>> super.channelDisconnected(ctx, e);
>>>> }
>>>>
>>>> @Override
>>>> public void channelClosed(ChannelHandlerContext ctx,
>>>> ChannelStateEvent
>>>> e) {
>>>> LOG.error("channelClosed");
>>>>
>>>> timer.newTimeout(new TimerTask() {
>>>> public void run(Timeout timeout) throws Exception {
>>>> timeout.cancel();
>>>> bootstrap.connect();
>>>> }
>>>> }, 10, TimeUnit.SECONDS);
>>>> }
>>>>
>>>> @Override
>>>> public void exceptionCaught(ChannelHandlerContext ctx,
>>>> ExceptionEvent
>>>> e) throws Exception {
>>>> if (e.getCause() instanceof ConnectException){
>>>> ctx.getChannel().close();
>>>> }
>>>> }
>>>> }
>>>>
>>>> TIA
>>>> Garry
>>>>
>>>
>>> --
>>> View this message in context: http://n2.nabble.com/HashedWheelTimer-is-leaking-threads-am-I-doing-something-wrong-tp3939379p3939502.html
>>> Sent from the Netty User Group mailing list archive at Nabble.com.
>>> _______________________________________________
>>> netty-users mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>> Hmm,
>>
>> 1. This is the reasons for leaks
>>
>>> public ChannelPipeline getPipeline() throws Exception {
>>> Timer timer = new HashedWheelTimer();
>>> pipeline.addLast("reconnect", new PLCDisconnectHandler
>>> (bootStrap,
>>> timer));
>>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5,
>>> 10, 0));
>>
>> You dont actually need a new instance for every pipeline, I suggest
>> that you create a single instance of timer and share it across all
>> channels as follows:
>>
>>
>> public class PassThroughMasterPipelineFactory implements
>> ChannelPipelineFactory {
>> private short unitId;
>> private Properties properties;
>> private ClientBootstrap bootStrap;
>> private Timer timer;
>> public PassThroughMasterPipelineFactory(short unitId, Properties
>> properties, ClientBootstrap bootStrap) {
>> this.unitId = unitId;
>> this.properties = properties;
>> this.bootStrap = bootStrap;
>> this.timer =new HashedWheelTimer();
>> }
>> public ChannelPipeline getPipeline() throws Exception {
>> pipeline.addLast("reconnect", new
>> PLCDisconnectHandler(bootStrap, timer));
>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10,
>> 0));
>> }
>>
>> And second:
>>
>>>> timer.newTimeout(new TimerTask() {
>>>> public void run(Timeout timeout) throws Exception {
>>>> timeout.cancel();
>>>> bootstrap.connect();
>>>> }
>>>> }, 10, TimeUnit.SECONDS);
>>
>> It basically creates a new thread every time a connection is closed.
>>
>> Thanks,
>>
>> Virat
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-03T08:36:25Z

Ok I fixed item one.

However for item two, shouldn't it just check threads out of the
thread pool and return the thread when it is done? We have clients
with terrible networks, and it really bugs me that it will leak
threads every time that the network disconnects my server.

Thanks for your help

Garry Watkins

Managing Director
DynaFOCUS, LLC
1097 Rivershore Rd.
Charleston, SC 29492

mobile: 843-276-2808
[hidden email]

On Nov 3, 2009, at 11:29 AM, Virat Gohil wrote:

> - Show quoted text -
> On Tue, Nov 3, 2009 at 9:51 PM, Garry Watkins <[hidden email]>
> wrote:
>>
>> Here is the pipeline factory code:
>>
>> public class PassThroughMasterPipelineFactory implements
>> ChannelPipelineFactory {
>> private short unitId;
>> private Properties properties;
>> private ClientBootstrap bootStrap;
>>
>> public PassThroughMasterPipelineFactory(short unitId, Properties
>> properties, ClientBootstrap bootStrap) {
>> this.unitId = unitId;
>> this.properties = properties;
>> this.bootStrap = bootStrap;
>> }
>>
>> public ChannelPipeline getPipeline() throws Exception {
>> ChannelPipeline pipeline = pipeline();
>>
>> // pipeline.addLast("logger", new
>> LoggingHandler(InternalLogLevel.INFO, false));
>>
>> Timer timer = new HashedWheelTimer();
>> pipeline.addLast("reconnect", new PLCDisconnectHandler
>> (bootStrap,
>> timer));
>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5,
>> 10, 0));
>>
>> UnitronicsLengthFieldBasedDecoder tcpFramer = new
>> UnitronicsLengthFieldBasedDecoder();
>> pipeline.addLast("tcpFramer", tcpFramer);
>>
>> pipeline.addLast("tcpCodec", new TCPFrameCodec());
>>
>> PLCStateMachine stateMachine = new PassThroughStateMachine
>> (unitId);
>> stateMachine.initializeScheduler(properties);
>> pipeline.addLast("handler", new
>> PLCStateMachineHandler(stateMachine));
>>
>> pipeline.addLast("test", new RequestStatisticHandler());
>>
>> return pipeline;
>> }
>>
>> }
>>
>>
>> Garry Watkins wrote:
>>>
>>> I have a reconnect handler that handles closed channels and it
>>> will try to
>>> connect up after 10 seconds. Every time that it opens the
>>> connection back
>>> up it creates a new thread. I am not sure if it is the wheel
>>> timer is
>>> causing the leak or if there is something else. I am willing to
>>> try and
>>> debug this if I can get a pointer in the right direction.
>>>
>>> Here is what my code looks like:
>>>
>>> @ChannelPipelineCoverage("one")
>>> public class PLCDisconnectHandler extends
>>> SimpleChannelUpstreamHandler
>>> {
>>> private static final Log LOG =
>>> LogFactory.getLog(PLCDisconnectHandler.class);
>>>
>>> final ClientBootstrap bootstrap;
>>> private final Timer timer;
>>>
>>> public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer
>>> timer) {
>>> this.bootstrap = bootstrap;
>>> this.timer = timer;
>>> }
>>>
>>> @Override
>>> public void channelOpen(ChannelHandlerContext ctx,
>>> ChannelStateEvent
>>> e) throws Exception
>>> {
>>> LOG.error("channelOpen");
>>>
>>> super.channelOpen(ctx, e);
>>> }
>>>
>>> @Override
>>> public void channelConnected(ChannelHandlerContext ctx,
>>> ChannelStateEvent e) throws Exception {
>>> LOG.error("channelConnected: " +
>>> ctx.getChannel().getRemoteAddress().toString());
>>>
>>> super.channelConnected(ctx, e);
>>> }
>>>
>>> @Override
>>> public void channelDisconnected(ChannelHandlerContext ctx,
>>> ChannelStateEvent e) throws Exception
>>> {
>>> LOG.error("Channel disconnected: " +
>>> ctx.getChannel().getRemoteAddress().toString());
>>>
>>> super.channelDisconnected(ctx, e);
>>> }
>>>
>>> @Override
>>> public void channelClosed(ChannelHandlerContext ctx,
>>> ChannelStateEvent
>>> e) {
>>> LOG.error("channelClosed");
>>>
>>> timer.newTimeout(new TimerTask() {
>>> public void run(Timeout timeout) throws Exception {
>>> timeout.cancel();
>>> bootstrap.connect();
>>> }
>>> }, 10, TimeUnit.SECONDS);
>>> }
>>>
>>> @Override
>>> public void exceptionCaught(ChannelHandlerContext ctx,
>>> ExceptionEvent
>>> e) throws Exception {
>>> if (e.getCause() instanceof ConnectException){
>>> ctx.getChannel().close();
>>> }
>>> }
>>> }
>>>
>>> TIA
>>> Garry
>>>
>>
>> --
>> View this message in context: http://n2.nabble.com/HashedWheelTimer-is-leaking-threads-am-I-doing-something-wrong-tp3939379p3939502.html
>> Sent from the Netty User Group mailing list archive at Nabble.com.
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> Hmm,
>
> 1. This is the reasons for leaks
>
>> public ChannelPipeline getPipeline() throws Exception {
>> Timer timer = new HashedWheelTimer();
>> pipeline.addLast("reconnect", new PLCDisconnectHandler
>> (bootStrap,
>> timer));
>> pipeline.addLast("timeout", new IdleStateHandler(timer, 5,
>> 10, 0));
>
> You dont actually need a new instance for every pipeline, I suggest
> that you create a single instance of timer and share it across all
> channels as follows:
>
>
> public class PassThroughMasterPipelineFactory implements
> ChannelPipelineFactory {
> private short unitId;
> private Properties properties;
> private ClientBootstrap bootStrap;
> private Timer timer;
> public PassThroughMasterPipelineFactory(short unitId, Properties
> properties, ClientBootstrap bootStrap) {
> this.unitId = unitId;
> this.properties = properties;
> this.bootStrap = bootStrap;
> this.timer =new HashedWheelTimer();
> }
> public ChannelPipeline getPipeline() throws Exception {
> pipeline.addLast("reconnect", new
> PLCDisconnectHandler(bootStrap, timer));
> pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10,
> 0));
> }
>
> And second:
>
>>> timer.newTimeout(new TimerTask() {
>>> public void run(Timeout timeout) throws Exception {
>>> timeout.cancel();
>>> bootstrap.connect();
>>> }
>>> }, 10, TimeUnit.SECONDS);
>
> It basically creates a new thread every time a connection is closed.
>
> Thanks,
>
> Virat
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-03T08:29:57Z

- Show quoted text -
On Tue, Nov 3, 2009 at 9:51 PM, Garry Watkins <[hidden email]> wrote:

>
> Here is the pipeline factory code:
>
> public class PassThroughMasterPipelineFactory implements
> ChannelPipelineFactory {
> private short unitId;
> private Properties properties;
> private ClientBootstrap bootStrap;
>
> public PassThroughMasterPipelineFactory(short unitId, Properties
> properties, ClientBootstrap bootStrap) {
> this.unitId = unitId;
> this.properties = properties;
> this.bootStrap = bootStrap;
> }
>
> public ChannelPipeline getPipeline() throws Exception {
> ChannelPipeline pipeline = pipeline();
>
> // pipeline.addLast("logger", new
> LoggingHandler(InternalLogLevel.INFO, false));
>
> Timer timer = new HashedWheelTimer();
> pipeline.addLast("reconnect", new PLCDisconnectHandler(bootStrap,
> timer));
> pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10, 0));
>
> UnitronicsLengthFieldBasedDecoder tcpFramer = new
> UnitronicsLengthFieldBasedDecoder();
> pipeline.addLast("tcpFramer", tcpFramer);
>
> pipeline.addLast("tcpCodec", new TCPFrameCodec());
>
> PLCStateMachine stateMachine = new PassThroughStateMachine(unitId);
> stateMachine.initializeScheduler(properties);
> pipeline.addLast("handler", new
> PLCStateMachineHandler(stateMachine));
>
> pipeline.addLast("test", new RequestStatisticHandler());
>
> return pipeline;
> }
>
> }
>
>
> Garry Watkins wrote:
>>
>> I have a reconnect handler that handles closed channels and it will try to
>> connect up after 10 seconds. Every time that it opens the connection back
>> up it creates a new thread. I am not sure if it is the wheel timer is
>> causing the leak or if there is something else. I am willing to try and
>> debug this if I can get a pointer in the right direction.
>>
>> Here is what my code looks like:
>>
>> @ChannelPipelineCoverage("one")
>> public class PLCDisconnectHandler extends SimpleChannelUpstreamHandler
>> {
>> private static final Log LOG =
>> LogFactory.getLog(PLCDisconnectHandler.class);
>>
>> final ClientBootstrap bootstrap;
>> private final Timer timer;
>>
>> public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer timer) {
>> this.bootstrap = bootstrap;
>> this.timer = timer;
>> }
>>
>> @Override
>> public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent
>> e) throws Exception
>> {
>> LOG.error("channelOpen");
>>
>> super.channelOpen(ctx, e);
>> }
>>
>> @Override
>> public void channelConnected(ChannelHandlerContext ctx,
>> ChannelStateEvent e) throws Exception {
>> LOG.error("channelConnected: " +
>> ctx.getChannel().getRemoteAddress().toString());
>>
>> super.channelConnected(ctx, e);
>> }
>>
>> @Override
>> public void channelDisconnected(ChannelHandlerContext ctx,
>> ChannelStateEvent e) throws Exception
>> {
>> LOG.error("Channel disconnected: " +
>> ctx.getChannel().getRemoteAddress().toString());
>>
>> super.channelDisconnected(ctx, e);
>> }
>>
>> @Override
>> public void channelClosed(ChannelHandlerContext ctx, ChannelStateEvent
>> e) {
>> LOG.error("channelClosed");
>>
>> timer.newTimeout(new TimerTask() {
>> public void run(Timeout timeout) throws Exception {
>> timeout.cancel();
>> bootstrap.connect();
>> }
>> }, 10, TimeUnit.SECONDS);
>> }
>>
>> @Override
>> public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent
>> e) throws Exception {
>> if (e.getCause() instanceof ConnectException){
>> ctx.getChannel().close();
>> }
>> }
>> }
>>
>> TIA
>> Garry
>>
>
> --
> View this message in context: http://n2.nabble.com/HashedWheelTimer-is-leaking-threads-am-I-doing-something-wrong-tp3939379p3939502.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

Hmm,

1. This is the reasons for leaks

> public ChannelPipeline getPipeline() throws Exception {
> Timer timer = new HashedWheelTimer();
> pipeline.addLast("reconnect", new PLCDisconnectHandler(bootStrap,
> timer));
> pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10, 0));

You dont actually need a new instance for every pipeline, I suggest
that you create a single instance of timer and share it across all
channels as follows:

public class PassThroughMasterPipelineFactory implements
ChannelPipelineFactory {
private short unitId;
private Properties properties;
private ClientBootstrap bootStrap;
private Timer timer;
public PassThroughMasterPipelineFactory(short unitId, Properties
properties, ClientBootstrap bootStrap) {
this.unitId = unitId;
this.properties = properties;
this.bootStrap = bootStrap;
this.timer =new HashedWheelTimer();
}
public ChannelPipeline getPipeline() throws Exception {
pipeline.addLast("reconnect", new
PLCDisconnectHandler(bootStrap, timer));
pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10, 0));
}

And second:

>> timer.newTimeout(new TimerTask() {
>> public void run(Timeout timeout) throws Exception {
>> timeout.cancel();
>> bootstrap.connect();
>> }
>> }, 10, TimeUnit.SECONDS);

It basically creates a new thread every time a connection is closed.

Thanks,

Virat
_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-03T08:28:51Z

On Tue, Nov 3, 2009 at 9:51 PM, Garry Watkins <[hidden email]> wrote:

Hmm,

1. This is the reasons for leaks

> public ChannelPipeline getPipeline() throws Exception {
> Timer timer = new HashedWheelTimer();
> pipeline.addLast("reconnect", new PLCDisconnectHandler(bootStrap,
> timer));
> pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10, 0));

You dont actually need a new instance for every pipeline, I suggest
that you create a single instance of timer and share it across all
channels as follows:

public class PassThroughMasterPipelineFactory implements
ChannelPipelineFactory {
private short unitId;
private Properties properties;
private ClientBootstrap bootStrap;
private Timer timer;
public PassThroughMasterPipelineFactory(short unitId, Properties
properties, ClientBootstrap bootStrap) {
this.unitId = unitId;
this.properties = properties;
this.bootStrap = bootStrap;
this.timer =new HashedWheelTimer();
}
public ChannelPipeline getPipeline() throws Exception {
pipeline.addLast("reconnect", new
PLCDisconnectHandler(bootStrap, timer));
pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10, 0));
}

And second:

>> timer.newTimeout(new TimerTask() {
>> public void run(Timeout timeout) throws Exception {
>> timeout.cancel();
>> bootstrap.connect();
>> }
>> }, 10, TimeUnit.SECONDS);

It basically creates a new thread every time a connection is closed.

Thanks,

Virat

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-03T08:21:56Z

Here is the pipeline factory code:

public class PassThroughMasterPipelineFactory implements ChannelPipelineFactory {
private short unitId;
private Properties properties;
private ClientBootstrap bootStrap;

public PassThroughMasterPipelineFactory(short unitId, Properties properties, ClientBootstrap bootStrap) {
this.unitId = unitId;
this.properties = properties;
this.bootStrap = bootStrap;
}

public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = pipeline();

// pipeline.addLast("logger", new LoggingHandler(InternalLogLevel.INFO, false));

Timer timer = new HashedWheelTimer();
pipeline.addLast("reconnect", new PLCDisconnectHandler(bootStrap, timer));
pipeline.addLast("timeout", new IdleStateHandler(timer, 5, 10, 0));

UnitronicsLengthFieldBasedDecoder tcpFramer = new UnitronicsLengthFieldBasedDecoder();
pipeline.addLast("tcpFramer", tcpFramer);

pipeline.addLast("tcpCodec", new TCPFrameCodec());

PLCStateMachine stateMachine = new PassThroughStateMachine(unitId);
stateMachine.initializeScheduler(properties);
pipeline.addLast("handler", new PLCStateMachineHandler(stateMachine));

pipeline.addLast("test", new RequestStatisticHandler());

return pipeline;
}

}

Garry Watkins wrote:

I have a reconnect handler that handles closed channels and it will try to connect up after 10 seconds. Every time that it opens the connection back up it creates a new thread. I am not sure if it is the wheel timer is causing the leak or if there is something else. I am willing to try and debug this if I can get a pointer in the right direction.

Here is what my code looks like:

@ChannelPipelineCoverage("one")
public class PLCDisconnectHandler extends SimpleChannelUpstreamHandler
{
private static final Log LOG = LogFactory.getLog(PLCDisconnectHandler.class);

final ClientBootstrap bootstrap;
private final Timer timer;

public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer timer) {
this.bootstrap = bootstrap;
this.timer = timer;
}

@Override
public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception
{
LOG.error("channelOpen");

super.channelOpen(ctx, e);
}

@Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception {
LOG.error("channelConnected: " + ctx.getChannel().getRemoteAddress().toString());

super.channelConnected(ctx, e);
}

@Override
public void channelDisconnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception
{
LOG.error("Channel disconnected: " + ctx.getChannel().getRemoteAddress().toString());

super.channelDisconnected(ctx, e);
}

@Override
public void channelClosed(ChannelHandlerContext ctx, ChannelStateEvent e) {
LOG.error("channelClosed");

timer.newTimeout(new TimerTask() {
public void run(Timeout timeout) throws Exception {
timeout.cancel();
bootstrap.connect();
}
}, 10, TimeUnit.SECONDS);
}

@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
if (e.getCause() instanceof ConnectException){
ctx.getChannel().close();
}
}
}

TIA
Garry

Re: HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-03T08:14:25Z

On Tue, Nov 3, 2009 at 9:34 PM, Garry Watkins <[hidden email]> wrote:

>
> I have a reconnect handler that handles closed channels and it will try to
> connect up after 10 seconds. Every time that it opens the connection back
> up it creates a new thread. I am not sure if it is the wheel timer is
> causing the leak or if there is something else. I am willing to try and
> debug this if I can get a pointer in the right direction.
>
> Here is what my code looks like:
>
> @ChannelPipelineCoverage("one")
> public class PLCDisconnectHandler extends SimpleChannelUpstreamHandler
> {
> private static final Log LOG =
> LogFactory.getLog(PLCDisconnectHandler.class);
>
> final ClientBootstrap bootstrap;
> private final Timer timer;
>
> public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer timer) {
> this.bootstrap = bootstrap;
> this.timer = timer;
> }
>
> @Override
> public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent e)
> throws Exception
> {
> LOG.error("channelOpen");
>
> super.channelOpen(ctx, e);
> }
>
> @Override
> public void channelConnected(ChannelHandlerContext ctx,
> ChannelStateEvent e) throws Exception {
> LOG.error("channelConnected: " +
> ctx.getChannel().getRemoteAddress().toString());
>
> super.channelConnected(ctx, e);
> }
>
> @Override
> public void channelDisconnected(ChannelHandlerContext ctx,
> ChannelStateEvent e) throws Exception
> {
> LOG.error("Channel disconnected: " +
> ctx.getChannel().getRemoteAddress().toString());
>
> super.channelDisconnected(ctx, e);
> }
>
> @Override
> public void channelClosed(ChannelHandlerContext ctx, ChannelStateEvent
> e) {
> LOG.error("channelClosed");
>
> timer.newTimeout(new TimerTask() {
> public void run(Timeout timeout) throws Exception {
> timeout.cancel();
> bootstrap.connect();
> }
> }, 10, TimeUnit.SECONDS);
> }
>
> @Override
> public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e)
> throws Exception {
> if (e.getCause() instanceof ConnectException){
> ctx.getChannel().close();
> }
> }
> }
>
> TIA
> Garry

Hi Garry,

Can you please post the code where you create a pipeline?

Thanks,

Virat

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

HashedWheelTimer is leaking threads, am I doing something wrong?

2009-11-03T08:04:37Z

I have a reconnect handler that handles closed channels and it will try to connect up after 10 seconds. Every time that it opens the connection back up it creates a new thread. I am not sure if it is the wheel timer is causing the leak or if there is something else. I am willing to try and debug this if I can get a pointer in the right direction.

Here is what my code looks like:

@ChannelPipelineCoverage("one")
public class PLCDisconnectHandler extends SimpleChannelUpstreamHandler
{
private static final Log LOG = LogFactory.getLog(PLCDisconnectHandler.class);

final ClientBootstrap bootstrap;
private final Timer timer;

public PLCDisconnectHandler(ClientBootstrap bootstrap, Timer timer) {
this.bootstrap = bootstrap;
this.timer = timer;
}

@Override
public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception
{
LOG.error("channelOpen");

super.channelOpen(ctx, e);
}

@Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception {
LOG.error("channelConnected: " + ctx.getChannel().getRemoteAddress().toString());

super.channelConnected(ctx, e);
}

@Override
public void channelDisconnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception
{
LOG.error("Channel disconnected: " + ctx.getChannel().getRemoteAddress().toString());

super.channelDisconnected(ctx, e);
}

@Override
public void channelClosed(ChannelHandlerContext ctx, ChannelStateEvent e) {
LOG.error("channelClosed");

timer.newTimeout(new TimerTask() {
public void run(Timeout timeout) throws Exception {
timeout.cancel();
bootstrap.connect();
}
}, 10, TimeUnit.SECONDS);
}

@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
if (e.getCause() instanceof ConnectException){
ctx.getChannel().close();
}
}
}

TIA
Garry

Re: Random connect exceptions

2009-11-08T02:22:26Z

Here is the client bootstrap initialization:

clientBootstrap = new ClientBootstrap(new
NioClientSocketChannelFactory(new VirtualExecutorService(threadPool),
new VirtualExecutorService(threadPool)));

ChannelPipelineFactory clientPipelineFactory = new
ChannelPipelineFactory() {
@Override public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("decoder", new SizeHeaderFrameDecoder());
pipeline.addLast("dispatcher", new
MessageChannelHandler(NettyTransport.this));
return pipeline;
}
};

clientBootstrap.setPipelineFactory(clientPipelineFactory);

clientBootstrap.setOption("connectTimeoutMillis",
connectTimeout.millis());

And here is the connect operation:

ChannelFuture channelFuture = clientBootstrap.connect(address);
channelFuture.awaitUninterruptibly();
if (!channelFuture.isSuccess()) {
// throw connect excep
}
return channelFuture.getChannel();

The server is a netty server bound to 0.0.0.0, and the address it to
connect to is the local ip address (as derived by java, for example,
192.168....).

Shay

On Sun, Nov 8, 2009 at 11:22 AM, Trustin Lee (이희승) <[hidden email]> wrote:

> Perhaps it might be a bug. I will investigate a little bit more.
> Could you paste the whole bootstrap initialization and connection
> attempt code?
>
> Thanks
>
> — Trustin Lee, http://gleamynode.net/
>
>
>
> On Sun, Nov 8, 2009 at 5:38 PM, Shay Banon <[hidden email]> wrote:
>> I see. Now, when I try with waiting without any timeout value
>> (channelFuture.awaitUninterruptibly()), then it simply hangs without
>> returning. I do set the connectionTimeoutMillis... :
>>
>> clientBootstrap.setOption("connectTimeoutMillis", connectTimeout.millis());
>>
>> and it is set to 1 second.
>>
>> Cheers,
>> Shay
>>
>> On Sat, Nov 7, 2009 at 5:09 PM, Trustin Lee (이희승) <[hidden email]> wrote:
>>> If you have specified connectTimeoutMillis option correctly, you
>>> should not specify the timeout value when you call
>>> ChannelFuture.await(). ChannelFuture will complete automatically on
>>> connection timeout.
>>>
>>> You cannot assume that the connection attempt has failed if
>>> ChannelFuture.isSuccess() returned false because it might also mean
>>> that the connection attempt is still in progress. To make sure that
>>> the connection attempt is finished, you have to await the future until
>>> its completion. If ChannelFuture is complete, ChannelFuture.isDone()
>>> returns true.
>>>
>>> HTH
>>>
>>> — Trustin Lee, http://gleamynode.net/
>>>
>>>
>>>
>>> On Tue, Nov 3, 2009 at 11:49 PM, Shay Banon <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> First of all, thanks for the reply. I have added the connectTimeout
>>>> option as you suggested, and it does not seem to help. Even when I set
>>>> it to a higher value, it simply waits there until the timeout passed,
>>>> and then the channelFuture#isSuccess is false. I still don't get the
>>>> actual cause as well. I am using netty version 3.1.5.
>>>>
>>>> What is the scenario that the cause of the future is null?
>>>>
>>>> Thanks,
>>>> Shay
>>>>
>>>> On Tue, Nov 3, 2009 at 1:41 PM, Christian Migowski <[hidden email]> wrote:
>>>>> Hi Shay,
>>>>>
>>>>> the code looks OK, but two things:
>>>>>
>>>>> 1) did you set the connectTimeout to the bootstrap, i.e.
>>>>>
>>>>> clientBootstrap.setOption("connectTimeoutMillis",connectTimeout.millis());
>>>>>
>>>>> if no that might be the reason: the connection attempt is just not finished yet
>>>>>
>>>>> 2) if yes do you use the latest Netty version? The NIO transport in
>>>>> versions prior to 3.1.4 (if i remember correctly) did not respect the
>>>>> connect timeout.
>>>>>
>>>>> hth,
>>>>> regards,
>>>>> christian!
>>>>>
>>>>>
>>>>> On Tue, Nov 3, 2009 at 12:19 PM, Shay Banon <[hidden email]> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am getting random connect failures when using netty, which I am
>>>>>> trying to track down the cause of it. First, here is the code I use to
>>>>>> connect:
>>>>>>
>>>>>> InetSocketAddress address = // obtain the address to connect to
>>>>>> ChannelFuture channelFuture = clientBootstrap.connect(address);
>>>>>> channelFuture.awaitUninterruptibly(connectTimeout.millis());
>>>>>> // connect timeout is 1 second
>>>>>> if (!channelFuture.isSuccess()) {
>>>>>> throw new ConnectTransportException(address, "",
>>>>>> channelFuture.getCause());
>>>>>> }
>>>>>>
>>>>>>
>>>>>> First, is this code fragment correct, or am I missing something?
>>>>>> Second, the strange thing I get is that the channelFuture.getCause()
>>>>>> is null .., so I don't really know the underlying problem here.
>>>>>>
>>>>>> Any idea?
>>>>>>
>>>>>> Cheers,
>>>>>> Shay
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Random connect exceptions

2009-11-08T01:22:40Z

Perhaps it might be a bug. I will investigate a little bit more.
Could you paste the whole bootstrap initialization and connection
attempt code?

Thanks

— Trustin Lee, http://gleamynode.net/

On Sun, Nov 8, 2009 at 5:38 PM, Shay Banon <[hidden email]> wrote:

> I see. Now, when I try with waiting without any timeout value
> (channelFuture.awaitUninterruptibly()), then it simply hangs without
> returning. I do set the connectionTimeoutMillis... :
>
> clientBootstrap.setOption("connectTimeoutMillis", connectTimeout.millis());
>
> and it is set to 1 second.
>
> Cheers,
> Shay
>
> On Sat, Nov 7, 2009 at 5:09 PM, Trustin Lee (이희승) <[hidden email]> wrote:
>> If you have specified connectTimeoutMillis option correctly, you
>> should not specify the timeout value when you call
>> ChannelFuture.await(). ChannelFuture will complete automatically on
>> connection timeout.
>>
>> You cannot assume that the connection attempt has failed if
>> ChannelFuture.isSuccess() returned false because it might also mean
>> that the connection attempt is still in progress. To make sure that
>> the connection attempt is finished, you have to await the future until
>> its completion. If ChannelFuture is complete, ChannelFuture.isDone()
>> returns true.
>>
>> HTH
>>
>> — Trustin Lee, http://gleamynode.net/
>>
>>
>>
>> On Tue, Nov 3, 2009 at 11:49 PM, Shay Banon <[hidden email]> wrote:
>>> Hi,
>>>
>>> First of all, thanks for the reply. I have added the connectTimeout
>>> option as you suggested, and it does not seem to help. Even when I set
>>> it to a higher value, it simply waits there until the timeout passed,
>>> and then the channelFuture#isSuccess is false. I still don't get the
>>> actual cause as well. I am using netty version 3.1.5.
>>>
>>> What is the scenario that the cause of the future is null?
>>>
>>> Thanks,
>>> Shay
>>>
>>> On Tue, Nov 3, 2009 at 1:41 PM, Christian Migowski <[hidden email]> wrote:
>>>> Hi Shay,
>>>>
>>>> the code looks OK, but two things:
>>>>
>>>> 1) did you set the connectTimeout to the bootstrap, i.e.
>>>>
>>>> clientBootstrap.setOption("connectTimeoutMillis",connectTimeout.millis());
>>>>
>>>> if no that might be the reason: the connection attempt is just not finished yet
>>>>
>>>> 2) if yes do you use the latest Netty version? The NIO transport in
>>>> versions prior to 3.1.4 (if i remember correctly) did not respect the
>>>> connect timeout.
>>>>
>>>> hth,
>>>> regards,
>>>> christian!
>>>>
>>>>
>>>> On Tue, Nov 3, 2009 at 12:19 PM, Shay Banon <[hidden email]> wrote:
>>>>> Hi,
>>>>>
>>>>> I am getting random connect failures when using netty, which I am
>>>>> trying to track down the cause of it. First, here is the code I use to
>>>>> connect:
>>>>>
>>>>> InetSocketAddress address = // obtain the address to connect to
>>>>> ChannelFuture channelFuture = clientBootstrap.connect(address);
>>>>> channelFuture.awaitUninterruptibly(connectTimeout.millis());
>>>>> // connect timeout is 1 second
>>>>> if (!channelFuture.isSuccess()) {
>>>>> throw new ConnectTransportException(address, "",
>>>>> channelFuture.getCause());
>>>>> }
>>>>>
>>>>>
>>>>> First, is this code fragment correct, or am I missing something?
>>>>> Second, the strange thing I get is that the channelFuture.getCause()
>>>>> is null .., so I don't really know the underlying problem here.
>>>>>
>>>>> Any idea?
>>>>>
>>>>> Cheers,
>>>>> Shay
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Re: Random connect exceptions

2009-11-08T00:38:47Z

I see. Now, when I try with waiting without any timeout value
(channelFuture.awaitUninterruptibly()), then it simply hangs without
returning. I do set the connectionTimeoutMillis... :

clientBootstrap.setOption("connectTimeoutMillis", connectTimeout.millis());

and it is set to 1 second.

Cheers,
Shay

On Sat, Nov 7, 2009 at 5:09 PM, Trustin Lee (이희승) <[hidden email]> wrote:

> If you have specified connectTimeoutMillis option correctly, you
> should not specify the timeout value when you call
> ChannelFuture.await(). ChannelFuture will complete automatically on
> connection timeout.
>
> You cannot assume that the connection attempt has failed if
> ChannelFuture.isSuccess() returned false because it might also mean
> that the connection attempt is still in progress. To make sure that
> the connection attempt is finished, you have to await the future until
> its completion. If ChannelFuture is complete, ChannelFuture.isDone()
> returns true.
>
> HTH
>
> — Trustin Lee, http://gleamynode.net/
>
>
>
> On Tue, Nov 3, 2009 at 11:49 PM, Shay Banon <[hidden email]> wrote:
>> Hi,
>>
>> First of all, thanks for the reply. I have added the connectTimeout
>> option as you suggested, and it does not seem to help. Even when I set
>> it to a higher value, it simply waits there until the timeout passed,
>> and then the channelFuture#isSuccess is false. I still don't get the
>> actual cause as well. I am using netty version 3.1.5.
>>
>> What is the scenario that the cause of the future is null?
>>
>> Thanks,
>> Shay
>>
>> On Tue, Nov 3, 2009 at 1:41 PM, Christian Migowski <[hidden email]> wrote:
>>> Hi Shay,
>>>
>>> the code looks OK, but two things:
>>>
>>> 1) did you set the connectTimeout to the bootstrap, i.e.
>>>
>>> clientBootstrap.setOption("connectTimeoutMillis",connectTimeout.millis());
>>>
>>> if no that might be the reason: the connection attempt is just not finished yet
>>>
>>> 2) if yes do you use the latest Netty version? The NIO transport in
>>> versions prior to 3.1.4 (if i remember correctly) did not respect the
>>> connect timeout.
>>>
>>> hth,
>>> regards,
>>> christian!
>>>
>>>
>>> On Tue, Nov 3, 2009 at 12:19 PM, Shay Banon <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> I am getting random connect failures when using netty, which I am
>>>> trying to track down the cause of it. First, here is the code I use to
>>>> connect:
>>>>
>>>> InetSocketAddress address = // obtain the address to connect to
>>>> ChannelFuture channelFuture = clientBootstrap.connect(address);
>>>> channelFuture.awaitUninterruptibly(connectTimeout.millis());
>>>> // connect timeout is 1 second
>>>> if (!channelFuture.isSuccess()) {
>>>> throw new ConnectTransportException(address, "",
>>>> channelFuture.getCause());
>>>> }
>>>>
>>>>
>>>> First, is this code fragment correct, or am I missing something?
>>>> Second, the strange thing I get is that the channelFuture.getCause()
>>>> is null .., so I don't really know the underlying problem here.
>>>>
>>>> Any idea?
>>>>
>>>> Cheers,
>>>> Shay
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Random connect exceptions

2009-11-07T07:09:07Z

If you have specified connectTimeoutMillis option correctly, you
should not specify the timeout value when you call
ChannelFuture.await(). ChannelFuture will complete automatically on
connection timeout.

You cannot assume that the connection attempt has failed if
ChannelFuture.isSuccess() returned false because it might also mean
that the connection attempt is still in progress. To make sure that
the connection attempt is finished, you have to await the future until
its completion. If ChannelFuture is complete, ChannelFuture.isDone()
returns true.

HTH

— Trustin Lee, http://gleamynode.net/

On Tue, Nov 3, 2009 at 11:49 PM, Shay Banon <[hidden email]> wrote:

> Hi,
>
> First of all, thanks for the reply. I have added the connectTimeout
> option as you suggested, and it does not seem to help. Even when I set
> it to a higher value, it simply waits there until the timeout passed,
> and then the channelFuture#isSuccess is false. I still don't get the
> actual cause as well. I am using netty version 3.1.5.
>
> What is the scenario that the cause of the future is null?
>
> Thanks,
> Shay
>
> On Tue, Nov 3, 2009 at 1:41 PM, Christian Migowski <[hidden email]> wrote:
>> Hi Shay,
>>
>> the code looks OK, but two things:
>>
>> 1) did you set the connectTimeout to the bootstrap, i.e.
>>
>> clientBootstrap.setOption("connectTimeoutMillis",connectTimeout.millis());
>>
>> if no that might be the reason: the connection attempt is just not finished yet
>>
>> 2) if yes do you use the latest Netty version? The NIO transport in
>> versions prior to 3.1.4 (if i remember correctly) did not respect the
>> connect timeout.
>>
>> hth,
>> regards,
>> christian!
>>
>>
>> On Tue, Nov 3, 2009 at 12:19 PM, Shay Banon <[hidden email]> wrote:
>>> Hi,
>>>
>>> I am getting random connect failures when using netty, which I am
>>> trying to track down the cause of it. First, here is the code I use to
>>> connect:
>>>
>>> InetSocketAddress address = // obtain the address to connect to
>>> ChannelFuture channelFuture = clientBootstrap.connect(address);
>>> channelFuture.awaitUninterruptibly(connectTimeout.millis());
>>> // connect timeout is 1 second
>>> if (!channelFuture.isSuccess()) {
>>> throw new ConnectTransportException(address, "",
>>> channelFuture.getCause());
>>> }
>>>
>>>
>>> First, is this code fragment correct, or am I missing something?
>>> Second, the strange thing I get is that the channelFuture.getCause()
>>> is null .., so I don't really know the underlying problem here.
>>>
>>> Any idea?
>>>
>>> Cheers,
>>> Shay
>>> _______________________________________________
>>> netty-users mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Re: Random connect exceptions

2009-11-03T06:49:02Z

Hi,

First of all, thanks for the reply. I have added the connectTimeout
option as you suggested, and it does not seem to help. Even when I set
it to a higher value, it simply waits there until the timeout passed,
and then the channelFuture#isSuccess is false. I still don't get the
actual cause as well. I am using netty version 3.1.5.

What is the scenario that the cause of the future is null?

Thanks,
Shay

On Tue, Nov 3, 2009 at 1:41 PM, Christian Migowski <[hidden email]> wrote:

> Hi Shay,
>
> the code looks OK, but two things:
>
> 1) did you set the connectTimeout to the bootstrap, i.e.
>
> clientBootstrap.setOption("connectTimeoutMillis",connectTimeout.millis());
>
> if no that might be the reason: the connection attempt is just not finished yet
>
> 2) if yes do you use the latest Netty version? The NIO transport in
> versions prior to 3.1.4 (if i remember correctly) did not respect the
> connect timeout.
>
> hth,
> regards,
> christian!
>
>
> On Tue, Nov 3, 2009 at 12:19 PM, Shay Banon <[hidden email]> wrote:
>> Hi,
>>
>> I am getting random connect failures when using netty, which I am
>> trying to track down the cause of it. First, here is the code I use to
>> connect:
>>
>> InetSocketAddress address = // obtain the address to connect to
>> ChannelFuture channelFuture = clientBootstrap.connect(address);
>> channelFuture.awaitUninterruptibly(connectTimeout.millis());
>> // connect timeout is 1 second
>> if (!channelFuture.isSuccess()) {
>> throw new ConnectTransportException(address, "",
>> channelFuture.getCause());
>> }
>>
>>
>> First, is this code fragment correct, or am I missing something?
>> Second, the strange thing I get is that the channelFuture.getCause()
>> is null .., so I don't really know the underlying problem here.
>>
>> Any idea?
>>
>> Cheers,
>> Shay
>> _______________________________________________
>> netty-users mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Re: Random connect exceptions

2009-11-03T03:41:26Z

Hi Shay,

the code looks OK, but two things:

1) did you set the connectTimeout to the bootstrap, i.e.

clientBootstrap.setOption("connectTimeoutMillis",connectTimeout.millis());

if no that might be the reason: the connection attempt is just not finished yet

2) if yes do you use the latest Netty version? The NIO transport in
versions prior to 3.1.4 (if i remember correctly) did not respect the
connect timeout.

hth,
regards,
christian!

On Tue, Nov 3, 2009 at 12:19 PM, Shay Banon <[hidden email]> wrote:

> Hi,
>
> I am getting random connect failures when using netty, which I am
> trying to track down the cause of it. First, here is the code I use to
> connect:
>
> InetSocketAddress address = // obtain the address to connect to
> ChannelFuture channelFuture = clientBootstrap.connect(address);
> channelFuture.awaitUninterruptibly(connectTimeout.millis());
> // connect timeout is 1 second
> if (!channelFuture.isSuccess()) {
> throw new ConnectTransportException(address, "",
> channelFuture.getCause());
> }
>
>
> First, is this code fragment correct, or am I missing something?
> Second, the strange thing I get is that the channelFuture.getCause()
> is null .., so I don't really know the underlying problem here.
>
> Any idea?
>
> Cheers,
> Shay
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

Random connect exceptions

2009-11-03T03:19:49Z

Hi,

I am getting random connect failures when using netty, which I am
trying to track down the cause of it. First, here is the code I use to
connect:

InetSocketAddress address = // obtain the address to connect to
ChannelFuture channelFuture = clientBootstrap.connect(address);
channelFuture.awaitUninterruptibly(connectTimeout.millis());
// connect timeout is 1 second
if (!channelFuture.isSuccess()) {
throw new ConnectTransportException(address, "",
channelFuture.getCause());
}

First, is this code fragment correct, or am I missing something?
Second, the strange thing I get is that the channelFuture.getCause()
is null .., so I don't really know the underlying problem here.

Any idea?

Cheers,
Shay
_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

couple of questions i come up with.

2009-11-02T23:30:06Z

Hi Netty Developer Group,

i am trying to get confortable with Netty(3.1.5). Here are a couple of questions i come up with.

1. Data comes into netty, it goes up and down through handlers. There may be many ChannelUpstreamHandler and ChannelDownstreamHandler handlers. In all those handlers,in general, should i use Channels.fireMessageReceived only once? if so when? what happens if Channels.fireMessageReceived is called more then once in up handlers?

2. What should i do when my decoder constructed by more than one handler? Is it required that Decoder must be implemented in one handler?

3. Why FrameDecoder.callDecode throws this Exception?

throw new IllegalStateException(
"decode() method must read at least one byte " +
"if it returned a frame (caused by: " + getClass() + ")");

Thank you very much.

Re: demultiplexing protobuf messages

2009-11-07T07:02:25Z

You are right. However, if the decoder is placed after
ExecutionHandler, decoder might be called from different threads while
it's not called simultaneously, as described in
OrderedMemoryAwareThreadPoolExecutor Javadoc. If your decoder is
placed before ExecutionHandler or you don't use ExecutionHandler at
all, it's safe to define the member variables as non-volatile.

HTH

— Trustin Lee, http://gleamynode.net/

On Wed, Nov 4, 2009 at 11:41 PM, sandstorm_sh <[hidden email]> wrote:

>
> Thanks to both of you.
>
> Why are those fields volatile ?
>
> Isnt there a handler/decoder instance per pipeline/connection ?
> --
> View this message in context: http://n2.nabble.com/demultiplexing-protobuf-messages-tp3931395p3945572.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Re: demultiplexing protobuf messages

2009-11-04T06:41:22Z

Thanks to both of you.

Why are those fields volatile ?

Isnt there a handler/decoder instance per pipeline/connection ?

Re: demultiplexing protobuf messages

2009-11-03T01:00:27Z

On Mon, Nov 2, 2009 at 9:04 PM, sandstorm_sh <[hidden email]> wrote:

>
> How can we demultiplex protobuf messages, given the fact that a message has
> not any type defining header information in itself, and the protocol we are
> trying to define is message-flexible so to say, i.e. has not got any fixed
> message sequence hardcoded into the protocol definition.
>
> one solution is to prepend the type header like a lengthfieldprepender
> prepends a length field, and to decode the message with an appropriate
> protobuf decoder.
>
> we can maintain a collection of decoders for the corresponding types, but
> the problem is that the decode method is protected, not public, and we can
> not invoke it.
> so we should use the pipeline somehow, and my question is what is the
> standard idiom for this particular situation: do we have to replace handlers
> in the pipeline, based on the incoming message type ?
>
> what is the standard thing to do ?

I would use DecoderEmbedder to decode the message whose type/length
prefix stripped out. It could look like this:

public class MyDecoder extends FrameDecoder {
private volatile boolean readingHeader = true;
private volatile int bodyLength;
private volatile DecoderEmbedder bodyDecoder;

protected Object decode(ctx, ch, buf) {
if (readingHeader) {
if (buf.readableByte() < 8) {
return null;
}
int messageType = buf.readInt();
bodyLength = buf.readInt();
switch (messageType) {
case 1:
bodyDecoder = new DecoderEmbedder(new MessageTypeOneDecoder());
break;
....
}
readingHeader = false;
}

if (!readingHeader) {
if (buf.readableBytes() < bodyLength) {
return;
}
bodyDecoder.offer(buf.readBytes(bodyLength));
readingHeader = true;
return bodyDecoder.poll();
}
}
}

I'm not sure the code above is perfect as I didn't run it by myself,
but it should show you some basic idea.

I also like Nicholas's idea, so I'd recommend it, too, if it's a
viable option for you.

HTH

— Trustin Lee, http://gleamynode.net/

>
>
> example code in the netty distribution contains protobuf examples, but there
> is one message type incoming and outgoing..
>
> im a bit newbie in netty, so i'd like some elaboration on this topic, thanks
> --
> View this message in context: http://n2.nabble.com/demultiplexing-protobuf-messages-tp3931395p3931395.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/netty-users
>

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

— Trustin Lee, http://gleamynode.net/

Re: demultiplexing protobuf messages

2009-11-02T06:20:10Z

Hi,

I'm not really an expert on Protocol Buffers, but I've been looking at them recently for a project of mine. One possibility is to have a fixed message type that has "optional" fields for each possible type of message that could come in. If the types of possible messages is too flexible to make this a reasonable option, protocol buffers does have an extension mechanism, although I am not too familiar with it, but from what I remember it should also be a possibility.

If you can't wrap all your messages in one container type, then I think the option you're suggesting is definitely another option.

Hope this helps a bit,
Nick

On Mon, Nov 2, 2009 at 2:04 PM, sandstorm_sh <[hidden email]> wrote:

How can we demultiplex protobuf messages, given the fact that a message has
not any type defining header information in itself, and the protocol we are
trying to define is message-flexible so to say, i.e. has not got any fixed
message sequence hardcoded into the protocol definition.

one solution is to prepend the type header like a lengthfieldprepender
prepends a length field, and to decode the message with an appropriate
protobuf decoder.

we can maintain a collection of decoders for the corresponding types, but
the problem is that the decode method is protected, not public, and we can
not invoke it.

so we should use the pipeline somehow, and my question is what is the
standard idiom for this particular situation: do we have to replace handlers
in the pipeline, based on the incoming message type ?

what is the standard thing to do ?

example code in the netty distribution contains protobuf examples, but there
is one message type incoming and outgoing..

im a bit newbie in netty, so i'd like some elaboration on this topic, thanks
--
View this message in context: http://n2.nabble.com/demultiplexing-protobuf-messages-tp3931395p3931395.html
Sent from the Netty User Group mailing list archive at Nabble.com.
_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users