POJOSE-4101: Allocated threads and blocking queue are full

[rjoshi]

When I ran a longevity test,  I see POJOSE throwing "POJOSE-4101: Allocated threads and blocking queue are full" error.

This error message happens to appear if load is too high. That is acceptable. But it also appears while longterm runs on a stable load causing in a short while system fully non-responsive and rejecting all messages. When this happens the CPU usage drops from 40-50% to 4%.

Here are the configurations.

ThreadPoolBlockingQueueSize=700
MaxThreadPoolSize=700
CoreThreadPoolSize=350

I use synchronous calls to communicate with BPEL.

[Girish Patil-2]

Some javascript/style in this post has been disabled (why?)

I am assuming "longeterm", you meant long running BPEL process.
POJO SE synch calls to long running BPEL Service/process will block the POJO SE threads, especially when POJO Service is waiting for BPEL process to respond with acknowledgment (for InOnly) or Output message (for InOut).

In this case async calls from POJO SE will block the the  thread, and not efficient. It is recommended to use asynch calls in this case.

We do not have sample on the Wiki yet, here is the driver test for reference[1].
-Girish

[1] http://fisheye5.cenqua.com/browse/open-jbi-components/driver-tests/pojose/asynch/InOuts

rjoshi wrote:

When I ran a longevity test,  I see POJOSE throwing "POJOSE-4101: Allocated
threads and blocking queue are full" error.

This error message happens to appear if load is too high. That is
acceptable. But it also appears while longterm runs on a stable load causing
in a short while system fully non-responsive and rejecting all messages.
When this happens the CPU usage drops from 40-50% to 4%.

Here are the configurations.

ThreadPoolBlockingQueueSize=700
MaxThreadPoolSize=700
CoreThreadPoolSize=350

I use synchronous calls to communicate with BPEL.
  

[Girish Patil-2]

Some javascript/style in this post has been disabled (why?)

Correction: Please  read below as "In this case synchronous calls from POJO SE will block the thread".

Girish Patil wrote:

I am assuming "longeterm", you meant long running BPEL process.
POJO SE synch calls to long running BPEL Service/process will block the POJO SE threads, especially when POJO Service is waiting for BPEL process to respond with acknowledgment (for InOnly) or Output message (for InOut).

In this case async calls from POJO SE will block the the  thread, and not efficient. It is recommended to use asynch calls in this case.

We do not have sample on the Wiki yet, here is the driver test for reference[1].
-Girish

[1] http://fisheye5.cenqua.com/browse/open-jbi-components/driver-tests/pojose/asynch/InOuts

rjoshi wrote:

When I ran a longevity test,  I see POJOSE throwing "POJOSE-4101: Allocated
threads and blocking queue are full" error.

This error message happens to appear if load is too high. That is
acceptable. But it also appears while longterm runs on a stable load causing
in a short while system fully non-responsive and rejecting all messages.
When this happens the CPU usage drops from 40-50% to 4%.

Here are the configurations.

ThreadPoolBlockingQueueSize=700
MaxThreadPoolSize=700
CoreThreadPoolSize=350

I use synchronous calls to communicate with BPEL.
  

[rjoshi]

Girish,
   Thanks for your quick response.  No, long term is related to the longevity test and not the long running BPEL process.  Our average request processing time (POJOSE<==>BPEL<==>BC) is less than 200 mill seconds.  Also number of requests per second is much lower than number of threads allocated.
This doesn't happen if I run test for less than an hour.  

Looking at the sample code for Asynchronous calls you provided, It seem it should be doable to implement asynchronous calls as router essentially implements a for loop and invokes multiple services. The exception for me is I can not invoke 2nd endpoint until response from 1st service comes back as 2nd BPEL service depends on the response from 1st BPEL service. So eventhough I invoke BPEL endpoints asynchronously, I have to synchronize and invokes services sequentially.

So comparing sample code, my for loop uses output of sendSyncInOut for input to the next service invocation.
e.g
Object in = nMsg;
for (int i=0; i< totalMsgsSent;i++){
                           
                     Object out = bpelSvc.sendSyncInOut(in);
                             in = out;
}

What is the best way to implement this where I can still use async calls (sendInOut) but use output of 1st service as input to the next.
Also few questions:


Question:  
1. Who triggers "OnDone" event? Because when it is done, I have some logic to execute billing log before I return the response.
2.  Below two functions are returning the response string.  Which function's return value will be used as a response to be sent to the client and what happens the the return data from other function?
1. public String theOperation(String input)
2. public String done()

3. When first endpoint request is invoked, what happens to that thread?  does it goes back to the pool for other request to be processed?

4. Is Onreply coming back to the same thread? If not than is the current context (ASynchInOutSvc class object) stored somewhere and when OnReply is invoked, the object is restored?

5. If answer to question 4 is not than how are we removing param from sendmsgs list in OnReply function which is local/notstatic variable to ?
this.sentMsgs.remove(param);


Thanks,
Rohit Joshi
 

[Girish Patil-2]

Some javascript/style in this post has been disabled (why?)

See inline for my comments.

rjoshi wrote:

Girish,
   Thanks for your quick response.  No, long term is related to the
longevity test and not the long running BPEL process.  Our average request
processing time (POJOSE<==>BPEL<==>BC) is less than 200 mill seconds.  Also
  

POJO Service is triggered by Http SOAP BC, like below?
Http Soap BC<==> POJOSE<==>BPEL<==>BC
If so then what is the processing time for a message triggering POJO service? 
Http Soap BC<==> POJOSE (multiple calls to BPEL) <==>BPEL<==>BC

number of requests per second is much lower than number of threads
allocated.
This doesn't happen if I run test for less than an hour.  So

1. do you think some garbage collection logic is not able to cleanup these
threads and eventually all threads are occupied? 
  

Not sure, seems like threads are not getting released somewhere, if POJO Service itself is taking less than minute and number of such requests are considerably less than number of threads.

2. Is there anyway to set timeout for synchronous request to avoid the
blocking of thread?
  

Currently no, I will add new method with timeout parameter.

3. Do you suggest any changes in threadpool/queue size configuration?

  

Could you file a bug, we need to investigate further what is the issue.

NOTE:  
*BPEL persistent is not enabled.
*Average request processing time is around 200 mill seconds and max
processing time is 1 second (on jmeter)
*All requests passing through NMR are two way.
*Here is the config:
ThreadPoolBlockingQueueSize=700
MaxThreadPoolSize=700
CoreThreadPoolSize=350
* POJO-SE is used as a complex router which invokes multiple BPEL services
synchronously and sequentially so it would be very difficult to alter the
logic to make it asynchronous calls. E.g POJO-SE invokes BPEL endpoints A, B
, C  using two ways requests and calls are synchronous which means it can
not invoke service B before response from A comes back.



Girish Patil wrote:
  

I am assuming "longeterm", you meant long running BPEL process.
POJO SE synch calls to long running BPEL Service/process will block 
the POJO SE threads, especially when POJO Service is waiting for BPEL 
process to respond with acknowledgment (for InOnly) or Output message 
(for InOut).

In this case async calls from POJO SE will block the the  thread, and 
not efficient. It is recommended to use asynch calls in this case.

We do not have sample on the Wiki yet, here is the driver test for 
reference[1].
-Girish

[1] 
http://fisheye5.cenqua.com/browse/open-jbi-components/driver-tests/pojose/asynch/InOuts

rjoshi wrote:
    

When I ran a longevity test,  I see POJOSE throwing "POJOSE-4101:
Allocated
threads and blocking queue are full" error.

This error message happens to appear if load is too high. That is
acceptable. But it also appears while longterm runs on a stable load
causing
in a short while system fully non-responsive and rejecting all messages.
When this happens the CPU usage drops from 40-50% to 4%.

Here are the configurations.

ThreadPoolBlockingQueueSize=700
MaxThreadPoolSize=700
CoreThreadPoolSize=350

I use synchronous calls to communicate with BPEL.
  
      

[Girish Patil]

rjoshi wrote:

e.g
Object in = nMsg;
for (int i=0; i< totalMsgsSent;i++){
                           
                     Object out = bpelSvc.sendSyncInOut(in);
                             in = out;
}

What is the best way to implement this where I can still use async calls (sendInOut) but use output of 1st service as input to the next.

Looks like usecase needs asynch calls with Sequential execution model[1].  Which is not yet implemented, we would like to have feedback on this. Please use issue #2065 to add your requirement related to this.

rjoshi wrote:

Question:  
1. Who triggers "OnDone" event? Because when it is done, I have some logic to execute billing log before I return the response.

It is called/executed by POJO SE engine when no more pending async response are there for the POJO Service instance.

rjoshi wrote:

2.  Below two functions are returning the response string.  Which function's return value will be used as a response to be sent to the client and what happens the the return data from other function?
1. public String theOperation(String input)
2. public String done()

OnDone's.  Currently Operation method's return value is ignored I think.

rjoshi wrote:

3. When first endpoint request is invoked, what happens to that thread?  does it goes back to the pool
for other request to be processed?

Yes, only after executing remaining logic after the call in the method.

rjoshi wrote:

4. Is Onreply coming back to the same thread? If not than is the current context (ASynchInOutSvc class object) stored somewhere and when OnReply is invoked, the object is restored?

Same POJO Service instance, not necessarily using the same thread. Yes, reference to your Service object is kept and used when response message arrived. Note currently POJO service objects are not serialized or persisted.

rjoshi wrote:

5. If answer to question 4 is not than how are we removing param from sendmsgs list in OnReply function which is local/notstatic variable to ?
this.sentMsgs.remove(param);

Same instance is used, so member variables/values will be the same.


[1] http://wiki.open-esb.java.net/Wiki.jsp?page=POJOSEConsumingAsynchronously
[2] https://open-esb.dev.java.net/issues/show_bug.cgi?id=2065