Out of memory issue - Glassfish ESB - server hangs up

3 messages

Out of memory issue - Glassfish ESB - server hangs up

Reply Threaded

Hi All,

we are using glassfish ESB as one of our Production servers. i am facing some out of memory exception in the server log due to which server is replying very slow. When i restarted the components then all the server hanged up and stopped working. Then i restarted the server and only then the problem was resolved.

Can any one please help me to find the root cause of the problem? Below are the details-

Server information-
-------------------
"Sun Glassfish Enterprise Server V2.1"
Version = Sun GlassFish Enterprise Server v2.1 (9.1.1) (build b60-fcs)
We are using a stand alone domain.

Hardware environment-
----------------------
AIX serverName 3 5 00CA883D4C00
JVM options- Xms/Xx both set to 2500m

Error log-
----------
java.lang.Exception: HTTPBC-E00753: HTTP POST request failed, portType {http://bpel.money.ge.com/wsdl/VisionPlus_CHE_V160}VisionPlusPort
URL: https://PCICCHGE:04178/
QUERY:
PATH_INFO:
Exception detail: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11

also i am getting the below IEP related error

[#|2009-10-30T17:17:13.574+1100|SEVERE|sun-appserver2.1|com.sun.jbi.engine.iep.core.runtime.IEPEngine|_ThreadID=22;_ThreadName=IEPSEHeartBeatT
hread;_RequestID=30d3aea9-4c3d-4739-94e7-1d84304de7a7;|IEPSEaja02000: DataSource not found in the context: jdbc/iepseDerbyNonXA|#]

But i have not configure the Derby nor i am using this.
Could it be reason for this?

Please let us know if you have any information for the same.
Thanks in advance.

Best regards,
~vishal bhardwaj.

vishal_with_me

Re: Out of memory issue - Glassfish ESB - server hangs up

Reply Threaded

Hi All,

Please send your suggestions for this as we are stucked here.
Today again we got the same issue in our Production server.

These are few lines from our server.log file
------------------------------------------------------------------------------------
[#|2009-11-03T22:10:04.902+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
[Fatal Error] :1:3: The markup in the document preceding the root element must be well-formed.
|#]

[#|2009-11-03T22:10:04.905+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=124;_ThreadName=HTTPBC-OutboundReceiver-9;_RequestID=0bdf9563-9f1e-4dfa-8604-91d1c0ba980a;|[Fatal Error] :1:3: The markup in the document preceding the root element must be well-formed.
|#]

[#|2009-11-03T22:10:04.917+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
java.util.logging.ErrorManager: 5: Error in formatting Logrecord|#]

[#|2009-11-03T22:10:04.917+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at java.text.MessageFormat.makeFormat(MessageFormat.java:1350)|#]

[#|2009-11-03T22:10:04.917+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at java.text.MessageFormat.applyPattern(MessageFormat.java:470)|#]

[#|2009-11-03T22:10:04.918+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at java.text.MessageFormat.<init>(MessageFormat.java:365)|#]

[#|2009-11-03T22:10:04.918+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at java.text.MessageFormat.format(MessageFormat.java:823)|#]

[#|2009-11-03T22:10:04.918+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:329)|#]

[#|2009-11-03T22:10:04.919+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at com.sun.jbi.httpsoapbc.OutboundMessageProcessor.processRequestReplyOutbound(OutboundMessageProcessor.java:676)|#]

[#|2009-11-03T22:10:04.919+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at com.sun.jbi.httpsoapbc.OutboundMessageProcessor.processMessage(OutboundMessageProcessor.java:249)|#]

[#|2009-11-03T22:10:04.923+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
at com.sun.enterprise.server.logging.AMXLoggingHook.publish(AMXLoggingHook.java:198)|#]
------------------------------------------------------------------------------------
Hardware environment-
OS - AIX
JVM - 32 Bit
8GB Memory RAM
4 Dual core processors

Glassfish ESB configurations-
HTTP BC threads(output/inbound) set to 100

I have captured the data from the JVisualVM and found that
Heap is in normal state and the size of used heap increases when server is in Hung state
HTTP BC worker threads all are in wait state
RMI TCP connection threads - all are in running state
HTTP JAX WS thread status – all are in monitor state

I have also configured the Hulp Profiler but didnt get any meaningful data, as when the server hung up - even the server.log file is not updated

I have captured the VMSTAT data also and it seems for the JAVA process there is an average use of 70-80 % of CPU, sometimes we found spikes and CPU usage went up to 95%.

when the server hungs, the admin console is accessible, from asadmin we cant perform any activity,
even when we tried to stop the server from asadmin, it didnt work. then finally we have to kill the glassfish process.

Please spare few minutes of your busy time and have a look into this.
Thanks in advance.

Best Regards,
~vishal bhardwaj.

Gabe-2

Re: Out of memory issue - Glassfish ESB - server hangs up

Reply Threaded

Vishal - if you have a support contract for GlassFish ESB you can engage
Sun support and services to help you out with this especially as it is a
production issue.
Thanks,
Gabe

vishal_with_me wrote:

> Hi All,
>
> Please send your suggestions for this as we are stucked here.
> Today again we got the same issue in our Production server.
>
> These are few lines from our server.log file
> ------------------------------------------------------------------------------------
> [#|2009-11-03T22:10:04.902+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> [Fatal Error] :1:3: The markup in the document preceding the root element
> must be well-formed.
> |#]
>
> [#|2009-11-03T22:10:04.905+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=124;_ThreadName=HTTPBC-OutboundReceiver-9;_RequestID=0bdf9563-9f1e-4dfa-8604-91d1c0ba980a;|[Fatal
> Error] :1:3: The markup in the document preceding the root element must be
> well-formed.
> |#]
>
> [#|2009-11-03T22:10:04.917+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> java.util.logging.ErrorManager: 5: Error in formatting Logrecord|#]
>
> [#|2009-11-03T22:10:04.917+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at java.text.MessageFormat.makeFormat(MessageFormat.java:1350)|#]
>
> [#|2009-11-03T22:10:04.917+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at java.text.MessageFormat.applyPattern(MessageFormat.java:470)|#]
>
> [#|2009-11-03T22:10:04.918+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at java.text.MessageFormat.<init>(MessageFormat.java:365)|#]
>
> [#|2009-11-03T22:10:04.918+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at java.text.MessageFormat.format(MessageFormat.java:823)|#]
>
> [#|2009-11-03T22:10:04.918+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at
> com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:329)|#]
>
> [#|2009-11-03T22:10:04.919+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at
> com.sun.jbi.httpsoapbc.OutboundMessageProcessor.processRequestReplyOutbound(OutboundMessageProcessor.java:676)|#]
>
> [#|2009-11-03T22:10:04.919+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at
> com.sun.jbi.httpsoapbc.OutboundMessageProcessor.processMessage(OutboundMessageProcessor.java:249)|#]
>
> [#|2009-11-03T22:10:04.923+1100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=123;_ThreadName=HTTPBC-OutboundReceiver-5;_RequestID=21382b18-352b-41f5-b8cd-3b0f0ef56b47;|
> at
> com.sun.enterprise.server.logging.AMXLoggingHook.publish(AMXLoggingHook.java:198)|#]
> ------------------------------------------------------------------------------------
> Hardware environment-
> OS - AIX
> JVM - 32 Bit
> 8GB Memory RAM
> 4 Dual core processors
>
> Glassfish ESB configurations-
> HTTP BC threads(output/inbound) set to 100
>
> I have captured the data from the JVisualVM and found that
> Heap is in normal state and the size of used heap increases when server is
> in Hung state
> HTTP BC worker threads all are in wait state
> RMI TCP connection threads - all are in running state
> HTTP JAX WS thread status – all are in monitor state
>
> I have also configured the Hulp Profiler but didnt get any meaningful data,
> as when the server hung up - even the server.log file is not updated
>
> I have captured the VMSTAT data also and it seems for the JAVA process there
> is an average use of 70-80 % of CPU, sometimes we found spikes and CPU usage
> went up to 95%.
>
> when the server hungs, the admin console is accessible, from asadmin we cant
> perform any activity,
> even when we tried to stop the server from asadmin, it didnt work. then
> finally we have to kill the glassfish process.
>
> Please spare few minutes of your busy time and have a look into this.
> Thanks in advance.
>
> Best Regards,
> ~vishal bhardwaj.
>
>

--
Gabriel Badescu
Open ESB Community (http://open-esb.org)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]