Too many open files

3 messages

Too many open files

Reply Threaded

Friends,

All of a sudden on our production box we started seeing the following exception. We have no clue why this is happening. Any idea? Any help really appreciated.

Regards,
Sridhar

[#|2009-06-28T06:57:53.003-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=86;_ThreadName=filebc-ib-Thread-53;_RequestID=ae3a7896-6220-4ba0-b692-d28154ddc035;|
java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
at com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
at java.lang.Thread.run(Thread.java:619)
|#]

[#|2009-06-28T06:57:53.005-0500|SEVERE|sun-appserver9.1|com.sun.jbi.filebc.InboundMessageProcessor|_ThreadID=337;_ThreadName=filebc-ib-Thread-141443;_RequestID=3257c59b-9155-4100-abb6-10934d92fda5;|FILEBC-E00720: inonly message for operation {http://ocfo.gsa.gov/2009/wsdl/FixedAssetAccountingResponseProcess/FixedAssetAccountingResponseProcessNS}fixedAssetAccountingResponse failed processing, an exception was raised. Too many open files
java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:883)
at com.sun.jbi.filebc.InboundMessageProcessor.acquireLockOnTargetDir(InboundMessageProcessor.java:327)
at com.sun.jbi.filebc.InboundMessageProcessor.execute(InboundMessageProcessor.java:363)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:212)
at java.lang.Thread.run(Thread.java:619)
|#]

[#|2009-06-28T06:57:53.005-0500|SEVERE|sun-appserver9.1|com.sun.jbi.filebc.InboundMessageProcessor|_ThreadID=95;_ThreadName=filebc-ib-Thread-57;_RequestID=6d32b55e-655e-4797-b7ed-ec7ab266ce1b;|FILEBC-E00720: inonly message for operation {http://ocfo.gsa.gov/2009/wsdl/BuildingAssetAccountingService/BuildingAssetAccountingServiceNS}poll failed processing, an exception was raised. Too many open files
java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:883)
at com.sun.jbi.filebc.InboundMessageProcessor.acquireLockOnTargetDir(InboundMessageProcessor.java:327)
at com.sun.jbi.filebc.InboundMessageProcessor.execute(InboundMessageProcessor.java:363)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:212)
at java.lang.Thread.run(Thread.java:619)
|#]

[#|2009-06-28T06:57:53.005-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=337;_ThreadName=filebc-ib-Thread-141443;_RequestID=3257c59b-9155-4100-abb6-10934d92fda5;|
java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
at com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
at java.lang.Thread.run(Thread.java:619)
|#]

[#|2009-06-28T06:57:53.005-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=95;_ThreadName=filebc-ib-Thread-57;_RequestID=6d32b55e-655e-4797-b7ed-ec7ab266ce1b;|
java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
at com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
at java.lang.Thread.run(Thread.java:619)
|#]

[#|2009-06-28T06:57:53.313-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=341;_ThreadName=Timer-1;_RequestID=9dad85e2-f66e-4ff3-991b-03eacbe8a934;|
java.lang.NullPointerException
at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:967)
at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
at javax.management.timer.Timer.sendNotification(Timer.java:1237)
at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
|#]

[#|2009-06-28T06:57:53.925-0500|SEVERE|sun-appserver9.1|com.sun.jbi.filebc.InboundMessageProcessor|_ThreadID=93;_ThreadName=filebc-ib-Thread-54;_RequestID=2c0fd25e-dd0f-4d4c-b21d-69cda1192279;|FILEBC-E00720: inonly message for operation {http://ocfo.gsa.gov/2009/wsdl/BuildingReferenceDataService/BuildingReferenceDataServiceNS}poll failed processing, an exception was raised. Too many open files
java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:883)
at com.sun.jbi.filebc.InboundMessageProcessor.acquireLockOnTargetDir(InboundMessageProcessor.java:327)
at com.sun.jbi.filebc.InboundMessageProcessor.execute(InboundMessageProcessor.java:363)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:212)
at java.lang.Thread.run(Thread.java:619)
|#]

[#|2009-06-28T06:57:53.926-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=93;_ThreadName=filebc-ib-Thread-54;_RequestID=2c0fd25e-dd0f-4d4c-b21d-69cda1192279;|
java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
at com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
at com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
at java.lang.Thread.run(Thread.java:619)
|#]

Jim Fu

Re: Too many open files

Reply Threaded

Hi Sridhar,

there is a limit on open file handles on a os - some are fixed (set when
file system created) and some are settable.

since the limit is a per machine thing, there are a lot factors can
cause this, ...

looking withing the application (and the components involved), there are
two possible causes:

(1) if there is no open file handler leak - then it is a system tuning
issue - either lift the os limit (if possible) - or - tune the
application, e.g. jbi components has QoS "Throttling" to limit the
number of pending message routings - this will help to avoid hitting the
OS specific resource cap.

(2) if there is defect in certain components used in the application
which caused the open handler leak (either under normal message routing
or when some other error triggered the leak), then we need further info
to (a) identify the component and (b) identify the scenario

you mentioned it is from production, so if it is possible to narrow it
down to a reasonable small project to repro the issue?

regards
--Jim

dvsridhar wrote:

> Friends,
>
> All of a sudden on our production box we started seeing the following
> exception. We have no clue why this is happening. Any idea? Any help really
> appreciated.
>
> Regards,
> Sridhar
>
> [#|2009-06-28T06:57:53.003-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=86;_ThreadName=filebc-ib-Thread-53;_RequestID=ae3a7896-6220-4ba0-b692-d28154ddc035;|
> java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
> at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
> at
> com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
> at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
> at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
> at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2009-06-28T06:57:53.005-0500|SEVERE|sun-appserver9.1|com.sun.jbi.filebc.InboundMessageProcessor|_ThreadID=337;_ThreadName=filebc-ib-Thread-141443;_RequestID=3257c59b-9155-4100-abb6-10934d92fda5;|FILEBC-E00720:
> inonly message for operation
> {http://ocfo.gsa.gov/2009/wsdl/FixedAssetAccountingResponseProcess/FixedAssetAccountingResponseProcessNS}fixedAssetAccountingResponse
> failed processing, an exception was raised. Too many open files
> java.io.IOException: Too many open files
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:883)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.acquireLockOnTargetDir(InboundMessageProcessor.java:327)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.execute(InboundMessageProcessor.java:363)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:212)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2009-06-28T06:57:53.005-0500|SEVERE|sun-appserver9.1|com.sun.jbi.filebc.InboundMessageProcessor|_ThreadID=95;_ThreadName=filebc-ib-Thread-57;_RequestID=6d32b55e-655e-4797-b7ed-ec7ab266ce1b;|FILEBC-E00720:
> inonly message for operation
> {http://ocfo.gsa.gov/2009/wsdl/BuildingAssetAccountingService/BuildingAssetAccountingServiceNS}poll
> failed processing, an exception was raised. Too many open files
> java.io.IOException: Too many open files
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:883)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.acquireLockOnTargetDir(InboundMessageProcessor.java:327)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.execute(InboundMessageProcessor.java:363)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:212)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2009-06-28T06:57:53.005-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=337;_ThreadName=filebc-ib-Thread-141443;_RequestID=3257c59b-9155-4100-abb6-10934d92fda5;|
> java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
> at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
> at
> com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
> at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
> at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
> at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2009-06-28T06:57:53.005-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=95;_ThreadName=filebc-ib-Thread-57;_RequestID=6d32b55e-655e-4797-b7ed-ec7ab266ce1b;|
> java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
> at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
> at
> com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
> at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
> at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
> at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2009-06-28T06:57:53.313-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=341;_ThreadName=Timer-1;_RequestID=9dad85e2-f66e-4ff3-991b-03eacbe8a934;|
> java.lang.NullPointerException
> at
> com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
> at
> com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
> at
> com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
> at
> com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:967)
> at
> com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
> at
> javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
> at
> javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
> at
> javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
> at
> javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
> at javax.management.timer.Timer.sendNotification(Timer.java:1237)
> at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
> at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> |#]
>
> [#|2009-06-28T06:57:53.925-0500|SEVERE|sun-appserver9.1|com.sun.jbi.filebc.InboundMessageProcessor|_ThreadID=93;_ThreadName=filebc-ib-Thread-54;_RequestID=2c0fd25e-dd0f-4d4c-b21d-69cda1192279;|FILEBC-E00720:
> inonly message for operation
> {http://ocfo.gsa.gov/2009/wsdl/BuildingReferenceDataService/BuildingReferenceDataServiceNS}poll
> failed processing, an exception was raised. Too many open files
> java.io.IOException: Too many open files
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:883)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.acquireLockOnTargetDir(InboundMessageProcessor.java:327)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.execute(InboundMessageProcessor.java:363)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:212)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2009-06-28T06:57:53.926-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=93;_ThreadName=filebc-ib-Thread-54;_RequestID=2c0fd25e-dd0f-4d4c-b21d-69cda1192279;|
> java.net.UnknownHostException: b06r6s-eaaprd01: b06r6s-eaaprd01
> at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
> at
> com.sun.jbi.alerter.AlerterImpl.getPhysicalHostName(AlerterImpl.java:400)
> at com.sun.jbi.alerter.AlerterImpl.createEventObject(AlerterImpl.java:365)
> at com.sun.jbi.alerter.AlerterImpl.sendAlert(AlerterImpl.java:243)
> at com.sun.jbi.alerter.AlerterImpl.critical(AlerterImpl.java:147)
> at
> com.sun.jbi.filebc.InboundMessageProcessor.run(InboundMessageProcessor.java:218)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Trond Strømme

Re: Too many open files

Reply Threaded

In reply to this post by dvsridhar

If you easily want to see which files are open for a particular process
on unix you should get lsof, (http://people.freebsd.org/~abe/) if it is
not already installed, for windows use the sysinternals tools (process
explorer)

in unix/linux use the ulimit command to see how many files a process can
have open.

As you say "all of a sudden" you started seeing Too many open files this
means you have a slowly accumulating file handle leak. Not closing
input/output streams (or as I've seen with weblogic, InitialContext
objects) will lead to the situation
.trond

dvsridhar wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]