OSGeo.org › OSGeo Software and Data Projects › GeoNetwork opensource › GeoNetwork developer

Re: [GeoNetwork-users] I-O Exception (Too many open files)

3 messages

Re: [GeoNetwork-users] I-O Exception (Too many open files)

Reply Threaded

Hi all, after some more tests with Sylvain on that
"java.io.FileNotFoundException" we could reproduce the problem on both 2.0.3
and trunk when stressing GeoNetwork (on linux, tomcat or jetty at least).

Problem occurs when too many search are made by lucene (you could try to use
apachebench "ab" to stress the app). Lucene opens index files but not close
thoses files in some cases. When the number of files opened by the java process
reach the system limit (ulimit -a), IOException occurs.

As said by Renaud, on option could be to increase the ulimit (see Lucene FAQ
also)
http://wiki.apache.org/lucene-java/LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82

Another (and better) option is to close the IndexSearcher/Reader in
LuceneSearcher once search ended.

A comment on the close method in LuceneSearcher said "RGFIX: check this" and
close only the reader. It looks like it's better to close both searcher and
reader once the search is done. Then the session use only the hits element.
Any comments on that?

Applying the patch attached will solve the problem. If that sounds good to
everyone, it could be applied for 2.3
as not so many users seem to face the problem for the time being ?

Ciao. Francois

PS: Thanks a lot to Sylvain for his help on tracking/solving that bug ...

[GeoNetwork_tooManyOpenFiles.patch]

Index: org/fao/geonet/kernel/search/LuceneSearcher.java =================================================================== --- org/fao/geonet/kernel/search/LuceneSearcher.java (revision 1210) +++ org/fao/geonet/kernel/search/LuceneSearcher.java (working copy) @@ -95,6 +95,7 @@ public void search(ServiceContext srvContext, Element request, ServiceConfig config) throws Exception { + computeQuery(srvContext, request, config); performQuery(request); initSearchRange(srvContext); @@ -179,11 +180,12 @@ public void close() { - try - { + try { + _searcher.close(); _reader.close(); - } - catch (IOException e) { e.printStackTrace(); } // DEBUG + } catch (IOException e) { + e.printStackTrace(); + } // DEBUG } //-------------------------------------------------------------------------------- @@ -267,11 +269,18 @@ _reader = IndexReader.open(_sm.getLuceneDir()); _searcher = new IndexSearcher(_reader); - _hits = _searcher.search(_query, sort); + + try { + _hits = _searcher.search(_query, sort); - Log.debug(Geonet.SEARCH_ENGINE, "Hits found : "+ _hits.length()); + Log.debug(Geonet.SEARCH_ENGINE, "Hits found : "+ _hits.length()); - makeSummary(); + makeSummary(); + + } finally { + close (); + } + setValid(true); }

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Jeroen Ticheler-3

Re: [GeoNetwork-users] I-O Exception (Too many open files)

Reply Threaded

+1 from me to fix that in 2.3. We can consider also applying it to
2.2.1 if it works well.
Ciao,
Jeroen

On Mar 26, 2008, at 6:10 PM, Francois-Xavier Prunayre wrote:

> Hi all, after some more tests with Sylvain on that
> "java.io.FileNotFoundException" we could reproduce the problem on
> both 2.0.3
> and trunk when stressing GeoNetwork (on linux, tomcat or jetty at
> least).
>
> Problem occurs when too many search are made by lucene (you could
> try to use
> apachebench "ab" to stress the app). Lucene opens index files but
> not close
> thoses files in some cases. When the number of files opened by the
> java process
> reach the system limit (ulimit -a), IOException occurs.
>
> As said by Renaud, on option could be to increase the ulimit (see
> Lucene FAQ
> also)
> http://wiki.apache.org/lucene-java/LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82
>
> Another (and better) option is to close the IndexSearcher/Reader in
> LuceneSearcher once search ended.
>
> A comment on the close method in LuceneSearcher said "RGFIX: check
> this" and
> close only the reader. It looks like it's better to close both
> searcher and
> reader once the search is done. Then the session use only the hits
> element.
> Any comments on that?
>
> Applying the patch attached will solve the problem. If that sounds
> good to
> everyone, it could be applied for 2.3
> as not so many users seem to face the problem for the time being ?
>
> Ciao. Francois
>
> PS: Thanks a lot to Sylvain for his help on tracking/solving that
> bug ...
> <
> GeoNetwork_tooManyOpenFiles
> .patch
> >
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Heikki Doeleman

Re: [GeoNetwork-users] I-O Exception (Too many open files)

Reply Threaded

hi,

we've had the "Too many open files" error on NGR. Though I'm not sure what part of the code caused it, after discussing with François I've applied the following fix to the code, in trunk and GN2.4 branches:

try {
computeQuery(srvContext, request, config);
performQuery(request, srvContext!=null?true:false);
initSearchRange(srvContext);
}
finally {
_hits = null;
_searcher.close();
_searcher = null;
setValid(false);
}

This forces the clean up of the resources after search, even if some exception occurred during it. Now I don't know if this fixes the problem at NGR, but at least it doesn't harm.

Kind regards
Heikki Doeleman

Jeroen Ticheler-3 wrote:

+1 from me to fix that in 2.3. We can consider also applying it to
2.2.1 if it works well.
Ciao,
Jeroen

On Mar 26, 2008, at 6:10 PM, Francois-Xavier Prunayre wrote:

> Hi all, after some more tests with Sylvain on that
> "java.io.FileNotFoundException" we could reproduce the problem on
> both 2.0.3
> and trunk when stressing GeoNetwork (on linux, tomcat or jetty at
> least).
>
> Problem occurs when too many search are made by lucene (you could
> try to use
> apachebench "ab" to stress the app). Lucene opens index files but
> not close
> thoses files in some cases. When the number of files opened by the
> java process
> reach the system limit (ulimit -a), IOException occurs.
>
> As said by Renaud, on option could be to increase the ulimit (see
> Lucene FAQ
> also)
> http://wiki.apache.org/lucene-java/LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82
>
> Another (and better) option is to close the IndexSearcher/Reader in
> LuceneSearcher once search ended.
>
> A comment on the close method in LuceneSearcher said "RGFIX: check
> this" and
> close only the reader. It looks like it's better to close both
> searcher and
> reader once the search is done. Then the session use only the hits
> element.
> Any comments on that?
>
> Applying the patch attached will solve the problem. If that sounds
> good to
> everyone, it could be applied for 2.3
> as not so many users seem to face the problem for the time being ?
>
> Ciao. Francois
>
> PS: Thanks a lot to Sylvain for his help on tracking/solving that
> bug ...
> <
> GeoNetwork_tooManyOpenFiles
> .patch
> >
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork