OSGeo.org › OSGeo Software and Data Projects › GeoNetwork opensource › GeoNetwork developer

Geonetwork concurrency errors

4 messages

James Wilson

Geonetwork concurrency errors

Reply Threaded

Hi,

I've been trying to do some load tests on geonetwork 2_4_1 csw getRecords service and I am getting java.util.ConcurrentModificationExceptions thrown by Jeeves.

What I've done is loaded up 1000 test records onto the server, and written a simple python client to sequentially fire CSW GetRecords queries at the server, asking for a randomly generated bounding box. For a single client, the server can handle ~ 60 requests / second.

If I fire up multiple instances of the client I occasionally get a java.util.ConcurrentModificationException error thrown and the client quits. I've spent some time looking through the code, and the error is thrown somewhere within CatalogDispatcher.java.

Although I doubt a production server would be hit this hard, it does cause some concern. Are there other subtle concurrency bugs out there waiting to bite, that have an effect on data integrity, for example.

The obvious (and painful) fix is to make the app single threaded, by synchronising the whole servlet (ie synchronize Jeeves doGet doPost). Of course, this introduces significant latency when the server gets loaded.

Has anyone else seen errors of this type or does anybody have suggestions as to how they could be tracked down - I don't have much expertise in concurrency issues.

Regards

James

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Heikki Doeleman

Re: Geonetwork concurrency errors

Reply Threaded

hi James,

is it possible you could post the complete stacktrace ?

I think there are solutions to the concurreny other than synchronizing the complete servlet. It will be interesting to find out where exactly the concurreny erorrors stem from.

Kind regards
Heikki Doeleman

On Tue, Oct 13, 2009 at 6:19 PM, James Wilson <[hidden email]> wrote:

Hi,

I've been trying to do some load tests on geonetwork 2_4_1 csw getRecords service and I am getting java.util.ConcurrentModificationExceptions thrown by Jeeves.

What I've done is loaded up 1000 test records onto the server, and written a simple python client to sequentially fire CSW GetRecords queries at the server, asking for a randomly generated bounding box. For a single client, the server can handle ~ 60 requests / second.

If I fire up multiple instances of the client I occasionally get a java.util.ConcurrentModificationException error thrown and the client quits. I've spent some time looking through the code, and the error is thrown somewhere within CatalogDispatcher.java.

Although I doubt a production server would be hit this hard, it does cause some concern. Are there other subtle concurrency bugs out there waiting to bite, that have an effect on data integrity, for example.

The obvious (and painful) fix is to make the app single threaded, by synchronising the whole servlet (ie synchronize Jeeves doGet doPost). Of course, this introduces significant latency when the server gets loaded.

Has anyone else seen errors of this type or does anybody have suggestions as to how they could be tracked down - I don't have much expertise in concurrency issues.

Regards

James

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Simon Pigot-2

Re: Geonetwork concurrency errors

Reply Threaded

Had a brief look at this in 2.4.1 using a couple of sessions that send
requests via curl.

Saw the following exception:

2009-10-14 14:19:23,058 DEBUG [jeeves.service] - Raised exception while
executing service
<error id="error">
<message />
<class>ConcurrentModificationException</class>
<stack>
<at class="java.util.HashMap$HashIterator" file="HashMap.java"
line="793" method="nextEntry" />
<at class="java.util.HashMap$KeyIterator" file="HashMap.java"
line="828" method="next" />
<at class="jeeves.resources.dbms.DbmsPool" file="DbmsPool.java"
line="173" method="close" />
<at class="jeeves.server.resources.ResourceManager"
file="ResourceManager.java" line="117" method="release" />
<at class="jeeves.server.resources.ResourceManager"
file="ResourceManager.java" line="83" method="close" />
<at class="jeeves.server.dispatchers.ServiceInfo"
file="ServiceInfo.java" line="245" method="execService" />
<at class="jeeves.server.dispatchers.ServiceInfo"
file="ServiceInfo.java" line="141" method="execServices" />
<at class="jeeves.server.dispatchers.ServiceManager"
file="ServiceManager.java" line="377" method="dispatch" />
<at class="jeeves.server.JeevesEngine" file="JeevesEngine.java"
line="621" method="dispatch" />
<at class="jeeves.server.sources.http.JeevesServlet"
file="JeevesServlet.java" line="174" method="execute" />
</stack>
<request>

Is this what you are seeing James?

Seems it can be fixed by making hsListeners into a Synchronized set in
DbmsPool.java - but this is a bit of shallow analysis because curiously
it doesn't seem to happen in an earlier version of the BlueNetMEST
(jetty 5 vs jetty 6 maybe?) which we've backported the CSW 2.0.2
implementation into.

Cheers,
Simon

heikki wrote:

> hi James,
>
> is it possible you could post the complete stacktrace ?
>
> I think there are solutions to the concurreny other than synchronizing
> the complete servlet. It will be interesting to find out where exactly
> the concurreny erorrors stem from.
>
> Kind regards
> Heikki Doeleman
>
>
>
> On Tue, Oct 13, 2009 at 6:19 PM, James Wilson
> <[hidden email] <mailto:[hidden email]>>
> wrote:
>
> Hi,
>
> I've been trying to do some load tests on geonetwork 2_4_1 csw
> getRecords service and I am getting
> java.util.ConcurrentModificationExceptions thrown by Jeeves.
>
> What I've done is loaded up 1000 test records onto the server, and
> written a simple python client to sequentially fire CSW GetRecords
> queries at the server, asking for a randomly generated bounding
> box. For a single client, the server can handle ~ 60 requests /
> second.
>
> If I fire up multiple instances of the client I occasionally get a
> java.util.ConcurrentModificationException error thrown and the
> client quits. I've spent some time looking through the code, and
> the error is thrown somewhere within CatalogDispatcher.java.
>
> Although I doubt a production server would be hit this hard, it
> does cause some concern. Are there other subtle concurrency bugs
> out there waiting to bite, that have an effect on data integrity,
> for example.
>
> The obvious (and painful) fix is to make the app single threaded,
> by synchronising the whole servlet (ie synchronize Jeeves doGet
> doPost). Of course, this introduces significant latency when the
> server gets loaded.
>
> Has anyone else seen errors of this type or does anybody have
> suggestions as to how they could be tracked down - I don't have
> much expertise in concurrency issues.
>
> Regards
>
> James
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year.
> Jumpstart your
> developing skills, take BlackBerry mobile applications to market
> and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> <mailto:[hidden email]>
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> ------------------------------------------------------------------------
>
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Simon Pigot-2

Re: Geonetwork concurrency errors

Reply Threaded

Or to put it more clearly: I've tested it and it doesn't happen when
hsListeners in DbmsPool.java is synchronized. But more analysis is required.

Simon Pigot wrote:

> Had a brief look at this in 2.4.1 using a couple of sessions that send
> requests via curl.
>
> Saw the following exception:
>
> 2009-10-14 14:19:23,058 DEBUG [jeeves.service] - Raised exception while
> executing service
> <error id="error">
> <message />
> <class>ConcurrentModificationException</class>
> <stack>
> <at class="java.util.HashMap$HashIterator" file="HashMap.java"
> line="793" method="nextEntry" />
> <at class="java.util.HashMap$KeyIterator" file="HashMap.java"
> line="828" method="next" />
> <at class="jeeves.resources.dbms.DbmsPool" file="DbmsPool.java"
> line="173" method="close" />
> <at class="jeeves.server.resources.ResourceManager"
> file="ResourceManager.java" line="117" method="release" />
> <at class="jeeves.server.resources.ResourceManager"
> file="ResourceManager.java" line="83" method="close" />
> <at class="jeeves.server.dispatchers.ServiceInfo"
> file="ServiceInfo.java" line="245" method="execService" />
> <at class="jeeves.server.dispatchers.ServiceInfo"
> file="ServiceInfo.java" line="141" method="execServices" />
> <at class="jeeves.server.dispatchers.ServiceManager"
> file="ServiceManager.java" line="377" method="dispatch" />
> <at class="jeeves.server.JeevesEngine" file="JeevesEngine.java"
> line="621" method="dispatch" />
> <at class="jeeves.server.sources.http.JeevesServlet"
> file="JeevesServlet.java" line="174" method="execute" />
> </stack>
> <request>
>
> Is this what you are seeing James?
>
> Seems it can be fixed by making hsListeners into a Synchronized set in
> DbmsPool.java - but this is a bit of shallow analysis because curiously
> it doesn't seem to happen in an earlier version of the BlueNetMEST
> (jetty 5 vs jetty 6 maybe?) which we've backported the CSW 2.0.2
> implementation into.
>
> Cheers,
> Simon
>
>
> heikki wrote:
>
>> hi James,
>>
>> is it possible you could post the complete stacktrace ?
>>
>> I think there are solutions to the concurreny other than synchronizing
>> the complete servlet. It will be interesting to find out where exactly
>> the concurreny erorrors stem from.
>>
>> Kind regards
>> Heikki Doeleman
>>
>>
>>
>> On Tue, Oct 13, 2009 at 6:19 PM, James Wilson
>> <[hidden email] <mailto:[hidden email]>>
>> wrote:
>>
>> Hi,
>>
>> I've been trying to do some load tests on geonetwork 2_4_1 csw
>> getRecords service and I am getting
>> java.util.ConcurrentModificationExceptions thrown by Jeeves.
>>
>> What I've done is loaded up 1000 test records onto the server, and
>> written a simple python client to sequentially fire CSW GetRecords
>> queries at the server, asking for a randomly generated bounding
>> box. For a single client, the server can handle ~ 60 requests /
>> second.
>>
>> If I fire up multiple instances of the client I occasionally get a
>> java.util.ConcurrentModificationException error thrown and the
>> client quits. I've spent some time looking through the code, and
>> the error is thrown somewhere within CatalogDispatcher.java.
>>
>> Although I doubt a production server would be hit this hard, it
>> does cause some concern. Are there other subtle concurrency bugs
>> out there waiting to bite, that have an effect on data integrity,
>> for example.
>>
>> The obvious (and painful) fix is to make the app single threaded,
>> by synchronising the whole servlet (ie synchronize Jeeves doGet
>> doPost). Of course, this introduces significant latency when the
>> server gets loaded.
>>
>> Has anyone else seen errors of this type or does anybody have
>> suggestions as to how they could be tracked down - I don't have
>> much expertise in concurrency issues.
>>
>> Regards
>>
>> James
>>
>>
>> ------------------------------------------------------------------------------
>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>> is the only developer event you need to attend this year.
>> Jumpstart your
>> developing skills, take BlackBerry mobile applications to market
>> and stay
>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>> http://p.sf.net/sfu/devconference
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> <mailto:[hidden email]>
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>>
>>
>> ------------------------------------------------------------------------
>>
>> ------------------------------------------------------------------------------
>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>> is the only developer event you need to attend this year. Jumpstart your
>> developing skills, take BlackBerry mobile applications to market and stay
>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>> http://p.sf.net/sfu/devconference
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
>>
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
>
>