OSGeo.org › OSGeo Software and Data Projects › GeoNetwork opensource › GeoNetwork developer

Geonetworks concurrency errors

2 messages Options
Embed this post
Permalink
James Wilson

Geonetworks concurrency errors

Reply Threaded More More options
Print post
Permalink
Hi,

I'm discovering more concurrency errors under load in geonetworks. BTW - thanks for CSW fix for error I reported earlier in week.

The errors are for geonetworks 2_4_1 in Tomcat 6.20 with 1000 iso records loaded:

- Massive Delete fails with java.util.ConcurrentModificationException - see stack trace below. This results in inconsistency between lucene index and database. Lucene index drops records up to point at which exception occurs. Since commit is never done in database, data still exists, and can be recovered by rebuilding lucene index.
- MassiveUpdatePriviledges fails with java.util.ConcurrentModificationException - see stack trace below
- MassiveUpdateCategories - got no stack trace recorded for this one.

Problem seems to be with HashSet declared in SelectionManager. These are wrapped as  synchronizedSet, which requires all iterators to be in synchronized blocks (according to Java doc).

Iterators over this map in MassiveDelete / etc are not synchronized. Explicitly synchronizing the iterators when these sets are accessed seems to fix the concurrency problems:

eg

synchronized(sm.getSelection("metadata")) {

        // use iterator here
}

Regards

James

Stack traces:

First the massive delete:


88562 [34658350@qtp0-6] ERROR jeeves.service  - Exception when executing servic
e
788562 [34658350@qtp0-6] ERROR jeeves.service  -  (C) Exc : java.util.Concurrent
ModificationException
788562 [34658350@qtp0-6] DEBUG jeeves.service  - Raised exception while executin
g service
<error id="error">
  <message />
  <class>ConcurrentModificationException</class>
  <stack>
    <at class="java.util.HashMap$HashIterator" file="HashMap.java" line="921" me
thod="nextEntry" />
    <at class="java.util.HashMap$KeyIterator" file="HashMap.java" line="956" met
hod="next" />
    <at class="org.fao.geonet.services.metadata.MassiveDelete" file="MassiveDele
te.java" line="87" method="exec" />
    <at class="jeeves.server.dispatchers.ServiceInfo" file="ServiceInfo.java" li
ne="238" method="execService" />
    <at class="jeeves.server.dispatchers.ServiceInfo" file="ServiceInfo.java" li
ne="141" method="execServices" />
    <at class="jeeves.server.dispatchers.ServiceManager" file="ServiceManager.ja
va" line="377" method="dispatch" />
    <at class="jeeves.server.JeevesEngine" file="JeevesEngine.java" line="621" m
ethod="dispatch" />
    <at class="jeeves.server.sources.http.JeevesServlet" file="JeevesServlet.jav
a" line="174" method="execute" />
    <at class="jeeves.server.sources.http.JeevesServlet" file="JeevesServlet.jav
a" line="89" method="doGet" />
    <at class="javax.servlet.http.HttpServlet" file="HttpServlet.java" line="707
" method="service" />
  </stack>
  <request>
    <language>en</language>
    <service>metadata.massive.delete</service>
  </request>
</error>

Then the massive Privs:

1454281 [http-8080-3] ERROR jeeves.service  - Exception when executing service
1454281 [http-8080-3] ERROR jeeves.service  -  (C) Exc : java.util.ConcurrentMod
ificationException
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:921)
        at java.util.HashMap$KeyIterator.next(HashMap.java:956)
        at org.fao.geonet.services.metadata.MassiveUpdatePrivileges.exec(Massive
UpdatePrivileges.java:85)
        at jeeves.server.dispatchers.ServiceInfo.execService(ServiceInfo.java:24
4)
        at jeeves.server.dispatchers.ServiceInfo.execServices(ServiceInfo.java:1
41)
        at jeeves.server.dispatchers.ServiceManager.dispatch(ServiceManager.java
:377)
        at jeeves.server.JeevesEngine.dispatch(JeevesEngine.java:621)
        at jeeves.server.sources.http.JeevesServlet.execute(JeevesServlet.java:1
74)
        at jeeves.server.sources.http.JeevesServlet.doGet(JeevesServlet.java:89)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:849)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:45
4)
        at java.lang.Thread.run(Thread.java:619)

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Simon Pigot-2

Re: Geonetworks concurrency errors

Reply Threaded More More options
Print post
Permalink
Thanks James - see ticket 161 - same problem would have turned up in a
few other places too!

Cheers,
Simon

James Wilson wrote:

> Hi,
>
> I'm discovering more concurrency errors under load in geonetworks. BTW
> - thanks for CSW fix for error I reported earlier in week.
>
> The errors are for geonetworks 2_4_1 in Tomcat 6.20 with 1000 iso
> records loaded:
>
> - Massive Delete fails with java.util.ConcurrentModificationException
> - see stack trace below. This results in inconsistency between lucene
> index and database. Lucene index drops records up to point at which
> exception occurs. Since commit is never done in database, data still
> exists, and can be recovered by rebuilding lucene index.
> - MassiveUpdatePriviledges fails with
> java.util.ConcurrentModificationException - see stack trace below
> - MassiveUpdateCategories - got no stack trace recorded for this one.
>
> Problem seems to be with HashSet declared in SelectionManager. These
> are wrapped as  synchronizedSet, which requires all iterators to be in
> synchronized blocks (according to Java doc).
>
> Iterators over this map in MassiveDelete / etc are not synchronized.
> Explicitly synchronizing the iterators when these sets are accessed
> seems to fix the concurrency problems:
>
> eg
>
> synchronized(sm.getSelection("metadata")) {
>
>         // use iterator here
> }
>
> Regards
>
> James
>
> Stack traces:
>
> First the massive delete:
>
>
> 88562 [34658350@qtp0-6] ERROR jeeves.service  - Exception when
> executing servic
> e
> 788562 [34658350@qtp0-6] ERROR jeeves.service  -  (C) Exc :
> java.util.Concurrent
> ModificationException
> 788562 [34658350@qtp0-6] DEBUG jeeves.service  - Raised exception
> while executin
> g service
> <error id="error">
>   <message />
>   <class>ConcurrentModificationException</class>
>   <stack>
>     <at class="java.util.HashMap$HashIterator" file="HashMap.java"
> line="921" me
> thod="nextEntry" />
>     <at class="java.util.HashMap$KeyIterator" file="HashMap.java"
> line="956" met
> hod="next" />
>     <at class="org.fao.geonet.services.metadata.MassiveDelete"
> file="MassiveDele
> te.java" line="87" method="exec" />
>     <at class="jeeves.server.dispatchers.ServiceInfo"
> file="ServiceInfo.java" li
> ne="238" method="execService" />
>     <at class="jeeves.server.dispatchers.ServiceInfo"
> file="ServiceInfo.java" li
> ne="141" method="execServices" />
>     <at class="jeeves.server.dispatchers.ServiceManager"
> file="ServiceManager.ja
> va" line="377" method="dispatch" />
>     <at class="jeeves.server.JeevesEngine" file="JeevesEngine.java"
> line="621" m
> ethod="dispatch" />
>     <at class="jeeves.server.sources.http.JeevesServlet"
> file="JeevesServlet.jav
> a" line="174" method="execute" />
>     <at class="jeeves.server.sources.http.JeevesServlet"
> file="JeevesServlet.jav
> a" line="89" method="doGet" />
>     <at class="javax.servlet.http.HttpServlet" file="HttpServlet.java"
> line="707
> " method="service" />
>   </stack>
>   <request>
>     <language>en</language>
>     <service>metadata.massive.delete</service>
>   </request>
> </error>
>
> Then the massive Privs:
>
> 1454281 [http-8080-3] ERROR jeeves.service  - Exception when executing
> service
> 1454281 [http-8080-3] ERROR jeeves.service  -  (C) Exc :
> java.util.ConcurrentMod
> ificationException
> java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:921)
>         at java.util.HashMap$KeyIterator.next(HashMap.java:956)
>         at
> org.fao.geonet.services.metadata.MassiveUpdatePrivileges.exec(Massive
> UpdatePrivileges.java:85)
>         at
> jeeves.server.dispatchers.ServiceInfo.execService(ServiceInfo.java:24
> 4)
>         at
> jeeves.server.dispatchers.ServiceInfo.execServices(ServiceInfo.java:1
> 41)
>         at
> jeeves.server.dispatchers.ServiceManager.dispatch(ServiceManager.java
> :377)
>         at jeeves.server.JeevesEngine.dispatch(JeevesEngine.java:621)
>         at
> jeeves.server.sources.http.JeevesServlet.execute(JeevesServlet.java:1
> 74)
>         at
> jeeves.server.sources.http.JeevesServlet.doGet(JeevesServlet.java:89)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
> icationFilterChain.java:290)
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
> ilterChain.java:206)
>         at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
> alve.java:233)
>         at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
> alve.java:191)
>         at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
> ava:128)
>         at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
> ava:102)
>         at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
> ve.java:109)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
> a:293)
>         at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
> :849)
>         at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
> ss(Http11Protocol.java:583)
>         at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:45
> 4)
>         at java.lang.Thread.run(Thread.java:619)
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> ------------------------------------------------------------------------
>
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Free Embeddable Forum Powered by Nabble Help