memoize and ConnectionStateErrors

[David Glick-2]

Is there any reason RAM caching using plone.memoize's @ram.cache  
decorator might result in sporadic ConnectionStateErrors?  I've been  
experiencing these lately, and they appeared around the same time I  
added memoization, so I'm suspicious.  (The methods I'm caching are  
returning a list of catalog brains, FWIW.)  If anyone knows a possible  
reason, or can give an explanation of what leads to this error in  
general, I'd be most grateful :)

This is with an older site running on Plone 3.0.3, which has a known  
PlacelessTranslationService issue that can result in  
ConnectionStateErrors, but I've switched to a newer PTS to incorporate  
the fix for that, so I think this is something else.

peace,

David Glick
Project Associate
ONE/Northwest

New tools and strategies for engaging people in protecting the  
environment

http://www.onenw.org
davidglick@...
[phone] x32

Subscribe to ONEList, our email newsletter!
Practical advice for effective online engagement
http://www.onenw.org/full_signup




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Alexander Pilz]

David Glick schrieb:
> Is there any reason RAM caching using plone.memoize's @ram.cache  
> decorator might result in sporadic ConnectionStateErrors?  I've been  
> experiencing these lately, and they appeared around the same time I  
> added memoization, so I'm suspicious.  (The methods I'm caching are  
> returning a list of catalog brains, FWIW.)  If anyone knows a possible  
> reason, or can give an explanation of what leads to this error in  
> general, I'd be most grateful :)
>  
I assume it is connected to the lazyness of the brains. If you cache
only the html result or a mapping which contains the values drawn from
the brains before you cache, it should work. Missing more knowledge on
the behaviour of the brains, I cannot tell you more.

br
alex

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Wichert Akkerman]

Previously David Glick wrote:
> Is there any reason RAM caching using plone.memoize's @ram.cache  
> decorator might result in sporadic ConnectionStateErrors?  I've been  
> experiencing these lately, and they appeared around the same time I  
> added memoization, so I'm suspicious.  (The methods I'm caching are  
> returning a list of catalog brains, FWIW.)  If anyone knows a possible  
> reason, or can give an explanation of what leads to this error in  
> general, I'd be most grateful :)

Only if you cache something that you should not cache, but the ram.cache
decorator uses RAMCache internally which should protect you from that.

I haven't seen any ConnectionState or other errors on deployments which
use ram.cache.

Wichert.

--
Wichert Akkerman <wichert@...>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Hanno Schlichting-2]

Hi.

David Glick wrote:
> Is there any reason RAM caching using plone.memoize's @ram.cache  
> decorator might result in sporadic ConnectionStateErrors?  I've been  
> experiencing these lately, and they appeared around the same time I  
> added memoization, so I'm suspicious.  (The methods I'm caching are  
> returning a list of catalog brains, FWIW.)  If anyone knows a possible  
> reason, or can give an explanation of what leads to this error in  
> general, I'd be most grateful :)

 From the docstring of zope.app.cache.ram:

The idea behind the `RAMCache` is that it should be shared between
threads, so that the same objects are not cached in each thread. This is
achieved by storing the cache data structure itself as a module level
variable (`RAMCache.caches`). This, of course, requires locking on
modifications of that data structure.

Now what does this tell you? Do not store objects in a RAM cache that
belong to a ZODB connection. Any peristent object including wrappers
around those, like catalog brains are a definitive no-no!

Often you won't see the connection state errors at first, since the
objects are all still in the thread local ZODB cache of all threads. Try
lowering your cache-size from the default 5000 to something like 100 and
see what you get. This should warn you a lot earlier (i.e. during
development) when you are doing something wonky ;)

Hanno


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Reinout van Rees]

Hanno Schlichting wrote:

> Now what does this tell you? Do not store objects in a RAM cache that
> belong to a ZODB connection. Any peristent object including wrappers
> around those, like catalog brains are a definitive no-no!

Can I have a definition of "belongs to a ZODB connection"?

This sounds like you cannot ramcache a list of brains, or a couple of
objects but only rendered results? Something like that?


Reinout

--
Reinout van Rees          Blog: http://vanrees.org/weblog/
reinout@...          Work: http://zestsoftware.nl/
http://vanrees.org          Video: http://reinout.blip.tv/


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Andreas Jung-5]

--On 17. Juli 2008 17:38:13 +0200 Reinout van Rees <reinout@...>
wrote:

> Hanno Schlichting wrote:
>
>> Now what does this tell you? Do not store objects in a RAM cache that
>> belong to a ZODB connection. Any peristent object including wrappers
>> around those, like catalog brains are a definitive no-no!
>
> Can I have a definition of "belongs to a ZODB connection"?

Basically everything that smells like an acquisition wrapped object.

>
> This sounds like you cannot ramcache a list of brains, or a couple of
> objects but only rendered results?

Basically no - at least no with a module-level cache.

> Something like that?

For result sets or similar beasts I would try to dictify the result set and
store the stuff in some cache. If you have deal in some one with persistent
objects I would assume that the ZODB cache takes care of the caching (in
most caches - depending on the cache settings).

Andreas

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Hanno Schlichting-2]

Reinout van Rees wrote:
> Hanno Schlichting wrote:
>
>> Now what does this tell you? Do not store objects in a RAM cache that
>> belong to a ZODB connection. Any peristent object including wrappers
>> around those, like catalog brains are a definitive no-no!
>
> Can I have a definition of "belongs to a ZODB connection"?

This is about the low-level stuff you usually don't have to care about.

If you invoke a ./bin/instance debug you can do the following:

 >>> app._p_jar
<Connection at 03726e30>
 >>> app._p_jar._db
<ZODB.DB.DB object at 0x1b92990>
 >>> app._p_jar.connections
{'main': <Connection at 03726e30>, 'temporary': <Connection at 0374ed70>}

So anything that has a non-empty _p_jar belongs to a connection. Every
persistent object you load from the ZODB is an example of that.

> This sounds like you cannot ramcache a list of brains, or a couple of
> objects but only rendered results? Something like that?

You cannot share persitent objects across threads, without loading them
from the ZODB over the connection from that thread.

So if you want to cache something, turn it into simple types, like
strings, dicts, list... obviously not lists of persistent objects - you
cannot cheat.

Hanno


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers@...
https://lists.sourceforge.net/lists/listinfo/plone-developers