Peter Koppatz wrote at 2008-7-20 16:04 -0000:
>> Es ist nicht im offiziellen Code.
>> Ich glaube, aber einen Patch zu haben.
>
>Ich wäre interessiert und würde den Patch gerne verwenden/testen.
Unser SVN hat mir gesagt, dass ich mit der Implementierung
noch nicht fertig geworden bin (es sagt mir
"partial -- feature implementation not yet finished").
Offenbar ist etwas mit höherer Priorität dazwischen gekommen.
Insbesondere gibt es noch keinen Test, der verifiziert,
dass auch tatsächlich alles funktioniert wie es soll.
Aber Du kannst ja mal probieren.
Hier ist der Patch so weit, wie er aktuell in unserem
SVN ist.
Index: feature_discard_cache_rather_verify.wiki
===================================================================
--- feature_discard_cache_rather_verify.wiki (revision 0)
+++ feature_discard_cache_rather_verify.wiki (revision 1592)
@@ -0,0 +1,60 @@
+|<bgcolor="#b0c0f4" style="text-align: center;" |2> ''' ZeoDiscardCacheRatherVerify''' || Ersteller: '' ["maurerd"]'' ||
+|| Erstellt am: [[Date(2008-05-06T11:48:14Z)]] ||
+||||<style="text-align: center;"> Discard the ZEO client cache rather than verify it. ||
+----
+
+= Table of Contents =
+[[TableOfContents]]
+
+= Introduction =
+The ZEO `ClientStorage` maintains a cache file (the so called ZEO client cache)
+in order to reduce network latency. As usual, larger caches have the chance
+to reduce mean latency.
+
+Under normal operation, the ZEO server informs all connected `ClientStorage`s
+about object modifications. The client invalidates its cache entries
+as necessary.
+However, when a `ClientStorage` looses its connection to the ZEO server,
+its client cache may get out of sync with the server.
+After reconnection, the client therefore needs to verify its cache.
+
+When the connection loss is caused by temporary network failures,
+the ZEO protocol usually provides for an efficient verification
+method: the server maintains the recently committed transactions
+and the objects thereby invalidated and can inform the client
+in an efficient way. However, when the connection loss was too far
+in the past or caused by a ZEO restart, then the client verifies
+each object in its cache. Especially, for problems caused by
+a ZEO server restart, this can take ages (in the order of hours)
+(as all clients concurrently
+verify their caches). During the cache verification, a client
+raises `DisconnectedError` and is unusable -- a very unsatisfactory
+situation.
+
+= Feature =
+Define a new `zeoclient` boolean configuration option
+`drop-cache-rather-verify` with default `false` to
+indicate that the cache should be dropped (i.e. a new empty one created)
+instead of an expensive verification.
+
+= Example Use Cases =
+Allows to use large client cache files without the fear that
+a ZEO server restart leads to hours of unavailability.
+
+[[BR]]
+[[BR]]
+[[BR]]
+[[BR]]
+[[BR]]
+[[BR]]
+[[BR]]
+[[BR]]
+
+------
+[wiki:/Diskussion Diskussionsseite][[BR]]
+[[AttachTable]]
+----
+
+[[BR]][[BR]]
+
+
Index: lib/python/ZEO/ClientStorage.py
===================================================================
--- lib/python/ZEO/ClientStorage.py (revision 1501)
+++ lib/python/ZEO/ClientStorage.py (revision 1592)
@@ -115,6 +115,10 @@
wait=None, wait_timeout=None,
read_only=0, read_only_fallback=0,
username='', password='', realm=None,
+ # DM 2006-06-12: when the ZEO server restarts, cache verification
+ # may require a really long time. We add an option to
+ # drop the cache instead
+ drop_cache_rather_verify=False,
blob_dir=None, shared_blob_dir=False):
"""ClientStorage constructor.
@@ -188,6 +192,9 @@
realm -- not documented.
+ drop_cache_rather_verify -- a flag indicating that the cache
+ should be dropped rather than expensively verified.
+
blob_dir -- directory path for blob data. 'blob data' is data that
is retrieved via the loadBlob API.
@@ -210,6 +217,17 @@
if debug:
log2("ClientStorage(): debug argument is no longer used")
+ # DM 2006-06-12: when the ZEO server restarts, cache verification
+ # may require a really long time. We add an option to
+ # drop the cache instead
+ # Remember some parameters for "_setupCache"
+ self._var_ = var
+ self._storage_ = storage
+ self._client_ = client
+ self._cache_size_ = cache_size
+
+ self._drop_cache_rather_verify = drop_cache_rather_verify
+
# wait defaults to True, but wait_for_server_on_startup overrides
# if not None
if wait_for_server_on_startup is not None:
@@ -333,15 +351,10 @@
else:
self.fshelper = None
- # Decide whether to use non-temporary files
- if client is not None:
- dir = var or os.getcwd()
- cache_path = os.path.join(dir, "%s-%s.zec" % (client, storage))
- else:
- cache_path = None
- self._cache = self.ClientCacheClass(cache_path, size=cache_size)
- # TODO: maybe there's a better time to open the cache? Unclear.
- self._cache.open()
+ # DM 2006-06-12: when the ZEO server restarts, cache verification
+ # may require a really long time. We add an option to
+ # drop the cache instead
+ self._setupCache()
self._rpc_mgr = self.ConnectionManagerClass(addr, self,
tmin=min_disconnect_poll,
@@ -595,6 +608,35 @@
self._ready.set()
return "quick verification"
+ # DM 2006-06-12: when the ZEO server restarts, cache verification
+ # may require a really long time. We add an option to
+ # drop the cache instead
+
+ if not self._cache: # an empty cache
+ log2("No verification necessary (cache is empty)")
+ self._server = server
+ self._ready.set()
+ return "no verification"
+
+ if self._client_ is None and self._drop_cache_rather_verify:
+ log2("dropping cache")
+ # ATT: there might be a conceptual flaw: if a Connection
+ # cache contains an object not in the client cache, then
+ # it may not be invalidated
+ # ATT: not sure that this cannot lead to a deadlock
+ # because '_process_invalidations' may not be able to
+ # acquire '_lock'.
+ invalidations = [(oid, version, None) for oid, tid, version in self._cache.contents()]
+ # the next two lines are for optimization
+ self._cache.close()
+ self._setupCache()
+ self._process_invalidations(invalidations)
+ self._server = server
+ self._ready.set()
+ return "cache dropped"
+
+ # DM 2006-06-12: end
+
log2("Verifying cache")
# setup tempfile to hold zeoVerify results
self._tfile = tempfile.TemporaryFile(suffix=".inv")
@@ -1316,6 +1358,26 @@
end = endVerify
Invalidate = invalidateTrans
+ # DM 2006-06-12: when the ZEO server restarts, cache verification
+ # may require a really long time. We add an option to
+ # drop the cache instead
+ def _setupCache(self):
+ '''create and open the cache.'''
+ # Decide whether to use non-temporary files
+ var = self._var_
+ storage = self._storage_
+ client = self._client_
+ cache_size = self._cache_size_
+ if client is not None:
+ dir = var or os.getcwd()
+ cache_path = os.path.join(dir, "%s-%s.zec" % (client, storage))
+ else:
+ cache_path = None
+ self._cache = self.ClientCacheClass(cache_path, size=cache_size)
+ # TODO: maybe there's a better time to open the cache? Unclear.
+ self._cache.open()
+
+
def InvalidationLogIterator(fileobj):
unpickler = cPickle.Unpickler(fileobj)
while 1:
Index: lib/python/ZEO/component.xml
===================================================================
--- lib/python/ZEO/component.xml (revision 1501)
+++ lib/python/ZEO/component.xml (revision 1592)
@@ -102,6 +102,16 @@
<metadefault>$INSTANCE/var/ZEO.pid (or $clienthome/ZEO.pid)</metadefault>
</key>
+ <!-- DM 2006-06-12: added option -->
+ <key name="drop-cache-rather-verify" datatype="boolean"
+ required="no" default="false">
+ <description>
+ indicates that the cache should be dropped rather than
+ verified when the verification optimization is not
+ available (e.g. when the ZEO server restarted).
+ </description>
+ </key>
+
</sectiontype>
</component>
Index: lib/python/ZODB/config.py
===================================================================
--- lib/python/ZODB/config.py (revision 1501)
+++ lib/python/ZODB/config.py (revision 1592)
@@ -162,6 +162,10 @@
max_disconnect_poll=self.config.max_disconnect_poll,
wait=self.config.wait,
read_only=self.config.read_only,
+ # DM 2006-06-12: when the ZEO server restarts, cache verification
+ # may require a really long time. We add an option to
+ # drop the cache instead
+ drop_cache_rather_verify=self.config.drop_cache_rather_verify,
read_only_fallback=self.config.read_only_fallback)
class BDBStorage(BaseConfig):
Index: lib/python/ZODB/component.xml
===================================================================
--- lib/python/ZODB/component.xml (revision 1501)
+++ lib/python/ZODB/component.xml (revision 1592)
@@ -153,6 +153,13 @@
that are accepted by this server.
</description>
</key>
+ <!-- DM 2008-05-15: added -->
+ <key name="drop-cache-rather-verify" datatype="boolean" default="off">
+ <description>
+ A flag indicating whether the client cache should be dropped
+ instead of an expensive verification.
+ </description>
+ </key>
</sectiontype>
<sectiontype name="demostorage" datatype=".DemoStorage"
--
Dieter
_______________________________________________
zope mailing list
zope@...
https://mail.dzug.org/mailman/listinfo/zope
