Z3 Event coverage in Plone and Zope core

5 messages Options
Embed this post
Permalink
Darryl Dixon - Winterhouse Consulting-2

Z3 Event coverage in Plone and Zope core

Reply Threaded More More options
Print post
Permalink
List,

We have recently found many places in Plone where things change (eg,
content item has a different display view chosen, or the order of the
items in a folder is changed) but the Dublin Core 'modified' attribute is
not updated (nor, really, would it be appropriate for it to be updated).
We have implemented a new IAnnotation to track these
'almost-modifications' and are finding many places that require hotfixes
in order to emit IObjectModifiedEvents at appropriate times so this
annotation can be updated. Are there any concrete plans to improve event
coverage in Plone and Zope core, or is it more of a laissez faire
situation where if the itch is felt by someone appropriate they are
scratching it?

More generally, what are the feelings around creating a PLIP for (a) a new
IAnnotation for 'metadata modifications' and/or (b) more pervasive
emitting of Events ?

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Hanno Schlichting-4

Re: Z3 Event coverage in Plone and Zope core

Reply Threaded More More options
Print post
Permalink
On Wed, Nov 4, 2009 at 4:01 AM, Darryl Dixon - Winterhouse Consulting
<[hidden email]> wrote:
> We have implemented a new IAnnotation to track these
> 'almost-modifications' and are finding many places that require hotfixes
> in order to emit IObjectModifiedEvents at appropriate times so this
> annotation can be updated. Are there any concrete plans to improve event
> coverage in Plone and Zope core, or is it more of a laissez faire
> situation where if the itch is felt by someone appropriate they are
> scratching it?

If you are interested in any kind of low-level data changes, you could
just use the persistent modification time as found in _p_mtime.

I'd be hesitant to put a gazillion of new event notifications into all
places, especially if it is such a general event as
IObjectModifiedEvent. We are already doing quite a bunch of work
during any such event and emitting it way more often will slow things
down considerably.

So far we have introduced new specific events whenever there was a
real concrete use-case for them. For example we introduced specific
workflow transition events into DCWorkflow at some point.

I feel that continuing on this road, where we introduce events only
for specific areas and where they either replace an existing hook
mechanism (as was the case in DCWorkflow) or there is a clear use-case
for them serves us better.

But maybe you have a concrete use-case in mind, that you might want to
share with us ;-)

Hanno

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Darryl Dixon - Winterhouse Consulting-2

Re: Z3 Event coverage in Plone and Zope core

Reply Threaded More More options
Print post
Permalink
> On Wed, Nov 4, 2009 at 4:01 AM, Darryl Dixon - Winterhouse Consulting
> <[hidden email]> wrote:
>> We have implemented a new IAnnotation to track these
>> 'almost-modifications' and are finding many places that require hotfixes
>> in order to emit IObjectModifiedEvents at appropriate times so this
>> annotation can be updated. Are there any concrete plans to improve event
>> coverage in Plone and Zope core, or is it more of a laissez faire
>> situation where if the itch is felt by someone appropriate they are
>> scratching it?
>
> If you are interested in any kind of low-level data changes, you could
> just use the persistent modification time as found in _p_mtime.
>
> I'd be hesitant to put a gazillion of new event notifications into all
> places, especially if it is such a general event as
> IObjectModifiedEvent.

In our case the events are a subclass of this.

> We are already doing quite a bunch of work
> during any such event and emitting it way more often will slow things
> down considerably.
>

Do you mean in terms of subscribers to this specific event?

> So far we have introduced new specific events whenever there was a
> real concrete use-case for them. For example we introduced specific
> workflow transition events into DCWorkflow at some point.
>
> I feel that continuing on this road, where we introduce events only
> for specific areas and where they either replace an existing hook
> mechanism (as was the case in DCWorkflow) or there is a clear use-case
> for them serves us better.

Well, for us events are simply a way of loosely and non-intrusively
coupling the existing core Plone code to some specific activities; to
whit, updating a 'metadata changed' annotation on the object. Arguably,
any code which changes something 'user facing' should possibly poke such
an annotation directly, if a standard implementation of it was agreed
upon, without needing to invoke events etc.

> But maybe you have a concrete use-case in mind, that you might want to
> share with us ;-)

Sure:
The use case for us is EnSimpleStaging, but equally anyone using perhaps
ContentMirror or some similar yet-to-be-implemented measure (eg, Limi's
hypothetical big green 'publish this' button) will at some point realise
they want this.

The reasoning is quite simple. If I have a site with 10,000 content
objects, doing a 'full publish' of every object in its complete glory to
some other representation (either an object clone, or RDBMS repr, or maybe
just XHTML) is expensive, both in terms of IO, CPU cycles, and wall-clock
time. It is nice to be able to do an 'incremental' publish that maybe only
captures and processes the items changed since the last time. But the
question then becomes 'what changed' and currently the Dublin Core
'modified' field does't accurately reflect this.

Here's a trivial example: changing the criteria of a Collection does not
update the modified time - that one is obviously a bug (a long-standing
one!), but there are as I noted in my earlier mail more subtle nuances;
eg, changing the chosen display view of the content item, or the order of
the items in a folder. These do not change the 'user supplied content' per
se, but they do alter the user-facing properties or aspects of the object
in some concrete way. Being able to easily find these without walking all
10,000 objects and performing some hash or low-level scrape of an
implementation detail of the Persistence machinery is therefore desirable.

For us, we are storing a float of epoch time and indexing that in
portal_catalog.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Alexander Limi

Re: Z3 Event coverage in Plone and Zope core

Reply Threaded More More options
Print post
Permalink
On Wed, Nov 4, 2009 at 1:33 PM, Darryl Dixon - Winterhouse Consulting <[hidden email]> wrote:
(eg, Limi's hypothetical big green 'publish this' button)

That was Martin, for the record. ;)

--
Alexander Limi · http://limi.net

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Alexander Limi · http://limi.net

Hanno Schlichting-4

Re: Z3 Event coverage in Plone and Zope core

Reply Threaded More More options
Print post
Permalink
In reply to this post by Darryl Dixon - Winterhouse Consulting-2
On Wed, Nov 4, 2009 at 10:33 PM, Darryl Dixon - Winterhouse Consulting
<[hidden email]> wrote:
>> I'd be hesitant to put a gazillion of new event notifications into all
>> places, especially if it is such a general event as
>> IObjectModifiedEvent.
>
> In our case the events are a subclass of this.

OK.

>> We are already doing quite a bunch of work
>> during any such event and emitting it way more often will slow things
>> down considerably.
>
> Do you mean in terms of subscribers to this specific event?

Subscribers to the general ObjectModified event.

> The use case for us is EnSimpleStaging, but equally anyone using perhaps
> ContentMirror or some similar yet-to-be-implemented measure (eg, Limi's
> hypothetical big green 'publish this' button) will at some point realise
> they want this.

Aha. So this is not some constrained functionality, but the rather
omnipresent "did anything change" use case. In practical terms I think
collective.indexing / solr have hooked into catalog updates as a
source of change information. IIRC c.solr keeps a queue of which
objects have been added/removed/updated based on catalog changes and
asynchronously pushes these into the Solr server. Should the Solr
server be unavailable for some time, it builds up a queue and
processes it after the Solr server is accessible again. The usual
"rebuild from scratch" button also exists if the two data sources have
come out of sync too much.

In the end that's essentially what the CMFSquidTool purge queue is
build on as well. For the most part it hooks into the catalog
operations, though it has some more specific event handlers that allow
to invalidate non-contentish URL's like image scales whenever an image
changes.

> Here's a trivial example: changing the criteria of a Collection does not
> update the modified time - that one is obviously a bug (a long-standing
> one!), but there are as I noted in my earlier mail more subtle nuances;
> eg, changing the chosen display view of the content item, or the order of
> the items in a folder. These do not change the 'user supplied content' per
> se, but they do alter the user-facing properties or aspects of the object
> in some concrete way. Being able to easily find these without walking all
> 10,000 objects and performing some hash or low-level scrape of an
> implementation detail of the Persistence machinery is therefore desirable.

The modification time we have today is in some loose way defined as
"when did an actual user change the real content of the item the last
time". It's specifically not a technical modification time that cares
about metadata, security, workflow or similar changes.

If we want to add extensive event coverage that is able to cover your
use-case I fear we end up with hundreds of them all over the place.
For example should adding a marker interface in the ZMI to a content
item fire some event? It might thanks to schemaextender or
p4a.subtypes change the schema of the item with some interesting
results. Should granting a user ownership permissions in a folder fire
some events for all contained items - where he now might have access?
If you update a criteria it's currently a separate object, so the
collection isn't touched at all, should it? What about the page that
references this collection via a collection portlet?

I'm not sure what kind of data it is that you are trying to sync, but
once you want more than the current catalog events give you, it's a
very unclear and open ended area. In order to do proper ESI caching we
tried several times to come up with a reliable mechanism that can tell
you what pages need to be invalidated when "anything" changes. So far
we haven't gotten any closer to this.

If you want really low-level data synchronization, another option is
to do it on the ZODB level. Mirror the Data.fs locally via some repozo
approach, remember the transaction you last looked at, analyze which
poid's have changed from the last time you synced and extract / push
this data into the other database. Or skip the intermediate Data.fs
mirror and incur some more load on the main database.

Hanno

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers