Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

9 messages Options
Embed this post
Permalink
Ross Patterson-2

Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
In Products.membrane, we're having an issue with our Plone 3.0-3.2
compatible branch.  Its quite a pain for us since the way we currently
get the tests to pass makes us depend on 3.3 meaning we have to have one
version whose tests pass with 3.0-3.2 and another whose tests pass with
3.3+.

Wichert spent some time with the original problem...

Wichert Akkerman <[hidden email]> writes:

> On 6/23/09 10:40 AM, Ross Patterson wrote:
>> Wichert described the parts of membrane that broke under 3.3 this way:
>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>> complex way in tests, and debugging that is a lot more effort".
>> Wichert, can you say in any more detail how this was related
>> specifically to tests?  Did the membrane test fixtures make assumptions
>> about the internals of indexable attributes they shouldn't?  If you
>> could help shed any light on this it could go a long way towards keeping
>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>
> registerIndexableAttribute in Plone 3.3 works via some special CA
> registrations for the indexers. For a mysterious reason I could not
> figure out after an hour of debugging those registration went missing
> in functional tests. Moving to the new API was a quick way to fix both
> that mysterious problem and get rid of the deprecation warnings.

I wonder then if it isn't an issue with the registerIndexableAttribute
magic happening at import time being cleared by the testing.cleanUp call
that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
breakage with 3.3?

Ross


------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Hi Ross,

> In Products.membrane, we're having an issue with our Plone 3.0-3.2
> compatible branch.  Its quite a pain for us since the way we currently
> get the tests to pass makes us depend on 3.3 meaning we have to have one
> version whose tests pass with 3.0-3.2 and another whose tests pass with
> 3.3+.
>
> Wichert spent some time with the original problem...
>
> Wichert Akkerman <[hidden email]> writes:
>
>> On 6/23/09 10:40 AM, Ross Patterson wrote:
>>> Wichert described the parts of membrane that broke under 3.3 this way:
>>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>>> complex way in tests, and debugging that is a lot more effort".
>>> Wichert, can you say in any more detail how this was related
>>> specifically to tests?  Did the membrane test fixtures make assumptions
>>> about the internals of indexable attributes they shouldn't?  If you
>>> could help shed any light on this it could go a long way towards keeping
>>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>> registerIndexableAttribute in Plone 3.3 works via some special CA
>> registrations for the indexers. For a mysterious reason I could not
>> figure out after an hour of debugging those registration went missing
>> in functional tests. Moving to the new API was a quick way to fix both
>> that mysterious problem and get rid of the deprecation warnings.
>
> I wonder then if it isn't an issue with the registerIndexableAttribute
> magic happening at import time being cleared by the testing.cleanUp call
> that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
> breakage with 3.3?

That sounds like a reasonable assumption.

Can you think of another way to fix this? We could try to emit
configuration actions instead of calling provideAdapter(). One way to do
that would be to have the registerIndexableAttribute() function just
build a list of indexers, and then have a ZCML directive that picks up
this list and registers components.

Do you think that'd work?

Martin


--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ross Patterson-2

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Martin Aspeli <[hidden email]>
writes:

> Hi Ross,
>
>> In Products.membrane, we're having an issue with our Plone 3.0-3.2
>> compatible branch.  Its quite a pain for us since the way we currently
>> get the tests to pass makes us depend on 3.3 meaning we have to have one
>> version whose tests pass with 3.0-3.2 and another whose tests pass with
>> 3.3+.
>>
>> Wichert spent some time with the original problem...
>>
>> Wichert Akkerman <[hidden email]> writes:
>>
>>> On 6/23/09 10:40 AM, Ross Patterson wrote:
>>>> Wichert described the parts of membrane that broke under 3.3 this way:
>>>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>>>> complex way in tests, and debugging that is a lot more effort".
>>>> Wichert, can you say in any more detail how this was related
>>>> specifically to tests?  Did the membrane test fixtures make assumptions
>>>> about the internals of indexable attributes they shouldn't?  If you
>>>> could help shed any light on this it could go a long way towards keeping
>>>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>>> registerIndexableAttribute in Plone 3.3 works via some special CA
>>> registrations for the indexers. For a mysterious reason I could not
>>> figure out after an hour of debugging those registration went missing
>>> in functional tests. Moving to the new API was a quick way to fix both
>>> that mysterious problem and get rid of the deprecation warnings.
>>
>> I wonder then if it isn't an issue with the registerIndexableAttribute
>> magic happening at import time being cleared by the testing.cleanUp call
>> that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
>> breakage with 3.3?
>
> That sounds like a reasonable assumption.
>
> Can you think of another way to fix this? We could try to emit
> configuration actions instead of calling provideAdapter(). One way to do
> that would be to have the registerIndexableAttribute() function just
> build a list of indexers, and then have a ZCML directive that picks up
> this list and registers components.
>
> Do you think that'd work?

That's certainly where my thoughts were headed.  But I'd like
confirmation of my assumption first.  As a workaround for membrane 1.1,
I was going to move our registerIndexableAttribute calls into a module
that is only imported by ZCML.  If that worked it would pretty much
confirm this assumption.

I'd still like to see if anyone else is experiencing this in the wild
in case it's just membrane.

Ross


------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ross Patterson-2

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Ross Patterson <[hidden email]> writes:

> Martin Aspeli <[hidden email]>
> writes:
>
>> Hi Ross,
>>
>>> In Products.membrane, we're having an issue with our Plone 3.0-3.2
>>> compatible branch.  Its quite a pain for us since the way we currently
>>> get the tests to pass makes us depend on 3.3 meaning we have to have one
>>> version whose tests pass with 3.0-3.2 and another whose tests pass with
>>> 3.3+.
>>>
>>> Wichert spent some time with the original problem...
>>>
>>> Wichert Akkerman <[hidden email]> writes:
>>>
>>>> On 6/23/09 10:40 AM, Ross Patterson wrote:
>>>>> Wichert described the parts of membrane that broke under 3.3 this way:
>>>>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>>>>> complex way in tests, and debugging that is a lot more effort".
>>>>> Wichert, can you say in any more detail how this was related
>>>>> specifically to tests?  Did the membrane test fixtures make assumptions
>>>>> about the internals of indexable attributes they shouldn't?  If you
>>>>> could help shed any light on this it could go a long way towards keeping
>>>>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>>>> registerIndexableAttribute in Plone 3.3 works via some special CA
>>>> registrations for the indexers. For a mysterious reason I could not
>>>> figure out after an hour of debugging those registration went missing
>>>> in functional tests. Moving to the new API was a quick way to fix both
>>>> that mysterious problem and get rid of the deprecation warnings.
>>>
>>> I wonder then if it isn't an issue with the registerIndexableAttribute
>>> magic happening at import time being cleared by the testing.cleanUp call
>>> that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
>>> breakage with 3.3?
>>
>> That sounds like a reasonable assumption.
>>
>> Can you think of another way to fix this? We could try to emit
>> configuration actions instead of calling provideAdapter(). One way to do
>> that would be to have the registerIndexableAttribute() function just
>> build a list of indexers, and then have a ZCML directive that picks up
>> this list and registers components.
>>
>> Do you think that'd work?
>
> That's certainly where my thoughts were headed.  But I'd like
> confirmation of my assumption first.  As a workaround for membrane 1.1,
> I was going to move our registerIndexableAttribute calls into a module
> that is only imported by ZCML.  If that worked it would pretty much
> confirm this assumption.

This workaround doesn't work because under PTC, site.zcml is actually
loaded *twice*.

I can't remember why or how and I'm certain it shouldn't be so, but I do
remember at some point in years past confirming that the ZCML does get
loaded twice.  It gets loaded at some point during instance startup and
then in the Products.PloneTestCase.layer.ZCML layer the
Products.PloneTestCase.five.safe_load_site clears the ZCA and runs the
ZCML again.

Of course, the second time around, the results of the
registerIndexableAttribute calls have already been cleared when the ZCA
was cleared but the module has already been imported and so the
registerIndexableAttribute registrations won't be repeated.

I confirmed this by moving the registerIndexableAttribute calls into a
separate module that isn't imported until after the ZCML layer runs and
the tests work again.  So yes, I think your solution would resolve
this.

Ross


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Ross Patterson wrote:

> Ross Patterson <[hidden email]> writes:
>
>> Martin Aspeli <[hidden email]>
>> writes:
>>
>>> Hi Ross,
>>>
>>>> In Products.membrane, we're having an issue with our Plone 3.0-3.2
>>>> compatible branch.  Its quite a pain for us since the way we currently
>>>> get the tests to pass makes us depend on 3.3 meaning we have to have one
>>>> version whose tests pass with 3.0-3.2 and another whose tests pass with
>>>> 3.3+.
>>>>
>>>> Wichert spent some time with the original problem...
>>>>
>>>> Wichert Akkerman <[hidden email]> writes:
>>>>
>>>>> On 6/23/09 10:40 AM, Ross Patterson wrote:
>>>>>> Wichert described the parts of membrane that broke under 3.3 this way:
>>>>>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>>>>>> complex way in tests, and debugging that is a lot more effort".
>>>>>> Wichert, can you say in any more detail how this was related
>>>>>> specifically to tests?  Did the membrane test fixtures make assumptions
>>>>>> about the internals of indexable attributes they shouldn't?  If you
>>>>>> could help shed any light on this it could go a long way towards keeping
>>>>>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>>>>> registerIndexableAttribute in Plone 3.3 works via some special CA
>>>>> registrations for the indexers. For a mysterious reason I could not
>>>>> figure out after an hour of debugging those registration went missing
>>>>> in functional tests. Moving to the new API was a quick way to fix both
>>>>> that mysterious problem and get rid of the deprecation warnings.
>>>> I wonder then if it isn't an issue with the registerIndexableAttribute
>>>> magic happening at import time being cleared by the testing.cleanUp call
>>>> that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
>>>> breakage with 3.3?
>>> That sounds like a reasonable assumption.
>>>
>>> Can you think of another way to fix this? We could try to emit
>>> configuration actions instead of calling provideAdapter(). One way to do
>>> that would be to have the registerIndexableAttribute() function just
>>> build a list of indexers, and then have a ZCML directive that picks up
>>> this list and registers components.
>>>
>>> Do you think that'd work?
>> That's certainly where my thoughts were headed.  But I'd like
>> confirmation of my assumption first.  As a workaround for membrane 1.1,
>> I was going to move our registerIndexableAttribute calls into a module
>> that is only imported by ZCML.  If that worked it would pretty much
>> confirm this assumption.
>
> This workaround doesn't work because under PTC, site.zcml is actually
> loaded *twice*.
>
> I can't remember why or how and I'm certain it shouldn't be so, but I do
> remember at some point in years past confirming that the ZCML does get
> loaded twice.  It gets loaded at some point during instance startup and
> then in the Products.PloneTestCase.layer.ZCML layer the
> Products.PloneTestCase.five.safe_load_site clears the ZCA and runs the
> ZCML again.
>
> Of course, the second time around, the results of the
> registerIndexableAttribute calls have already been cleared when the ZCA
> was cleared but the module has already been imported and so the
> registerIndexableAttribute registrations won't be repeated.
>
> I confirmed this by moving the registerIndexableAttribute calls into a
> separate module that isn't imported until after the ZCML layer runs and
> the tests work again.  So yes, I think your solution would resolve
> this.

Okay. Are you able to help look into this? I think what we need to do is:

  - change registerIndexableAttribute() to store its arguments in a
module level list

  - add a new ZCML directive like plone:bbbIndexers with no options,
just a handler; it should probably have a late order

  - have this handler zip through the list in
registerIndexableAttribute() and do what it does now: register adapters

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Ross Patterson-2

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Martin Aspeli <[hidden email]>
writes:

> Ross Patterson wrote:
>> Ross Patterson <[hidden email]> writes:
>>
>>> Martin Aspeli <[hidden email]>
>>> writes:
>>>
>
>>>> Hi Ross,
>>>>
>>>>> In Products.membrane, we're having an issue with our Plone 3.0-3.2
>>>>> compatible branch.  Its quite a pain for us since the way we currently
>>>>> get the tests to pass makes us depend on 3.3 meaning we have to have one
>>>>> version whose tests pass with 3.0-3.2 and another whose tests pass with
>>>>> 3.3+.
>>>>>
>>>>> Wichert spent some time with the original problem...
>>>>>
>>>>> Wichert Akkerman <[hidden email]> writes:
>>>>>
>>>>>> On 6/23/09 10:40 AM, Ross Patterson wrote:
>>>>>>> Wichert described the parts of membrane that broke under 3.3 this way:
>>>>>>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>>>>>>> complex way in tests, and debugging that is a lot more effort".
>>>>>>> Wichert, can you say in any more detail how this was related
>>>>>>> specifically to tests?  Did the membrane test fixtures make assumptions
>>>>>>> about the internals of indexable attributes they shouldn't?  If you
>>>>>>> could help shed any light on this it could go a long way towards keeping
>>>>>>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>>>>>> registerIndexableAttribute in Plone 3.3 works via some special CA
>>>>>> registrations for the indexers. For a mysterious reason I could not
>>>>>> figure out after an hour of debugging those registration went missing
>>>>>> in functional tests. Moving to the new API was a quick way to fix both
>>>>>> that mysterious problem and get rid of the deprecation warnings.
>>>>> I wonder then if it isn't an issue with the registerIndexableAttribute
>>>>> magic happening at import time being cleared by the testing.cleanUp call
>>>>> that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
>>>>> breakage with 3.3?
>>>> That sounds like a reasonable assumption.
>>>>
>>>> Can you think of another way to fix this? We could try to emit
>>>> configuration actions instead of calling provideAdapter(). One way to do
>>>> that would be to have the registerIndexableAttribute() function just
>>>> build a list of indexers, and then have a ZCML directive that picks up
>>>> this list and registers components.
>>>>
>>>> Do you think that'd work?
>>> That's certainly where my thoughts were headed.  But I'd like
>>> confirmation of my assumption first.  As a workaround for membrane 1.1,
>>> I was going to move our registerIndexableAttribute calls into a module
>>> that is only imported by ZCML.  If that worked it would pretty much
>>> confirm this assumption.
>>
>> This workaround doesn't work because under PTC, site.zcml is actually
>> loaded *twice*.
>>
>> I can't remember why or how and I'm certain it shouldn't be so, but I
>> do remember at some point in years past confirming that the ZCML does
>> get loaded twice.  It gets loaded at some point during instance
>> startup and then in the Products.PloneTestCase.layer.ZCML layer the
>> Products.PloneTestCase.five.safe_load_site clears the ZCA and runs
>> the ZCML again.
>>
>> Of course, the second time around, the results of the
>> registerIndexableAttribute calls have already been cleared when the
>> ZCA was cleared but the module has already been imported and so the
>> registerIndexableAttribute registrations won't be repeated.
>>
>> I confirmed this by moving the registerIndexableAttribute calls into
>> a separate module that isn't imported until after the ZCML layer runs
>> and the tests work again.  So yes, I think your solution would
>> resolve this.
>
> Okay. Are you able to help look into this? I think what we need to do
> is:

No, sorry, I've already put all the time I can afford into this.  :(

>   - change registerIndexableAttribute() to store its arguments in a
> module level list
>
>   - add a new ZCML directive like plone:bbbIndexers with no options,
> just a handler; it should probably have a late order
>
>   - have this handler zip through the list in
> registerIndexableAttribute() and do what it does now: register adapters

Yup, sounds right.  Sorry I can't help more with the impl.

Ross


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Wichert Akkerman

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Ross Patterson-2
On 6/23/09 2:53 PM, Ross Patterson wrote:

> In Products.membrane, we're having an issue with our Plone 3.0-3.2
> compatible branch.  Its quite a pain for us since the way we currently
> get the tests to pass makes us depend on 3.3 meaning we have to have one
> version whose tests pass with 3.0-3.2 and another whose tests pass with
> 3.3+.
>
> Wichert spent some time with the original problem...
>
> Wichert Akkerman<[hidden email]>  writes:
>
>> On 6/23/09 10:40 AM, Ross Patterson wrote:
>>> Wichert described the parts of membrane that broke under 3.3 this way:
>>> "the indexing changesin 3.3 break registerIndexableAttribute in some
>>> complex way in tests, and debugging that is a lot more effort".
>>> Wichert, can you say in any more detail how this was related
>>> specifically to tests?  Did the membrane test fixtures make assumptions
>>> about the internals of indexable attributes they shouldn't?  If you
>>> could help shed any light on this it could go a long way towards keeping
>>> a Plone 3.0-3.x compatible 1.1 branch which I'd certainly like.
>> registerIndexableAttribute in Plone 3.3 works via some special CA
>> registrations for the indexers. For a mysterious reason I could not
>> figure out after an hour of debugging those registration went missing
>> in functional tests. Moving to the new API was a quick way to fix both
>> that mysterious problem and get rid of the deprecation warnings.
>
> I wonder then if it isn't an issue with the registerIndexableAttribute
> magic happening at import time being cleared by the testing.cleanUp call
> that happens in the ZCML layer, IIRC.  Is anyone else experiencing test
> breakage with 3.3?

This sounds like a blocker bug for 3.3: this breaks valid existing
functional test cases, with no way to good way fix that without dropping
support for older Plone versions.

Wichert.

--
Wichert Akkerman <[hidden email]>   It is simple to make things.
http://www.wiggy.net/                  It is hard to make things simple.

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Wichert Akkerman wrote:

> This sounds like a blocker bug for 3.3: this breaks valid existing
> functional test cases, with no way to good way fix that without dropping
> support for older Plone versions.

You're probably right. Would you be happy with the fix I proposed?

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli

Re: Plone 3.3, plone.indexer, and registerIndexableAttribute break functional tests?

Reply Threaded More More options
Print post
Permalink
Martin Aspeli wrote:
> Wichert Akkerman wrote:
>
>> This sounds like a blocker bug for 3.3: this breaks valid existing
>> functional test cases, with no way to good way fix that without dropping
>> support for older Plone versions.
>
> You're probably right. Would you be happy with the fix I proposed?

Okay, I've implemented this now. I haven't tested it against membrane,
but there's at least a test for the "only register on ZCML load" behaviour.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers