Downsides of moving from Products to lib/python?

15 Messages Forum Options Options
Embed this topic
Permalink
Maurits van Rees-3
Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
Hi,

The lib/python directory of a Zope instance seems to become a popular
place for putting things and rightfully so.  I can imagine that in a
few years time the Products dir becomes totally empty and that zope
only uses packages in lib/python.

But I wonder if there are (perhaps temporarily) downsides to that.
I bumped my toe the past week with two such issues:


1. zcml files are not loaded automatically.  This is no big deal
   really, as there are a few ways to solve this:

   - You can do an <include package="..." /> in a zcml file somewhere
     in the Products dir.

   - I heard ploneout does some hacking to the path to make sure these
     zcmls in lib/python get loaded too.

   - You can put zcml "slugs" (one-liners) in etc/site.zcml or
     etc/package-includes/


2. Translations from i18n and locales directories do not get loaded.
   You can add <i18n:registerTranslations directory="locales" /> and
   then that will get loaded, but this is done in a Zope 3 way that is
   not getting picked by the Zope 2 or Plone translation machinery.
   For starters, the po files do not get listed in the PTS in the
   Zope Control Panel.

   Plone 3.0 of course has templates in lib/python that need
   translations.  But those translations are in
   Products/PloneTranslations, so they get picked up as usual.

   So if you want to have a package in lib/python that needs
   translation, then you will also have to make a Product for the
   Products dir that contains the actual translations.


So I have two questions really:

A. Is there a way to have po files in lib/python get picked up by
   Zope 2 after all?

B. Are there more of these gotchas that developers should know?


Thanks,

--
Maurits van Rees | http://maurits.vanrees.org/ [NL]
            Work | http://zestsoftware.nl/
"Do not worry about your difficulties in computers,
 I can assure you mine are still greater."


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Martin Aspeli-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
Maurits van Rees wrote:
> Hi,
>
> The lib/python directory of a Zope instance seems to become a popular
> place for putting things and rightfully so.  I can imagine that in a
> few years time the Products dir becomes totally empty and that zope
> only uses packages in lib/python.

I hope so. :)

> But I wonder if there are (perhaps temporarily) downsides to that.
> I bumped my toe the past week with two such issues:
>
>
> 1. zcml files are not loaded automatically.  This is no big deal
>    really, as there are a few ways to solve this:
>
>    - You can do an <include package="..." /> in a zcml file somewhere
>      in the Products dir.
>
>    - I heard ploneout does some hacking to the path to make sure these
>      zcmls in lib/python get loaded too.

Nope.

But the plone.recipe.zope2instance recipe (like it's zope 3 equivalent)
has an option 'zcml' where you can list package names for which ZCML
slugs get installed automatically. This is not a hack. :)

The wider point here is that if you're working with eggs (and you will
be, since third party products are starting to move to eggs) you
probably want something a bit more egg-aware than dumb old instances. I
strongly favour zc.buildout here, because it meets all those needs and
makes sure the right packages (and the right packages only) are
installed for Zope to run.

I keep promising more documentation on this (and I will, once I'm done
with the book, which also covers it), but see this post:

http://www.nabble.com/forum/ViewPost.jtp?post=12284509&framed=y&skin=6741

>    - You can put zcml "slugs" (one-liners) in etc/site.zcml or
>      etc/package-includes/

The approach I prefer and advocate is to have *one* slug for a "policy"
product (Wichert calls this a "deployment product"). This has a
configure.zcml that includes other dependencies as and when needed. It
also has an installation step that installs other packages as
dependencies when the policy is installed from the quickinstaller.

Trust me - automatically loading things can lead to just as many
headaches. Obviously, you want the core Plone stuff to load in
lock-step, but sometimes other packages can do things that e.g. affect
your package's tests or setup.

The main benefit I see of using eggs and non-flat namespaces is that it
makes it much easier to manage things as sets of related packages,
rather than big monoliths. $INSTANCE_HOME/Products/ was becoming *very*
crowded, and people didn't split things up into more reusable, managable
chunks. We saw that happen *a lot more* in the Plone 3 release, when
people could create a package with paster. :)

See also: http://plone.org/documentation/how-to/use-paster

> 2. Translations from i18n and locales directories do not get loaded.
>    You can add <i18n:registerTranslations directory="locales" /> and
>    then that will get loaded, but this is done in a Zope 3 way that is
>    not getting picked by the Zope 2 or Plone translation machinery.
>    For starters, the po files do not get listed in the PTS in the
>    Zope Control Panel.
>
>    Plone 3.0 of course has templates in lib/python that need
>    translations.  But those translations are in
>    Products/PloneTranslations, so they get picked up as usual.
>
>    So if you want to have a package in lib/python that needs
>    translation, then you will also have to make a Product for the
>    Products dir that contains the actual translations.

I'm not really too familiar with this. It sounds like a bug to me. :)

> So I have two questions really:
>
> A. Is there a way to have po files in lib/python get picked up by
>    Zope 2 after all?
>
> B. Are there more of these gotchas that developers should know?

When writing tests for packages that are also Zope 2 products, you can't
use ZopeTestCase.installProduct('my.package'); the pattern I prefer is
as follows:

http://dev.plone.org/collective/browser/borg/components/borg.project/trunk/borg/project/tests.py

That is:

  - Any old-style products that need to be loaded for the test goes at
module level as before:

ZopeTestCase.installProduct('RichDocument')

  - After that, a *deferred method* is used to load packages. Note that
this ONLY necessary if the package is a product, e.g. it uses
<five:registerPackage /> to get skins, GS profiles or other product-like
features. This can also be used to load specific ZCML files.

  - Then, we call setupPloneSite() as normal, telling it to install any
products (including packages that are products). Note that in Plone 3,
this also works for packages without an Install.py, if they have a GS
profile, you don't need to do a separate extension_profiles=[] argument.

This approach ensures that the package is loaded and installed only once
for the test fixture, making tests much faster.

Martin

--
Acquisition is a jealous mistress


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Hanno Schlichting-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Maurits van Rees-3
Hi,

I'll just comment on the translation aspect :)

Maurits van Rees wrote:

> 2. Translations from i18n and locales directories do not get loaded.
>    You can add <i18n:registerTranslations directory="locales" /> and
>    then that will get loaded, but this is done in a Zope 3 way that is
>    not getting picked by the Zope 2 or Plone translation machinery.
>    For starters, the po files do not get listed in the PTS in the
>    Zope Control Panel.
>
>    Plone 3.0 of course has templates in lib/python that need
>    translations.  But those translations are in
>    Products/PloneTranslations, so they get picked up as usual.
>
>    So if you want to have a package in lib/python that needs
>    translation, then you will also have to make a Product for the
>    Products dir that contains the actual translations.

Just because po files aren't listed in the Control_Panel/PTS doesn't
mean they won't get picked up. Starting in Plone 2.5 PTS is not the main
translation service anymore but the one from Five is. This one will load
a translation registered for Zope3 first and fall back to PTS if none is
found.

This means that as soon as you register your translations in the Zope3
way with locales folders and the configure zcml snippet, everything
works fine both in pure Zope3 code as well as in Zope2 code.

If you use the i18n folder layout your translations will not be
available in pure Zope3 code (like ViewPageTemplateFiles used a lot for
viewlets for example). We have introduced a simple proxy object in Plone
3.0 here, which you can use to register i18n-folder-based translations
for Zope3 as well (see CMFPlone/i18nl10n.py PTSTranslationDomain).

In Plone 3.5 PTS itself is mostly deprecated and the mo file cache as
well as the Control_Panel entry are gone. Instead it will register all
po files found in a i18n folder as Zope3 ITranslationDomain utilities in
the same way the ones found in locales folders are treated.

> So I have two questions really:
>
> A. Is there a way to have po files in lib/python get picked up by
>    Zope 2 after all?

Simple answer: Just use the locales folder layout and register it via
zcml. This should be the preferred way for all add-on products or
packages targeting Plone 3.0 or newer. In Plone 4.0 or 4.5 the support
for the i18n folder layout will probably be removed.

Hanno


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Maurits van Rees-3
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Martin Aspeli-2
Martin Aspeli, on 2007-08-25:
> Maurits van Rees wrote:
>>    - I heard ploneout does some hacking to the path to make sure these
>>      zcmls in lib/python get loaded too.
>
> Nope.
>
> But the plone.recipe.zope2instance recipe (like it's zope 3 equivalent)
> has an option 'zcml' where you can list package names for which ZCML
> slugs get installed automatically. This is not a hack. :)

Mu information was wrong then.  It looks like buildout does this
correctly and in a non-hackish way.

>>    - You can put zcml "slugs" (one-liners) in etc/site.zcml or
>>      etc/package-includes/
>
> The approach I prefer and advocate is to have *one* slug for a "policy"
> product (Wichert calls this a "deployment product"). This has a
> configure.zcml that includes other dependencies as and when needed. It
> also has an installation step that installs other packages as
> dependencies when the policy is installed from the quickinstaller.

I ususally also make some setup Product that makes sure all
QuickInstaller and GenericSetup dependencies are installed, skin
layers are ordered correctly, etc.  Making this one also responsible
for loading the zcml files of its dependent python packages seems
logical yes.

>> B. Are there more of these gotchas that developers should know?
>
> When writing tests for packages that are also Zope 2 products, you can't
> use ZopeTestCase.installProduct('my.package'); the pattern I prefer is
> as follows:
>
> http://dev.plone.org/collective/browser/borg/components/borg.project/trunk/borg/project/tests.py

Nice, thanks for sharing.  I will keep that in mind.

--
Maurits van Rees | http://maurits.vanrees.org/ [NL]
            Work | http://zestsoftware.nl/
"Do not worry about your difficulties in computers,
 I can assure you mine are still greater."


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Maurits van Rees-3
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Hanno Schlichting-2
Hanno Schlichting, on 2007-08-25:
> Just because po files aren't listed in the Control_Panel/PTS doesn't
> mean they won't get picked up. Starting in Plone 2.5 PTS is not the main
> translation service anymore but the one from Five is. This one will load
> a translation registered for Zope3 first and fall back to PTS if none is
> found.

Ah, good to know.

> This means that as soon as you register your translations in the Zope3
> way with locales folders and the configure zcml snippet, everything
> works fine both in pure Zope3 code as well as in Zope2 code.

Right, I see I made a mistake.  For the Dutch language I copied
i18n/blah-nl.po to locales/nl/LC_MESSAGES/blah-nl.po.  Of course since
'nl' is already the folder name the po file should just be named
blah.po.  Then it works in Plone 2.5 (and I am sure that would work in
3.0 as well).

>> A. Is there a way to have po files in lib/python get picked up by
>>    Zope 2 after all?
>
> Simple answer: Just use the locales folder layout and register it via
> zcml. This should be the preferred way for all add-on products or
> packages targeting Plone 3.0 or newer. In Plone 4.0 or 4.5 the support
> for the i18n folder layout will probably be removed.
>
> Hanno

Thanks,

--
Maurits van Rees | http://maurits.vanrees.org/ [NL]
            Work | http://zestsoftware.nl/
"Do not worry about your difficulties in computers,
 I can assure you mine are still greater."


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Kai Diefenbach
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Maurits van Rees-3
Hi,

Maurits van Rees <m.van.rees@...>
wrote:

> The lib/python directory of a Zope instance seems to become a popular
> place for putting things and rightfully so.  I can imagine that in a
> few years time the Products dir becomes totally empty and that zope
> only uses packages in lib/python.

[Snip]

> B. Are there more of these gotchas that developers should know?

Today I run into following issue:

>From a pure package based product (within lib/python) the initialize
method (aka: "Intializer called when used as a Zope 2 product.") within
__init__.py is not called when I run tests: zopectl test -m ...  (It is
when the instance is started normally: e.g. zopectl fg)

This means - since I use AT based types - that the types are not
registered appropriate, hence they can not added within the portal:

   [...] raise AccessControl_Unauthorized('Cannot create %s' %    
   self.getId()) Unauthorized: Cannot create Company

I wonder about two things now:

1.
How can I make Zope call the initialize method for tests as it does when
it starts the normal way?

2.
When Zope calls the initialize method it passes an instance of
App.ProductContext.ProductContext, which is needed to initalize the
content_types.

As I'm able to call the initialize method explicitely within my TestCase
I would like to know how can I get on this ProductContext instance? Or
how can I create one, which is appropriate?

Thanks
Kai∆

--
Kai Diefenbach - http://diefenba.ch
iqplusplus - http://iqpp.de


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Reinout van Rees
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
Kai Diefenbach wrote:

> How can I make Zope call the initialize method for tests as it does when
> it starts the normal way?

Copy-pasted from one of Maurits' testcases:



from Products.Five import pythonproducts

class MainTestCase(testcase):
     """Base TestCase for plonehrm."""

     def afterSetUp(self):
         pythonproducts.applyPatches()

     def afterClear(self):
         pythonproducts.removePatches()



It works just fine for me if I add this. Make sure you don't override
the afterSetUp() in subsequent testcases (in case you're inheriting from
a base testcase).

Reinout


--
Reinout van Rees  - Programmer at http://zestsoftware.nl/
http://vanrees.org/weblog/     mailto:reinout@...
"White space, the opposite of black hole?" -- Joris slob


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Martin Aspeli-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Kai Diefenbach

Kai Diefenbach wrote:
Today I run into following issue:

From a pure package based product (within lib/python) the initialize
method (aka: "Intializer called when used as a Zope 2 product.") within
__init__.py is not called when I run tests: zopectl test -m ...  (It is
when the instance is started normally: e.g. zopectl fg)
This is because you're not doing the test setup right. Look at http://dev.plone.org/collective/browser/borg/components/borg.project/trunk/borg/project/tests.py.

Martin
Kai Diefenbach
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
Hi Martin,

Martin Aspeli <optilude@...> wrote:

> Kai Diefenbach wrote: > > Today I run into following issue: > > From a
> pure package based product (within lib/python) the initialize > method
> (aka: "Intializer called when used as a Zope 2 product.") within >
> __init__.py is not called when I run tests: zopectl test -m ...  (It is >
> when the instance is started normally: e.g. zopectl fg) >
>
> This is because you're not doing the test setup right. Look at
> http://dev.plone.org/collective/browser/borg/components/borg.project/trunk
> /borg/project/tests.py.

I tried out an adapted version of your setup for my product before I
posted and it didn't help.

Now I tried it again with a fresh buildout
(http://pastie.textmate.org/96457), borg and
membrane (branch for 3.0) checked out in the products directory, running
bin/instance test -m borg.project I got:
   
    installProduct() got an unexpected keyword argument 'package'

Whole Traceback:

    http://pastie.textmate.org/pastes/96455

The initialize methode is not called. When I run the tests for
borg.localrole initialize is not called, too.

If I start the instance with fg both initialize methods - from
borg.project and borg.localrole - are called.

Thanks
Kai

--
Kai Diefenbach - http://diefenba.ch
iqplusplus - http://iqpp.de


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Kai Diefenbach
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Reinout van Rees
Hi Reinout,

Reinout van Rees <reinout@...>
wrote:

> Kai Diefenbach wrote:
>
> > How can I make Zope call the initialize method for tests as it does when
> > it starts the normal way?
>
> Copy-pasted from one of Maurits' testcases:
>
> from Products.Five import pythonproducts

[Snip]

I should have said, that I using Plone 3.0 / Zope 2.10.4. In the Five
version (1.5.5) which comes with Zope 2.10.4 there is no pythonproducts.

FWIW, I copied pythonproducts from 1.4.2 (IIRC) but this doesn't change
the fact that initialize method isn't called.

Thanks
Kai

--
Kai Diefenbach - http://diefenba.ch
iqplusplus - http://iqpp.de


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Martin Aspeli-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink

Kai Diefenbach wrote:
Hi Reinout,

Reinout van Rees <reinout@vanrees.org>
wrote:

> Kai Diefenbach wrote:
>
> > How can I make Zope call the initialize method for tests as it does when
> > it starts the normal way?
>
> Copy-pasted from one of Maurits' testcases:
>
> from Products.Five import pythonproducts

[Snip]

I should have said, that I using Plone 3.0 / Zope 2.10.4. In the Five
version (1.5.5) which comes with Zope 2.10.4 there is no pythonproducts.

FWIW, I copied pythonproducts from 1.4.2 (IIRC) but this doesn't change
the fact that initialize method isn't called.
Pythonproducts is *not* needed on Zope 2.10 (and may be harmful, not sure). If you do the test setup right, it should work (it does for me ;-))

Martin
Martin Aspeli-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Reinout van Rees

Reinout van Rees wrote:
Kai Diefenbach wrote:

> How can I make Zope call the initialize method for tests as it does when
> it starts the normal way?

Copy-pasted from one of Maurits' testcases:



from Products.Five import pythonproducts

class MainTestCase(testcase):
     """Base TestCase for plonehrm."""

     def afterSetUp(self):
         pythonproducts.applyPatches()

     def afterClear(self):
         pythonproducts.removePatches()



It works just fine for me if I add this. Make sure you don't override
the afterSetUp() in subsequent testcases (in case you're inheriting from
a base testcase).
You should not be using Pythonproducts (or its monkey patches) on Zope 2.10/Plone 3.

Martin
Ross Patterson-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Kai Diefenbach
usenet@... (Kai Diefenbach) writes:

> Hi Martin,
>
> Martin Aspeli <optilude@...> wrote:
>
>> Kai Diefenbach wrote: > > Today I run into following issue: > > From a
>> pure package based product (within lib/python) the initialize > method
>> (aka: "Intializer called when used as a Zope 2 product.") within >
>> __init__.py is not called when I run tests: zopectl test -m ...  (It is >
>> when the instance is started normally: e.g. zopectl fg) >
>>
>> This is because you're not doing the test setup right. Look at
>> http://dev.plone.org/collective/browser/borg/components/borg.project/trunk> /borg/project/tests.py.
>
> I tried out an adapted version of your setup for my product before I
> posted and it didn't help.
>
> Now I tried it again with a fresh buildout
> (http://pastie.textmate.org/96457), borg and
> membrane (branch for 3.0) checked out in the products directory, running
> bin/instance test -m borg.project I got:
>    
>     installProduct() got an unexpected keyword argument 'package'
>
> Whole Traceback:
>
>     http://pastie.textmate.org/pastes/96455
>
> The initialize methode is not called. When I run the tests for
> borg.localrole initialize is not called, too.
>
> If I start the instance with fg both initialize methods - from
> borg.project and borg.localrole - are called.

Not sure I understand the context fully, but ZTC has an installPackage
as well as an installProduct.  Maybe that will help?

Ross


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Kai Diefenbach
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink
Hi Ross,

Ross Patterson <me@...> wrote:

> usenet@... (Kai Diefenbach) writes:
>
> > Hi Martin,
> >
> > Martin Aspeli <optilude@...> wrote:
> >
> >> Kai Diefenbach wrote: > > Today I run into following issue: > > From a
> >> pure package based product (within lib/python) the initialize > method
> >> (aka: "Intializer called when used as a Zope 2 product.") within
> >> __init__.py is not called when I run tests: zopectl test -m ...  (It is
> >> when the instance is started normally: e.g. zopectl fg)
> >>
> >> This is because you're not doing the test setup right. Look at
> >> http://dev.plone.org/collective/browser/borg/components/borg.project/tr
> >> unk> /borg/project/tests.py.
> >
> > I tried out an adapted version of your setup for my product before I
> > posted and it didn't help.
> >
> > Now I tried it again with a fresh buildout
> > (http://pastie.textmate.org/96457), borg and membrane (branch for 3.0)
> > checked out in the products directory, running bin/instance test -m
> > borg.project I got:
> >    
> >     installProduct() got an unexpected keyword argument 'package'
> >
> > Whole Traceback:
> >
> >     http://pastie.textmate.org/pastes/96455
> >
> > The initialize methode is not called. When I run the tests for
> > borg.localrole initialize is not called, too.
> >
> > If I start the instance with fg both initialize methods - from
> > borg.project and borg.localrole - are called.
>
> Not sure I understand the context fully, but ZTC has an installPackage as
> well as an installProduct.  Maybe that will help?

Bingo Ross!

    ztc.installPackage('borg.project')

is the solution (means the initialize method is called and all tests are
passing).

Thanks
Kai

--
Kai Diefenbach - http://diefenba.ch
iqplusplus - http://iqpp.de


_______________________________________________
Product-Developers mailing list
Product-Developers@...
http://lists.plone.org/mailman/listinfo/product-developers
Martin Aspeli-2
Re: Downsides of moving from Products to lib/python?
Reply Threaded MoreMore options
Print post
Permalink

Kai Diefenbach wrote:
> Not sure I understand the context fully, but ZTC has an installPackage as
> well as an installProduct.  Maybe that will help?

Bingo Ross!

    ztc.installPackage('borg.project')

is the solution (means the initialize method is called and all tests are
passing).
Ah, sorry, I sent you a duff example. :) It is indeed installPackage(). For a while, it was installProduct(..., package=True) in Zope svn, but stefan changed it. :)

Martin