eggs/zc.buildout releasing and deploying

[Tarek Ziadé]

Hi all,

just to let you know: we have discussed at the Paris Sprint of seeing if we could have common tools
to release and deploy egg-based packages and zc-buildout based applications, at the entreprise level.

I summarized it here, if someone is interested in this topic:

http://tarekziade.wordpress.com/2008/04/29/plone-paris-sprint-wrapup-part-1-how-do-you-use-eggs-and-zcbuildout-in-your-projects/

Cheers

Tarek

[Kapil Thangavelu-5]

reading through that blog post reads like utilities and templates not enterprise requirements for deployments of eggs.  for pushing eggs out, i find pushing them to a directory and adding a find links script works well, and doesn't force publication to the cheeseshop. if someone wants that, its builtin to setuptools. i'm not sure what the value add of making additional tools for pushing bits into directories...

one thing of immense value that any such discussion of enterprise deployments with eggs, should include known good set (kgs) facilities, so you can get repeatable builds at latter points in time. 

-kapil

On Tue, Apr 29, 2008 at 9:17 AM, Tarek Ziadé <[hidden email]> wrote:

Hi all,

just to let you know: we have discussed at the Paris Sprint of seeing if we
could have common tools
to release and deploy egg-based packages and zc-buildout based applications,
at the entreprise level.

I summarized it here, if someone is interested in this topic:

http://tarekziade.wordpress.com/2008/04/29/plone-paris-sprint-wrapup-part-1-how-do-you-use-eggs-and-zcbuildout-in-your-projects/

Cheers

Tarek

--
View this message in context: http://www.nabble.com/eggs-zc.buildout-releasing-and-deploying-tp16960238p16960238.html
Sent from the Enterprise mailing list archive at Nabble.com.


_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

[Tarek Ziadé]

On Wed, Apr 30, 2008 at 4:34 AM, Kapil Thangavelu <[hidden email]> wrote:
>
> reading through that blog post reads like utilities and templates not
> enterprise requirements for deployments of eggs.  for pushing eggs out, i
> find pushing them to a directory and adding a find links script works well,
> and doesn't force publication to the cheeseshop. if someone wants that, its
> builtin to setuptools. i'm not sure what the value add of making additional
> tools for pushing bits into directories...

But how do you prepare and deliver a release of a zc.buildout based
app to a customer that
asks you for an installer ? or an upgrade ? and which server does not
have any access to
internet or intranet.

For some of our customers we don't even have access on their servers
and need to provide
something simple that can be installed by them. Same thing for an upgrade.

So you have to "prepare" those releases to ship them. That is what
those tools are meant
for: they automate the preparation of a release or an upgrade from any buildout.

>
> one thing of immense value that any such discussion of enterprise
> deployments with eggs, should include known good set (kgs) facilities, so
> you can get repeatable builds at latter points in time.

We use the versions for that in buildout, to make sure we freeze what is being
delivered.


++
Tarek

>
>
> -kapil
>
>
>
> On Tue, Apr 29, 2008 at 9:17 AM, Tarek Ziadé <[hidden email]> wrote:
> >
> > Hi all,
> >
> > just to let you know: we have discussed at the Paris Sprint of seeing if
> we
> > could have common tools
> > to release and deploy egg-based packages and zc-buildout based
> applications,
> > at the entreprise level.
> >
> > I summarized it here, if someone is interested in this topic:
> >
> >
> http://tarekziade.wordpress.com/2008/04/29/plone-paris-sprint-wrapup-part-1-how-do-you-use-eggs-and-zcbuildout-in-your-projects/
> >
> > Cheers
> >
> > Tarek
> >
> > --
> > View this message in context:
> http://www.nabble.com/eggs-zc.buildout-releasing-and-deploying-tp16960238p16960238.html
> > Sent from the Enterprise mailing list archive at Nabble.com.
> >
> >
> > _______________________________________________
> > Enterprise mailing list
> > [hidden email]
> > http://lists.plone.org/mailman/listinfo/enterprise
> >
>
>
> _______________________________________________
>  Enterprise mailing list
>  [hidden email]
>  http://lists.plone.org/mailman/listinfo/enterprise
>
>

--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/

_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

[ajung]

--On 29. April 2008 06:17:08 -0700 Tarek Ziadé <[hidden email]>
wrote:

>
> Hi all,
>
> just to let you know: we have discussed at the Paris Sprint of seeing if
> we could have common tools
> to release and deploy egg-based packages and zc-buildout based
> applications, at the entreprise level.
>
> I summarized it here, if someone is interested in this topic:
>
> http://tarekziade.wordpress.com/2008/04/29/plone-paris-sprint-wrapup-part
> -1-how-do-you-use-eggs-and-zcbuildout-in-your-projects/
>
>

My 2 cents on using buildout within a big enterprise:

<http://www.zopyx.com/blog/on-enterprise-level-buildouts>

Andreas

_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

[Kapil Thangavelu-5]

On Wed, Apr 30, 2008 at 4:00 AM, Tarek Ziadé <[hidden email]> wrote:

On Wed, Apr 30, 2008 at 4:34 AM, Kapil Thangavelu <[hidden email]> wrote:
>
> reading through that blog post reads like utilities and templates not
> enterprise requirements for deployments of eggs.  for pushing eggs out, i
> find pushing them to a directory and adding a find links script works well,
> and doesn't force publication to the cheeseshop. if someone wants that, its
> builtin to setuptools. i'm not sure what the value add of making additional
> tools for pushing bits into directories...

But how do you prepare and deliver a release of a zc.buildout based
app to a customer that
asks you for an installer ? or an upgrade ? and which server does not
have any access to
internet or intranet.

hi tarek, 

so for deployment from zc.buildout, i setup a download-cache, and push them a tarball of the buildout minus parts. this assumes the deployment server has a compiler setup. on certain paranoid security environments its not or no network access, so i ask them or setup an identical machine with compiler chain, run the buildout tarball there, and give them a binary image tarball. 

in some sense there's always going to be a tradeoff between flexibility and repeatability, from pure egg based /virtualenv development with compoze deployments, to using buildout, to adding convention/structure on top of buildout with iw.releaser, to own all development approach of http://trac.minitage.org/trac/, to pushing virtual machines images. 

i personally prefer the simplicity of not having tools doing potentially magic things underneath the hood in the name of automation, part of that is just familiarity with tools and using things that are well documented/known and in wide use. that said there are definitely parts of iw.releaser that look interesting to me, and i'm playing with it now, to see if it makes sense for my projects. however, for the most part it just seems like a veneer/gloss to the underlying tools. 

 

For some of our customers we don't even have access on their servers
and need to provide
something simple that can be installed by them. Same thing for an upgrade.

pretty much the same as deployment (ie ship a tarball) except its typically just the eggs/dev packages, sometimes with an upgrade script for instance data manipulation if needed. which at least for zodb based systems isn't automate-able. for upgrades on compiled parts, zope, webserver, db, etc require additional manipulation of the buildout environment or reshipping a complete build. for network accessible buildout, a scm update works fine.

 

So you have to "prepare" those releases to ship them. That is what
those tools are meant
for: they automate the preparation of a release or an upgrade from any buildout.

its not clear to me that this is what i would call an upgrade.  actual enterprise deployments need upgrade/migration scripts, vetting of updates etc. the update facility within  iw.releaser is just making an alias to making a tarball.. 

 

>
> one thing of immense value that any such discussion of enterprise
> deployments with eggs, should include known good set (kgs) facilities, so
> you can get repeatable builds at latter points in time.

We use the versions for that in buildout, to make sure we freeze what is being
delivered.

besides kgs,  another nice tool for genarating a frozen index from a working set is  compoze.

http://repoze.org/viewcvs/compoze/trunk/

cheers,

kapil

Tarek

>
>
> -kapil
>
>
>
> On Tue, Apr 29, 2008 at 9:17 AM, Tarek Ziadé <[hidden email]> wrote:
> >
> > Hi all,
> >
> > just to let you know: we have discussed at the Paris Sprint of seeing if
> we
> > could have common tools
> > to release and deploy egg-based packages and zc-buildout based
> applications,
> > at the entreprise level.
> >
> > I summarized it here, if someone is interested in this topic:
> >
> >
> http://tarekziade.wordpress.com/2008/04/29/plone-paris-sprint-wrapup-part-1-how-do-you-use-eggs-and-zcbuildout-in-your-projects/
> >
> > Cheers
> >
> > Tarek
> >
> > --
> > View this message in context:
> http://www.nabble.com/eggs-zc.buildout-releasing-and-deploying-tp16960238p16960238.html
> > Sent from the Enterprise mailing list archive at Nabble.com.
> >
> >
> > _______________________________________________
> > Enterprise mailing list
> > [hidden email]
> > http://lists.plone.org/mailman/listinfo/enterprise
> >
>
>
> _______________________________________________
>  Enterprise mailing list
>  [hidden email]
>  http://lists.plone.org/mailman/listinfo/enterprise
>
>

--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/

_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

[Tarek Ziadé]

On Wed, Apr 30, 2008 at 5:22 PM, Andreas Jung <[hidden email]> wrote:
>  My 2 cents on using buildout within a big enterprise:
>
>  <http://www.zopyx.com/blog/on-enterprise-level-buildouts>
>

Thanks Andreas, that is exactly what I was hoping to have :
people sharing their experience in this area.

So if I understand it correctly, your strategy is to set egg mirrors systems in
production servers, to be able to deploy your apps,

how do you manage upgrades ? (that change both the cfg files, and the code)
Looking forward for part 2 ;)

Tarek

btw:
Some people also reacted on my blog to present a tool called 'minitage', which
seems to be a system-wide tool that takes care of library environments.
They told me they would provide a testable version soon

_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

[Tarek Ziadé]

On Wed, Apr 30, 2008 at 10:12 PM, Kapil Thangavelu <[hidden email]> wrote:
>
> hi tarek,
>
> so for deployment from zc.buildout, i setup a download-cache, and push them
> a tarball of the buildout minus parts. this assumes the deployment server
> has a compiler setup.

Yes same here, we are trying to have a clone server with the same OS, to prepare
the buildout and send the tarball.

> on certain paranoid security environments its not or
> no network access, so i ask them or setup an identical machine with compiler
> chain, run the buildout tarball there, and give them a binary image tarball.

Ok, similar practices here, except that I don't know yet how to pre-built
something that does not depend on the buildout path.

>
> in some sense there's always going to be a tradeoff between flexibility and
> repeatability, from pure egg based /virtualenv development with compoze
> deployments, to using buildout, to adding convention/structure on top of
> buildout with iw.releaser, to own all development approach of
> http://trac.minitage.org/trac/, to pushing virtual machines images.
>
> i personally prefer the simplicity of not having tools doing potentially
> magic things underneath the hood in the name of automation, part of that is
> just familiarity with tools and using things that are well documented/known
> and in wide use. that said there are definitely parts of iw.releaser that
> look interesting to me, and i'm playing with it now, to see if it makes
> sense for my projects. however, for the most part it just seems like a
> veneer/gloss to the underlying tools.
>

Yes, from what you have described, I don't think it will bring you a
lot of help.
besides the project_copy_eggs script

> pretty much the same as deployment (ie ship a tarball) except its typically
> just the eggs/dev packages, sometimes with an upgrade script for instance
> data manipulation if needed. which at least for zodb based systems isn't
> automate-able. for upgrades on compiled parts, zope, webserver, db, etc
> require additional manipulation of the buildout environment or reshipping a
> complete build. for network accessible buildout, a scm update works fine.
>

Ok,

>
> >
> > So you have to "prepare" those releases to ship them. That is what
> > those tools are meant
> > for: they automate the preparation of a release or an upgrade from any
> buildout.
> >
> >
>
> its not clear to me that this is what i would call an upgrade.  actual
> enterprise deployments need upgrade/migration scripts, vetting of updates
> etc. the update facility within  iw.releaser is just making an alias to
> making a tarball..
>

At software level, the migration part is kept in upgrade steps in GS for us,
and we can decide wether to ship a full "eggs" folder or not, depending on
changes applied in the .cfg structure.

When the structure does not change, and this happens for many updates
where just a few eggs are changes, we can use the project_copy_eggs script,

It will let you collect a subset of eggs by scanning the .cfg
files, and doesn't not call a full "bin/buildout", so it creates a small tarball
very quickly.  (but does the dependency work as if the buildout was called)

This is probably overkill for smaller project.

Now I realized that this script misses something : after the
dependency work has been
done, it needs to add dependencies pulled in the tarball as well, so :

$ project_eggs buildout.cfg /tmp/upgrade.tgz "acme.*"

will put plone.some.stuff 1.3 in the tarball if its a dependent

Will fix it.

> besides kgs,  another nice tool for genarating a frozen index from a working
> set is  compoze.
> http://repoze.org/viewcvs/compoze/trunk/

Need to check it

Thanks for your feedback !

Cheers

Tarek

_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise

[ajung]

--On 30. April 2008 22:12:35 +0200 Tarek Ziadé <[hidden email]>
wrote:

> On Wed, Apr 30, 2008 at 5:22 PM, Andreas Jung <[hidden email]> wrote:
>>  My 2 cents on using buildout within a big enterprise:
>>
>>  <http://www.zopyx.com/blog/on-enterprise-level-buildouts>
>>
>
> Thanks Andreas, that is exactly what I was hoping to have :
> people sharing their experience in this area.
>
> So if I understand it correctly, your strategy is to set egg mirrors
> systems in production servers, to be able to deploy your apps,

We don't have mirrors. It is our expectation that the related servers
(zope.org, plone.org. PyPI) will seen an improvement when it comes to
reliability. Redundancy is a must for the future. Although we have our own
big repositories (often with customized code), it is sometimes necessary to
to e.g. to PyPI for picking up an updated recipe.

>
> how do you manage upgrades ? (that change both the cfg files, and the
> code) Looking forward for part 2 ;)


Our deployment is historically driven through CVS tag+branches. We are in
the transition phase to SVN. Eggs play a minor role right now (but this
will hopefully improve over time). Right now lots of things are still
changing on our side and we are trying to figure out how to use the various
options we have with buildout in best and most simple way. At the moment
each project has a hierarchical configuration file that keeps a list of
products/modules (can be SVN checkouts, eggs, distros)...based on paster
variables and some magic we can pull the various sources out of SVN (having
different tags/branches if needed). For doing releases we plan to svn copy
all checkout
modules to a dedicated  directory in SVN and the release configuration for
a particular project would only refer to this dedicated directory within
SVN...sounds complex...it is complex..it is risky because Zope plays a
major role within our company (250 users working with Zope apps internally,
>100 sites running on Zope for external customers, shop system, business
community, internal production workflows running on Zope/Python, a huge
SGML CMS on top of Zope)....moving all this stuff to buildout within a
short period of time is huge undertaking.

Andreas

>
> Tarek
>
> btw:
> Some people also reacted on my blog to present a tool called 'minitage',
> which seems to be a system-wide tool that takes care of library
> environments. They told me they would provide a testable version soon



--
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany
Web: www.zopyx.com - Email: [hidden email] - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting


_______________________________________________
Enterprise mailing list
[hidden email]
http://lists.plone.org/mailman/listinfo/enterprise