Next generation multilingual support

[Jean Jordaan]

> If you would want to revive i18nlayer, i am sure there would be some
> people that would be happy. — But my experience is that we spent too
> many weeks hacking on what turned out to be an obviously flawed idea
> (i18nLayer) to ever go down that path again.

I've been explaining what I like about I18NLayer, which means that I'm
clearly missing the obvious flaws that put it out of the running. In
fact, I wouldn't be surprised if some of the things I like about it
count as obvious flaws to you.

--
jean                                              . .. .... //\\\oo///\\

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Martin Aspeli]

Jean Jordaan wrote:
>> If you would want to revive i18nlayer, i am sure there would be some
>> people that would be happy. — But my experience is that we spent too
>> many weeks hacking on what turned out to be an obviously flawed idea
>> (i18nLayer) to ever go down that path again.
>
> I've been explaining what I like about I18NLayer, which means that I'm
> clearly missing the obvious flaws that put it out of the running. In
> fact, I wouldn't be surprised if some of the things I like about it
> count as obvious flaws to you.

This is probably true.

I do see a few rather important flaws in this approach though.

I think when I18NLayer was used, we didn't use the catalog to build a
navtree or tabs or folder listings. Now we do, and that makes something
like this very hard.

For example, let's say we have this tree:

  /song
  /song/en
  /song/de
  /dance
  /dance/en
  /dance/de
  /dance/waltz
  /dance/waltz/en
  /dance/waltz/de

Now, let's say we use the catalog to build the top level navigation
tabs. The search finds everything in the root of the site. So it'll find
/song and /dance. It won't find /en or /de. And the catalog metadata on
/song and /dance (which are i18nlayer objects) can by definition only
contain information in one language - probably whichever language the
user was in when that particular object was cataloged.

For tabs (and folder listings, which is the same problem: a search with
depth 1 from a given root) it's probably solvable. You can look for
depth 2 and do some kind of filtering.

For navtrees, it gets hellish. The navtree search from the
ExtendedPathIndex would find the following if you're viewing
/dance/waltz/en:

   /song
   /dance
   /dance/en
   /dance/de
   /dance/waltz
   /dance/waltz/en
   /dance/waltz/de

We could filter out the 'de' ones, of course. But you'd have the same
problem getting language dependent information for /song and /dance and
/dance/waltz. Fixing that problem is not at all trivial.

There are some other issues too. These days, most views are registered
for the interface of the type they pertain to. So /alpha/@@view and
/beta/@@view can be two different things. You need to trick the
i18nlayer object at /song into providing the same interfaces as /song/en
for /song/@@view to even find the right object.

Then there's the traversal issue: If you are traversing to /song
directly, it's pretty straightforward: you can go to /song/en in the
same way we go to a default-page or an index_html object (that's
basically what I18NLayer does: it implements an index_html method). But
when you're going to /song/@@view, the index_html thing doesn't kick in.
The publisher will try to look up the view on /song and it'll fail.

That problem is actually solvable (and I'm experimenting with solutions
in collective.alias that may apply to this as well, although my interest
is something a bit different). But it gets messy and it breaks down more
quickly.

There's a performance problem as well: every time you traverse to
anything that's an i18nlayer object, you need to load two persistent
objects: the i18nlayer object, and the 'real' object, plus of course
whatever separate objects hang off either of those. That means your ZODB
cache fills more quickly and scaling becomes harder. It probably isn't a
huge problem, but it's worth bearing in mind.

One final issue I can think of: I think it'd be relatively common to
want to assign local roles for a given subtree in a given language. I
want user John to be able to edit everything in /dance in English, and
user Wolfgang to edit everything that's German in the same folder. This
is actually impossible: I can assign a local role to /dance, but it'll
apply to all content, in all languages. I can also assign a role to
/dance/en, but it won't inherit into /dance/waltz/en since that's not a
child of /dance/en.

Those, I think, are the "hard" problems. I also think it's hard to get
the UI right for this kind of thing, given that we these days have a lot
more stuff going on in the UI, and I'm sure there would be other things
that'd need to be adjusted or worked around I haven't thought about.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Jean Jordaan]

Hi there

> I think when I18NLayer was used, we didn't use the catalog to build a
> navtree or tabs or folder listings. Now we do, and that makes
> something like this very hard.

Agreed, catalog-based navigation won't Just Work. At the least, in a
site with translations, you'd have to query n+1 levels deep if you only
want to see n levels (you want to see /dance/waltz, but you have to
query down to /dance/waltz/{en,de}). Then you'd have to filter
i18nlayers out of the results (drop /dance/waltz). And then you'd have
to munge the paths shown to that of the layer (i.e. while displaying the
catalog data from the record of /dance/waltz/en, link to /dance/waltz)
(or not -- link to /dance/walts/en if you like, as discussed in my
previous post). And, again as mentioned previously, if you want fallback
and have both 'en' and 'de', you'd drop 'de'. If you don't want
fallback, drop all 'de'.

> Now, let's say we use the catalog to build the top level navigation
> tabs. The search finds everything in the root of the site. So it'll find
> /song and /dance. It won't find /en or /de. And the catalog metadata on
> /song and /dance (which are i18nlayer objects) can by definition only
> contain information in one language - probably whichever language the
> user was in when that particular object was cataloged.

Per the above: search root and one level down. Don't use metadata from
i18nlayer objects. Show data from the record of the appropriate actual
document, not the i18nlayer.

I honestly don't think that a multilingual site should be expected to
work exactly the same as a monolingual site. I think it's quite fair to
do some work behind the scenes to accommodate the presence of
translations.

> We could filter out the 'de' ones, of course. But you'd have the same
> problem getting language dependent information for /song and /dance and
> /dance/waltz. Fixing that problem is not at all trivial.

Getting language dependent information for /song and /dance is a red
herring. They aren't content objects and shouldn't show up in
navigation.

> There are some other issues too. These days, most views are registered
> for the interface of the type they pertain to. So /alpha/@@view and
> /beta/@@view can be two different things. You need to trick the
> i18nlayer object at /song into providing the same interfaces as
> /song/en for /song/@@view to even find the right object.

Now this is quite another kettle of fish. If we can substitute tricks,
and -- instead of making /song duplicate /song/en's interface -- rather
make traversing to /song return /song/en so that the @@view gets looked
up on the proper object, that would be great.

> But when you're going to /song/@@view, the index_html thing doesn't
> kick in. The publisher will try to look up the view on /song and it'll
> fail.

Traversal retrieves /song before applying the view, right? Isn't it
possible to make /song fetch /song/en and stick it in front of itself?

__before_publishing_traverse__: "the request object also provides
methods that change the path segments still to be traversed. Therefore,
the hook can drastically change the traversal. This is used for example
to implement virtual hosting or to facilitate internationalization.
Products that use this hook are for example SiteAccess2 (for virtual
hosting) and Localizer (a localization tool)."
(I see that adapting to IPublishTraverse has replaced
__before_publishing_traverse__, but I'm sure you know more about that
than I do.)

__bobo_traverse__: "this function is called with the request object and
the segment as parameters to determine the next object [...] replacing
the current object. This hook is used, for example, to implement
advanced features of ZSQL methods such as direct traversal."

http://wiki.zope.org/zope2/ZPublisher

> There's a performance problem as well: every time you traverse to
> anything that's an i18nlayer object, you need to load two persistent
> objects: the i18nlayer object, and the 'real' object, plus of course
> whatever separate objects hang off either of those. That means your ZODB
> cache fills more quickly and scaling becomes harder. It probably isn't a
> huge problem, but it's worth bearing in mind.

I seriously doubt that this problem is unique to this approach to
translation.

> One final issue I can think of: I think it'd be relatively common to
> want to assign local roles for a given subtree in a given language. I
> want user John to be able to edit everything in /dance in English, and
> user Wolfgang to edit everything that's German in the same folder. This
> is actually impossible: I can assign a local role to /dance, but it'll
> apply to all content, in all languages. I can also assign a role to
> /dance/en, but it won't inherit into /dance/waltz/en since that's not a
> child of /dance/en.

Add a guard expression to the 'edit' action which checks
'<language>-editors-group' membership.

Or add a content rule (not that I've used one!) running after save which
shares any document with '<language>-editors-group', give editing rights
to those groups, and put people into them.

Whatever route we choose, it won't suit all comers. Personally I think
your strawman is relatively less common than this one: assign the dance
instructors to supply the contents of /dance. In the case of separate
language trees, you'd have to assign the same local role to each tree,
boring. I don't think Wolfgang would be able to get John to update the
German translations for him even if he offered to buy beer ;-)

> I think, are the "hard" problems. I also think it's hard to get the UI
> right for this kind of thing, given that we these days have a lot more
> stuff going on in the UI, and I'm sure there would be other things
> that'd need to be adjusted or worked around I haven't thought about.

I absolutely agree that it's hard to get the UI right, which is one of
the reasons I'm pushing the I18NLayer style, as it felt quite OK to me.
And other things you or me haven't thought about can't really be counted
as a drawback of I18NLayer ..

--
jean                                              . .. .... //\\\oo///\\

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Martin Aspeli]

Hi Jean,

> Agreed, catalog-based navigation won't Just Work.

Or "just won't work"? :-)

> At the least, in a
> site with translations, you'd have to query n+1 levels deep if you only
> want to see n levels (you want to see /dance/waltz, but you have to
> query down to /dance/waltz/{en,de}). Then you'd have to filter
> i18nlayers out of the results (drop /dance/waltz). And then you'd have
> to munge the paths shown to that of the layer (i.e. while displaying the
> catalog data from the record of /dance/waltz/en, link to /dance/waltz)
> (or not -- link to /dance/walts/en if you like, as discussed in my
> previous post). And, again as mentioned previously, if you want fallback
> and have both 'en' and 'de', you'd drop 'de'. If you don't want
> fallback, drop all 'de'.

Right. I think once you start pulling on the threads, you'll be sitting
with a lot of the sweater in your lap.

> Per the above: search root and one level down. Don't use metadata from
> i18nlayer objects. Show data from the record of the appropriate actual
> document, not the i18nlayer.

Yes, you have to do that.

> I honestly don't think that a multilingual site should be expected to
> work exactly the same as a monolingual site. I think it's quite fair to
> do some work behind the scenes to accommodate the presence of
> translations.

Sure, but we need a solution that works without requiring a lot of
customisation and integration, and which is maintainable.

> Getting language dependent information for /song and /dance is a red
> herring. They aren't content objects and shouldn't show up in
> navigation.

Well, but this is the problem. Plone isn't really geared up to work with
'non-content' interleaved with content.

> Now this is quite another kettle of fish. If we can substitute tricks,
> and -- instead of making /song duplicate /song/en's interface -- rather
> make traversing to /song return /song/en so that the @@view gets looked
> up on the proper object, that would be great.

Yes, but this kind of traversal trickery is not easy to do without
monkey patching the publisher and working around a lot of issues with
acquisition and security and urls and paths.

> Traversal retrieves /song before applying the view, right? Isn't it
> possible to make /song fetch /song/en and stick it in front of itself?

Maybe. :)

> __before_publishing_traverse__: "the request object also provides
> methods that change the path segments still to be traversed. Therefore,
> the hook can drastically change the traversal. This is used for example
> to implement virtual hosting or to facilitate internationalization.
> Products that use this hook are for example SiteAccess2 (for virtual
> hosting) and Localizer (a localization tool)."
> (I see that adapting to IPublishTraverse has replaced
> __before_publishing_traverse__, but I'm sure you know more about that
> than I do.)
>
> __bobo_traverse__: "this function is called with the request object and
> the segment as parameters to determine the next object [...] replacing
> the current object. This hook is used, for example, to implement
> advanced features of ZSQL methods such as direct traversal."
>
> http://wiki.zope.org/zope2/ZPublisher

Yeah, the solution is somewhere in those things. I'm not sure it's going
to work seamlessly, though. I guess someone will need to try it. And
then they'll also need to try to rewrite the way Plone navigation works. :)

> I seriously doubt that this problem is unique to this approach to
> translation.
>
>> One final issue I can think of: I think it'd be relatively common to
>> want to assign local roles for a given subtree in a given language. I
>> want user John to be able to edit everything in /dance in English, and
>> user Wolfgang to edit everything that's German in the same folder. This
>> is actually impossible: I can assign a local role to /dance, but it'll
>> apply to all content, in all languages. I can also assign a role to
>> /dance/en, but it won't inherit into /dance/waltz/en since that's not a
>> child of /dance/en.
>
> Add a guard expression to the 'edit' action which checks
> '<language>-editors-group' membership.

That's not the same thing. It's a hacky workaround for this particular
example, and it means you can't use standard security features.

> Or add a content rule (not that I've used one!) running after save which
> shares any document with '<language>-editors-group', give editing rights
> to those groups, and put people into them.

Ouch.

> Whatever route we choose, it won't suit all comers. Personally I think
> your strawman is relatively less common than this one: assign the dance
> instructors to supply the contents of /dance. In the case of separate
> language trees, you'd have to assign the same local role to each tree,
> boring. I don't think Wolfgang would be able to get John to update the
> German translations for him even if he offered to buy beer ;-)

Huh? The dance instructors will own the /dance tree in the canonical
language. When they make updates, translation workflow should kick in.
John owns the English content. Wolfgang owns the German content.

> I absolutely agree that it's hard to get the UI right, which is one of
> the reasons I'm pushing the I18NLayer style, as it felt quite OK to me.
> And other things you or me haven't thought about can't really be counted
> as a drawback of I18NLayer ..

I'm not saying this isn't the right approach. But my gut feel (and the
advice of the people who've probably done the most work with large scale
multilingual sites - Jarn) tells me that the approach with separate
language trees will be more in line with how Plone works today, and
easier to make easy to use, and more consistent with how other
multilingual systems work. I may be wrong. But I think if *I* am to
invest any time in implementing this, it's the solution that feels least
like it's going to eat a lot of my time before we find out it won't work
for one reason or another. ;)

That said, I think it's important that we have a solution that allows
people to build something like i18nlayer as a policy option without
throwing away everything and starting again.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Jean Jordaan]

Hi there

>> Agreed, catalog-based navigation won't Just Work.
>
> Or "just won't work"? :-)

That's a cheap shot. There's a whole summer of code project on making
navigation work for large folders of perfectly normal content.

There are certainly other ways of approaching this. Perhaps a
translation-aware Index. I just verbosely described how even the
current system could deal with it.

> Right. I think once you start pulling on the threads, you'll be sitting
> with a lot of the sweater in your lap.

Well, I did it once about 5 years ago, and it worked OK as far as I remember ..

>> I honestly don't think that a multilingual site should be expected to
>> work exactly the same as a monolingual site. I think it's quite fair to
>> do some work behind the scenes to accommodate the presence of
>> translations.
>
> Sure, but we need a solution that works without requiring a lot of
> customisation and integration, and which is maintainable.

Those properties are always desirable. It would be helpful if you said
how the approach I'm advocating is not maintainable, and imposes an
excessive customisation and integration burden.

It needs support in the navigation code, but that is because
navigation in a multilingual site is different from that in a
monolingual site. Similarly, permissions need support in the
navigation code, so that only accessible items are listed.

I think ATContentTypes work unchanged, but I don't know about custom
applications built on Plone. AT-based types should work. I think even
PloneArticle (which is a folder masquerading as a page) should work,
though you'd have a copy of article images per translation. Should
Plone aim to cater for all 3rd party content types? I think a solution
which only allows "plain" content to be translated is OK (e.g.
folder/page/event/review/image/file/...).

>> Getting language dependent information for /song and /dance is a red
>> herring. They aren't content objects and shouldn't show up in
>> navigation.
>
> Well, but this is the problem. Plone isn't really geared up to work with
> 'non-content' interleaved with content.

That's taking an easy way out. Is a portlet content or not? Is a
content rule? Or smart folder criteria? Or comments? Or images (Geir
suggested that they shouldn't be content objects unless in an album)?

The answer is that it depends on the point of view of the user. From
the point of view of a member of the public, it's not. From the point
of view of a translator, it is (it's the container that keeps
translations together). My use of the term 'content' was sloppy.

> Yes, but this kind of traversal trickery is not easy to do without
> monkey patching the publisher and working around a lot of issues with
> acquisition and security and urls and paths.

It's not trivial, but you seem to be suggesting it's something of a
derelict mineshaft, to be approached only by officials in order to
erect a "no entry" sign before they back away hurriedly.

>>> One final issue I can think of: I think it'd be relatively common to
>>> want to assign local roles for a given subtree in a given language. I
>>> want user John to be able to edit everything in /dance in English, and
>>> user Wolfgang to edit everything that's German in the same folder.

[...]

> Huh? The dance instructors will own the /dance tree in the canonical
> language. When they make updates, translation workflow should kick in.
> John owns the English content. Wolfgang owns the German content.

So Wolfgang is not a dance instructor? Because if he were, he'd also
be an owner of the /dance tree in the canonical language, right?

I admit the solutions I suggested were hackish. Perhaps I should just
have stuck with:
"I think there are well defined answers to all of those that make sense.
I don't want to elaborate them all here." ;->

Note that the i18nlayer approach doesn't require a canonical language.
You can add documents in any site language. It may be useful to track
which document is a translation of which other document. Even if the
canonical language is German, the Afrikaans translator most probably
used the English translation as source. Even if the canonical language
of the site is English, it may be that some document exists in German
and needs to be translated in order to be added to the site in the
first place, where the German original will feature as a translation.
However, tracking how translations relate to each other is a different
story.

> I'm not saying this isn't the right approach. But my gut feel (and the
> advice of the people who've probably done the most work with large scale
> multilingual sites - Jarn) tells me that the approach with separate
> language trees will be more in line with how Plone works today, and
> easier to make easy to use, and more consistent with how other
> multilingual systems work.

I appreciate that. I only wanted to try and state the i18nlayer case.
The reason I did so is that Jarn categorically denies the validity of
any use case but that supported by LinguaPlone (except to recommend
using Plone as is). And it isn't my use case.

There are already some examples of other use cases in this thread.
Perhaps it would help if the people who brought them up gave more
detail.

> That said, I think it's important that we have a solution that allows
> people to build something like i18nlayer as a policy option without
> throwing away everything and starting again.

In that case, I've succeeded in what I wanted to do :-)

Regards,
--
jean                                              . .. .... //\\\oo///\\

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Martin Aspeli]

Jean Jordaan wrote:
> Hi there
>
>>> Agreed, catalog-based navigation won't Just Work.
>> Or "just won't work"? :-)
>
> That's a cheap shot.

Or a joke?

> It's not trivial, but you seem to be suggesting it's something of a
> derelict mineshaft, to be approached only by officials in order to
> erect a "no entry" sign before they back away hurriedly.

No, I'm not. Look, if you want to work on this, I'm not going to stop
you. If I have given that impression, I'm sorry. But I don't believe
it's going to be a good default approach. That's me making a judgement
based on my experience writing weird things for Plone. I'd be happy to
be proven wrong.

> I admit the solutions I suggested were hackish. Perhaps I should just
> have stuck with:
> "I think there are well defined answers to all of those that make sense.
> I don't want to elaborate them all here." ;->

I'm not really interested in mud slinging here, so I'm going to stop
this now. The approach and pros and cons are well explored and
documented at this point.

> There are already some examples of other use cases in this thread.
> Perhaps it would help if the people who brought them up gave more
> detail.

I think I've tried pretty hard to bring out the various use cases and
approaches. If you feel differently, I've clearly failed.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Jean Jordaan]

Hi Martin

I agree that we've thrashed out the pros and cons of this approach
about as well as can be done at this point.

I'm sorry that you seem to be getting a negative vibe. I don't think I
was being unfair in pointing out that you declined to address my list
of questions (by quoting your response, admittedly cheekily). Not that
you are under any obligation to do so, of course.

>> There are already some examples of other use cases in this thread.
>> Perhaps it would help if the people who brought them up gave more
>> detail.
>
> I think I've tried pretty hard to bring out the various use cases and
> approaches. If you feel differently, I've clearly failed.

Well, the thing which led me to bring this up, and to persist this
far, is the specific use case of sites where not everything is
translated, and where not every document originates in the same
canonical language, which is not supported by LinguaPlone.

I didn't say you deny other use cases, I said that Jarn does. You set
out the various use cases very clearly.

--
jean                                              . .. .... //\\\oo///\\

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers