i18ndude "sync" error?

[John Trammell]

Hi all:

Is there an instance of trac or poi set up for i18ndude?  I think I've found an error in its 'sync' behavior when the msgid contains line breaks.

Thanks,
JT

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Hanno Schlichting-4]

On Wed, Aug 26, 2009 at 10:56 PM, John Trammell<[hidden email]> wrote:
> Is there an instance of trac or poi set up for i18ndude?  I think I've found an error in its 'sync' behavior when the msgid contains line breaks.

It used to have a Poi tracker on plone.org, but that got spammed too
badly, so I had to shut it down.

So this mailing list is the best place to report things for now.

Hanno

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[John Trammell]

> Hanno Schlichting wrote:
> It used to have a Poi tracker on plone.org, but that got spammed too
> badly, so I had to shut it down.
>
> So this mailing list is the best place to report things for now.

OK, I have a number of multiline phrases in some python source, e.g.

return _('''This is a really really really
really long phrase.''')

This ends up (via i18ndude rebuild-pot) in a .pot file looking like this:

#: foo/bar.py:123
msgid "This is a really really really
really long phrase."
msgstr ""

When I call i18ndude sync, this phrase does not even make it into the .po file, but other phrases (with msgids not having newlines) do.

Some other features:

* if the source string has a "\" at the end of each internal line, then the .pot msgids do not contain newlines.
* Editing the .pot file to look like the following makes the sync succeed:

msgid "This is a really really really "
"really long phrase."
msgstr ""

Let me know how you want to proceed with this.

Thanks,
J
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[askvictor]

On Thu, Aug 27, 2009 at 7:18 AM, Hanno Schlichting<[hidden email]> wrote:
> On Wed, Aug 26, 2009 at 10:56 PM, John Trammell<[hidden email]> wrote:
>> Is there an instance of trac or poi set up for i18ndude?  I think I've found an error in its 'sync' behavior when the msgid contains line breaks.
>
> It used to have a Poi tracker on plone.org, but that got spammed too
> badly, so I had to shut it down.
>
> So this mailing list is the best place to report things for now.

OK, I got one too!

I have a .pt file for creating RSS2 feeds. RSS2 is XML, but is not
HTML. In particular, the <link> tag in HTML is an empty tag (i.e.
never has any content), while in RSS2 it needs content (the URI of
some item). The problem is, i18ndude treats .pt (in fact .*pt) files
as HTML, and puts them through the HTMLTALParser, which fails when it
gets a non-empty <link> tag. This occurs around line 472 in
i18ndude/extract.py.

Possible solutions include using the TALParser for .*pt files rather
than the HTMLParser, or using a smarter method than the file extension
to determine the filetype (e.g. checking for an html or xml string
near the start of the file, or using python-magic)

ciao

vik

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Vincent Fretin]

On Thu, Aug 27, 2009 at 7:40 AM, victor rajewski <[hidden email]> wrote:

On Thu, Aug 27, 2009 at 7:18 AM, Hanno Schlichting<[hidden email]> wrote:
> On Wed, Aug 26, 2009 at 10:56 PM, John Trammell<[hidden email]> wrote:
>> Is there an instance of trac or poi set up for i18ndude?  I think I've found an error in its 'sync' behavior when the msgid contains line breaks.
>
> It used to have a Poi tracker on plone.org, but that got spammed too
> badly, so I had to shut it down.
>
> So this mailing list is the best place to report things for now.

OK, I got one too!

I have a .pt file for creating RSS2 feeds. RSS2 is XML, but is not
HTML. In particular, the <link> tag in HTML is an empty tag (i.e.
never has any content), while in RSS2 it needs content (the URI of
some item). The problem is, i18ndude treats .pt (in fact .*pt) files
as HTML, and puts them through the HTMLTALParser, which fails when it
gets a non-empty <link> tag. This occurs around line 472 in
i18ndude/extract.py.

I'm not sure, but do your link is closed like this:
<link attr1="" attr2="" />
?
If this don't resolve the problem, can you send me your file so I can take a look at it?

Vincent

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Vincent Fretin]

On Thu, Aug 27, 2009 at 12:09 AM, John Trammell <[hidden email]> wrote:

> Hanno Schlichting wrote:
> It used to have a Poi tracker on plone.org, but that got spammed too
> badly, so I had to shut it down.
>
> So this mailing list is the best place to report things for now.

OK, I have a number of multiline phrases in some python source, e.g.

return _('''This is a really really really
really long phrase.''')

This ends up (via i18ndude rebuild-pot) in a .pot file looking like this:

#: foo/bar.py:123
msgid "This is a really really really
really long phrase."
msgstr ""

When I call i18ndude sync, this phrase does not even make it into the .po file, but other phrases (with msgids not having newlines) do.

Some other features:

* if the source string has a "\" at the end of each internal line, then the .pot msgids do not contain newlines.
* Editing the .pot file to look like the following makes the sync succeed:

msgid "This is a really really really "
"really long phrase."
msgstr ""

Let me know how you want to proceed with this.

Thanks,

J

Hi,

Can you try with a msgid like this

_('text-long-long-phrase', default="""This is a really really really
really long phrase.""")

Vincent


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Maurits van Rees-3]

Hanno Schlichting, on 2009-08-26:
> On Wed, Aug 26, 2009 at 10:56 PM, John Trammell<[hidden email]> wrote:
>> Is there an instance of trac or poi set up for i18ndude?  I think I've found an error in its 'sync' behavior when the msgid contains line breaks.
>
> It used to have a Poi tracker on plone.org, but that got spammed too
> badly, so I had to shut it down.
>
> So this mailing list is the best place to report things for now.

The Poi version currently used on plone.org now has a workflow state
where only members can add issues and responses; that should help if
we want to resurrect the issue tracker for i18ndude.  It is the
default state for new trackers now and it looks like all trackers have
been put in that state (at least the tracker for Poi itself is in this
state, which I did not do myself).


BTW, the i18ndude products page gives an internal server error.  The
suggested old.plone.org page does not work either.  I have to add
base_view to the end to see something useful:

http://plone.org/products/i18ndude/base_view

and that page still shows an error while rendering a portlet.

Sounds like the display setting is not correct.  Can you take a look
at that?

--
Maurits van Rees | http://maurits.vanrees.org/
            Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Hanno Schlichting-4]

On Thu, Aug 27, 2009 at 9:46 AM, Maurits van
Rees<[hidden email]> wrote:

> BTW, the i18ndude products page gives an internal server error.  The
> suggested old.plone.org page does not work either.  I have to add
> base_view to the end to see something useful:
>
> http://plone.org/products/i18ndude/base_view
>
> and that page still shows an error while rendering a portlet.
>
> Sounds like the display setting is not correct.  Can you take a look
> at that?

I tried to delete the i18ndude and i18ntestcase entries from the
products section, as these hadn't been updated in more than three
years. They still showed the 2.1.1 version as most recent which is
severely outdated.

plone.org produced lots of errors while trying to do anything with it
(Varnish errors, not found, timeout) ... so it ended up in some
half-deleted state. Today I still wasn't able to get rid of them
entirely, but removed enough of them, so they don't show up anymore.

Hanno

P.S. I know there is some value in publishing add-ons on plone.org
instead of only PyPi, but the overhead is too much for me ;)

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[askvictor]

On Thu, Aug 27, 2009 at 8:04 AM, Vincent Fretin<[hidden email]> wrote:

> On Thu, Aug 27, 2009 at 7:40 AM, victor rajewski <[hidden email]>
>> I have a .pt file for creating RSS2 feeds. RSS2 is XML, but is not
>> HTML. In particular, the <link> tag in HTML is an empty tag (i.e.
>> never has any content), while in RSS2 it needs content (the URI of
>> some item). The problem is, i18ndude treats .pt (in fact .*pt) files
>> as HTML, and puts them through the HTMLTALParser, which fails when it
>> gets a non-empty <link> tag. This occurs around line 472 in
>> i18ndude/extract.py.
>
> I'm not sure, but do your link is closed like this:
> <link attr1="" attr2="" />
> ?

The link tag is not closed, because it is not meant to be. In RSS, the
link tag looks like:
<link>http://foo.bar/blah.html</link>
which is what my template has. However, this is not valid HTML, which expects
<link />

The issue is with i18ndude expecting all .*pt files to be HTML. It is
possible they are XML.

Cheers,

vik

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Vincent Fretin]

On Fri, Aug 28, 2009 at 12:40 AM, victor rajewski <[hidden email]> wrote:

On Thu, Aug 27, 2009 at 8:04 AM, Vincent Fretin<[hidden email]> wrote:
> On Thu, Aug 27, 2009 at 7:40 AM, victor rajewski <[hidden email]>

>> I have a .pt file for creating RSS2 feeds. RSS2 is XML, but is not
>> HTML. In particular, the <link> tag in HTML is an empty tag (i.e.
>> never has any content), while in RSS2 it needs content (the URI of
>> some item). The problem is, i18ndude treats .pt (in fact .*pt) files
>> as HTML, and puts them through the HTMLTALParser, which fails when it
>> gets a non-empty <link> tag. This occurs around line 472 in
>> i18ndude/extract.py.
>
> I'm not sure, but do your link is closed like this:
> <link attr1="" attr2="" />
> ?

The link tag is not closed, because it is not meant to be. In RSS, the
link tag looks like:
<link>http://foo.bar/blah.html</link>
which is what my template has. However, this is not valid HTML, which expects
<link />

The issue is with i18ndude expecting all .*pt files to be HTML. It is
possible they are XML.

Cheers,

vik

Oh yes, I understand. Sorry, I read too quickly your previous post.

And if you replace HTMLParser by TALParser like you said, you solved the problem, right?
What's the difference between the two parsers by the way?

Vincent

 

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[John Trammell]

> From: Vincent Fretin [[hidden email]]
> Sent: Thursday, August 27, 2009 2:07 AM
> To: John Trammell
> Cc: Hanno Schlichting; [hidden email]
> Subject: Re: [Plone-i18n] i18ndude "sync" error?
>
> Hi,
>
> Can you try with a msgid like this
>
> _('text-long-long-phrase', default="""This is a really really really
> really long phrase.""")

That appears to be a suitable workaround for this bug.  Thanks for the suggestion!

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[askvictor]

On Fri, Aug 28, 2009 at 5:33 PM, Vincent Fretin<[hidden email]> wrote:

> On Fri, Aug 28, 2009 at 12:40 AM, victor rajewski <[hidden email]>
> wrote:
>>
>> On Thu, Aug 27, 2009 at 8:04 AM, Vincent Fretin<[hidden email]>
>> wrote:
>> > On Thu, Aug 27, 2009 at 7:40 AM, victor rajewski <[hidden email]>
>> >> I have a .pt file for creating RSS2 feeds. RSS2 is XML, but is not
>> >> HTML. In particular, the <link> tag in HTML is an empty tag (i.e.
>> >> never has any content), while in RSS2 it needs content (the URI of
>> >> some item). The problem is, i18ndude treats .pt (in fact .*pt) files
>> >> as HTML, and puts them through the HTMLTALParser, which fails when it
>> >> gets a non-empty <link> tag. This occurs around line 472 in
>> >> i18ndude/extract.py.
>> >
>> > I'm not sure, but do your link is closed like this:
>> > <link attr1="" attr2="" />
>> > ?
>>
>> The link tag is not closed, because it is not meant to be. In RSS, the
>> link tag looks like:
>> <link>http://foo.bar/blah.html</link>
>> which is what my template has. However, this is not valid HTML, which
>> expects
>> <link />
>>
>> The issue is with i18ndude expecting all .*pt files to be HTML. It is
>> possible they are XML.
>>
>> Cheers,
>>
>> vik
>
> Oh yes, I understand. Sorry, I read too quickly your previous post.
>
> And if you replace HTMLParser by TALParser like you said, you solved the
> problem, right?
> What's the difference between the two parsers by the way?

TALParser parses XML files, HTMLTALParser parses HTML files. I tried
getting rid of HTMLTALParser and using just TALParser, but since HTML
is not (necessarily) valid XML, this fails on some .pt files. So I
tried using python-magic to determine the filetype magically. This
works fine if the HTML template has <html> tags, but included
templates might not, so I opted for this approach, which for files
.*pt tries to work out if they are XML, otherwise uses TALParser. Not
a great approach, and I don't think 'magic' is available for windows
systems, but it worked for me.

Here is the diff to extract.py:

458,462d457
<     #for determining filetype based on more than extension
<     import magic
<     filetypes = magic.open(magic.MAGIC_NONE)
<     filetypes.load()
<
478,481c473
<                 if filetypes.file(filename) == 'XML':
<                     p = TALParser()
<                 else:
<                     p = HTMLTALParser()
---
>                 p = HTMLTALParser()


ciao

vik

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n