Sys.setlocale for GRASS6.4

[Alexandre VILLERS]

Good afternoon,

Sorry bothering you with this on the list, but one of you might know how to list the locale available on my computer.
In order to try running GRASS6.4 on my machine (with windows XP), Markus Neteler advised me I needed to set locale to "French_France.UTF8"

I tried Sys.setlocale("LC_ALL", locale="French_France.UTF8") but R doesn't seem to find this setting
Warning message:
In Sys.setlocale(category = "LC_ALL", locale = "French_France.UTF8") :
  la requête OS pour spécifier la localisation à "French_France.UTF8" n'a pas pu être honorée, the warning message meaning the request was not available.

How do I check I can use the locale proposed ? Shall I download it from somewhere or change something on windows ?

Thanks for any hints

Alex



Alexandre Villers
PhD. Student
Team Biodiversity
CEBC CNRS
79360 Beauvoir sur Niort


__________ Information from ESET Mail Security, version of virus signature database 4395 (20090904) __________

The message was checked by ESET Mail Security.
http://www.eset.com


_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Markus Neteler]

On Fri, Sep 4, 2009 at 4:06 PM, alexandre
villers<[hidden email]> wrote:
> Good afternoon,
>
> Sorry bothering you with this on the list, but one of you might know how to list the locale available on my computer.
> In order to try running GRASS6.4 on my machine (with windows XP), Markus Neteler advised me I needed to set locale to "French_France.UTF8"

... to give you a bit more background, here excerpts of my related
offlist mail to Alexandre:


On Fri, Sep 4, 2009 at 12:25 AM, Markus Neteler<[hidden email]> wrote:

>> *g.region --interface-description
>>
>> *<?xml version="1.0" encoding="UTF-8"?>
>> <!DOCTYPE task SYSTEM "grass-interface.dtd">
>
> -> it wants UTF8.. but it does not get it:
>
>> > loc<-initGRASS("C:/GRASS", home="C:/temp/", override=T) #intialise GRASS
>> > dans un dossier de travail dans tempdir
>> Input is not proper UTF-8, indicate encoding !
>> Bytes: 0xE9 0x66 0x69 0x6E
>> Error : 1: Input is not proper UTF-8, indicate encoding !
>> Bytes: 0xE9 0x66 0x69 0x6E
>
> because in your system:
>>
>> locale:
>> LC_COLLATE=French_France.1252;LC_CTYPE=French_France.1252;LC_MONETARY=French_France.1252;LC_NUMERIC=C;LC_TIME=French_France.1252
>
> ... this seems to be a candidate.

What I mean: apparently .UTF8 output by GRASS conflicts with expected .1252
input in R.

best
Markus
_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Roger Bivand]

OK, don't worry about that. I'll try to send an updated spgrass6 when I have adequate net access. The problem is that GRASS writes UTF-8 as the encoding into the XML header from --nterface-description, but the French translations seem (on my XP machine) to be in latin1, that is the 0x.. spell eacute (\'{e}) in latex, then "fin" from "défine...", the first non-ASCII string in the output for g.region.

 

The fix is to insert "latin1" into the header, so I've updated the package to allow the user to do this from within R

 

Roger

 

--- Roger Bivand, NHH, Helleveien 30, N-5045 Bergen, [hidden email]

---

Fra: [hidden email] på vegne av alexandre villers
Sendt: fr 04.09.2009 16:06
Til: [hidden email]
Emne: [GRASS-stats] Sys.setlocale for GRASS6.4

Good afternoon,

Sorry bothering you with this on the list, but one of you might know how to list the locale available on my computer.
In order to try running GRASS6.4 on my machine (with windows XP), Markus Neteler advised me I needed to set locale to "French_France.UTF8"

I tried Sys.setlocale("LC_ALL", locale="French_France.UTF8") but R doesn't seem to find this setting
Warning message:
In Sys.setlocale(category = "LC_ALL", locale = "French_France.UTF8") :
  la requête OS pour spécifier la localisation à "French_France.UTF8" n'a pas pu être honorée, the warning message meaning the request was not available.

How do I check I can use the locale proposed ? Shall I download it from somewhere or change something on windows ?

Thanks for any hints

Alex



Alexandre Villers
PhD. Student
Team Biodiversity
CEBC CNRS
79360 Beauvoir sur Niort


__________ Information from ESET Mail Security, version of virus signature database 4395 (20090904) __________

The message was checked by ESET Mail Security.
http://www.eset.com


_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Markus Neteler]

(cc grass-dev)

@grass-dev: There are encoding issues with --interface-description
   which states UTF-8 also then the actual language encoding is different:

2009/9/4 Roger Bivand <[hidden email]>:
> OK, don't worry about that. I'll try to send an updated spgrass6 when I have
> adequate net access. The problem is that GRASS writes UTF-8 as the encoding
> into the XML header from --nterface-description, but the French translations
> seem (on my XP machine) to be in latin1, that is the 0x.. spell eacute
> (\'{e}) in latex, then "fin" from "défine...", the first non-ASCII string in
> the output for g.region.
>
> The fix is to insert "latin1" into the header, so I've updated the package
> to allow the user to do this from within R

This gave me another idea to check the .po files in GRASS 6.4:

grep charset locale/po/grass*_fr.po
locale/po/grasslibs_fr.po:"Content-Type: text/plain; charset=ISO-8859-1\n"
locale/po/grassmods_fr.po:"Content-Type: text/plain; charset=ISO-8859-1\n"
locale/po/grasstcl_fr.po:"Content-Type: text/plain; charset=UTF-8\n"
locale/po/grasswxpy_fr.po:"Content-Type: text/plain; charset=UTF-8\n"

Apparently the translators mixed several charsets instead of using one,
this is valid for various languages supported in GRASS.

@grass-dev: Should we harmonize the encodings to UTF-8 for all/subset
  of languages? If yes, how? With iconv?

@grass-stats: perhaps we need to do our homework first and fix the mess
  if it is a mess...

Markus
_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Glynn Clements]

Markus Neteler wrote:

> @grass-dev: There are encoding issues with --interface-description
>    which states UTF-8 also then the actual language encoding is different:
>
> 2009/9/4 Roger Bivand <[hidden email]>:
> > OK, don't worry about that. I'll try to send an updated spgrass6 when I have
> > adequate net access. The problem is that GRASS writes UTF-8 as the encoding
> > into the XML header from --nterface-description, but the French translations
> > seem (on my XP machine) to be in latin1, that is the 0x.. spell eacute
> > (\'{e}) in latex, then "fin" from "défine...", the first non-ASCII string in
> > the output for g.region.
> >
> > The fix is to insert "latin1" into the header, so I've updated the package
> > to allow the user to do this from within R
>
> This gave me another idea to check the .po files in GRASS 6.4:
>
> grep charset locale/po/grass*_fr.po
> locale/po/grasslibs_fr.po:"Content-Type: text/plain; charset=ISO-8859-1\n"
> locale/po/grassmods_fr.po:"Content-Type: text/plain; charset=ISO-8859-1\n"
> locale/po/grasstcl_fr.po:"Content-Type: text/plain; charset=UTF-8\n"
> locale/po/grasswxpy_fr.po:"Content-Type: text/plain; charset=UTF-8\n"
>
> Apparently the translators mixed several charsets instead of using one,
> this is valid for various languages supported in GRASS.
>
> @grass-dev: Should we harmonize the encodings to UTF-8 for all/subset
>   of languages? If yes, how? With iconv?

No.

For systems using GNU libc, the .mo files use unicode, which is
converted to the locale's encoding automatically at run time. If your
locale uses ISO-8859-1, that's what the program will get regardless of
the encoding used in the .po files.

Additionally, using "historical" encodings provides better
compatibility. Systems which support unicode will typically convert
the .po files to unicode in the .mo files, then convert this to the
locale's encoding at run time, so it doesn't matter which encoding is
used. Systems which don't support unicode will require the .po files
to use the locale's historical encoding (ISO-8859-*, EUC-JP, etc).

The --interface description option needs to either determine the
locale's encoding via e.g. nl_langinfo() and use that in the header,
or convert the data to UTF-8. The latter has the advantage of
relieving the reader of the burden of handling multiple encodings.

--
Glynn Clements <[hidden email]>
_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Markus Neteler]

On Fri, Sep 4, 2009 at 9:26 PM, Glynn Clements<[hidden email]> wrote:
> Markus Neteler wrote:
>> @grass-dev: There are encoding issues with --interface-description
>>    which states UTF-8 also then the actual language encoding is different:

[then -> when]

...
> The --interface description option needs to either determine the
> locale's encoding via e.g. nl_langinfo() and use that in the header,

It seems to be already there?

lib/gis/parser.c:

static void G_usage_xml(void)
{
...
#if defined(HAVE_LANGINFO_H)
    encoding = nl_langinfo(CODESET);
    if (!encoding || strlen(encoding) == 0) {
        encoding = "UTF-8";
    }
#else
    encoding = "UTF-8";
#endif
...
    fprintf(stdout, "<?xml version=\"1.0\" encoding=\"%s\"?>\n", encoding);
    fprintf(stdout, "<!DOCTYPE task SYSTEM \"grass-interface.dtd\">\n");

Apparently the Windows built was missing HAVE_LANGINFO_H or it
isn't properly set on Windows.

?
Markus
_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Roger Bivand]

On Fri, 4 Sep 2009, Markus Neteler wrote:

> On Fri, Sep 4, 2009 at 9:26 PM, Glynn Clements<[hidden email]> wrote:
>> Markus Neteler wrote:
>>> @grass-dev: There are encoding issues with --interface-description
>>>    which states UTF-8 also then the actual language encoding is different:
>
> [then -> when]
>
> ...
>> The --interface description option needs to either determine the
>> locale's encoding via e.g. nl_langinfo() and use that in the header,
>
> It seems to be already there?
>
> lib/gis/parser.c:
>
> static void G_usage_xml(void)
> {
> ...
> #if defined(HAVE_LANGINFO_H)
>    encoding = nl_langinfo(CODESET);
>    if (!encoding || strlen(encoding) == 0) {
>        encoding = "UTF-8";
>    }
> #else
>    encoding = "UTF-8";
> #endif
> ...
>    fprintf(stdout, "<?xml version=\"1.0\" encoding=\"%s\"?>\n", encoding);
>    fprintf(stdout, "<!DOCTYPE task SYSTEM \"grass-interface.dtd\">\n");
>
> Apparently the Windows built was missing HAVE_LANGINFO_H or it
> isn't properly set on Windows.

As a work-around, I've submitted to CRAN a revised version of spgrass6
allowing the user to manipulate the encoding string in the XML data
directly, since Windows users of binary releases cannot get at this
themselves on the GRASS side.

The user would typically run a parseGRASS(<whatever>) command, see a UTF-8
error message, and then try inserting usual suspects with setXMLencoding()
- typically "latin1" - and retry parseGRASS(<whatever>). Users of
initGRASS() will see the UTF-8 error because parseGRASS("g.region") is run
when the function completes.

Roger

>
> ?
> Markus
>

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: [phone]; fax [phone]
e-mail: [hidden email]

_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Glynn Clements]

Markus Neteler wrote:

> >> @grass-dev: There are encoding issues with --interface-description
> >>    which states UTF-8 also then the actual language encoding is different:
>
> [then -> when]
>
> ...
> > The --interface description option needs to either determine the
> > locale's encoding via e.g. nl_langinfo() and use that in the header,
>
> It seems to be already there?
>
> lib/gis/parser.c:
>
> static void G_usage_xml(void)
> {
> ...
> #if defined(HAVE_LANGINFO_H)
>     encoding = nl_langinfo(CODESET);
>     if (!encoding || strlen(encoding) == 0) {
>         encoding = "UTF-8";
>     }

That would explain the sense of déjà vu ;)

> Apparently the Windows built was missing HAVE_LANGINFO_H or it
> isn't properly set on Windows.

This probably won't exist on Windows. I think that we can get the
information from locale_charset(), declared in localcharset.h and
defined in both libintl and libgettext.

Can someone try the attached patch?

--
Glynn Clements <[hidden email]>


Index: lib/gis/parser.c
===================================================================
--- lib/gis/parser.c (revision 39007)
+++ lib/gis/parser.c (working copy)
@@ -77,6 +77,9 @@
 #if defined(HAVE_LANGINFO_H)
 #include <langinfo.h>
 #endif
+#if defined(__MINGW32__) && defined(USE_NLS)
+#include <localcharset.h>
+#endif
 
 #include <stdio.h>
 #include <stdlib.h>
@@ -1311,6 +1314,11 @@
     if (!encoding || strlen(encoding) == 0) {
  encoding = "UTF-8";
     }
+#elif defined(__MINGW32__) && defined(USE_NLS)
+    encoding = locale_charset();
+    if (!encoding || strlen(encoding) == 0) {
+ encoding = "UTF-8";
+    }
 #else
     encoding = "UTF-8";
 #endif

_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats

[Markus Neteler]

(for the record)

On Sat, Sep 5, 2009 at 8:55 PM, Glynn Clements <[hidden email]> wrote:
> Markus Neteler wrote:
>> >> @grass-dev: There are encoding issues with --interface-description
...
>> Apparently the Windows built was missing HAVE_LANGINFO_H or it
>> isn't properly set on Windows.
>
> This probably won't exist on Windows. I think that we can get the
> information from locale_charset(), declared in localcharset.h and
> defined in both libintl and libgettext.
>
> Can someone try the attached patch?

Glynn's patch has been backported to 6.5.svn and 6.4.svn. Hopefully
the problem is gone now.

Markus
_______________________________________________
grass-stats mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-stats