Re: [Pharo-dev] Which Character Encodings do you want/need in Pharo ?

Sept. 12, 2014

      Kilon,

On 12 Sep 2014, at 16:51, kilon alios <kilon.alios@gmail.com> wrote:
...
Ok this is strange the ISO 8857-7 is available with Chrome as an encoding , I tried it and it turns Greek to weird signs. Actually I tried also several others available for Greek by Chrome and none works apart from the default which is UTF-8 ....
You misunderstand how this works, let me try to explain using Pharo ;-)

When a website serves a page containing non-ascii content, like ÎµÎ»Î»Î·Î½Î¹ÎºÎ¬, it needs to convert from characters to bytes using an encoding. This is normally done by adding a parameter to the content-type of the page being served. Modern servers default to utf-8.

This is how you start your own HTTP server in Pharo:

(ZnServer defaultOn: 1701)
  onRequestRespond: [ :req |
    ZnResponse ok: (ZnEntity with: 'Do you speak ÎµÎ»Î»Î·Î½Î¹ÎºÎ¬ ?' type: ZnMimeType textPlain) ];
  start.

It will respond with the same text to any request. Try browsing http://localhost:1701

In code, you can do it like this:

ZnClient new get: ZnServer default localUrl.

Here is how you check the content-type being used:

(ZnClient new get: ZnServer default localUrl; response) contentType.

You will see that it is text/plain;charset=utf-8 which is the default used by Zinc when serving textual content. Asking/forcing your browser to interpret these bytes using a different encoding is wrong, since the data was not encoded as such.

But the server could also be using a Greek specific byte encoding, like this:

(ZnServer defaultOn: 1701)
  onRequestRespond: [ :req |
    ZnResponse ok: (ZnEntity 
                      with: 'Do you speak ÎµÎ»Î»Î·Î½Î¹ÎºÎ¬ ?' 
                      type: (ZnMimeType textPlain charSet: 'iso-8859-7')) ];
  start.

Now the page is served differently and browsers/clients know what to do, they will switch their encoding accordingly. Explicitly switching to utf-8 will now fail, as the content is *not* utf-8 encoded.

So why is the option there in the browser ?

Because sometimes, the server is lazy (and technically, wrong) by not explicitly setting the charset/encoding. That leaves the browser/client with a problem of course. For this there is a default, often utf-8, sometimes ascii, ..

Zinc uses the null encoder as default (ZnNullEncoder) when the server fails to set one, you can override this default as follows:

ZnDefaultCharacterEncoder 
  value: ZnUTF8Encoder new
  during: [ ZnClient new get: ZnServer default localUrl ].

ZnDefaultCharacterEncoder 
  value: (ZnCharacterEncoder newForEncoding: 'iso-8859-7')
  during: [ ZnClient new get: ZnServer default localUrl ].

Note however that Zinc will always follow the encoding set by the response (i.e. you cannot force another one, only use one if the encoding is missing).

HTH,

Sven

PS: If you are on Mac OS X or Linux, you can try using curl as a client. ZnClient can return a command line that you can copy/paste into a terminal:

ZnClient new get: ZnServer default localUrl; curl.

$ curl http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP Components 1.0" -H"Host:localhost:1701"

Do you speak ÎµÎ»Î»Î·Î½Î¹ÎºÎ¬ ?

By add the -D - option the incoming HTTP headers will be dumped

$ curl -D - http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP Components 1.0" -H"Host:localhost:1701"

HTTP/1.1 200 OK
Content-Type: text/plain;charset=utf-8
Content-Length: 31
Date: Fri, 12 Sep 2014 20:15:53 GMT
Server: Zinc HTTP Components 1.0

Do you speak ÎµÎ»Î»Î·Î½Î¹ÎºÎ¬ ?
...
Also you can ignore my request as I tried Pharo with True Type fonts with Arial and it can display Greeks just fine. Well done Pharoers , once again Pharo surprises me in a good way :)
On Fri, Sep 12, 2014 at 5:16 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote:
...
I am not in any urgent need as I rarely use (like almost never) my native language in reading or writing  when it comes to anything related to computers, apart from facebook friends. But I would like to see it added to pharo , its ÎÎ»Î»Î·Î½Î¹ÎºÎ¬ or as you know it "Greek".
So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7 - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ?
Is this nice to have or really necessary (as in used by everybody using the Greek language) ?
...
I assume the most popular encondings to have would be Chinese and Hindu ?
I think this something that (native) speakers should help with ;-)
Remember we have full Unicode Character/String capable classes, as well as good UTF-8. Those cover all languages, in theory.
The question comes down to which older encodings are still really used/necessary today.
...
On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding
- calculation of encoding size per Character/String
- proper handling of holes in tables with exceptions (strict/lenient)
- properly backing up one character on a binary stream
- simple, efficient, documented
- convenience API
- optimisations for some encodings
Together, they support the following encodings:
- utf-8
- utf-16 (big/little endian, bom detection)
- null
- iso-8859-1 (latin-1)
- iso-8859-2 (latin-2)
- mac-roman
- cp1250
- cp1252
- cp1253
- koi8r
- iso-8859-7
- iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven