On Fri, Sep 12, 2014 at 11:18 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

Kilon,

On 12 Sep 2014, at 16:51, kilon alios <kilon.alios@gmail.com> wrote:

> Ok this is strange the ISO 8857-7 is available with Chrome as an encoding , I tried it and it turns Greek to weird signs. Actually I tried also several others available for Greek by Chrome and none works apart from the default which is UTF-8 ....

You misunderstand how this works, let me try to explain using Pharo ;-)

When a website serves a page containing non-ascii content, like ��, it needs to convert from characters to bytes using an encoding. This is normally done by adding a parameter to the content-type of the page being served. Modern servers default to utf-8.

This is how you start your own HTTP server in Pharo:

(ZnServer defaultOn: 1701)
�� onRequestRespond: [ :req |
�� ZnResponse ok: (ZnEntity with: 'Do you speak �� ?' type: ZnMimeType textPlain) ];
�� start.

It will respond with the same text to any request. Try browsing http://localhost:1701

In code, you can do it like this:

ZnClient new get: ZnServer default localUrl.

Here is how you check the content-type being used:

(ZnClient new get: ZnServer default localUrl; response) contentType.

You will see that it is text/plain;charset=utf-8 which is the default used by Zinc when serving textual content. Asking/forcing your browser to interpret these bytes using a different encoding is wrong, since the data was not encoded as such.

But the server could also be using a Greek specific byte encoding, like this:

(ZnServer defaultOn: 1701)
�� onRequestRespond: [ :req |
�� ZnResponse ok: (ZnEntity
�� with: 'Do you speak �� ?'
�� type: (ZnMimeType textPlain charSet: 'iso-8859-7')) ];
�� start.

Now the page is served differently and browsers/clients know what to do, they will switch their encoding accordingly. Explicitly switching to utf-8 will now fail, as the content is *not* utf-8 encoded.

So why is the option there in the browser ?

Because sometimes, the server is lazy (and technically, wrong) by not explicitly setting the charset/encoding. That leaves the browser/client with a problem of course. For this there is a default, often utf-8, sometimes ascii, ..

Zinc uses the null encoder as default (ZnNullEncoder) when the server fails to set one, you can override this default as follows:

ZnDefaultCharacterEncoder
�� value: ZnUTF8Encoder new
�� during: [ ZnClient new get: ZnServer default localUrl ].

ZnDefaultCharacterEncoder
�� value: (ZnCharacterEncoder newForEncoding: 'iso-8859-7')
�� during: [ ZnClient new get: ZnServer default localUrl ].

Note however that Zinc will always follow the encoding set by the response (i.e. you cannot force another one, only use one if the encoding is missing).

HTH,

Sven

PS: If you are on Mac OS X or Linux, you can try using curl as a client. ZnClient can return a command line that you can copy/paste into a terminal:

ZnClient new get: ZnServer default localUrl; curl.

$ curl http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP Components 1.0" -H"Host:localhost:1701"

Do you speak �� ?

By add the -D - option the incoming HTTP headers will be dumped

$ curl -D - http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP Components 1.0" -H"Host:localhost:1701"

HTTP/1.1 200 OK
Content-Type: text/plain;charset=utf-8
Content-Length: 31
Date: Fri, 12 Sep 2014 20:15:53 GMT
Server: Zinc HTTP Components 1.0

Do you speak �� ?

> Also you can ignore my request as I tried Pharo with True Type fonts with Arial and it can display Greeks just fine. Well done Pharoers , once again Pharo surprises me in a good way :)
>
> On Fri, Sep 12, 2014 at 5:16 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
>
> On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote:
>
> > I am not in any urgent need as I rarely use (like almost never) my native language in reading or writing�� when it comes to anything related to computers, apart from facebook friends. But I would like to see it added to pharo , its �� or as you know it "Greek".
>
> So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7 - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ?
>
> Is this nice to have or really necessary (as in used by everybody using the Greek language) ?
>
> > I assume the most popular encondings to have would be Chinese and Hindu ?
>
> I think this something that (native) speakers should help with ;-)
>
> Remember we have full Unicode Character/String capable classes, as well as good UTF-8. Those cover all languages, in theory.
>
> The question comes down to which older encodings are still really used/necessary today.
>
> > On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
> > Hi,
> >
> > ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
> >
> > - proper Unicode Character/String to/from byte(s) encoding/decoding
> > - calculation of encoding size per Character/String
> > - proper handling of holes in tables with exceptions (strict/lenient)
> > - properly backing up one character on a binary stream
> > - simple, efficient, documented
> > - convenience API
> > - optimisations for some encodings
> >
> > Together, they support the following encodings:
> >
> > - utf-8
> > - utf-16 (big/little endian, bom detection)
> > - null
> > - iso-8859-1 (latin-1)
> > - iso-8859-2 (latin-2)
> > - mac-roman
> > - cp1250
> > - cp1252
> > - cp1253
> > - koi8r
> > - iso-8859-7
> > - iso-8859-15
> >
> > The question is, what other encodings do you want/need ?
> >
> > There is already a request to add CP1251.
> >
> > New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
> >
> > Please help us make Pharo more international friendly.
> >
> > Sven
> >
> >
> >
> >
>
>
>