Re: [Pharo-dev] Which Character Encodings do you want/need in Pharo ?

I see thank you very much for the detailed explanation. I suspected it was
working a bit like you described but now I know the juicy details :)

Maybe you should add that text to an Encoding chapter in Pharo For The
Enterprise so its widely available for people like me that are not familiar
with Encodings.

On Fri, Sep 12, 2014 at 11:18 PM, Sven Van Caekenberghe <sven@stfx.eu>
wrote:

> Kilon,
>
> On 12 Sep 2014, at 16:51, kilon alios <kilon.alios@gmail.com> wrote:
>
> > Ok this is strange the ISO 8857-7 is available with Chrome as an
> encoding , I tried it and it turns Greek to weird signs. Actually I tried
> also several others available for Greek by Chrome and none works apart from
> the default which is UTF-8 ....
>
> You misunderstand how this works, let me try to explain using Pharo ;-)
>
> When a website serves a page containing non-ascii content, like ελληνικά,
> it needs to convert from characters to bytes using an encoding. This is
> normally done by adding a parameter to the content-type of the page being
> served. Modern servers default to utf-8.
>
> This is how you start your own HTTP server in Pharo:
>
> (ZnServer defaultOn: 1701)
>   onRequestRespond: [ :req |
>     ZnResponse ok: (ZnEntity with: 'Do you speak ελληνικά ?' type:
> ZnMimeType textPlain) ];
>   start.
>
> It will respond with the same text to any request. Try browsing
> http://localhost:1701
>
> In code, you can do it like this:
>
> ZnClient new get: ZnServer default localUrl.
>
> Here is how you check the content-type being used:
>
> (ZnClient new get: ZnServer default localUrl; response) contentType.
>
> You will see that it is text/plain;charset=utf-8 which is the default used
> by Zinc when serving textual content. Asking/forcing your browser to
> interpret these bytes using a different encoding is wrong, since the data
> was not encoded as such.
>
> But the server could also be using a Greek specific byte encoding, like
> this:
>
> (ZnServer defaultOn: 1701)
>   onRequestRespond: [ :req |
>     ZnResponse ok: (ZnEntity
>                       with: 'Do you speak ελληνικά ?'
>                       type: (ZnMimeType textPlain charSet: 'iso-8859-7'))
> ];
>   start.
>
> Now the page is served differently and browsers/clients know what to do,
> they will switch their encoding accordingly. Explicitly switching to utf-8
> will now fail, as the content is *not* utf-8 encoded.
>
> So why is the option there in the browser ?
>
> Because sometimes, the server is lazy (and technically, wrong) by not
> explicitly setting the charset/encoding. That leaves the browser/client
> with a problem of course. For this there is a default, often utf-8,
> sometimes ascii, ..
>
> Zinc uses the null encoder as default (ZnNullEncoder) when the server
> fails to set one, you can override this default as follows:
>
> ZnDefaultCharacterEncoder
>   value: ZnUTF8Encoder new
>   during: [ ZnClient new get: ZnServer default localUrl ].
>
> ZnDefaultCharacterEncoder
>   value: (ZnCharacterEncoder newForEncoding: 'iso-8859-7')
>   during: [ ZnClient new get: ZnServer default localUrl ].
>
> Note however that Zinc will always follow the encoding set by the response
> (i.e. you cannot force another one, only use one if the encoding is
> missing).
>
> HTH,
>
> Sven
>
> PS: If you are on Mac OS X or Linux, you can try using curl as a client.
> ZnClient can return a command line that you can copy/paste into a terminal:
>
> ZnClient new get: ZnServer default localUrl; curl.
>
> $ curl http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP
> Components 1.0" -H"Host:localhost:1701"
>
> Do you speak ελληνικά ?
>
> By add the -D - option the incoming HTTP headers will be dumped
>
> $ curl -D - http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP
> Components 1.0" -H"Host:localhost:1701"
>
> HTTP/1.1 200 OK
> Content-Type: text/plain;charset=utf-8
> Content-Length: 31
> Date: Fri, 12 Sep 2014 20:15:53 GMT
> Server: Zinc HTTP Components 1.0
>
> Do you speak ελληνικά ?
>
> > Also you can ignore my request as I tried Pharo with True Type fonts
> with Arial and it can display Greeks just fine. Well done Pharoers , once
> again Pharo surprises me in a good way :)
> >
> > On Fri, Sep 12, 2014 at 5:16 PM, Sven Van Caekenberghe <sven@stfx.eu>
> wrote:
> >
> > On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote:
> >
> > > I am not in any urgent need as I rarely use (like almost never) my
> native language in reading or writing  when it comes to anything related to
> computers, apart from facebook friends. But I would like to see it added to
> pharo , its Ελληνικά or as you know it "Greek".
> >
> > So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7
> - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ?
> >
> > Is this nice to have or really necessary (as in used by everybody using
> the Greek language) ?
> >
> > > I assume the most popular encondings to have would be Chinese and
> Hindu ?
> >
> > I think this something that (native) speakers should help with ;-)
> >
> > Remember we have full Unicode Character/String capable classes, as well
> as good UTF-8. Those cover all languages, in theory.
> >
> > The question comes down to which older encodings are still really
> used/necessary today.
> >
> > > On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu>
> wrote:
> > > Hi,
> > >
> > > ZnCharacterEncoder and its subclasses in the package
> Zinc-Character-Encoding-Core (a standard part of Pharo) have the following
> features:
> > >
> > > - proper Unicode Character/String to/from byte(s) encoding/decoding
> > > - calculation of encoding size per Character/String
> > > - proper handling of holes in tables with exceptions (strict/lenient)
> > > - properly backing up one character on a binary stream
> > > - simple, efficient, documented
> > > - convenience API
> > > - optimisations for some encodings
> > >
> > > Together, they support the following encodings:
> > >
> > > - utf-8
> > > - utf-16 (big/little endian, bom detection)
> > > - null
> > > - iso-8859-1 (latin-1)
> > > - iso-8859-2 (latin-2)
> > > - mac-roman
> > > - cp1250
> > > - cp1252
> > > - cp1253
> > > - koi8r
> > > - iso-8859-7
> > > - iso-8859-15
> > >
> > > The question is, what other encodings do you want/need ?
> > >
> > > There is already a request to add CP1251.
> > >
> > > New byte encoders are easy to add, based on the official Unicode
> mappings that can be found here: http://unicode.org/Public/MAPPINGS
> (these are automagically parsed).
> > >
> > > Please help us make Pharo more international friendly.
> > >
> > > Sven
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>