Which Character Encodings do you want/need in Pharo ?
Hi, ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features: - proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings Together, they support the following encodings: - utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15 The question is, what other encodings do you want/need ? There is already a request to add CP1251. New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed). Please help us make Pharo more international friendly. Sven
I am not in any urgent need as I rarely use (like almost never) my native language in reading or writing when it comes to anything related to computers, apart from facebook friends. But I would like to see it added to pharo , its Îλληνικά or as you know it "Greek". I assume the most popular encondings to have would be Chinese and Hindu ? On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote:
I am not in any urgent need as I rarely use (like almost never) my native language in reading or writing when it comes to anything related to computers, apart from facebook friends. But I would like to see it added to pharo , its Îλληνικά or as you know it "Greek".
So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7 - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ? Is this nice to have or really necessary (as in used by everybody using the Greek language) ?
I assume the most popular encondings to have would be Chinese and Hindu ?
I think this something that (native) speakers should help with ;-) Remember we have full Unicode Character/String capable classes, as well as good UTF-8. Those cover all languages, in theory. The question comes down to which older encodings are still really used/necessary today.
On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
Ok this is strange the ISO 8857-7 is available with Chrome as an encoding , I tried it and it turns Greek to weird signs. Actually I tried also several others available for Greek by Chrome and none works apart from the default which is UTF-8 .... Also you can ignore my request as I tried Pharo with True Type fonts with Arial and it can display Greeks just fine. Well done Pharoers , once again Pharo surprises me in a good way :) On Fri, Sep 12, 2014 at 5:16 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote:
I am not in any urgent need as I rarely use (like almost never) my native language in reading or writing when it comes to anything related to computers, apart from facebook friends. But I would like to see it added to pharo , its Îλληνικά or as you know it "Greek".
So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7 - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ?
Is this nice to have or really necessary (as in used by everybody using the Greek language) ?
I assume the most popular encondings to have would be Chinese and Hindu ?
I think this something that (native) speakers should help with ;-)
Remember we have full Unicode Character/String capable classes, as well as good UTF-8. Those cover all languages, in theory.
The question comes down to which older encodings are still really used/necessary today.
On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
Ok this is strange the ISO 8857-7 is available with Chrome as an encoding , I tried it and it turns Greek to weird signs. Actually I tried also several others available for Greek by Chrome and none works apart from the default which is UTF-8 ....
Also you can ignore my request as I tried Pharo with True Type fonts with Arial and it can display Greeks just fine. Well done Pharoers , once again Pharo surprises me in a good way :) Us too. :)
Kilon, On 12 Sep 2014, at 16:51, kilon alios <kilon.alios@gmail.com> wrote:
Ok this is strange the ISO 8857-7 is available with Chrome as an encoding , I tried it and it turns Greek to weird signs. Actually I tried also several others available for Greek by Chrome and none works apart from the default which is UTF-8 ....
You misunderstand how this works, let me try to explain using Pharo ;-) When a website serves a page containing non-ascii content, like ελληνικά, it needs to convert from characters to bytes using an encoding. This is normally done by adding a parameter to the content-type of the page being served. Modern servers default to utf-8. This is how you start your own HTTP server in Pharo: (ZnServer defaultOn: 1701) onRequestRespond: [ :req | ZnResponse ok: (ZnEntity with: 'Do you speak ελληνικά ?' type: ZnMimeType textPlain) ]; start. It will respond with the same text to any request. Try browsing http://localhost:1701 In code, you can do it like this: ZnClient new get: ZnServer default localUrl. Here is how you check the content-type being used: (ZnClient new get: ZnServer default localUrl; response) contentType. You will see that it is text/plain;charset=utf-8 which is the default used by Zinc when serving textual content. Asking/forcing your browser to interpret these bytes using a different encoding is wrong, since the data was not encoded as such. But the server could also be using a Greek specific byte encoding, like this: (ZnServer defaultOn: 1701) onRequestRespond: [ :req | ZnResponse ok: (ZnEntity with: 'Do you speak ελληνικά ?' type: (ZnMimeType textPlain charSet: 'iso-8859-7')) ]; start. Now the page is served differently and browsers/clients know what to do, they will switch their encoding accordingly. Explicitly switching to utf-8 will now fail, as the content is *not* utf-8 encoded. So why is the option there in the browser ? Because sometimes, the server is lazy (and technically, wrong) by not explicitly setting the charset/encoding. That leaves the browser/client with a problem of course. For this there is a default, often utf-8, sometimes ascii, .. Zinc uses the null encoder as default (ZnNullEncoder) when the server fails to set one, you can override this default as follows: ZnDefaultCharacterEncoder value: ZnUTF8Encoder new during: [ ZnClient new get: ZnServer default localUrl ]. ZnDefaultCharacterEncoder value: (ZnCharacterEncoder newForEncoding: 'iso-8859-7') during: [ ZnClient new get: ZnServer default localUrl ]. Note however that Zinc will always follow the encoding set by the response (i.e. you cannot force another one, only use one if the encoding is missing). HTH, Sven PS: If you are on Mac OS X or Linux, you can try using curl as a client. ZnClient can return a command line that you can copy/paste into a terminal: ZnClient new get: ZnServer default localUrl; curl. $ curl http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP Components 1.0" -H"Host:localhost:1701" Do you speak ελληνικά ? By add the -D - option the incoming HTTP headers will be dumped $ curl -D - http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP Components 1.0" -H"Host:localhost:1701" HTTP/1.1 200 OK Content-Type: text/plain;charset=utf-8 Content-Length: 31 Date: Fri, 12 Sep 2014 20:15:53 GMT Server: Zinc HTTP Components 1.0 Do you speak ελληνικά ?
Also you can ignore my request as I tried Pharo with True Type fonts with Arial and it can display Greeks just fine. Well done Pharoers , once again Pharo surprises me in a good way :)
On Fri, Sep 12, 2014 at 5:16 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote:
I am not in any urgent need as I rarely use (like almost never) my native language in reading or writing when it comes to anything related to computers, apart from facebook friends. But I would like to see it added to pharo , its Îλληνικά or as you know it "Greek".
So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7 - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ?
Is this nice to have or really necessary (as in used by everybody using the Greek language) ?
I assume the most popular encondings to have would be Chinese and Hindu ?
I think this something that (native) speakers should help with ;-)
Remember we have full Unicode Character/String capable classes, as well as good UTF-8. Those cover all languages, in theory.
The question comes down to which older encodings are still really used/necessary today.
On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
I see thank you very much for the detailed explanation. I suspected it was working a bit like you described but now I know the juicy details :) Maybe you should add that text to an Encoding chapter in Pharo For The Enterprise so its widely available for people like me that are not familiar with Encodings. On Fri, Sep 12, 2014 at 11:18 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: > Kilon, > > On 12 Sep 2014, at 16:51, kilon alios <kilon.alios@gmail.com> wrote: > > > Ok this is strange the ISO 8857-7 is available with Chrome as an > encoding , I tried it and it turns Greek to weird signs. Actually I tried > also several others available for Greek by Chrome and none works apart from > the default which is UTF-8 .... > > You misunderstand how this works, let me try to explain using Pharo ;-) > > When a website serves a page containing non-ascii content, like ελληνικά, > it needs to convert from characters to bytes using an encoding. This is > normally done by adding a parameter to the content-type of the page being > served. Modern servers default to utf-8. > > This is how you start your own HTTP server in Pharo: > > (ZnServer defaultOn: 1701) > onRequestRespond: [ :req | > ZnResponse ok: (ZnEntity with: 'Do you speak ελληνικά ?' type: > ZnMimeType textPlain) ]; > start. > > It will respond with the same text to any request. Try browsing > http://localhost:1701 > > In code, you can do it like this: > > ZnClient new get: ZnServer default localUrl. > > Here is how you check the content-type being used: > > (ZnClient new get: ZnServer default localUrl; response) contentType. > > You will see that it is text/plain;charset=utf-8 which is the default used > by Zinc when serving textual content. Asking/forcing your browser to > interpret these bytes using a different encoding is wrong, since the data > was not encoded as such. > > But the server could also be using a Greek specific byte encoding, like > this: > > (ZnServer defaultOn: 1701) > onRequestRespond: [ :req | > ZnResponse ok: (ZnEntity > with: 'Do you speak ελληνικά ?' > type: (ZnMimeType textPlain charSet: 'iso-8859-7')) > ]; > start. > > Now the page is served differently and browsers/clients know what to do, > they will switch their encoding accordingly. Explicitly switching to utf-8 > will now fail, as the content is *not* utf-8 encoded. > > So why is the option there in the browser ? > > Because sometimes, the server is lazy (and technically, wrong) by not > explicitly setting the charset/encoding. That leaves the browser/client > with a problem of course. For this there is a default, often utf-8, > sometimes ascii, .. > > Zinc uses the null encoder as default (ZnNullEncoder) when the server > fails to set one, you can override this default as follows: > > ZnDefaultCharacterEncoder > value: ZnUTF8Encoder new > during: [ ZnClient new get: ZnServer default localUrl ]. > > ZnDefaultCharacterEncoder > value: (ZnCharacterEncoder newForEncoding: 'iso-8859-7') > during: [ ZnClient new get: ZnServer default localUrl ]. > > Note however that Zinc will always follow the encoding set by the response > (i.e. you cannot force another one, only use one if the encoding is > missing). > > HTH, > > Sven > > PS: If you are on Mac OS X or Linux, you can try using curl as a client. > ZnClient can return a command line that you can copy/paste into a terminal: > > ZnClient new get: ZnServer default localUrl; curl. > > $ curl http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP > Components 1.0" -H"Host:localhost:1701" > > Do you speak ελληνικά ? > > By add the -D - option the incoming HTTP headers will be dumped > > $ curl -D - http://localhost:1701/ -H"Accept:*/*" -H"User-Agent:Zinc HTTP > Components 1.0" -H"Host:localhost:1701" > > HTTP/1.1 200 OK > Content-Type: text/plain;charset=utf-8 > Content-Length: 31 > Date: Fri, 12 Sep 2014 20:15:53 GMT > Server: Zinc HTTP Components 1.0 > > Do you speak ελληνικά ? > > > Also you can ignore my request as I tried Pharo with True Type fonts > with Arial and it can display Greeks just fine. Well done Pharoers , once > again Pharo surprises me in a good way :) > > > > On Fri, Sep 12, 2014 at 5:16 PM, Sven Van Caekenberghe <sven@stfx.eu> > wrote: > > > > On 12 Sep 2014, at 16:05, kilon alios <kilon.alios@gmail.com> wrote: > > > > > I am not in any urgent need as I rarely use (like almost never) my > native language in reading or writing when it comes to anything related to > computers, apart from facebook friends. But I would like to see it added to > pharo , its Îλληνικά or as you know it "Greek". > > > > So that would be ISO 8857-9 (http://en.wikipedia.org/wiki/ISO/IEC_8859-7 > - http://unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT) then, right ? > > > > Is this nice to have or really necessary (as in used by everybody using > the Greek language) ? > > > > > I assume the most popular encondings to have would be Chinese and > Hindu ? > > > > I think this something that (native) speakers should help with ;-) > > > > Remember we have full Unicode Character/String capable classes, as well > as good UTF-8. Those cover all languages, in theory. > > > > The question comes down to which older encodings are still really > used/necessary today. > > > > > On Fri, Sep 12, 2014 at 4:10 PM, Sven Van Caekenberghe <sven@stfx.eu> > wrote: > > > Hi, > > > > > > ZnCharacterEncoder and its subclasses in the package > Zinc-Character-Encoding-Core (a standard part of Pharo) have the following > features: > > > > > > - proper Unicode Character/String to/from byte(s) encoding/decoding > > > - calculation of encoding size per Character/String > > > - proper handling of holes in tables with exceptions (strict/lenient) > > > - properly backing up one character on a binary stream > > > - simple, efficient, documented > > > - convenience API > > > - optimisations for some encodings > > > > > > Together, they support the following encodings: > > > > > > - utf-8 > > > - utf-16 (big/little endian, bom detection) > > > - null > > > - iso-8859-1 (latin-1) > > > - iso-8859-2 (latin-2) > > > - mac-roman > > > - cp1250 > > > - cp1252 > > > - cp1253 > > > - koi8r > > > - iso-8859-7 > > > - iso-8859-15 > > > > > > The question is, what other encodings do you want/need ? > > > > > > There is already a request to add CP1251. > > > > > > New byte encoders are easy to add, based on the official Unicode > mappings that can be found here: http://unicode.org/Public/MAPPINGS > (these are automagically parsed). > > > > > > Please help us make Pharo more international friendly. > > > > > > Sven > > > > > > > > > > > > > > > > > > > > >
Sven thank you. I love your code and spirit! We should clone you. Stef
Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
On 12 Sep 2014, at 21:43, stepharo <stepharo@free.fr> wrote:
Sven thank you. I love your code and spirit! We should clone you.
Thanks, but Pharo is a group effort, we all contribute in our area of expertise. I am especially happy that we have some new people that recently became active.
Stef
Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
On 12 Sep 2014, at 15:10, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
ZnCharacterEncoder and its subclasses in the package Zinc-Character-Encoding-Core (a standard part of Pharo) have the following features:
- proper Unicode Character/String to/from byte(s) encoding/decoding - calculation of encoding size per Character/String - proper handling of holes in tables with exceptions (strict/lenient) - properly backing up one character on a binary stream - simple, efficient, documented - convenience API - optimisations for some encodings
Together, they support the following encodings:
- utf-8 - utf-16 (big/little endian, bom detection) - null - iso-8859-1 (latin-1) - iso-8859-2 (latin-2) - mac-roman - cp1250 - cp1252 - cp1253 - koi8r - iso-8859-7 - iso-8859-15
The question is, what other encodings do you want/need ?
There is already a request to add CP1251.
I decided to add a comprehensive set of encodings, based on http://encoding.spec.whatwg.org/#legacy-single-byte-encodings (a list which more or less corresponds to the encodings list supported by popular browsers). The mappings themselves are still taken from http://unicode.org/Public/MAPPINGS So now Pharo (Zinc-Character-Encoding-Core) supports the following byte encodings: ISO/IEC 8859 1, 2, 3, 4, 5, 6, 7, 8, 10, 13, 14, 15 and 16, Windows Codepages 866, 874, 1250, 1251, 1252, 1253, 1253, 1254, 1255, 1256, 1257, 1258, KOI8 R & U as well as Mac Roman & Cyrillic - each of these with a number of aliases like latin1, latin2, latin3, latin4, latin5, latin6, cyrillic, arabic, greek and hebrew. Furthermore, identifier mapping has been made more liberal (case and separator insensitive). For now, this is in Zinc #bleedingEdge
New byte encoders are easy to add, based on the official Unicode mappings that can be found here: http://unicode.org/Public/MAPPINGS (these are automagically parsed).
Please help us make Pharo more international friendly.
Sven
participants (4)
-
Ben Coman -
kilon alios -
stepharo -
Sven Van Caekenberghe