Because, who doesn't want to spend an evening with a workspace like this: http://pastebin.com/gdxCrTgM trying to figure out why some of the last 7 out of 189k official collate tests fail? Seriously though, personally I was just tired of having to answer "Sort of, but not really" every time the question of Unicode support is raised, specifically the last discussion that lead nowhere back in December. Better to show in code how you think a decent implementation might look like, and Sven made it easier to get rolling when he wrote import code for many of the data tables needed, parsing text is one of the things I dislike most... Cheers, Henry P.S. Unicode Collation is more of an intellectual exercise. For practical collation you need the locale tailorings from CLDR, which is kept in a, frankly, abominable format. The object model / actual collate algorithms should carry over nicely though, so it's not a complete waste. On Wed, Feb 17, 2016 at 7:56 PM, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
I read the whole Article, seems like a tricky, to not say hard, subject. The article is very detailed and well written though.
What is the rationale behind embracing such a challenging feature like supporting Unicode?
Regards!
Esteban A. Maringolo
2016-02-17 15:24 GMT-03:00 Max Leske <maxleske@gmail.com>:
Good stuff guys!
On 17 Feb 2016, at 10:16, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
In Pharo we can deal with and represent any Unicode character and string, but there are still some important pieces of functionality missing.
The goal of the Pharo Unicode project is to gradually improve and expand Unicode support in Pharo. We started in December last year to lay the foundation and are now ready to go public with what we built.
The first delivery is an implementation of Unicode Normalization, together with an implementation of the Unicode Character Database.
Please read the following article for more information (the appendix explains how to get the code).
An Implementation of Unicode Normalisation
Streaming NFC, NFD, NFKC & NFKD, normalization QC and normalization preserving concatenation.
https://medium.com/concerning-pharo/an-implementation-of-unicode-normalizati...
The development branch also contains a work in progress implementation
of Unicode Collation.
Much work remains to be done and contributions are more than welcome.
Sven & Henrik