Re: [Pharo-dev] [Cuis] Sorting Unicode strings (Re: [Unicode] collation sequences (Re: [squeak-dev] Unicode Support))
Dale Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray. I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk. I am currently looking into what the ICU library provides. I found as well a Ruby library [2] which implements CLDR [3] It has methods like this "Alphabetize a list using regular Ruby sort:" $> ["Art", "Wasa", "Ãlg", "Ved"].sort $> ["Art", "Ved", "Wasa", "Ãlg"] Alphabetize a list using TwitterCLDRâs locale-aware sort: $> ["Art", "Wasa", "Ãlg", "Ved"].localize(:de).sort.to_a $> ["Ãlg", "Art", "Ved", "Wasa"] I hope that given such an example it would not be too difficult to reimplement a similar sort algorithm in Squeak/Cuis/Pharo. Currently the interest is in getting sorting done in a cross-dialect-way. --Hannes [2] https://blog.twitter.com/2012/twittercldr-improving-internationalization-sup... [3] Unicode Common Locale Data Repository http://cldr.unicode.org/index On 12/7/15, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
Hannes,
For GemStone, we are using the ICU library[1]. We have Unicode7, Unicode16 and Unicode32 classes (subclasses of CharacterCollection) for internal Strings and the class Utf8 (a subclass of ByteArray) for Utf8 encoded strings ...
The ICU library provides the primitive implementations for working with the Unicode* and Utf8 classes
When we started considering Unicode support, we looked at what it would take to support collation - our main reason for looking at Unicode in the first place) -- and we saw just how complicated the collation rules can be[2], we were glad to see that someone had already done the hard work[1]...
Reconciling our legacy String implementations (String, DoubleByteString, and QuadByteString) with the Unicode* classes was also interesting, because the rules for Unicode equality and our legacy equality implementation were not quite compatible.
If you are interested in more information, I can share additional details ...
Dale
[1] http://site.icu-project.org/ [2] http://unicode.org/reports/tr10/
On 12/07/2015 11:54 AM, H. Hirzel wrote:
Hello
According to http://www.unicode.org/cldr/charts/27/collation/de.html the German phonebook sort order is
a A ä Ã Ä Ì ÄÌ Ç Ç áº¡Ì áº Ì á¸Ì á¸Ì b B c C d D e E f F g G h H i I j J k K l L m M n N o O ö Ã Ç«Ì ÇªÌ È« Ȫ Æ¡Ì Æ Ì á»£Ì á»¢Ì á»Ì á»Ì p P q Q r R s S ss à t T u U ü Ã Ç Ç Ç Ç Ç Ç Å³Ì Å²Ì Ç Ç Æ°Ì Æ¯Ì á»±Ì á»°Ì á»¥Ì á»¤Ì á¹³Ì á¹²Ì á¹·Ì á¹¶Ì á¹µÌ á¹´Ì v V w W x X y Y z Z
I wonder why it looks like this. A lot of characters which never appear in a German text.
For Spanish there is 'traditional' and 'standard'
http://www.unicode.org/cldr/charts/27/collation/es.html
standard a A á à b B c C d D e E é à f F g G h H i I à à j J k K l L m M n N ñ à ÅÌ Å Ì á¹Ì á¹Ì á¹Ì á¹Ì á¹Ì á¹Ì o O ó à p P q Q r R s S t T u U ú à ü à v V w W x X y Y z Z
traditional a A á à b B c C ch Ch CH cÄ¥ CÄ¥ CĤ cÈ CÈ CÈ cḧ Cḧ CḦ cḣ Cḣ CḢ cḩ Cḩ CḨ cḥ Cḥ CḤ cḫ Cḫ CḪ cẠCẠd D e E é à f F g G h H i I à à j J k K l L ll Ll LL lĺ Lĺ LĹ lľ Lľ LĽ lļ Lļ LÄ» lḷ Lḷ LḶ lḹ Lḹ LḸ lḽ Lḽ LḼ lḻ Lḻ LḺ m M n N ñ à ÅÌ Å Ì á¹Ì á¹Ì á¹Ì á¹Ì á¹Ì á¹Ì o O ó à p P q Q r R s S t T u U ú à ü à v V w W x X y Y z Z
And French is not easily found http://www.unicode.org/cldr/charts/27/collation/index.html or seems to be defined elsewhere
http://unicode.org/repos/cldr/tags/release-27/common/collation/fr.xml
Suggestions and hints are welcome
--Hannes _______________________________________________ Cuis mailing list Cuis@jvuletich.org http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
_______________________________________________ Cuis mailing list Cuis@jvuletich.org http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
I found as well a Ruby library [2] which implements CLDR [3]
It has methods like this
"Alphabetize a list using regular Ruby sort:"
$> ["Art", "Wasa", "Ãlg", "Ved"].sort $> ["Art", "Ved", "Wasa", "Ãlg"]
Alphabetize a list using TwitterCLDRâs locale-aware sort:
$> ["Art", "Wasa", "Ãlg", "Ved"].localize(:de).sort.to_a $> ["Ãlg", "Art", "Ved", "Wasa"]
I hope that given such an example it would not be too difficult to reimplement a similar sort algorithm in Squeak/Cuis/Pharo. Currently the interest is in getting sorting done in a cross-dialect-way.
I think that the issue (from a performance perspective) is that you can't depend upon the value of the code point when doing collation --- the main algorithm[5] is pretty much table based --- In addition to the different sort orders based on characters there are even more arcane sort rules where characters at the end of a word can affect the sort order of the word (for more info see[4]). It is worth looking at the Conformance section of the Unicode spec[1] as there are different levels of collation conformance ..... ICU conforms[2] to to UTS #10[3], the highest level of conformance ... It looks like TwitterCLDR[6] uses the Main Algorithm[5] with tailoring[7]. They don't claim to be conformant to the Unicode Collation Algorithm[3], but they are covering a big chunk of the standard use cases .... Dale [1] http://unicode.org/reports/tr10/#Conformance [2] http://userguide.icu-project.org/collation [3] http://www.unicode.org/reports/tr10/ [4] http://www.unicode.org/reports/tr10/#Introduction [5] http://www.unicode.org/reports/tr10/#Main_Algorithm [6] https://blog.twitter.com/2012/twittercldr-improving-internationalization-sup... [7] http://unicode.org/reports/tr10/#Tailoring
Dale - is that you can't depend on the value of a codepoint *unless the string is either in fully-composed form (or has just been fully-decomposed from a fully-composed form) * OR are there circumstances where even those two cases cannot be relied upon? On 8 December 2015 at 19:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
I found as well a Ruby library [2] which implements CLDR [3]
It has methods like this
"Alphabetize a list using regular Ruby sort:"
$> ["Art", "Wasa", "Ãlg", "Ved"].sort $> ["Art", "Ved", "Wasa", "Ãlg"]
Alphabetize a list using TwitterCLDRâs locale-aware sort:
$> ["Art", "Wasa", "Ãlg", "Ved"].localize(:de).sort.to_a $> ["Ãlg", "Art", "Ved", "Wasa"]
I hope that given such an example it would not be too difficult to reimplement a similar sort algorithm in Squeak/Cuis/Pharo. Currently the interest is in getting sorting done in a cross-dialect-way.
I think that the issue (from a performance perspective) is that you can't depend upon the value of the code point when doing collation --- the main algorithm[5] is pretty much table based --- In addition to the different sort orders based on characters there are even more arcane sort rules where characters at the end of a word can affect the sort order of the word (for more info see[4]).
It is worth looking at the Conformance section of the Unicode spec[1] as there are different levels of collation conformance .....
ICU conforms[2] to to UTS #10[3], the highest level of conformance ...
It looks like TwitterCLDR[6] uses the Main Algorithm[5] with tailoring[7]. They don't claim to be conformant to the Unicode Collation Algorithm[3], but they are covering a big chunk of the standard use cases ....
Dale
[1] http://unicode.org/reports/tr10/#Conformance [2] http://userguide.icu-project.org/collation [3] http://www.unicode.org/reports/tr10/ [4] http://www.unicode.org/reports/tr10/#Introduction [5] http://www.unicode.org/reports/tr10/#Main_Algorithm [6] https://blog.twitter.com/2012/twittercldr-improving-internationalization-sup... [7] http://unicode.org/reports/tr10/#Tailoring
Euan, What I meant is that you can't _always_ use the code point for collation, i.e., sorting based on the value of code points is not always correct[1]. If I'm not mistaken the fully-composed and fully-decomposed forms can only be used for testing the equivalence of two strings[2] ... The Main Algorithm[3], starts by producing a normalized form of the string, but the subsequent steps (produce array, form sort key and compare) involves table lookups among other things .... Once you've produced a sort key for a string, the sort key does use "binary comparison" for collating , which is a byte by byte numeric comparison ... Dale [1] http://www.unicode.org/reports/tr10/#Common_Misperceptions [2] http://unicode.org/reports/tr15/pdtr15.html [3] http://www.unicode.org/reports/tr10/#Main_Algorithm On 12/08/2015 12:22 PM, EuanM wrote:
Dale - is that you can't depend on the value of a codepoint *unless the string is either in fully-composed form (or has just been fully-decomposed from a fully-composed form) *
OR are there circumstances where even those two cases cannot be relied upon?
On 8 December 2015 at 19:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
I found as well a Ruby library [2] which implements CLDR [3]
It has methods like this
"Alphabetize a list using regular Ruby sort:"
$> ["Art", "Wasa", "Ãlg", "Ved"].sort $> ["Art", "Ved", "Wasa", "Ãlg"]
Alphabetize a list using TwitterCLDRâs locale-aware sort:
$> ["Art", "Wasa", "Ãlg", "Ved"].localize(:de).sort.to_a $> ["Ãlg", "Art", "Ved", "Wasa"]
I hope that given such an example it would not be too difficult to reimplement a similar sort algorithm in Squeak/Cuis/Pharo. Currently the interest is in getting sorting done in a cross-dialect-way.
I think that the issue (from a performance perspective) is that you can't depend upon the value of the code point when doing collation --- the main algorithm[5] is pretty much table based --- In addition to the different sort orders based on characters there are even more arcane sort rules where characters at the end of a word can affect the sort order of the word (for more info see[4]).
It is worth looking at the Conformance section of the Unicode spec[1] as there are different levels of collation conformance .....
ICU conforms[2] to to UTS #10[3], the highest level of conformance ...
It looks like TwitterCLDR[6] uses the Main Algorithm[5] with tailoring[7]. They don't claim to be conformant to the Unicode Collation Algorithm[3], but they are covering a big chunk of the standard use cases ....
Dale
[1] http://unicode.org/reports/tr10/#Conformance [2] http://userguide.icu-project.org/collation [3] http://www.unicode.org/reports/tr10/ [4] http://www.unicode.org/reports/tr10/#Introduction [5] http://www.unicode.org/reports/tr10/#Main_Algorithm [6] https://blog.twitter.com/2012/twittercldr-improving-internationalization-sup... [7] http://unicode.org/reports/tr10/#Tailoring
Reading up http://www.unicode.org/reports/tr15/#Examples The Unicode standard seems to require you never to make aGermanStrasse equivalent in a sort order to the ligatured version, aGermanStraÃe This seems counter-intuitive to me. Is there a reason for this? Have I just simply picked this up wrongly? On 8 December 2015 at 21:35, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
Euan,
What I meant is that you can't _always_ use the code point for collation, i.e., sorting based on the value of code points is not always correct[1].
If I'm not mistaken the fully-composed and fully-decomposed forms can only be used for testing the equivalence of two strings[2] ...
The Main Algorithm[3], starts by producing a normalized form of the string, but the subsequent steps (produce array, form sort key and compare) involves table lookups among other things ....
Once you've produced a sort key for a string, the sort key does use "binary comparison" for collating , which is a byte by byte numeric comparison ...
Dale
[1] http://www.unicode.org/reports/tr10/#Common_Misperceptions [2] http://unicode.org/reports/tr15/pdtr15.html [3] http://www.unicode.org/reports/tr10/#Main_Algorithm
On 12/08/2015 12:22 PM, EuanM wrote:
Dale - is that you can't depend on the value of a codepoint *unless the string is either in fully-composed form (or has just been fully-decomposed from a fully-composed form) *
OR are there circumstances where even those two cases cannot be relied upon?
On 8 December 2015 at 19:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
I found as well a Ruby library [2] which implements CLDR [3]
It has methods like this
"Alphabetize a list using regular Ruby sort:"
$> ["Art", "Wasa", "Ãlg", "Ved"].sort $> ["Art", "Ved", "Wasa", "Ãlg"]
Alphabetize a list using TwitterCLDRâs locale-aware sort:
$> ["Art", "Wasa", "Ãlg", "Ved"].localize(:de).sort.to_a $> ["Ãlg", "Art", "Ved", "Wasa"]
I hope that given such an example it would not be too difficult to reimplement a similar sort algorithm in Squeak/Cuis/Pharo. Currently the interest is in getting sorting done in a cross-dialect-way.
I think that the issue (from a performance perspective) is that you can't depend upon the value of the code point when doing collation --- the main algorithm[5] is pretty much table based --- In addition to the different sort orders based on characters there are even more arcane sort rules where characters at the end of a word can affect the sort order of the word (for more info see[4]).
It is worth looking at the Conformance section of the Unicode spec[1] as there are different levels of collation conformance .....
ICU conforms[2] to to UTS #10[3], the highest level of conformance ...
It looks like TwitterCLDR[6] uses the Main Algorithm[5] with tailoring[7]. They don't claim to be conformant to the Unicode Collation Algorithm[3], but they are covering a big chunk of the standard use cases ....
Dale
[1] http://unicode.org/reports/tr10/#Conformance [2] http://userguide.icu-project.org/collation [3] http://www.unicode.org/reports/tr10/ [4] http://www.unicode.org/reports/tr10/#Introduction [5] http://www.unicode.org/reports/tr10/#Main_Algorithm [6]
https://blog.twitter.com/2012/twittercldr-improving-internationalization-sup... [7] http://unicode.org/reports/tr10/#Tailoring
On 12/08/2015 04:33 PM, EuanM wrote:
Reading up http://www.unicode.org/reports/tr15/#Examples
The Unicode standard seems to require you never to make aGermanStrasse equivalent in a sort order to the ligatured version, aGermanStraÃe I didn't see anything in the examples related to this ... could you be a bit more specific?
Dale
http://www.unicode.org/reports/tr15/#Stable_Code_Points Table 7, the discussion of Ligatures, (which uses the ligature of "ffi" as its example) Every time I think I'm about to grokk this standard, something like this crops up. On 9 December 2015 at 00:43, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/08/2015 04:33 PM, EuanM wrote:
Reading up http://www.unicode.org/reports/tr15/#Examples
The Unicode standard seems to require you never to make aGermanStrasse equivalent in a sort order to the ligatured version, aGermanStraÃe
I didn't see anything in the examples related to this ... could you be a bit more specific?
Dale
I've just discovered the ICU's official listing for an ICU wrapper for Smalltalk: https://schrievkrom.wordpress.com/tag/icu/ Listed on http://site.icu-project.org/related In turn, it mentions: ICU: Initial source code for Squeak/Pharo. http://ss3.gemstone.com/ss/ICU.html by Marten Feldtmann, Jan van de Sandt ICU-V2 Unicode support http://ss3.gemstone.com/ss/ICU-V2.html also by Marten Feldtmann, Jan van de Sandt On 9 December 2015 at 00:59, EuanM <euanmee@gmail.com> wrote:
http://www.unicode.org/reports/tr15/#Stable_Code_Points Table 7, the discussion of Ligatures, (which uses the ligature of "ffi" as its example)
Every time I think I'm about to grokk this standard, something like this crops up.
On 9 December 2015 at 00:43, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/08/2015 04:33 PM, EuanM wrote:
Reading up http://www.unicode.org/reports/tr15/#Examples
The Unicode standard seems to require you never to make aGermanStrasse equivalent in a sort order to the ligatured version, aGermanStraÃe
I didn't see anything in the examples related to this ... could you be a bit more specific?
Dale
"Accessing ICU library from Squeak/Pharo via NativeBoost interface FFI. This is my port to Squeak/Pharo - to have same interface available under VASmalltalk and Pharo/Squeak." On 9 December 2015 at 01:30, EuanM <euanmee@gmail.com> wrote:
I've just discovered the ICU's official listing for an ICU wrapper for Smalltalk: https://schrievkrom.wordpress.com/tag/icu/
Listed on http://site.icu-project.org/related
In turn, it mentions:
ICU: Initial source code for Squeak/Pharo. http://ss3.gemstone.com/ss/ICU.html by Marten Feldtmann, Jan van de Sandt
ICU-V2 Unicode support http://ss3.gemstone.com/ss/ICU-V2.html also by Marten Feldtmann, Jan van de Sandt
On 9 December 2015 at 00:59, EuanM <euanmee@gmail.com> wrote:
http://www.unicode.org/reports/tr15/#Stable_Code_Points Table 7, the discussion of Ligatures, (which uses the ligature of "ffi" as its example)
Every time I think I'm about to grokk this standard, something like this crops up.
On 9 December 2015 at 00:43, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/08/2015 04:33 PM, EuanM wrote:
Reading up http://www.unicode.org/reports/tr15/#Examples
The Unicode standard seems to require you never to make aGermanStrasse equivalent in a sort order to the ligatured version, aGermanStraÃe
I didn't see anything in the examples related to this ... could you be a bit more specific?
Dale
On 12/08/2015 04:59 PM, EuanM wrote:
http://www.unicode.org/reports/tr15/#Stable_Code_Points Table 7, the discussion of Ligatures, (which uses the ligature of "ffi" as its example)
Every time I think I'm about to grokk this standard, something like this crops up.
Table 7 and Table 8 are showing different normalization forms and I think Table 8 is an example where the two strings are equivalent (the are two categories of normalization: canonical equivalent [table 7] and compatibility equivalent [table 8]) .... it seems (I haven't delved this far myself) that the normalization form is chosen based on the actual set of characters and possibly some application specific choices .... So I don't think there is a fixed one-size fits all interpretation ... which makes duplicating the functionality ICU challenging.... Dale
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software. I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it. -Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
Dale, yes - sorting based on the value of codepoints is almost always guaranteed to be wrong. Sorting is an application-specific issue, not a technical Unicode issue, as there is more than one canonical sort order per culture, and there is often more than one culture per writing system. e.g. ISO Latin 1 / Latin 9 covers these cultures (amongst others) English (2 sort orders); Spanish; French (2 sort orders); German (2 sort orders); Swedish; etc German sort order differs from Swedish for the same characters, etc Todd, My thinking is that if we implement fully-composed strings as heterogenous arrays, we sidestep a lot of the complexity of the ICU. If it turns out that the performance is terrible, we can then seek to incorporate the ICU. On 8 December 2015 at 22:36, Todd Blanchard <tblanchard@mac.com> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
Equally old are the NextStep Object C functions which are now embodied within MacOS X. It might also be informative for us to see what's done there. On 8 December 2015 at 23:50, EuanM <euanmee@gmail.com> wrote:
Dale,
yes - sorting based on the value of codepoints is almost always guaranteed to be wrong. Sorting is an application-specific issue, not a technical Unicode issue, as there is more than one canonical sort order per culture, and there is often more than one culture per writing system.
e.g. ISO Latin 1 / Latin 9 covers these cultures (amongst others) English (2 sort orders); Spanish; French (2 sort orders); German (2 sort orders); Swedish; etc
German sort order differs from Swedish for the same characters, etc
Todd,
My thinking is that if we implement fully-composed strings as heterogenous arrays, we sidestep a lot of the complexity of the ICU.
If it turns out that the performance is terrible, we can then seek to incorporate the ICU.
On 8 December 2015 at 22:36, Todd Blanchard <tblanchard@mac.com> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
They are practically the same thing. ICU was developed by Taligent which was a joint venture between Apple and IBM. Makes sense that NSString and ICU's UnicodeString are pretty close in implementation. ICU was also ported to Java for Sun by IBM. The point is - this is a very elaborate chunk of code with far reach. If ICU is wrong on some point - it is universally wrong and thus likely to be taken as "right" as it is at least consistent. I think re-implementing it is folly TBH. Just use it.
On Dec 8, 2015, at 15:52, EuanM <euanmee@gmail.com> wrote:
Equally old are the NextStep Object C functions which are now embodied within MacOS X.
It seems that people at Twitter just did this for Ruby -- reimplementation from scratch with an API oriented towards Ruby users. A summary http://wiki.squeak.org/squeak/6263 Of course it depends on what you expect to achieve. This thread starts with aiming at getting German, French and Spanish sorting done. Plus some other similar cases. The algorithms are table driven and the tables are read into the Smalltalk image as is. As of now the Unicode Character Database is in the Squeak/Pharo image http://wiki.squeak.org/squeak/6244 Getting the sorting done does not seem to be extraordinary hard. However making use of the Smalltalk ICU wrapper is surely an option. --Hannes On 12/9/15, Todd Blanchard <tblanchard@mac.com> wrote:
They are practically the same thing.
ICU was developed by Taligent which was a joint venture between Apple and IBM. Makes sense that NSString and ICU's UnicodeString are pretty close in implementation. ICU was also ported to Java for Sun by IBM. The point is - this is a very elaborate chunk of code with far reach. If ICU is wrong on some point - it is universally wrong and thus likely to be taken as "right" as it is at least consistent. I think re-implementing it is folly TBH. Just use it.
On Dec 8, 2015, at 15:52, EuanM <euanmee@gmail.com> wrote:
Equally old are the NextStep Object C functions which are now embodied within MacOS X.
On Thu, Dec 10, 2015 at 12:37 AM, Todd Blanchard <tblanchard@mac.com> wrote:
They are practically the same thing.
ICU was developed by Taligent which was a joint venture between Apple and IBM. Makes sense that NSString and ICU's UnicodeString are pretty close in implementation. ICU was also ported to Java for Sun by IBM. The point is - this is a very elaborate chunk of code with far reach. If ICU is wrong on some point - it is universally wrong and thus likely to be taken as "right" as it is at least consistent. I think re-implementing it is folly TBH. Just use it.
Apple seem to have moved on from NSString to support Unicode in a different way in Switft...
On Dec 8, 2015, at 15:52, EuanM <euanmee@gmail.com> wrote:
Equally old are the NextStep Object C functions which are now embodied within MacOS X.
I wouldn't say that necessarily. There's a whole lot of API that is dependent on objective c - I suspect they have a toll free bridging in there where swift/objective c strings share enough common protocol to be fully substitutable like with CFString. Anyhow swifts step away from smalltalk ideals and the added complexity of language bridging has me avoiding it. Sent from the road
On Dec 15, 2015, at 00:49, Ben Coman <btc@openinworld.com> wrote:
Apple seem to have moved on from NSString to support Unicode in a different way in Switft...
On 12/15/15, Ben Coman <btc@openinworld.com> wrote:
On Thu, Dec 10, 2015 at 12:37 AM, Todd Blanchard <tblanchard@mac.com> wrote:
They are practically the same thing.
ICU was developed by Taligent which was a joint venture between Apple and IBM. Makes sense that NSString and ICU's UnicodeString are pretty close in implementation. ICU was also ported to Java for Sun by IBM. The point is - this is a very elaborate chunk of code with far reach. If ICU is wrong on some point - it is universally wrong and thus likely to be taken as "right" as it is at least consistent. I think re-implementing it is folly TBH. Just use it.
Apple seem to have moved on from NSString to support Unicode in a different way in Switft...
Could you please give some more details? I have read https://www.objc.io/issues/9-strings/unicode/#nsstring-and-unicode so far. It says that an NSString object actually represents an array of UTF-16-encoded code units. This in contrast to Squeak / Pharo where a String is an ArrayedCollection of 21 bit Unicode code points (transparently optimizing to a ByteArray if the string only contains values of the first code page).
On Dec 8, 2015, at 15:52, EuanM <euanmee@gmail.com> wrote:
Equally old are the NextStep Object C functions which are now embodied within MacOS X.
On 15/12/2015 16:05, H. Hirzel wrote:
On 12/15/15, Ben Coman <btc@openinworld.com> wrote:
On Thu, Dec 10, 2015 at 12:37 AM, Todd Blanchard <tblanchard@mac.com> wrote:
They are practically the same thing.
ICU was developed by Taligent which was a joint venture between Apple and IBM. Makes sense that NSString and ICU's UnicodeString are pretty close in implementation. ICU was also ported to Java for Sun by IBM. The point is - this is a very elaborate chunk of code with far reach. If ICU is wrong on some point - it is universally wrong and thus likely to be taken as "right" as it is at least consistent. I think re-implementing it is folly TBH. Just use it.
I suspect the history is wrong as NSString was first written at NeXT when they were not Apple and was not unicode aware at first, but the end result does seem to be NSString now uses ICU
Apple seem to have moved on from NSString to support Unicode in a different way in Switft...
Could you please give some more details?
I have read https://www.objc.io/issues/9-strings/unicode/#nsstring-and-unicode so far.
It says that an NSString object actually represents an array of UTF-16-encoded code units.
This in contrast to Squeak / Pharo where a String is an ArrayedCollection of 21 bit Unicode code points (transparently optimizing to a ByteArray if the string only contains values of the first code page).
The Swift code (well for the next version) is open source see https://github.com/apple/swift-corelibs-foundation for the Foundation lib which includes Strings. Foundation is being rewritten in Swift rather than Objective C so should be Swift->C but I wonder if it is really clean yet, i.e. do they need all the current code. A quick scan gives the low level as Stringing (same as current OSX_. The encodings and conversions seem to use ICU to do the work (which is UTF-16 based) Also in NSString.swift is the line public typealias unichar = UInt16
On Dec 8, 2015, at 15:52, EuanM <euanmee@gmail.com> wrote:
Equally old are the NextStep Object C functions which are now embodied within MacOS X.
-- Mark
Hi Todd, it's taken me til now to put my thoughts into words on this issue. I think we should make it work first. This will allow us to gain more insight into the issues, and create documentation about the process that we, as a community, understand. If ICU is the right way to go, we can *then* "make it right". On 8 December 2015 at 22:36, Todd Blanchard <tblanchard@mac.com> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
EuanM wrote
Hi Todd, it's taken me til now to put my thoughts into words on this issue.
I think we should make it work first. This will allow us to gain more insight into the issues, and create documentation about the process that we, as a community, understand.
If ICU is the right way to go, we can *then* "make it right".
One way to cross the Grand Canyon is to climb down the wall, assemble the materials to build a bridge, build the bridge, cross it, then clamber up the wall. One would learn a lot about climbing and bridge building, worthwhile if the goal is to learn about those subjects. Alternatively, you could use the highway bridge that was built long ago and just cross to the other side. This is definitely my inclination.
On 8 December 2015 at 22:36, Todd Blanchard <
tblanchard@
> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <
dale.henrichs@
>
wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
-- View this message in context: http://forum.world.st/Sorting-Unicode-strings-Re-Unicode-collation-sequences... Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
Hi All,
On Dec 14, 2015, at 7:23 AM, Richard Sargent <richard.sargent@gemtalksystems.com> wrote:
EuanM wrote
Hi Todd, it's taken me til now to put my thoughts into words on this issue.
I think we should make it work first. This will allow us to gain more insight into the issues, and create documentation about the process that we, as a community, understand.
If ICU is the right way to go, we can *then* "make it right".
One way to cross the Grand Canyon is to climb down the wall, assemble the materials to build a bridge, build the bridge, cross it, then clamber up the wall. One would learn a lot about climbing and bridge building, worthwhile if the goal is to learn about those subjects.
Alternatively, you could use the highway bridge that was built long ago and just cross to the other side. This is definitely my inclination.
Using ICU is a disaster at least for the VM. It means we cannot simulate an important part of the system without being able to call ICU from within the simulator. It means bugs in Unicode can potentially cause hard crashes in c++ code, destroying the perfectly safe VM simulation environment /and/ the perfectly safe Smalltalk string library. Goodbye debugger hello gdb. It means having a dependency on a large c++ library for a core part of the system, with all that entails about more complex builds, portability, etc. It means opacity in a core part of the system. This move will be a terrible mistake.
On 8 December 2015 at 22:36, Todd Blanchard <
tblanchard@
> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <
dale.henrichs@
>
wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
-- View this message in context: http://forum.world.st/Sorting-Unicode-strings-Re-Unicode-collation-sequences... Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
On Dec 14, 2015, at 7:23 AM, Richard Sargent <richard.sargent@gemtalksystems.com> wrote:
EuanM wrote
Hi Todd, it's taken me til now to put my thoughts into words on this issue.
I think we should make it work first. This will allow us to gain more insight into the issues, and create documentation about the process that we, as a community, understand.
If ICU is the right way to go, we can *then* "make it right".
One way to cross the Grand Canyon is to climb down the wall, assemble the materials to build a bridge, build the bridge, cross it, then clamber up the wall. One would learn a lot about climbing and bridge building, worthwhile if the goal is to learn about those subjects.
Alternatively, you could use the highway bridge that was built long ago and just cross to the other side. This is definitely my inclination.
and given that translating c++ to Smalltalk is doable since Smalltalk is a more powerful language, whereas going the other way is decidedly non-trivial, the right way to use the highway bridge is to point moose at it, extract the plans for the bridge, and reconstruct it out of quality materials.
On 8 December 2015 at 22:36, Todd Blanchard <
tblanchard@
> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
On Dec 8, 2015, at 11:20, Dale Henrichs <
dale.henrichs@
>
wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
-- View this message in context: http://forum.world.st/Sorting-Unicode-strings-Re-Unicode-collation-sequences... Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
On 12/8/15, Todd Blanchard <tblanchard@mac.com> wrote:
I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
ICU already seems to be available for Smalltalk accessed through wrappers http://wiki.squeak.org/squeak/6234 --Hannes
On Dec 8, 2015, at 11:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote:
Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
So we should start with that. Fwiw - icu also provides time and date format internationalization and some other utilities. It is a very comprehensive library. Sent from the road
On Dec 9, 2015, at 07:03, H. Hirzel <hannes.hirzel@gmail.com> wrote:
On 12/8/15, Todd Blanchard <tblanchard@mac.com> wrote: I just want to second Dale's endorsement of the ICU library. It has been around a long time (originally developed by Taligent) and it provides the base unicode capabilities for an awful lot of software.
I think it would make more sense to bring icu into Smalltalk as a NativeBoost library than to spend resources reimplementing and maintaining it.
-Todd Blanchard
ICU already seems to be available for Smalltalk accessed through wrappers
http://wiki.squeak.org/squeak/6234
--Hannes
On Dec 8, 2015, at 11:20, Dale Henrichs <dale.henrichs@gemtalksystems.com> wrote:
On 12/07/2015 11:31 PM, H. Hirzel wrote: Dale
Thank you for your answer with links to the ICU library and the notes about classes in Gemstone. Noteworthy that you have a class Utf8 as a subclass of ByteArray.
I understand that Gemstone uses the ICU library and thus does not implement the algorithms in Smalltalk.
I am currently looking into what the ICU library provides.
participants (8)
-
Ben Coman -
Dale Henrichs -
Eliot Miranda -
EuanM -
H. Hirzel -
Mark Bestley -
Richard Sargent -
Todd Blanchard