Pharo-dev

By thread

pharo-dev@lists.pharo.org

3 participants
144636 messages

Re: [Pharo-dev] Unicode Support

by EuanM

And indeed, in principle. On 7 December 2015 at 10:51, EuanM <euanmee(a)gmail.com> wrote: > Verifying assumptions is the key reason why you should documents like > this out for review. > > Sven - > > Cuis is encoded with ISO 8859-15 (aka ISO Latin 9) > > Sven, this is *NOT* as you state, ISO 99591, (and not as I stated, 8859-1). > > We caught the right specification bug for the wrong reason. > > Juan: "Cuis: Chose not to use Squeak approach. Chose to make the base > image include and use only 1-byte strings. Chose to use ISO-8859-15" > > I have double-checked - each character encoded in ISO Latin 15 (ISO > 8859-15) is exactly the character represented by the corresponding > 1-byte codepoint in Unicode 0000 to 00FF, > > with the following exceptions: > > codepoint 20ac - Euro symbol > character code a4 (replaces codepoint 00a4 generic currency symbol) > > codepoint 0160 Latin Upper Case S with Caron > character code a6 (replaces codepoint 00A6 was | Unix pipe character) > > codepoint 0161 Latin Lower Case s with Caron > character code a8 (replaces codepoint 00A8 was dierisis) > > codepoint 017d Latin Upper Case Z with Caron > character code b4 (replaces codepoint 00b4 was Acute accent) > > codepoint 017e Latin Lower Case Z with Caron > character code b8 (replaces codepoint 00b8 was cedilla) > > codepoint 0152 Upper Case OE ligature = Ethel > character code bc (replaces codepoint 00bc was 1/4 symbol) > > codepoint 0153 Lower Case oe ligature = ethel > character code bd (replaces codepoint 00bd was 1/2 symbol) > > codepoint 0178 Upper Case Y diaeresis > character code be (replaces codepoint 00be was 3/4 symbol) > > Juan - I don't suppose we could persuade you to change to ISO Latin-1 > from ISO Latin-9 ? > > It means we could run the same 1 byte string encoding across Cuis, > Squeak, Pharo, and, as far as I can make out so far, Dolphin Smalltalk > and Gnu Smalltalk. > > The downside would be that French Y diaeresis would lose the ability > to use that character, along with users of oe, OE, and s, S, z, Z with > caron. Along with the Euro. > > https://en.wikipedia.org/wiki/ISO/IEC_8859-15. > > I'm confident I understand the use of UTF-8 in principal. > > > On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> I am sorry but one of your basic assumptions is completely wrong: >> >> 'Les Ã©lÃ¨ves FranÃ§ais' encodeWith: #iso99591. >> >> => #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115] >> >> 'Les Ã©lÃ¨ves FranÃ§ais' utf8Encoded. >> >> => #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115] >> >> ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !! >> >> Or shorter, $Ã© is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8. >> >> So more than half the points you make, or the facts that you state, are thus plain wrong. >> >> The only thing that is correct is that the code points are equal, but that is not the same as the encoding ! >> >> From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String. >> >> Sorry. >> >> PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world. >> >>> On 07 Dec 2015, at 04:21, EuanM <euanmee(a)gmail.com> wrote: >>> >>> This a long email. A *lot* of it is encapsulated in the Venn diagram both: >>> http://smalltalk.uk.to/unicode-utf8.html >>> and my Smalltalk in Small Steps blog at: >>> http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… >>> >>> My current thinking, and understanding. >>> ============================== >>> >>> 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. >>> b) UTF-8 can encode all of those characters in 1 byte, but can >>> prefer some of them to be encoded as sequences of multiple bytes. And >>> can encode additional characters as sequences of multiple bytes. >>> >>> 1) Smalltalk has long had multiple String classes. >>> >>> 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex >>> is encoded as a UTF-8 codepoint of nn hex. >>> >>> 3) All valid ISO-8859-1 characters have a character code between 20 >>> hex and 7E hex, or between A0 hex and FF hex. >>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1 >>> >>> 4) All valid ASCII characters have a character code between 00 hex and 7E hex. >>> https://en.wikipedia.org/wiki/ASCII >>> >>> >>> 5) a) All character codes which are defined within ISO-8859-1 and also >>> defined within ASCII. (i.e. character codes 20 hex to 7E hex) are >>> defined identically in both. >>> >>> b) All printable ASCII characters are defined identically in both >>> ASCII and ISO-8859-1 >>> >>> 6) All character codes defined in ASCII (00 hex to 7E hex) are >>> defined identically in Unicode UTF-8. >>> >>> 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex >>> - FF hex ) are defined identically in UTF-8. >>> >>> 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. >>> all ASCII maps 1:1 to Unicode UTF-8 >>> all ISO-8859-1 maps 1:1 to Unicode UTF-8 >>> >>> 9) All ByteStrings elements which are either a valid ISO-8859-1 >>> character or a valid ASCII character are *also* a valid UTF-8 >>> character. >>> >>> 10) ISO-8859-1 characters representing a character with a diacritic, >>> or a two-character ligature, have no ASCII equivalent. In Unicode >>> UTF-8, those character codes which are representing compound glyphs, >>> are called "compatibility codepoints". >>> >>> 11) The preferred Unicode representation of the characters which have >>> compatibility codepoints is as a a short set of codepoints >>> representing the characters which are combined together to form the >>> glyph of the convenience codepoint, as a sequence of bytes >>> representing the component characters. >>> >>> >>> 12) Some concrete examples: >>> >>> A - aka Upper Case A >>> In ASCII, in ISO 8859-1 >>> ASCII A - 41 hex >>> ISO-8859-1 A - 41 hex >>> UTF-8 A - 41 hex >>> >>> BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) >>> In ASCII, not in ISO 8859-1 >>> ASCII : BEL - 07 hex >>> ISO-8859-1 : 07 hex is not a valid character code >>> UTF-8 : BEL - 07 hex >>> >>> Â£ (GBP currency symbol) >>> In ISO-8859-1, not in ASCII >>> ASCII : A3 hex is not a valid ASCII code >>> UTF-8: Â£ - A3 hex >>> ISO-8859-1: Â£ - A3 hex >>> >>> Upper Case C cedilla >>> In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint >>> *and* a composed set of codepoints >>> ASCII : C7 hex is not a valid ASCII character code >>> ISO-8859-1 : Upper Case C cedilla - C7 hex >>> UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex >>> Unicode preferred Upper Case C cedilla (composed set of codepoints) >>> Upper case C 0043 hex (Upper case C) >>> followed by >>> cedilla 00B8 hex (cedilla) >>> >>> 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, >>> aByteString is completely adequate for editing and display. >>> >>> 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 >>> string, upper and lower case versions of the same character will be >>> treated differently. >>> >>> 15) When sorting any valid ISO-8859-1 string containing >>> letter+diacritic combination glyphs or ligature combination glyphs, >>> the glyphs in combination will treated differently to a "plain" glyph >>> of the character >>> i.e. "C" and "C cedilla" will be treated very differently. "Ã" and >>> "fs" will be treated very differently. >>> >>> 16) Different nations have different rules about where diacritic-ed >>> characted and ligature pairs should be placed when in alphabetical >>> order. >>> >>> 17) Some nations even have multiple standards - e.g. surnames >>> beginning either "M superscript-c" or "M superscript-a superscript-c" >>> are treated as beginning equivalently in UK phone directories, but not >>> in other situations. >>> >>> >>> Some practical upshots >>> ================== >>> >>> 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, >>> for any single character it considers valid, or any ByteString it has >>> made up of characters it considers valid. >>> >>> 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any >>> other Smalltalk with a single byte ByteString following ASCII or >>> ISO-8859-1. >>> >>> 3) Any Smalltalk (or derivative language) using ByteString can >>> immediately consider it's ByteString as valid UTF-8, as long as it >>> also considers the ByteSring as valid ASCII and/or ISO-8859-1. >>> >>> 4) All of those can be successfully exported to any system using UTF-8 >>> (e.g. HTML). >>> >>> 5) To successfully *accept* all UTF-8 we much be able to do either: >>> a) accept UTF-8 strings with composed characters >>> b) convert UTF-8 strings with composed characters into UTF-8 strings >>> that use *only* compatibility codepoints. >>> >>> >>> Class + protocol proposals >>> >>> >>> >>> a Utf8CompatibilityString class. >>> >>> asByteString - ensure only compatibility codepoints are used. >>> Ensure it doews not encode characters above 00FF hex. >>> >>> asIso8859String - ensures only compatibility codepoints are used, >>> and that the characters are each valid ISO 8859-1 >>> >>> asAsciiString - ensures only characters 00hex - 7F hex are used. >>> >>> asUtf8ComposedIso8859String - ensures all compatibility codepoints >>> are expanded into small OrderedCollections of codepoints >>> >>> a Utf8ComposedIso8859String class - will provide sortable and >>> comparable UTF8 strings of all ASCII and ISO 8859-1 strings. >>> >>> Then a Utf8SortableCollection class - a collection of >>> Utf8ComposedIso8859Strings words and phrases. >>> >>> Custom sortBlocks will define the applicable sort order. >>> >>> We can create a collection... a Dictionary, thinking about it, of >>> named, prefabricated sortBlocks. >>> >>> This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. >>> >>> If anyone has better names for the classes, please let me know. >>> >>> If anyone else wants to help >>> - build these, >>> - create SUnit tests for these >>> - write documentation for these >>> Please let me know. >>> >>> n.b. I have had absolutely no experience of Ropes. >>> >>> My own background with this stuff: In the early 90's as a Project >>> Manager implementing office automation systems across a global >>> company, with offices in the Americas, Western, Eastern and Central >>> Europe, (including Slavic and Cyrillic users) nations, Japan and >>> China. The mission-critical application was word-processing. >>> >>> Our offices were spread around the globe, and we needed those offices >>> to successfully exchange documents with their sister offices, and with >>> the customers in each region the offices were in. >>> >>> Unicode was then new, and our platform supplier was the NeXT >>> Corporation, who had been founder members in of the Unicode Consortium >>> in 1990. >>> >>> So far: I've read the latest version of the Unicode Standard (v8.0). >>> This is freely downloadable. >>> I've purchased a paper copy of an earlier release. New releases >>> typically consist additional codespaces (i.e. alphabets). So old >>> copies are useful, as well as cheap. (Paper copies of version 4.0 >>> are available second-hand for < $10 / â¬10). >>> >>> The typical change with each release is the addition of further >>> codespaces (i.e alphabets (more or less) ), so you don't lose a lot. >>> (I'll be going through my V4.0 just to make sure) >>> >>> Cheers, >>> Euan >>> >>> >>> >>> >>> On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: >>>> Hi EuanM >>>> >>>> Le 4/12/15 12:42, EuanM a Ã©crit : >>>>> >>>>> I'm currently groping my way to seeing how feature-complete our >>>>> Unicode support is. I am doing this to establish what still needs to >>>>> be done to provide full Unicode support. >>>> >>>> >>>> this is great. Thanks for pushing this. I wrote and collected some roadmap >>>> (analyses on different topics) >>>> on the pharo github project feel free to add this one there. >>>>> >>>>> >>>>> This seems to me to be an area where it would be best to write it >>>>> once, and then have the same codebase incorporated into the Smalltalks >>>>> that most share a common ancestry. >>>>> >>>>> I am keen to get: equality-testing for strings; sortability for >>>>> strings which have ligatures and diacritic characters; and correct >>>>> round-tripping of data. >>>> >>>> Go! >>>> My suggestion is >>>> start small >>>> make steady progress >>>> write tests >>>> commit often :) >>>> >>>> Stef >>>> >>>> What is the french phoneBook ordering because this is the first time I hear >>>> about it. >>>> >>>>> >>>>> Call to action: >>>>> ========== >>>>> >>>>> If you have comments on these proposals - such as "but we already have >>>>> that facility" or "the reason we do not have these facilities is >>>>> because they are dog-slow" - please let me know them. >>>>> >>>>> If you would like to help out, please let me know. >>>>> >>>>> If you have Unicode experience and expertise, and would like to be, or >>>>> would be willing to be, in the 'council of experts' for this project, >>>>> please let me know. >>>>> >>>>> If you have comments or ideas on anything mentioned in this email >>>>> >>>>> In the first instance, the initiative's website will be: >>>>> http://smalltalk.uk.to/unicode.html >>>>> >>>>> I have created a SqueakSource.com project called UnicodeSupport >>>>> >>>>> I want to avoid re-inventing any facilities which already exist. >>>>> Except where they prevent us reaching the goals of: >>>>> - sortable UTF8 strings >>>>> - sortable UTF16 strings >>>>> - equivalence testing of 2 UTF8 strings >>>>> - equivalence testing of 2 UTF16 strings >>>>> - round-tripping UTF8 strings through Smalltalk >>>>> - roundtripping UTF16 strings through Smalltalk. >>>>> As I understand it, we have limited Unicode support atm. >>>>> >>>>> Current state of play >>>>> =============== >>>>> ByteString gets converted to WideString when need is automagically >>>>> detected. >>>>> >>>>> Is there anything else that currently exists? >>>>> >>>>> Definition of Terms >>>>> ============== >>>>> A quick definition of terms before I go any further: >>>>> >>>>> Standard terms from the Unicode standard >>>>> =============================== >>>>> a compatibility character : an additional encoding of a *normal* >>>>> character, for compatibility and round-trip conversion purposes. For >>>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>>> >>>>> Made-up terms >>>>> ============ >>>>> a convenience codepoint : a single codepoint which represents an item >>>>> that is also encoded as a string of codepoints. >>>>> >>>>> (I tend to use the terms compatibility character and compatibility >>>>> codepoint interchangably. The standard only refers to them as >>>>> compatibility characters. However, the standard is determined to >>>>> emphasise that characters are abstract and that codepoints are >>>>> concrete. So I think it is often more useful and productive to think >>>>> of compatibility or convenience codepoints). >>>>> >>>>> a composed character : a character made up of several codepoints >>>>> >>>>> Unicode encoding explained >>>>> ===================== >>>>> A convenience codepoint can therefore be thought of as a code point >>>>> used for a character which also has a composed form. >>>>> >>>>> The way Unicode works is that sometimes you can encode a character in >>>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>>> sometimes not. >>>>> >>>>> You can therefore have a long stream of ASCII which is single-byte >>>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>>> stream, it would be represented either by a compatibility character or >>>>> by a multi-byte combination. >>>>> >>>>> Using compatibility characters can prevent proper sorting and >>>>> equivalence testing. >>>>> >>>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>>> and round-tripping probelms. Although avoiding them can *also* cause >>>>> compatibility issues and round-tripping problems. >>>>> >>>>> Currently my thinking is: >>>>> >>>>> a Utf8String class >>>>> an Ordered collection, with 1 byte characters as the modal element, >>>>> but short arrays of wider strings where necessary >>>>> a Utf16String class >>>>> an Ordered collection, with 2 byte characters as the modal element, >>>>> but short arrays of wider strings >>>>> beginning with a 2-byte endianness indicator. >>>>> >>>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>>> compatible. >>>>> >>>>> So my thinking is that Utf8String will contain convenience codepoints, >>>>> for round-tripping. And where there are multiple convenience >>>>> codepoints for a character, that it standardises on one. >>>>> >>>>> And that there is a Utf8SortableString which uses *only* normal >>>>> characters. >>>>> >>>>> We then need methods to convert between the two. >>>>> >>>>> aUtf8String asUtf8SortableString >>>>> >>>>> and >>>>> >>>>> aUtf8SortableString asUtf8String >>>>> >>>>> >>>>> Sort orders are culture and context dependent - Sweden and Germany >>>>> have different sort orders for the same diacritic-ed characters. Some >>>>> countries have one order in general usage, and another for specific >>>>> usages, such as phone directories (e.g. UK and France) >>>>> >>>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>>> conversion methods >>>>> >>>>> A list of sorted words would be a SortedCollection, and there could be >>>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>>> seOrder, ukOrder, etc >>>>> >>>>> along the lines of >>>>> aListOfWords := SortedCollection sortBlock: deOrder >>>>> >>>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>>> then we can perform equivalence testing on them trivially. >>>>> >>>>> To make sure a Utf8String is well formed, we would need to have a way >>>>> of cleaning up any convenience codepoints which were valid, but which >>>>> were for a character which has multiple equally-valid alternative >>>>> convenience codepoints, and for which the string currently had the >>>>> "wrong" convenience codepoint. (i.e for any character with valid >>>>> alternative convenience codepoints, we would choose one to be in the >>>>> well-formed Utf8String, and we would need a method for cleaning the >>>>> alternative convenience codepoints out of the string, and replacing >>>>> them with the chosen approved convenience codepoint. >>>>> >>>>> aUtf8String cleanUtf8String >>>>> >>>>> With WideString, a lot of the issues disappear - except >>>>> round-tripping(although I'm sure I have seen something recently about >>>>> 4-byte strings that also have an additional bit. Which would make >>>>> some Unicode characters 5-bytes long.) >>>>> >>>>> >>>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>>> subtle, or somewhere in between, please let me know) >>>>> >>>>> Cheers, >>>>> Euan >>>>> >>>>> >>>> >>>> >> >>> On 07 Dec 2015, at 04:21, EuanM <euanmee(a)gmail.com> wrote: >>> >>> This a long email. A *lot* of it is encapsulated in the Venn diagram both: >>> http://smalltalk.uk.to/unicode-utf8.html >>> and my Smalltalk in Small Steps blog at: >>> http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… >>> >>> My current thinking, and understanding. >>> ============================== >>> >>> 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. >>> b) UTF-8 can encode all of those characters in 1 byte, but can >>> prefer some of them to be encoded as sequences of multiple bytes. And >>> can encode additional characters as sequences of multiple bytes. >>> >>> 1) Smalltalk has long had multiple String classes. >>> >>> 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex >>> is encoded as a UTF-8 codepoint of nn hex. >>> >>> 3) All valid ISO-8859-1 characters have a character code between 20 >>> hex and 7E hex, or between A0 hex and FF hex. >>> https://en.wikipedia.org/wiki/ISO/IEC_8859-1 >>> >>> 4) All valid ASCII characters have a character code between 00 hex and 7E hex. >>> https://en.wikipedia.org/wiki/ASCII >>> >>> >>> 5) a) All character codes which are defined within ISO-8859-1 and also >>> defined within ASCII. (i.e. character codes 20 hex to 7E hex) are >>> defined identically in both. >>> >>> b) All printable ASCII characters are defined identically in both >>> ASCII and ISO-8859-1 >>> >>> 6) All character codes defined in ASCII (00 hex to 7E hex) are >>> defined identically in Unicode UTF-8. >>> >>> 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex >>> - FF hex ) are defined identically in UTF-8. >>> >>> 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. >>> all ASCII maps 1:1 to Unicode UTF-8 >>> all ISO-8859-1 maps 1:1 to Unicode UTF-8 >>> >>> 9) All ByteStrings elements which are either a valid ISO-8859-1 >>> character or a valid ASCII character are *also* a valid UTF-8 >>> character. >>> >>> 10) ISO-8859-1 characters representing a character with a diacritic, >>> or a two-character ligature, have no ASCII equivalent. In Unicode >>> UTF-8, those character codes which are representing compound glyphs, >>> are called "compatibility codepoints". >>> >>> 11) The preferred Unicode representation of the characters which have >>> compatibility codepoints is as a a short set of codepoints >>> representing the characters which are combined together to form the >>> glyph of the convenience codepoint, as a sequence of bytes >>> representing the component characters. >>> >>> >>> 12) Some concrete examples: >>> >>> A - aka Upper Case A >>> In ASCII, in ISO 8859-1 >>> ASCII A - 41 hex >>> ISO-8859-1 A - 41 hex >>> UTF-8 A - 41 hex >>> >>> BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) >>> In ASCII, not in ISO 8859-1 >>> ASCII : BEL - 07 hex >>> ISO-8859-1 : 07 hex is not a valid character code >>> UTF-8 : BEL - 07 hex >>> >>> Â£ (GBP currency symbol) >>> In ISO-8859-1, not in ASCII >>> ASCII : A3 hex is not a valid ASCII code >>> UTF-8: Â£ - A3 hex >>> ISO-8859-1: Â£ - A3 hex >>> >>> Upper Case C cedilla >>> In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint >>> *and* a composed set of codepoints >>> ASCII : C7 hex is not a valid ASCII character code >>> ISO-8859-1 : Upper Case C cedilla - C7 hex >>> UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex >>> Unicode preferred Upper Case C cedilla (composed set of codepoints) >>> Upper case C 0043 hex (Upper case C) >>> followed by >>> cedilla 00B8 hex (cedilla) >>> >>> 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, >>> aByteString is completely adequate for editing and display. >>> >>> 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 >>> string, upper and lower case versions of the same character will be >>> treated differently. >>> >>> 15) When sorting any valid ISO-8859-1 string containing >>> letter+diacritic combination glyphs or ligature combination glyphs, >>> the glyphs in combination will treated differently to a "plain" glyph >>> of the character >>> i.e. "C" and "C cedilla" will be treated very differently. "Ã" and >>> "fs" will be treated very differently. >>> >>> 16) Different nations have different rules about where diacritic-ed >>> characted and ligature pairs should be placed when in alphabetical >>> order. >>> >>> 17) Some nations even have multiple standards - e.g. surnames >>> beginning either "M superscript-c" or "M superscript-a superscript-c" >>> are treated as beginning equivalently in UK phone directories, but not >>> in other situations. >>> >>> >>> Some practical upshots >>> ================== >>> >>> 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, >>> for any single character it considers valid, or any ByteString it has >>> made up of characters it considers valid. >>> >>> 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any >>> other Smalltalk with a single byte ByteString following ASCII or >>> ISO-8859-1. >>> >>> 3) Any Smalltalk (or derivative language) using ByteString can >>> immediately consider it's ByteString as valid UTF-8, as long as it >>> also considers the ByteSring as valid ASCII and/or ISO-8859-1. >>> >>> 4) All of those can be successfully exported to any system using UTF-8 >>> (e.g. HTML). >>> >>> 5) To successfully *accept* all UTF-8 we much be able to do either: >>> a) accept UTF-8 strings with composed characters >>> b) convert UTF-8 strings with composed characters into UTF-8 strings >>> that use *only* compatibility codepoints. >>> >>> >>> Class + protocol proposals >>> >>> >>> >>> a Utf8CompatibilityString class. >>> >>> asByteString - ensure only compatibility codepoints are used. >>> Ensure it doews not encode characters above 00FF hex. >>> >>> asIso8859String - ensures only compatibility codepoints are used, >>> and that the characters are each valid ISO 8859-1 >>> >>> asAsciiString - ensures only characters 00hex - 7F hex are used. >>> >>> asUtf8ComposedIso8859String - ensures all compatibility codepoints >>> are expanded into small OrderedCollections of codepoints >>> >>> a Utf8ComposedIso8859String class - will provide sortable and >>> comparable UTF8 strings of all ASCII and ISO 8859-1 strings. >>> >>> Then a Utf8SortableCollection class - a collection of >>> Utf8ComposedIso8859Strings words and phrases. >>> >>> Custom sortBlocks will define the applicable sort order. >>> >>> We can create a collection... a Dictionary, thinking about it, of >>> named, prefabricated sortBlocks. >>> >>> This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. >>> >>> If anyone has better names for the classes, please let me know. >>> >>> If anyone else wants to help >>> - build these, >>> - create SUnit tests for these >>> - write documentation for these >>> Please let me know. >>> >>> n.b. I have had absolutely no experience of Ropes. >>> >>> My own background with this stuff: In the early 90's as a Project >>> Manager implementing office automation systems across a global >>> company, with offices in the Americas, Western, Eastern and Central >>> Europe, (including Slavic and Cyrillic users) nations, Japan and >>> China. The mission-critical application was word-processing. >>> >>> Our offices were spread around the globe, and we needed those offices >>> to successfully exchange documents with their sister offices, and with >>> the customers in each region the offices were in. >>> >>> Unicode was then new, and our platform supplier was the NeXT >>> Corporation, who had been founder members in of the Unicode Consortium >>> in 1990. >>> >>> So far: I've read the latest version of the Unicode Standard (v8.0). >>> This is freely downloadable. >>> I've purchased a paper copy of an earlier release. New releases >>> typically consist additional codespaces (i.e. alphabets). So old >>> copies are useful, as well as cheap. (Paper copies of version 4.0 >>> are available second-hand for < $10 / â¬10). >>> >>> The typical change with each release is the addition of further >>> codespaces (i.e alphabets (more or less) ), so you don't lose a lot. >>> (I'll be going through my V4.0 just to make sure) >>> >>> Cheers, >>> Euan >>> >>> >>> >>> >>> On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: >>>> Hi EuanM >>>> >>>> Le 4/12/15 12:42, EuanM a Ã©crit : >>>>> >>>>> I'm currently groping my way to seeing how feature-complete our >>>>> Unicode support is. I am doing this to establish what still needs to >>>>> be done to provide full Unicode support. >>>> >>>> >>>> this is great. Thanks for pushing this. I wrote and collected some roadmap >>>> (analyses on different topics) >>>> on the pharo github project feel free to add this one there. >>>>> >>>>> >>>>> This seems to me to be an area where it would be best to write it >>>>> once, and then have the same codebase incorporated into the Smalltalks >>>>> that most share a common ancestry. >>>>> >>>>> I am keen to get: equality-testing for strings; sortability for >>>>> strings which have ligatures and diacritic characters; and correct >>>>> round-tripping of data. >>>> >>>> Go! >>>> My suggestion is >>>> start small >>>> make steady progress >>>> write tests >>>> commit often :) >>>> >>>> Stef >>>> >>>> What is the french phoneBook ordering because this is the first time I hear >>>> about it. >>>> >>>>> >>>>> Call to action: >>>>> ========== >>>>> >>>>> If you have comments on these proposals - such as "but we already have >>>>> that facility" or "the reason we do not have these facilities is >>>>> because they are dog-slow" - please let me know them. >>>>> >>>>> If you would like to help out, please let me know. >>>>> >>>>> If you have Unicode experience and expertise, and would like to be, or >>>>> would be willing to be, in the 'council of experts' for this project, >>>>> please let me know. >>>>> >>>>> If you have comments or ideas on anything mentioned in this email >>>>> >>>>> In the first instance, the initiative's website will be: >>>>> http://smalltalk.uk.to/unicode.html >>>>> >>>>> I have created a SqueakSource.com project called UnicodeSupport >>>>> >>>>> I want to avoid re-inventing any facilities which already exist. >>>>> Except where they prevent us reaching the goals of: >>>>> - sortable UTF8 strings >>>>> - sortable UTF16 strings >>>>> - equivalence testing of 2 UTF8 strings >>>>> - equivalence testing of 2 UTF16 strings >>>>> - round-tripping UTF8 strings through Smalltalk >>>>> - roundtripping UTF16 strings through Smalltalk. >>>>> As I understand it, we have limited Unicode support atm. >>>>> >>>>> Current state of play >>>>> =============== >>>>> ByteString gets converted to WideString when need is automagically >>>>> detected. >>>>> >>>>> Is there anything else that currently exists? >>>>> >>>>> Definition of Terms >>>>> ============== >>>>> A quick definition of terms before I go any further: >>>>> >>>>> Standard terms from the Unicode standard >>>>> =============================== >>>>> a compatibility character : an additional encoding of a *normal* >>>>> character, for compatibility and round-trip conversion purposes. For >>>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>>> >>>>> Made-up terms >>>>> ============ >>>>> a convenience codepoint : a single codepoint which represents an item >>>>> that is also encoded as a string of codepoints. >>>>> >>>>> (I tend to use the terms compatibility character and compatibility >>>>> codepoint interchangably. The standard only refers to them as >>>>> compatibility characters. However, the standard is determined to >>>>> emphasise that characters are abstract and that codepoints are >>>>> concrete. So I think it is often more useful and productive to think >>>>> of compatibility or convenience codepoints). >>>>> >>>>> a composed character : a character made up of several codepoints >>>>> >>>>> Unicode encoding explained >>>>> ===================== >>>>> A convenience codepoint can therefore be thought of as a code point >>>>> used for a character which also has a composed form. >>>>> >>>>> The way Unicode works is that sometimes you can encode a character in >>>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>>> sometimes not. >>>>> >>>>> You can therefore have a long stream of ASCII which is single-byte >>>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>>> stream, it would be represented either by a compatibility character or >>>>> by a multi-byte combination. >>>>> >>>>> Using compatibility characters can prevent proper sorting and >>>>> equivalence testing. >>>>> >>>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>>> and round-tripping probelms. Although avoiding them can *also* cause >>>>> compatibility issues and round-tripping problems. >>>>> >>>>> Currently my thinking is: >>>>> >>>>> a Utf8String class >>>>> an Ordered collection, with 1 byte characters as the modal element, >>>>> but short arrays of wider strings where necessary >>>>> a Utf16String class >>>>> an Ordered collection, with 2 byte characters as the modal element, >>>>> but short arrays of wider strings >>>>> beginning with a 2-byte endianness indicator. >>>>> >>>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>>> compatible. >>>>> >>>>> So my thinking is that Utf8String will contain convenience codepoints, >>>>> for round-tripping. And where there are multiple convenience >>>>> codepoints for a character, that it standardises on one. >>>>> >>>>> And that there is a Utf8SortableString which uses *only* normal >>>>> characters. >>>>> >>>>> We then need methods to convert between the two. >>>>> >>>>> aUtf8String asUtf8SortableString >>>>> >>>>> and >>>>> >>>>> aUtf8SortableString asUtf8String >>>>> >>>>> >>>>> Sort orders are culture and context dependent - Sweden and Germany >>>>> have different sort orders for the same diacritic-ed characters. Some >>>>> countries have one order in general usage, and another for specific >>>>> usages, such as phone directories (e.g. UK and France) >>>>> >>>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>>> conversion methods >>>>> >>>>> A list of sorted words would be a SortedCollection, and there could be >>>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>>> seOrder, ukOrder, etc >>>>> >>>>> along the lines of >>>>> aListOfWords := SortedCollection sortBlock: deOrder >>>>> >>>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>>> then we can perform equivalence testing on them trivially. >>>>> >>>>> To make sure a Utf8String is well formed, we would need to have a way >>>>> of cleaning up any convenience codepoints which were valid, but which >>>>> were for a character which has multiple equally-valid alternative >>>>> convenience codepoints, and for which the string currently had the >>>>> "wrong" convenience codepoint. (i.e for any character with valid >>>>> alternative convenience codepoints, we would choose one to be in the >>>>> well-formed Utf8String, and we would need a method for cleaning the >>>>> alternative convenience codepoints out of the string, and replacing >>>>> them with the chosen approved convenience codepoint. >>>>> >>>>> aUtf8String cleanUtf8String >>>>> >>>>> With WideString, a lot of the issues disappear - except >>>>> round-tripping(although I'm sure I have seen something recently about >>>>> 4-byte strings that also have an additional bit. Which would make >>>>> some Unicode characters 5-bytes long.) >>>>> >>>>> >>>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>>> subtle, or somewhere in between, please let me know) >>>>> >>>>> Cheers, >>>>> Euan >>>>> >>>>> >>>> >>>> >>> >> >>

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

Verifying assumptions is the key reason why you should documents like this out for review. Sven - Cuis is encoded with ISO 8859-15 (aka ISO Latin 9) Sven, this is *NOT* as you state, ISO 99591, (and not as I stated, 8859-1). We caught the right specification bug for the wrong reason. Juan: "Cuis: Chose not to use Squeak approach. Chose to make the base image include and use only 1-byte strings. Chose to use ISO-8859-15" I have double-checked - each character encoded in ISO Latin 15 (ISO 8859-15) is exactly the character represented by the corresponding 1-byte codepoint in Unicode 0000 to 00FF, with the following exceptions: codepoint 20ac - Euro symbol character code a4 (replaces codepoint 00a4 generic currency symbol) codepoint 0160 Latin Upper Case S with Caron character code a6 (replaces codepoint 00A6 was | Unix pipe character) codepoint 0161 Latin Lower Case s with Caron character code a8 (replaces codepoint 00A8 was dierisis) codepoint 017d Latin Upper Case Z with Caron character code b4 (replaces codepoint 00b4 was Acute accent) codepoint 017e Latin Lower Case Z with Caron character code b8 (replaces codepoint 00b8 was cedilla) codepoint 0152 Upper Case OE ligature = Ethel character code bc (replaces codepoint 00bc was 1/4 symbol) codepoint 0153 Lower Case oe ligature = ethel character code bd (replaces codepoint 00bd was 1/2 symbol) codepoint 0178 Upper Case Y diaeresis character code be (replaces codepoint 00be was 3/4 symbol) Juan - I don't suppose we could persuade you to change to ISO Latin-1 from ISO Latin-9 ? It means we could run the same 1 byte string encoding across Cuis, Squeak, Pharo, and, as far as I can make out so far, Dolphin Smalltalk and Gnu Smalltalk. The downside would be that French Y diaeresis would lose the ability to use that character, along with users of oe, OE, and s, S, z, Z with caron. Along with the Euro. https://en.wikipedia.org/wiki/ISO/IEC_8859-15. I'm confident I understand the use of UTF-8 in principal. On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > I am sorry but one of your basic assumptions is completely wrong: > > 'Les Ã©lÃ¨ves FranÃ§ais' encodeWith: #iso99591. > > => #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115] > > 'Les Ã©lÃ¨ves FranÃ§ais' utf8Encoded. > > => #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115] > > ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !! > > Or shorter, $Ã© is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8. > > So more than half the points you make, or the facts that you state, are thus plain wrong. > > The only thing that is correct is that the code points are equal, but that is not the same as the encoding ! > > From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String. > > Sorry. > > PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world. > >> On 07 Dec 2015, at 04:21, EuanM <euanmee(a)gmail.com> wrote: >> >> This a long email. A *lot* of it is encapsulated in the Venn diagram both: >> http://smalltalk.uk.to/unicode-utf8.html >> and my Smalltalk in Small Steps blog at: >> http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… >> >> My current thinking, and understanding. >> ============================== >> >> 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. >> b) UTF-8 can encode all of those characters in 1 byte, but can >> prefer some of them to be encoded as sequences of multiple bytes. And >> can encode additional characters as sequences of multiple bytes. >> >> 1) Smalltalk has long had multiple String classes. >> >> 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex >> is encoded as a UTF-8 codepoint of nn hex. >> >> 3) All valid ISO-8859-1 characters have a character code between 20 >> hex and 7E hex, or between A0 hex and FF hex. >> https://en.wikipedia.org/wiki/ISO/IEC_8859-1 >> >> 4) All valid ASCII characters have a character code between 00 hex and 7E hex. >> https://en.wikipedia.org/wiki/ASCII >> >> >> 5) a) All character codes which are defined within ISO-8859-1 and also >> defined within ASCII. (i.e. character codes 20 hex to 7E hex) are >> defined identically in both. >> >> b) All printable ASCII characters are defined identically in both >> ASCII and ISO-8859-1 >> >> 6) All character codes defined in ASCII (00 hex to 7E hex) are >> defined identically in Unicode UTF-8. >> >> 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex >> - FF hex ) are defined identically in UTF-8. >> >> 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. >> all ASCII maps 1:1 to Unicode UTF-8 >> all ISO-8859-1 maps 1:1 to Unicode UTF-8 >> >> 9) All ByteStrings elements which are either a valid ISO-8859-1 >> character or a valid ASCII character are *also* a valid UTF-8 >> character. >> >> 10) ISO-8859-1 characters representing a character with a diacritic, >> or a two-character ligature, have no ASCII equivalent. In Unicode >> UTF-8, those character codes which are representing compound glyphs, >> are called "compatibility codepoints". >> >> 11) The preferred Unicode representation of the characters which have >> compatibility codepoints is as a a short set of codepoints >> representing the characters which are combined together to form the >> glyph of the convenience codepoint, as a sequence of bytes >> representing the component characters. >> >> >> 12) Some concrete examples: >> >> A - aka Upper Case A >> In ASCII, in ISO 8859-1 >> ASCII A - 41 hex >> ISO-8859-1 A - 41 hex >> UTF-8 A - 41 hex >> >> BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) >> In ASCII, not in ISO 8859-1 >> ASCII : BEL - 07 hex >> ISO-8859-1 : 07 hex is not a valid character code >> UTF-8 : BEL - 07 hex >> >> Â£ (GBP currency symbol) >> In ISO-8859-1, not in ASCII >> ASCII : A3 hex is not a valid ASCII code >> UTF-8: Â£ - A3 hex >> ISO-8859-1: Â£ - A3 hex >> >> Upper Case C cedilla >> In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint >> *and* a composed set of codepoints >> ASCII : C7 hex is not a valid ASCII character code >> ISO-8859-1 : Upper Case C cedilla - C7 hex >> UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex >> Unicode preferred Upper Case C cedilla (composed set of codepoints) >> Upper case C 0043 hex (Upper case C) >> followed by >> cedilla 00B8 hex (cedilla) >> >> 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, >> aByteString is completely adequate for editing and display. >> >> 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 >> string, upper and lower case versions of the same character will be >> treated differently. >> >> 15) When sorting any valid ISO-8859-1 string containing >> letter+diacritic combination glyphs or ligature combination glyphs, >> the glyphs in combination will treated differently to a "plain" glyph >> of the character >> i.e. "C" and "C cedilla" will be treated very differently. "Ã" and >> "fs" will be treated very differently. >> >> 16) Different nations have different rules about where diacritic-ed >> characted and ligature pairs should be placed when in alphabetical >> order. >> >> 17) Some nations even have multiple standards - e.g. surnames >> beginning either "M superscript-c" or "M superscript-a superscript-c" >> are treated as beginning equivalently in UK phone directories, but not >> in other situations. >> >> >> Some practical upshots >> ================== >> >> 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, >> for any single character it considers valid, or any ByteString it has >> made up of characters it considers valid. >> >> 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any >> other Smalltalk with a single byte ByteString following ASCII or >> ISO-8859-1. >> >> 3) Any Smalltalk (or derivative language) using ByteString can >> immediately consider it's ByteString as valid UTF-8, as long as it >> also considers the ByteSring as valid ASCII and/or ISO-8859-1. >> >> 4) All of those can be successfully exported to any system using UTF-8 >> (e.g. HTML). >> >> 5) To successfully *accept* all UTF-8 we much be able to do either: >> a) accept UTF-8 strings with composed characters >> b) convert UTF-8 strings with composed characters into UTF-8 strings >> that use *only* compatibility codepoints. >> >> >> Class + protocol proposals >> >> >> >> a Utf8CompatibilityString class. >> >> asByteString - ensure only compatibility codepoints are used. >> Ensure it doews not encode characters above 00FF hex. >> >> asIso8859String - ensures only compatibility codepoints are used, >> and that the characters are each valid ISO 8859-1 >> >> asAsciiString - ensures only characters 00hex - 7F hex are used. >> >> asUtf8ComposedIso8859String - ensures all compatibility codepoints >> are expanded into small OrderedCollections of codepoints >> >> a Utf8ComposedIso8859String class - will provide sortable and >> comparable UTF8 strings of all ASCII and ISO 8859-1 strings. >> >> Then a Utf8SortableCollection class - a collection of >> Utf8ComposedIso8859Strings words and phrases. >> >> Custom sortBlocks will define the applicable sort order. >> >> We can create a collection... a Dictionary, thinking about it, of >> named, prefabricated sortBlocks. >> >> This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. >> >> If anyone has better names for the classes, please let me know. >> >> If anyone else wants to help >> - build these, >> - create SUnit tests for these >> - write documentation for these >> Please let me know. >> >> n.b. I have had absolutely no experience of Ropes. >> >> My own background with this stuff: In the early 90's as a Project >> Manager implementing office automation systems across a global >> company, with offices in the Americas, Western, Eastern and Central >> Europe, (including Slavic and Cyrillic users) nations, Japan and >> China. The mission-critical application was word-processing. >> >> Our offices were spread around the globe, and we needed those offices >> to successfully exchange documents with their sister offices, and with >> the customers in each region the offices were in. >> >> Unicode was then new, and our platform supplier was the NeXT >> Corporation, who had been founder members in of the Unicode Consortium >> in 1990. >> >> So far: I've read the latest version of the Unicode Standard (v8.0). >> This is freely downloadable. >> I've purchased a paper copy of an earlier release. New releases >> typically consist additional codespaces (i.e. alphabets). So old >> copies are useful, as well as cheap. (Paper copies of version 4.0 >> are available second-hand for < $10 / â¬10). >> >> The typical change with each release is the addition of further >> codespaces (i.e alphabets (more or less) ), so you don't lose a lot. >> (I'll be going through my V4.0 just to make sure) >> >> Cheers, >> Euan >> >> >> >> >> On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: >>> Hi EuanM >>> >>> Le 4/12/15 12:42, EuanM a Ã©crit : >>>> >>>> I'm currently groping my way to seeing how feature-complete our >>>> Unicode support is. I am doing this to establish what still needs to >>>> be done to provide full Unicode support. >>> >>> >>> this is great. Thanks for pushing this. I wrote and collected some roadmap >>> (analyses on different topics) >>> on the pharo github project feel free to add this one there. >>>> >>>> >>>> This seems to me to be an area where it would be best to write it >>>> once, and then have the same codebase incorporated into the Smalltalks >>>> that most share a common ancestry. >>>> >>>> I am keen to get: equality-testing for strings; sortability for >>>> strings which have ligatures and diacritic characters; and correct >>>> round-tripping of data. >>> >>> Go! >>> My suggestion is >>> start small >>> make steady progress >>> write tests >>> commit often :) >>> >>> Stef >>> >>> What is the french phoneBook ordering because this is the first time I hear >>> about it. >>> >>>> >>>> Call to action: >>>> ========== >>>> >>>> If you have comments on these proposals - such as "but we already have >>>> that facility" or "the reason we do not have these facilities is >>>> because they are dog-slow" - please let me know them. >>>> >>>> If you would like to help out, please let me know. >>>> >>>> If you have Unicode experience and expertise, and would like to be, or >>>> would be willing to be, in the 'council of experts' for this project, >>>> please let me know. >>>> >>>> If you have comments or ideas on anything mentioned in this email >>>> >>>> In the first instance, the initiative's website will be: >>>> http://smalltalk.uk.to/unicode.html >>>> >>>> I have created a SqueakSource.com project called UnicodeSupport >>>> >>>> I want to avoid re-inventing any facilities which already exist. >>>> Except where they prevent us reaching the goals of: >>>> - sortable UTF8 strings >>>> - sortable UTF16 strings >>>> - equivalence testing of 2 UTF8 strings >>>> - equivalence testing of 2 UTF16 strings >>>> - round-tripping UTF8 strings through Smalltalk >>>> - roundtripping UTF16 strings through Smalltalk. >>>> As I understand it, we have limited Unicode support atm. >>>> >>>> Current state of play >>>> =============== >>>> ByteString gets converted to WideString when need is automagically >>>> detected. >>>> >>>> Is there anything else that currently exists? >>>> >>>> Definition of Terms >>>> ============== >>>> A quick definition of terms before I go any further: >>>> >>>> Standard terms from the Unicode standard >>>> =============================== >>>> a compatibility character : an additional encoding of a *normal* >>>> character, for compatibility and round-trip conversion purposes. For >>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>> >>>> Made-up terms >>>> ============ >>>> a convenience codepoint : a single codepoint which represents an item >>>> that is also encoded as a string of codepoints. >>>> >>>> (I tend to use the terms compatibility character and compatibility >>>> codepoint interchangably. The standard only refers to them as >>>> compatibility characters. However, the standard is determined to >>>> emphasise that characters are abstract and that codepoints are >>>> concrete. So I think it is often more useful and productive to think >>>> of compatibility or convenience codepoints). >>>> >>>> a composed character : a character made up of several codepoints >>>> >>>> Unicode encoding explained >>>> ===================== >>>> A convenience codepoint can therefore be thought of as a code point >>>> used for a character which also has a composed form. >>>> >>>> The way Unicode works is that sometimes you can encode a character in >>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>> sometimes not. >>>> >>>> You can therefore have a long stream of ASCII which is single-byte >>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>> stream, it would be represented either by a compatibility character or >>>> by a multi-byte combination. >>>> >>>> Using compatibility characters can prevent proper sorting and >>>> equivalence testing. >>>> >>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>> and round-tripping probelms. Although avoiding them can *also* cause >>>> compatibility issues and round-tripping problems. >>>> >>>> Currently my thinking is: >>>> >>>> a Utf8String class >>>> an Ordered collection, with 1 byte characters as the modal element, >>>> but short arrays of wider strings where necessary >>>> a Utf16String class >>>> an Ordered collection, with 2 byte characters as the modal element, >>>> but short arrays of wider strings >>>> beginning with a 2-byte endianness indicator. >>>> >>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>> compatible. >>>> >>>> So my thinking is that Utf8String will contain convenience codepoints, >>>> for round-tripping. And where there are multiple convenience >>>> codepoints for a character, that it standardises on one. >>>> >>>> And that there is a Utf8SortableString which uses *only* normal >>>> characters. >>>> >>>> We then need methods to convert between the two. >>>> >>>> aUtf8String asUtf8SortableString >>>> >>>> and >>>> >>>> aUtf8SortableString asUtf8String >>>> >>>> >>>> Sort orders are culture and context dependent - Sweden and Germany >>>> have different sort orders for the same diacritic-ed characters. Some >>>> countries have one order in general usage, and another for specific >>>> usages, such as phone directories (e.g. UK and France) >>>> >>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>> conversion methods >>>> >>>> A list of sorted words would be a SortedCollection, and there could be >>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>> seOrder, ukOrder, etc >>>> >>>> along the lines of >>>> aListOfWords := SortedCollection sortBlock: deOrder >>>> >>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>> then we can perform equivalence testing on them trivially. >>>> >>>> To make sure a Utf8String is well formed, we would need to have a way >>>> of cleaning up any convenience codepoints which were valid, but which >>>> were for a character which has multiple equally-valid alternative >>>> convenience codepoints, and for which the string currently had the >>>> "wrong" convenience codepoint. (i.e for any character with valid >>>> alternative convenience codepoints, we would choose one to be in the >>>> well-formed Utf8String, and we would need a method for cleaning the >>>> alternative convenience codepoints out of the string, and replacing >>>> them with the chosen approved convenience codepoint. >>>> >>>> aUtf8String cleanUtf8String >>>> >>>> With WideString, a lot of the issues disappear - except >>>> round-tripping(although I'm sure I have seen something recently about >>>> 4-byte strings that also have an additional bit. Which would make >>>> some Unicode characters 5-bytes long.) >>>> >>>> >>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>> subtle, or somewhere in between, please let me know) >>>> >>>> Cheers, >>>> Euan >>>> >>>> >>> >>> > >> On 07 Dec 2015, at 04:21, EuanM <euanmee(a)gmail.com> wrote: >> >> This a long email. A *lot* of it is encapsulated in the Venn diagram both: >> http://smalltalk.uk.to/unicode-utf8.html >> and my Smalltalk in Small Steps blog at: >> http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… >> >> My current thinking, and understanding. >> ============================== >> >> 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. >> b) UTF-8 can encode all of those characters in 1 byte, but can >> prefer some of them to be encoded as sequences of multiple bytes. And >> can encode additional characters as sequences of multiple bytes. >> >> 1) Smalltalk has long had multiple String classes. >> >> 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex >> is encoded as a UTF-8 codepoint of nn hex. >> >> 3) All valid ISO-8859-1 characters have a character code between 20 >> hex and 7E hex, or between A0 hex and FF hex. >> https://en.wikipedia.org/wiki/ISO/IEC_8859-1 >> >> 4) All valid ASCII characters have a character code between 00 hex and 7E hex. >> https://en.wikipedia.org/wiki/ASCII >> >> >> 5) a) All character codes which are defined within ISO-8859-1 and also >> defined within ASCII. (i.e. character codes 20 hex to 7E hex) are >> defined identically in both. >> >> b) All printable ASCII characters are defined identically in both >> ASCII and ISO-8859-1 >> >> 6) All character codes defined in ASCII (00 hex to 7E hex) are >> defined identically in Unicode UTF-8. >> >> 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex >> - FF hex ) are defined identically in UTF-8. >> >> 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. >> all ASCII maps 1:1 to Unicode UTF-8 >> all ISO-8859-1 maps 1:1 to Unicode UTF-8 >> >> 9) All ByteStrings elements which are either a valid ISO-8859-1 >> character or a valid ASCII character are *also* a valid UTF-8 >> character. >> >> 10) ISO-8859-1 characters representing a character with a diacritic, >> or a two-character ligature, have no ASCII equivalent. In Unicode >> UTF-8, those character codes which are representing compound glyphs, >> are called "compatibility codepoints". >> >> 11) The preferred Unicode representation of the characters which have >> compatibility codepoints is as a a short set of codepoints >> representing the characters which are combined together to form the >> glyph of the convenience codepoint, as a sequence of bytes >> representing the component characters. >> >> >> 12) Some concrete examples: >> >> A - aka Upper Case A >> In ASCII, in ISO 8859-1 >> ASCII A - 41 hex >> ISO-8859-1 A - 41 hex >> UTF-8 A - 41 hex >> >> BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) >> In ASCII, not in ISO 8859-1 >> ASCII : BEL - 07 hex >> ISO-8859-1 : 07 hex is not a valid character code >> UTF-8 : BEL - 07 hex >> >> Â£ (GBP currency symbol) >> In ISO-8859-1, not in ASCII >> ASCII : A3 hex is not a valid ASCII code >> UTF-8: Â£ - A3 hex >> ISO-8859-1: Â£ - A3 hex >> >> Upper Case C cedilla >> In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint >> *and* a composed set of codepoints >> ASCII : C7 hex is not a valid ASCII character code >> ISO-8859-1 : Upper Case C cedilla - C7 hex >> UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex >> Unicode preferred Upper Case C cedilla (composed set of codepoints) >> Upper case C 0043 hex (Upper case C) >> followed by >> cedilla 00B8 hex (cedilla) >> >> 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, >> aByteString is completely adequate for editing and display. >> >> 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 >> string, upper and lower case versions of the same character will be >> treated differently. >> >> 15) When sorting any valid ISO-8859-1 string containing >> letter+diacritic combination glyphs or ligature combination glyphs, >> the glyphs in combination will treated differently to a "plain" glyph >> of the character >> i.e. "C" and "C cedilla" will be treated very differently. "Ã" and >> "fs" will be treated very differently. >> >> 16) Different nations have different rules about where diacritic-ed >> characted and ligature pairs should be placed when in alphabetical >> order. >> >> 17) Some nations even have multiple standards - e.g. surnames >> beginning either "M superscript-c" or "M superscript-a superscript-c" >> are treated as beginning equivalently in UK phone directories, but not >> in other situations. >> >> >> Some practical upshots >> ================== >> >> 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, >> for any single character it considers valid, or any ByteString it has >> made up of characters it considers valid. >> >> 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any >> other Smalltalk with a single byte ByteString following ASCII or >> ISO-8859-1. >> >> 3) Any Smalltalk (or derivative language) using ByteString can >> immediately consider it's ByteString as valid UTF-8, as long as it >> also considers the ByteSring as valid ASCII and/or ISO-8859-1. >> >> 4) All of those can be successfully exported to any system using UTF-8 >> (e.g. HTML). >> >> 5) To successfully *accept* all UTF-8 we much be able to do either: >> a) accept UTF-8 strings with composed characters >> b) convert UTF-8 strings with composed characters into UTF-8 strings >> that use *only* compatibility codepoints. >> >> >> Class + protocol proposals >> >> >> >> a Utf8CompatibilityString class. >> >> asByteString - ensure only compatibility codepoints are used. >> Ensure it doews not encode characters above 00FF hex. >> >> asIso8859String - ensures only compatibility codepoints are used, >> and that the characters are each valid ISO 8859-1 >> >> asAsciiString - ensures only characters 00hex - 7F hex are used. >> >> asUtf8ComposedIso8859String - ensures all compatibility codepoints >> are expanded into small OrderedCollections of codepoints >> >> a Utf8ComposedIso8859String class - will provide sortable and >> comparable UTF8 strings of all ASCII and ISO 8859-1 strings. >> >> Then a Utf8SortableCollection class - a collection of >> Utf8ComposedIso8859Strings words and phrases. >> >> Custom sortBlocks will define the applicable sort order. >> >> We can create a collection... a Dictionary, thinking about it, of >> named, prefabricated sortBlocks. >> >> This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. >> >> If anyone has better names for the classes, please let me know. >> >> If anyone else wants to help >> - build these, >> - create SUnit tests for these >> - write documentation for these >> Please let me know. >> >> n.b. I have had absolutely no experience of Ropes. >> >> My own background with this stuff: In the early 90's as a Project >> Manager implementing office automation systems across a global >> company, with offices in the Americas, Western, Eastern and Central >> Europe, (including Slavic and Cyrillic users) nations, Japan and >> China. The mission-critical application was word-processing. >> >> Our offices were spread around the globe, and we needed those offices >> to successfully exchange documents with their sister offices, and with >> the customers in each region the offices were in. >> >> Unicode was then new, and our platform supplier was the NeXT >> Corporation, who had been founder members in of the Unicode Consortium >> in 1990. >> >> So far: I've read the latest version of the Unicode Standard (v8.0). >> This is freely downloadable. >> I've purchased a paper copy of an earlier release. New releases >> typically consist additional codespaces (i.e. alphabets). So old >> copies are useful, as well as cheap. (Paper copies of version 4.0 >> are available second-hand for < $10 / â¬10). >> >> The typical change with each release is the addition of further >> codespaces (i.e alphabets (more or less) ), so you don't lose a lot. >> (I'll be going through my V4.0 just to make sure) >> >> Cheers, >> Euan >> >> >> >> >> On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: >>> Hi EuanM >>> >>> Le 4/12/15 12:42, EuanM a Ã©crit : >>>> >>>> I'm currently groping my way to seeing how feature-complete our >>>> Unicode support is. I am doing this to establish what still needs to >>>> be done to provide full Unicode support. >>> >>> >>> this is great. Thanks for pushing this. I wrote and collected some roadmap >>> (analyses on different topics) >>> on the pharo github project feel free to add this one there. >>>> >>>> >>>> This seems to me to be an area where it would be best to write it >>>> once, and then have the same codebase incorporated into the Smalltalks >>>> that most share a common ancestry. >>>> >>>> I am keen to get: equality-testing for strings; sortability for >>>> strings which have ligatures and diacritic characters; and correct >>>> round-tripping of data. >>> >>> Go! >>> My suggestion is >>> start small >>> make steady progress >>> write tests >>> commit often :) >>> >>> Stef >>> >>> What is the french phoneBook ordering because this is the first time I hear >>> about it. >>> >>>> >>>> Call to action: >>>> ========== >>>> >>>> If you have comments on these proposals - such as "but we already have >>>> that facility" or "the reason we do not have these facilities is >>>> because they are dog-slow" - please let me know them. >>>> >>>> If you would like to help out, please let me know. >>>> >>>> If you have Unicode experience and expertise, and would like to be, or >>>> would be willing to be, in the 'council of experts' for this project, >>>> please let me know. >>>> >>>> If you have comments or ideas on anything mentioned in this email >>>> >>>> In the first instance, the initiative's website will be: >>>> http://smalltalk.uk.to/unicode.html >>>> >>>> I have created a SqueakSource.com project called UnicodeSupport >>>> >>>> I want to avoid re-inventing any facilities which already exist. >>>> Except where they prevent us reaching the goals of: >>>> - sortable UTF8 strings >>>> - sortable UTF16 strings >>>> - equivalence testing of 2 UTF8 strings >>>> - equivalence testing of 2 UTF16 strings >>>> - round-tripping UTF8 strings through Smalltalk >>>> - roundtripping UTF16 strings through Smalltalk. >>>> As I understand it, we have limited Unicode support atm. >>>> >>>> Current state of play >>>> =============== >>>> ByteString gets converted to WideString when need is automagically >>>> detected. >>>> >>>> Is there anything else that currently exists? >>>> >>>> Definition of Terms >>>> ============== >>>> A quick definition of terms before I go any further: >>>> >>>> Standard terms from the Unicode standard >>>> =============================== >>>> a compatibility character : an additional encoding of a *normal* >>>> character, for compatibility and round-trip conversion purposes. For >>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>> >>>> Made-up terms >>>> ============ >>>> a convenience codepoint : a single codepoint which represents an item >>>> that is also encoded as a string of codepoints. >>>> >>>> (I tend to use the terms compatibility character and compatibility >>>> codepoint interchangably. The standard only refers to them as >>>> compatibility characters. However, the standard is determined to >>>> emphasise that characters are abstract and that codepoints are >>>> concrete. So I think it is often more useful and productive to think >>>> of compatibility or convenience codepoints). >>>> >>>> a composed character : a character made up of several codepoints >>>> >>>> Unicode encoding explained >>>> ===================== >>>> A convenience codepoint can therefore be thought of as a code point >>>> used for a character which also has a composed form. >>>> >>>> The way Unicode works is that sometimes you can encode a character in >>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>> sometimes not. >>>> >>>> You can therefore have a long stream of ASCII which is single-byte >>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>> stream, it would be represented either by a compatibility character or >>>> by a multi-byte combination. >>>> >>>> Using compatibility characters can prevent proper sorting and >>>> equivalence testing. >>>> >>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>> and round-tripping probelms. Although avoiding them can *also* cause >>>> compatibility issues and round-tripping problems. >>>> >>>> Currently my thinking is: >>>> >>>> a Utf8String class >>>> an Ordered collection, with 1 byte characters as the modal element, >>>> but short arrays of wider strings where necessary >>>> a Utf16String class >>>> an Ordered collection, with 2 byte characters as the modal element, >>>> but short arrays of wider strings >>>> beginning with a 2-byte endianness indicator. >>>> >>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>> compatible. >>>> >>>> So my thinking is that Utf8String will contain convenience codepoints, >>>> for round-tripping. And where there are multiple convenience >>>> codepoints for a character, that it standardises on one. >>>> >>>> And that there is a Utf8SortableString which uses *only* normal >>>> characters. >>>> >>>> We then need methods to convert between the two. >>>> >>>> aUtf8String asUtf8SortableString >>>> >>>> and >>>> >>>> aUtf8SortableString asUtf8String >>>> >>>> >>>> Sort orders are culture and context dependent - Sweden and Germany >>>> have different sort orders for the same diacritic-ed characters. Some >>>> countries have one order in general usage, and another for specific >>>> usages, such as phone directories (e.g. UK and France) >>>> >>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>> conversion methods >>>> >>>> A list of sorted words would be a SortedCollection, and there could be >>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>> seOrder, ukOrder, etc >>>> >>>> along the lines of >>>> aListOfWords := SortedCollection sortBlock: deOrder >>>> >>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>> then we can perform equivalence testing on them trivially. >>>> >>>> To make sure a Utf8String is well formed, we would need to have a way >>>> of cleaning up any convenience codepoints which were valid, but which >>>> were for a character which has multiple equally-valid alternative >>>> convenience codepoints, and for which the string currently had the >>>> "wrong" convenience codepoint. (i.e for any character with valid >>>> alternative convenience codepoints, we would choose one to be in the >>>> well-formed Utf8String, and we would need a method for cleaning the >>>> alternative convenience codepoints out of the string, and replacing >>>> them with the chosen approved convenience codepoint. >>>> >>>> aUtf8String cleanUtf8String >>>> >>>> With WideString, a lot of the issues disappear - except >>>> round-tripping(although I'm sure I have seen something recently about >>>> 4-byte strings that also have an additional bit. Which would make >>>> some Unicode characters 5-bytes long.) >>>> >>>> >>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>> subtle, or somewhere in between, please let me know) >>>> >>>> Cheers, >>>> Euan >>>> >>>> >>> >>> >> > >

Dec. 7, 2015

Re: [Pharo-dev] [squeak-dev] Unicode Support

by Max Leske

> On 06 Dec 2015, at 22:07, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Ah yes, Go is a good example, thanks Mark. > > After reading these two blog articles: > > http://blog.golang.org/strings > http://blog.golang.org/normalization > > And especially after looking through their libraries/APIs, the conclusion is: this is not simple. > > I am also not so sure they managed to offer such an API to their users so that it is easier for them not to make mistakes (given that they leave a lot of things 'open'). > > Note that they also say that "In practice, 99.98% of the web's HTML page content is in NFC form (not counting markup, in which case it would be more)." I must say that I have never come across anything else myself, let alone something that gave a problem, but that probably depends on the situation. > > @Max: you seem to suggest that you do see non-normalised unicode, where does it come from, how does it happen ? We do have problems occasionally with pasted strings that come, at least in the cases weâve been able to determine, from PDF documents (pasted into a web browser). Some generators apparently donât normalise UTF-8 and then we end up with characters (in our case those are mostly umlauts) that are made up of two distinct characters (e.g. âuâ + âcombining diaeresisâ). Max > > Sven > >> On 06 Dec 2015, at 20:27, Mark Bestley <st(a)bestley.co.uk> wrote: >> >> On 06/12/2015 19:08, Sven Van Caekenberghe wrote: >>> BTW, does anyone know of any programming language that did go that way or has a library that directly implements 'storing all strings as utf-8' ? >> Java is UTF-16 >> >> Python3, Go and Swift are UTF-8 as I suspect are other new languages not based on .Net or the JVM >> >> Mark >> >>> >> >> >>>> On 06 Dec 2015, at 18:45, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>> >>>> Well written, Todd. I agree, the loss of indexing might not be that big a problem in practice. The only way to find out it to try an experiment I guess. >>>> >>>> Sven >>>> >>>>> On 06 Dec 2015, at 17:37, Todd Blanchard <tblanchard(a)mac.com> wrote: >>>>> >>>>> (Resent because of bounce notification (email handling in osx is really beginning to annoy me). Sorry if its a dup) >>>>> >>>>> I used to worry a lot about strings being indexable. And then I eventually let go of that and realized that it isn't a particularly important property for them to have. >>>>> >>>>> I think you will find that UTF8 is generally the most convenient for a lot of things but its a bit like light in that you treat it alternately as a wave or particle depending on what you are trying to do. >>>>> >>>>> So goes strings - they can be treated alternately as streams or byte arrays (not character arrays - stop thinking in characters). In practice, this tends to not be a problem since a lot of the times when you want to replace a character or pick out the nth one you are doing something very computerish and the characters you are working with are the single byte (ASCII legacy) variety. You generally know when you can get away with that and when you can't. >>>>> >>>>> Otherwise you are most likely doing things that are best dealt with in a streaming paradigm. For most computation, you come to realize you don't generally care how many characters but how much space (bytes) you need to store your chunk of text. Collation is tricky and complicated in unicode in general but it isn't any worse in UTF8 than any other encoding. You are still going to scan each sortable item from front to back to determine its order, regardless. >>>>> >>>>> Most of the outside world has settled on UTF8 and any ASCII file is already UTF8 - which is why it ends up being so convenient. Most of our old text handling infrastructure can still handle UTF8 while it tends to choke on wider encodings. >>>>> >>>>> -Todd Blanchard >>>>> >>>>>> On Dec 6, 2015, at 07:23, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>>>>> >>>>>>> We do the same thing, but that doesn't mean it's a good idea to create a >>>>>>> new String-like class having its content encoded in UTF-8, because >>>>>>> UTF-8-encoded strings can't be modified like regular strings. While it >>>>>>> would be possible to implement all operations, such implementation would >>>>>>> become the next SortedCollection (bad performance due to misuse). >>>>> >>>> >>> >>> >>> >> >> >> -- >> Mark > >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by Sven Van Caekenberghe

I am sorry but one of your basic assumptions is completely wrong: 'Les Ã©lÃ¨ves FranÃ§ais' encodeWith: #iso99591. => #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115] 'Les Ã©lÃ¨ves FranÃ§ais' utf8Encoded. => #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115] ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !! Or shorter, $Ã© is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8. So more than half the points you make, or the facts that you state, are thus plain wrong. The only thing that is correct is that the code points are equal, but that is not the same as the encoding ! >From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String. Sorry. PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world. > On 07 Dec 2015, at 04:21, EuanM <euanmee(a)gmail.com> wrote: > > This a long email. A *lot* of it is encapsulated in the Venn diagram both: > http://smalltalk.uk.to/unicode-utf8.html > and my Smalltalk in Small Steps blog at: > http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… > > My current thinking, and understanding. > ============================== > > 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. > b) UTF-8 can encode all of those characters in 1 byte, but can > prefer some of them to be encoded as sequences of multiple bytes. And > can encode additional characters as sequences of multiple bytes. > > 1) Smalltalk has long had multiple String classes. > > 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex > is encoded as a UTF-8 codepoint of nn hex. > > 3) All valid ISO-8859-1 characters have a character code between 20 > hex and 7E hex, or between A0 hex and FF hex. > https://en.wikipedia.org/wiki/ISO/IEC_8859-1 > > 4) All valid ASCII characters have a character code between 00 hex and 7E hex. > https://en.wikipedia.org/wiki/ASCII > > > 5) a) All character codes which are defined within ISO-8859-1 and also > defined within ASCII. (i.e. character codes 20 hex to 7E hex) are > defined identically in both. > > b) All printable ASCII characters are defined identically in both > ASCII and ISO-8859-1 > > 6) All character codes defined in ASCII (00 hex to 7E hex) are > defined identically in Unicode UTF-8. > > 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex > - FF hex ) are defined identically in UTF-8. > > 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. > all ASCII maps 1:1 to Unicode UTF-8 > all ISO-8859-1 maps 1:1 to Unicode UTF-8 > > 9) All ByteStrings elements which are either a valid ISO-8859-1 > character or a valid ASCII character are *also* a valid UTF-8 > character. > > 10) ISO-8859-1 characters representing a character with a diacritic, > or a two-character ligature, have no ASCII equivalent. In Unicode > UTF-8, those character codes which are representing compound glyphs, > are called "compatibility codepoints". > > 11) The preferred Unicode representation of the characters which have > compatibility codepoints is as a a short set of codepoints > representing the characters which are combined together to form the > glyph of the convenience codepoint, as a sequence of bytes > representing the component characters. > > > 12) Some concrete examples: > > A - aka Upper Case A > In ASCII, in ISO 8859-1 > ASCII A - 41 hex > ISO-8859-1 A - 41 hex > UTF-8 A - 41 hex > > BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) > In ASCII, not in ISO 8859-1 > ASCII : BEL - 07 hex > ISO-8859-1 : 07 hex is not a valid character code > UTF-8 : BEL - 07 hex > > Â£ (GBP currency symbol) > In ISO-8859-1, not in ASCII > ASCII : A3 hex is not a valid ASCII code > UTF-8: Â£ - A3 hex > ISO-8859-1: Â£ - A3 hex > > Upper Case C cedilla > In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint > *and* a composed set of codepoints > ASCII : C7 hex is not a valid ASCII character code > ISO-8859-1 : Upper Case C cedilla - C7 hex > UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex > Unicode preferred Upper Case C cedilla (composed set of codepoints) > Upper case C 0043 hex (Upper case C) > followed by > cedilla 00B8 hex (cedilla) > > 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, > aByteString is completely adequate for editing and display. > > 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 > string, upper and lower case versions of the same character will be > treated differently. > > 15) When sorting any valid ISO-8859-1 string containing > letter+diacritic combination glyphs or ligature combination glyphs, > the glyphs in combination will treated differently to a "plain" glyph > of the character > i.e. "C" and "C cedilla" will be treated very differently. "Ã" and > "fs" will be treated very differently. > > 16) Different nations have different rules about where diacritic-ed > characted and ligature pairs should be placed when in alphabetical > order. > > 17) Some nations even have multiple standards - e.g. surnames > beginning either "M superscript-c" or "M superscript-a superscript-c" > are treated as beginning equivalently in UK phone directories, but not > in other situations. > > > Some practical upshots > ================== > > 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, > for any single character it considers valid, or any ByteString it has > made up of characters it considers valid. > > 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any > other Smalltalk with a single byte ByteString following ASCII or > ISO-8859-1. > > 3) Any Smalltalk (or derivative language) using ByteString can > immediately consider it's ByteString as valid UTF-8, as long as it > also considers the ByteSring as valid ASCII and/or ISO-8859-1. > > 4) All of those can be successfully exported to any system using UTF-8 > (e.g. HTML). > > 5) To successfully *accept* all UTF-8 we much be able to do either: > a) accept UTF-8 strings with composed characters > b) convert UTF-8 strings with composed characters into UTF-8 strings > that use *only* compatibility codepoints. > > > Class + protocol proposals > > > > a Utf8CompatibilityString class. > > asByteString - ensure only compatibility codepoints are used. > Ensure it doews not encode characters above 00FF hex. > > asIso8859String - ensures only compatibility codepoints are used, > and that the characters are each valid ISO 8859-1 > > asAsciiString - ensures only characters 00hex - 7F hex are used. > > asUtf8ComposedIso8859String - ensures all compatibility codepoints > are expanded into small OrderedCollections of codepoints > > a Utf8ComposedIso8859String class - will provide sortable and > comparable UTF8 strings of all ASCII and ISO 8859-1 strings. > > Then a Utf8SortableCollection class - a collection of > Utf8ComposedIso8859Strings words and phrases. > > Custom sortBlocks will define the applicable sort order. > > We can create a collection... a Dictionary, thinking about it, of > named, prefabricated sortBlocks. > > This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. > > If anyone has better names for the classes, please let me know. > > If anyone else wants to help > - build these, > - create SUnit tests for these > - write documentation for these > Please let me know. > > n.b. I have had absolutely no experience of Ropes. > > My own background with this stuff: In the early 90's as a Project > Manager implementing office automation systems across a global > company, with offices in the Americas, Western, Eastern and Central > Europe, (including Slavic and Cyrillic users) nations, Japan and > China. The mission-critical application was word-processing. > > Our offices were spread around the globe, and we needed those offices > to successfully exchange documents with their sister offices, and with > the customers in each region the offices were in. > > Unicode was then new, and our platform supplier was the NeXT > Corporation, who had been founder members in of the Unicode Consortium > in 1990. > > So far: I've read the latest version of the Unicode Standard (v8.0). > This is freely downloadable. > I've purchased a paper copy of an earlier release. New releases > typically consist additional codespaces (i.e. alphabets). So old > copies are useful, as well as cheap. (Paper copies of version 4.0 > are available second-hand for < $10 / â¬10). > > The typical change with each release is the addition of further > codespaces (i.e alphabets (more or less) ), so you don't lose a lot. > (I'll be going through my V4.0 just to make sure) > > Cheers, > Euan > > > > > On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: >> Hi EuanM >> >> Le 4/12/15 12:42, EuanM a Ã©crit : >>> >>> I'm currently groping my way to seeing how feature-complete our >>> Unicode support is. I am doing this to establish what still needs to >>> be done to provide full Unicode support. >> >> >> this is great. Thanks for pushing this. I wrote and collected some roadmap >> (analyses on different topics) >> on the pharo github project feel free to add this one there. >>> >>> >>> This seems to me to be an area where it would be best to write it >>> once, and then have the same codebase incorporated into the Smalltalks >>> that most share a common ancestry. >>> >>> I am keen to get: equality-testing for strings; sortability for >>> strings which have ligatures and diacritic characters; and correct >>> round-tripping of data. >> >> Go! >> My suggestion is >> start small >> make steady progress >> write tests >> commit often :) >> >> Stef >> >> What is the french phoneBook ordering because this is the first time I hear >> about it. >> >>> >>> Call to action: >>> ========== >>> >>> If you have comments on these proposals - such as "but we already have >>> that facility" or "the reason we do not have these facilities is >>> because they are dog-slow" - please let me know them. >>> >>> If you would like to help out, please let me know. >>> >>> If you have Unicode experience and expertise, and would like to be, or >>> would be willing to be, in the 'council of experts' for this project, >>> please let me know. >>> >>> If you have comments or ideas on anything mentioned in this email >>> >>> In the first instance, the initiative's website will be: >>> http://smalltalk.uk.to/unicode.html >>> >>> I have created a SqueakSource.com project called UnicodeSupport >>> >>> I want to avoid re-inventing any facilities which already exist. >>> Except where they prevent us reaching the goals of: >>> - sortable UTF8 strings >>> - sortable UTF16 strings >>> - equivalence testing of 2 UTF8 strings >>> - equivalence testing of 2 UTF16 strings >>> - round-tripping UTF8 strings through Smalltalk >>> - roundtripping UTF16 strings through Smalltalk. >>> As I understand it, we have limited Unicode support atm. >>> >>> Current state of play >>> =============== >>> ByteString gets converted to WideString when need is automagically >>> detected. >>> >>> Is there anything else that currently exists? >>> >>> Definition of Terms >>> ============== >>> A quick definition of terms before I go any further: >>> >>> Standard terms from the Unicode standard >>> =============================== >>> a compatibility character : an additional encoding of a *normal* >>> character, for compatibility and round-trip conversion purposes. For >>> instance, a 1-byte encoding of a Latin character with a diacritic. >>> >>> Made-up terms >>> ============ >>> a convenience codepoint : a single codepoint which represents an item >>> that is also encoded as a string of codepoints. >>> >>> (I tend to use the terms compatibility character and compatibility >>> codepoint interchangably. The standard only refers to them as >>> compatibility characters. However, the standard is determined to >>> emphasise that characters are abstract and that codepoints are >>> concrete. So I think it is often more useful and productive to think >>> of compatibility or convenience codepoints). >>> >>> a composed character : a character made up of several codepoints >>> >>> Unicode encoding explained >>> ===================== >>> A convenience codepoint can therefore be thought of as a code point >>> used for a character which also has a composed form. >>> >>> The way Unicode works is that sometimes you can encode a character in >>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>> sometimes not. >>> >>> You can therefore have a long stream of ASCII which is single-byte >>> Unicode. If there is an occasional Cyrillic or Greek character in the >>> stream, it would be represented either by a compatibility character or >>> by a multi-byte combination. >>> >>> Using compatibility characters can prevent proper sorting and >>> equivalence testing. >>> >>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>> and round-tripping probelms. Although avoiding them can *also* cause >>> compatibility issues and round-tripping problems. >>> >>> Currently my thinking is: >>> >>> a Utf8String class >>> an Ordered collection, with 1 byte characters as the modal element, >>> but short arrays of wider strings where necessary >>> a Utf16String class >>> an Ordered collection, with 2 byte characters as the modal element, >>> but short arrays of wider strings >>> beginning with a 2-byte endianness indicator. >>> >>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>> compatible. >>> >>> So my thinking is that Utf8String will contain convenience codepoints, >>> for round-tripping. And where there are multiple convenience >>> codepoints for a character, that it standardises on one. >>> >>> And that there is a Utf8SortableString which uses *only* normal >>> characters. >>> >>> We then need methods to convert between the two. >>> >>> aUtf8String asUtf8SortableString >>> >>> and >>> >>> aUtf8SortableString asUtf8String >>> >>> >>> Sort orders are culture and context dependent - Sweden and Germany >>> have different sort orders for the same diacritic-ed characters. Some >>> countries have one order in general usage, and another for specific >>> usages, such as phone directories (e.g. UK and France) >>> >>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>> conversion methods >>> >>> A list of sorted words would be a SortedCollection, and there could be >>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>> seOrder, ukOrder, etc >>> >>> along the lines of >>> aListOfWords := SortedCollection sortBlock: deOrder >>> >>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>> then we can perform equivalence testing on them trivially. >>> >>> To make sure a Utf8String is well formed, we would need to have a way >>> of cleaning up any convenience codepoints which were valid, but which >>> were for a character which has multiple equally-valid alternative >>> convenience codepoints, and for which the string currently had the >>> "wrong" convenience codepoint. (i.e for any character with valid >>> alternative convenience codepoints, we would choose one to be in the >>> well-formed Utf8String, and we would need a method for cleaning the >>> alternative convenience codepoints out of the string, and replacing >>> them with the chosen approved convenience codepoint. >>> >>> aUtf8String cleanUtf8String >>> >>> With WideString, a lot of the issues disappear - except >>> round-tripping(although I'm sure I have seen something recently about >>> 4-byte strings that also have an additional bit. Which would make >>> some Unicode characters 5-bytes long.) >>> >>> >>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>> subtle, or somewhere in between, please let me know) >>> >>> Cheers, >>> Euan >>> >>> >> >> > On 07 Dec 2015, at 04:21, EuanM <euanmee(a)gmail.com> wrote: > > This a long email. A *lot* of it is encapsulated in the Venn diagram both: > http://smalltalk.uk.to/unicode-utf8.html > and my Smalltalk in Small Steps blog at: > http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… > > My current thinking, and understanding. > ============================== > > 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. > b) UTF-8 can encode all of those characters in 1 byte, but can > prefer some of them to be encoded as sequences of multiple bytes. And > can encode additional characters as sequences of multiple bytes. > > 1) Smalltalk has long had multiple String classes. > > 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex > is encoded as a UTF-8 codepoint of nn hex. > > 3) All valid ISO-8859-1 characters have a character code between 20 > hex and 7E hex, or between A0 hex and FF hex. > https://en.wikipedia.org/wiki/ISO/IEC_8859-1 > > 4) All valid ASCII characters have a character code between 00 hex and 7E hex. > https://en.wikipedia.org/wiki/ASCII > > > 5) a) All character codes which are defined within ISO-8859-1 and also > defined within ASCII. (i.e. character codes 20 hex to 7E hex) are > defined identically in both. > > b) All printable ASCII characters are defined identically in both > ASCII and ISO-8859-1 > > 6) All character codes defined in ASCII (00 hex to 7E hex) are > defined identically in Unicode UTF-8. > > 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex > - FF hex ) are defined identically in UTF-8. > > 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. > all ASCII maps 1:1 to Unicode UTF-8 > all ISO-8859-1 maps 1:1 to Unicode UTF-8 > > 9) All ByteStrings elements which are either a valid ISO-8859-1 > character or a valid ASCII character are *also* a valid UTF-8 > character. > > 10) ISO-8859-1 characters representing a character with a diacritic, > or a two-character ligature, have no ASCII equivalent. In Unicode > UTF-8, those character codes which are representing compound glyphs, > are called "compatibility codepoints". > > 11) The preferred Unicode representation of the characters which have > compatibility codepoints is as a a short set of codepoints > representing the characters which are combined together to form the > glyph of the convenience codepoint, as a sequence of bytes > representing the component characters. > > > 12) Some concrete examples: > > A - aka Upper Case A > In ASCII, in ISO 8859-1 > ASCII A - 41 hex > ISO-8859-1 A - 41 hex > UTF-8 A - 41 hex > > BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) > In ASCII, not in ISO 8859-1 > ASCII : BEL - 07 hex > ISO-8859-1 : 07 hex is not a valid character code > UTF-8 : BEL - 07 hex > > Â£ (GBP currency symbol) > In ISO-8859-1, not in ASCII > ASCII : A3 hex is not a valid ASCII code > UTF-8: Â£ - A3 hex > ISO-8859-1: Â£ - A3 hex > > Upper Case C cedilla > In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint > *and* a composed set of codepoints > ASCII : C7 hex is not a valid ASCII character code > ISO-8859-1 : Upper Case C cedilla - C7 hex > UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex > Unicode preferred Upper Case C cedilla (composed set of codepoints) > Upper case C 0043 hex (Upper case C) > followed by > cedilla 00B8 hex (cedilla) > > 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, > aByteString is completely adequate for editing and display. > > 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 > string, upper and lower case versions of the same character will be > treated differently. > > 15) When sorting any valid ISO-8859-1 string containing > letter+diacritic combination glyphs or ligature combination glyphs, > the glyphs in combination will treated differently to a "plain" glyph > of the character > i.e. "C" and "C cedilla" will be treated very differently. "Ã" and > "fs" will be treated very differently. > > 16) Different nations have different rules about where diacritic-ed > characted and ligature pairs should be placed when in alphabetical > order. > > 17) Some nations even have multiple standards - e.g. surnames > beginning either "M superscript-c" or "M superscript-a superscript-c" > are treated as beginning equivalently in UK phone directories, but not > in other situations. > > > Some practical upshots > ================== > > 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, > for any single character it considers valid, or any ByteString it has > made up of characters it considers valid. > > 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any > other Smalltalk with a single byte ByteString following ASCII or > ISO-8859-1. > > 3) Any Smalltalk (or derivative language) using ByteString can > immediately consider it's ByteString as valid UTF-8, as long as it > also considers the ByteSring as valid ASCII and/or ISO-8859-1. > > 4) All of those can be successfully exported to any system using UTF-8 > (e.g. HTML). > > 5) To successfully *accept* all UTF-8 we much be able to do either: > a) accept UTF-8 strings with composed characters > b) convert UTF-8 strings with composed characters into UTF-8 strings > that use *only* compatibility codepoints. > > > Class + protocol proposals > > > > a Utf8CompatibilityString class. > > asByteString - ensure only compatibility codepoints are used. > Ensure it doews not encode characters above 00FF hex. > > asIso8859String - ensures only compatibility codepoints are used, > and that the characters are each valid ISO 8859-1 > > asAsciiString - ensures only characters 00hex - 7F hex are used. > > asUtf8ComposedIso8859String - ensures all compatibility codepoints > are expanded into small OrderedCollections of codepoints > > a Utf8ComposedIso8859String class - will provide sortable and > comparable UTF8 strings of all ASCII and ISO 8859-1 strings. > > Then a Utf8SortableCollection class - a collection of > Utf8ComposedIso8859Strings words and phrases. > > Custom sortBlocks will define the applicable sort order. > > We can create a collection... a Dictionary, thinking about it, of > named, prefabricated sortBlocks. > > This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. > > If anyone has better names for the classes, please let me know. > > If anyone else wants to help > - build these, > - create SUnit tests for these > - write documentation for these > Please let me know. > > n.b. I have had absolutely no experience of Ropes. > > My own background with this stuff: In the early 90's as a Project > Manager implementing office automation systems across a global > company, with offices in the Americas, Western, Eastern and Central > Europe, (including Slavic and Cyrillic users) nations, Japan and > China. The mission-critical application was word-processing. > > Our offices were spread around the globe, and we needed those offices > to successfully exchange documents with their sister offices, and with > the customers in each region the offices were in. > > Unicode was then new, and our platform supplier was the NeXT > Corporation, who had been founder members in of the Unicode Consortium > in 1990. > > So far: I've read the latest version of the Unicode Standard (v8.0). > This is freely downloadable. > I've purchased a paper copy of an earlier release. New releases > typically consist additional codespaces (i.e. alphabets). So old > copies are useful, as well as cheap. (Paper copies of version 4.0 > are available second-hand for < $10 / â¬10). > > The typical change with each release is the addition of further > codespaces (i.e alphabets (more or less) ), so you don't lose a lot. > (I'll be going through my V4.0 just to make sure) > > Cheers, > Euan > > > > > On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: >> Hi EuanM >> >> Le 4/12/15 12:42, EuanM a Ã©crit : >>> >>> I'm currently groping my way to seeing how feature-complete our >>> Unicode support is. I am doing this to establish what still needs to >>> be done to provide full Unicode support. >> >> >> this is great. Thanks for pushing this. I wrote and collected some roadmap >> (analyses on different topics) >> on the pharo github project feel free to add this one there. >>> >>> >>> This seems to me to be an area where it would be best to write it >>> once, and then have the same codebase incorporated into the Smalltalks >>> that most share a common ancestry. >>> >>> I am keen to get: equality-testing for strings; sortability for >>> strings which have ligatures and diacritic characters; and correct >>> round-tripping of data. >> >> Go! >> My suggestion is >> start small >> make steady progress >> write tests >> commit often :) >> >> Stef >> >> What is the french phoneBook ordering because this is the first time I hear >> about it. >> >>> >>> Call to action: >>> ========== >>> >>> If you have comments on these proposals - such as "but we already have >>> that facility" or "the reason we do not have these facilities is >>> because they are dog-slow" - please let me know them. >>> >>> If you would like to help out, please let me know. >>> >>> If you have Unicode experience and expertise, and would like to be, or >>> would be willing to be, in the 'council of experts' for this project, >>> please let me know. >>> >>> If you have comments or ideas on anything mentioned in this email >>> >>> In the first instance, the initiative's website will be: >>> http://smalltalk.uk.to/unicode.html >>> >>> I have created a SqueakSource.com project called UnicodeSupport >>> >>> I want to avoid re-inventing any facilities which already exist. >>> Except where they prevent us reaching the goals of: >>> - sortable UTF8 strings >>> - sortable UTF16 strings >>> - equivalence testing of 2 UTF8 strings >>> - equivalence testing of 2 UTF16 strings >>> - round-tripping UTF8 strings through Smalltalk >>> - roundtripping UTF16 strings through Smalltalk. >>> As I understand it, we have limited Unicode support atm. >>> >>> Current state of play >>> =============== >>> ByteString gets converted to WideString when need is automagically >>> detected. >>> >>> Is there anything else that currently exists? >>> >>> Definition of Terms >>> ============== >>> A quick definition of terms before I go any further: >>> >>> Standard terms from the Unicode standard >>> =============================== >>> a compatibility character : an additional encoding of a *normal* >>> character, for compatibility and round-trip conversion purposes. For >>> instance, a 1-byte encoding of a Latin character with a diacritic. >>> >>> Made-up terms >>> ============ >>> a convenience codepoint : a single codepoint which represents an item >>> that is also encoded as a string of codepoints. >>> >>> (I tend to use the terms compatibility character and compatibility >>> codepoint interchangably. The standard only refers to them as >>> compatibility characters. However, the standard is determined to >>> emphasise that characters are abstract and that codepoints are >>> concrete. So I think it is often more useful and productive to think >>> of compatibility or convenience codepoints). >>> >>> a composed character : a character made up of several codepoints >>> >>> Unicode encoding explained >>> ===================== >>> A convenience codepoint can therefore be thought of as a code point >>> used for a character which also has a composed form. >>> >>> The way Unicode works is that sometimes you can encode a character in >>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>> sometimes not. >>> >>> You can therefore have a long stream of ASCII which is single-byte >>> Unicode. If there is an occasional Cyrillic or Greek character in the >>> stream, it would be represented either by a compatibility character or >>> by a multi-byte combination. >>> >>> Using compatibility characters can prevent proper sorting and >>> equivalence testing. >>> >>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>> and round-tripping probelms. Although avoiding them can *also* cause >>> compatibility issues and round-tripping problems. >>> >>> Currently my thinking is: >>> >>> a Utf8String class >>> an Ordered collection, with 1 byte characters as the modal element, >>> but short arrays of wider strings where necessary >>> a Utf16String class >>> an Ordered collection, with 2 byte characters as the modal element, >>> but short arrays of wider strings >>> beginning with a 2-byte endianness indicator. >>> >>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>> compatible. >>> >>> So my thinking is that Utf8String will contain convenience codepoints, >>> for round-tripping. And where there are multiple convenience >>> codepoints for a character, that it standardises on one. >>> >>> And that there is a Utf8SortableString which uses *only* normal >>> characters. >>> >>> We then need methods to convert between the two. >>> >>> aUtf8String asUtf8SortableString >>> >>> and >>> >>> aUtf8SortableString asUtf8String >>> >>> >>> Sort orders are culture and context dependent - Sweden and Germany >>> have different sort orders for the same diacritic-ed characters. Some >>> countries have one order in general usage, and another for specific >>> usages, such as phone directories (e.g. UK and France) >>> >>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>> conversion methods >>> >>> A list of sorted words would be a SortedCollection, and there could be >>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>> seOrder, ukOrder, etc >>> >>> along the lines of >>> aListOfWords := SortedCollection sortBlock: deOrder >>> >>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>> then we can perform equivalence testing on them trivially. >>> >>> To make sure a Utf8String is well formed, we would need to have a way >>> of cleaning up any convenience codepoints which were valid, but which >>> were for a character which has multiple equally-valid alternative >>> convenience codepoints, and for which the string currently had the >>> "wrong" convenience codepoint. (i.e for any character with valid >>> alternative convenience codepoints, we would choose one to be in the >>> well-formed Utf8String, and we would need a method for cleaning the >>> alternative convenience codepoints out of the string, and replacing >>> them with the chosen approved convenience codepoint. >>> >>> aUtf8String cleanUtf8String >>> >>> With WideString, a lot of the issues disappear - except >>> round-tripping(although I'm sure I have seen something recently about >>> 4-byte strings that also have an additional bit. Which would make >>> some Unicode characters 5-bytes long.) >>> >>> >>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>> subtle, or somewhere in between, please let me know) >>> >>> Cheers, >>> Euan >>> >>> >> >> >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

Todd, As long as others are using it, it's useful to be able to send UTF16, and to successfully import it. I like systems that play well with others. :-) On 5 December 2015 at 16:35, Todd Blanchard <tblanchard(a)mac.com> wrote: > would suggest that the only worthwhile encoding is UTF8 - the rest are > distractions except for being able to read and convert from other encodings > to UTF8. UTF16 is a complete waste of time. > > Read http://utf8everywhere.org/ > > I have extensive Unicode chops from around 1999 to 2004 and my experience > leads me to strongly agree with the views on that site. > > > Sent from the road > > On Dec 5, 2015, at 05:08, stepharo <stepharo(a)free.fr> wrote: > > Hi EuanM > > Le 4/12/15 12:42, EuanM a Ã©crit : > > I'm currently groping my way to seeing how feature-complete our > > Unicode support is. I am doing this to establish what still needs to > > be done to provide full Unicode support. > > > this is great. Thanks for pushing this. I wrote and collected some roadmap > (analyses on different topics) > on the pharo github project feel free to add this one there. > > > This seems to me to be an area where it would be best to write it > > once, and then have the same codebase incorporated into the Smalltalks > > that most share a common ancestry. > > > I am keen to get: equality-testing for strings; sortability for > > strings which have ligatures and diacritic characters; and correct > > round-tripping of data. > > Go! > My suggestion is > start small > make steady progress > write tests > commit often :) > > Stef > > What is the french phoneBook ordering because this is the first time I hear > about it. > > > Call to action: > > ========== > > > If you have comments on these proposals - such as "but we already have > > that facility" or "the reason we do not have these facilities is > > because they are dog-slow" - please let me know them. > > > If you would like to help out, please let me know. > > > If you have Unicode experience and expertise, and would like to be, or > > would be willing to be, in the 'council of experts' for this project, > > please let me know. > > > If you have comments or ideas on anything mentioned in this email > > > In the first instance, the initiative's website will be: > > http://smalltalk.uk.to/unicode.html > > > I have created a SqueakSource.com project called UnicodeSupport > > > I want to avoid re-inventing any facilities which already exist. > > Except where they prevent us reaching the goals of: > > - sortable UTF8 strings > > - sortable UTF16 strings > > - equivalence testing of 2 UTF8 strings > > - equivalence testing of 2 UTF16 strings > > - round-tripping UTF8 strings through Smalltalk > > - roundtripping UTF16 strings through Smalltalk. > > As I understand it, we have limited Unicode support atm. > > > Current state of play > > =============== > > ByteString gets converted to WideString when need is automagically detected. > > > Is there anything else that currently exists? > > > Definition of Terms > > ============== > > A quick definition of terms before I go any further: > > > Standard terms from the Unicode standard > > =============================== > > a compatibility character : an additional encoding of a *normal* > > character, for compatibility and round-trip conversion purposes. For > > instance, a 1-byte encoding of a Latin character with a diacritic. > > > Made-up terms > > ============ > > a convenience codepoint : a single codepoint which represents an item > > that is also encoded as a string of codepoints. > > > (I tend to use the terms compatibility character and compatibility > > codepoint interchangably. The standard only refers to them as > > compatibility characters. However, the standard is determined to > > emphasise that characters are abstract and that codepoints are > > concrete. So I think it is often more useful and productive to think > > of compatibility or convenience codepoints). > > > a composed character : a character made up of several codepoints > > > Unicode encoding explained > > ===================== > > A convenience codepoint can therefore be thought of as a code point > > used for a character which also has a composed form. > > > The way Unicode works is that sometimes you can encode a character in > > one byte, sometimes not. Sometimes you can encode it in two bytes, > > sometimes not. > > > You can therefore have a long stream of ASCII which is single-byte > > Unicode. If there is an occasional Cyrillic or Greek character in the > > stream, it would be represented either by a compatibility character or > > by a multi-byte combination. > > > Using compatibility characters can prevent proper sorting and > > equivalence testing. > > > Using "pure" Unicode, ie. "normal encodings", can cause compatibility > > and round-tripping probelms. Although avoiding them can *also* cause > > compatibility issues and round-tripping problems. > > > Currently my thinking is: > > > a Utf8String class > > an Ordered collection, with 1 byte characters as the modal element, > > but short arrays of wider strings where necessary > > a Utf16String class > > an Ordered collection, with 2 byte characters as the modal element, > > but short arrays of wider strings > > beginning with a 2-byte endianness indicator. > > > Utf8Strings sometimes need to be sortable, and sometimes need to be > compatible. > > > So my thinking is that Utf8String will contain convenience codepoints, > > for round-tripping. And where there are multiple convenience > > codepoints for a character, that it standardises on one. > > > And that there is a Utf8SortableString which uses *only* normal characters. > > > We then need methods to convert between the two. > > > aUtf8String asUtf8SortableString > > > and > > > aUtf8SortableString asUtf8String > > > > Sort orders are culture and context dependent - Sweden and Germany > > have different sort orders for the same diacritic-ed characters. Some > > countries have one order in general usage, and another for specific > > usages, such as phone directories (e.g. UK and France) > > > Similarly for Utf16 : Utf16String and Utf16SortableString and > > conversion methods > > > A list of sorted words would be a SortedCollection, and there could be > > pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, > > seOrder, ukOrder, etc > > > along the lines of > > aListOfWords := SortedCollection sortBlock: deOrder > > > If a word is either a Utf8SortableString, or a well-formed Utf8String, > > then we can perform equivalence testing on them trivially. > > > To make sure a Utf8String is well formed, we would need to have a way > > of cleaning up any convenience codepoints which were valid, but which > > were for a character which has multiple equally-valid alternative > > convenience codepoints, and for which the string currently had the > > "wrong" convenience codepoint. (i.e for any character with valid > > alternative convenience codepoints, we would choose one to be in the > > well-formed Utf8String, and we would need a method for cleaning the > > alternative convenience codepoints out of the string, and replacing > > them with the chosen approved convenience codepoint. > > > aUtf8String cleanUtf8String > > > With WideString, a lot of the issues disappear - except > > round-tripping(although I'm sure I have seen something recently about > > 4-byte strings that also have an additional bit. Which would make > > some Unicode characters 5-bytes long.) > > > > (I'm starting to zone out now - if I've overlooked anything - obvious, > > subtle, or somewhere in between, please let me know) > > > Cheers, > > Euan > > > > >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

Steph - I'll dig out the Fr phone book ordering from wherever it was I read about it! I thought I ghad it to hand, but I haven;t found it tonight. It can't be far away. On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: > Hi EuanM > > Le 4/12/15 12:42, EuanM a Ã©crit : >> >> I'm currently groping my way to seeing how feature-complete our >> Unicode support is. I am doing this to establish what still needs to >> be done to provide full Unicode support. > > > this is great. Thanks for pushing this. I wrote and collected some roadmap > (analyses on different topics) > on the pharo github project feel free to add this one there. >> >> >> This seems to me to be an area where it would be best to write it >> once, and then have the same codebase incorporated into the Smalltalks >> that most share a common ancestry. >> >> I am keen to get: equality-testing for strings; sortability for >> strings which have ligatures and diacritic characters; and correct >> round-tripping of data. > > Go! > My suggestion is > start small > make steady progress > write tests > commit often :) > > Stef > > What is the french phoneBook ordering because this is the first time I hear > about it. > >> >> Call to action: >> ========== >> >> If you have comments on these proposals - such as "but we already have >> that facility" or "the reason we do not have these facilities is >> because they are dog-slow" - please let me know them. >> >> If you would like to help out, please let me know. >> >> If you have Unicode experience and expertise, and would like to be, or >> would be willing to be, in the 'council of experts' for this project, >> please let me know. >> >> If you have comments or ideas on anything mentioned in this email >> >> In the first instance, the initiative's website will be: >> http://smalltalk.uk.to/unicode.html >> >> I have created a SqueakSource.com project called UnicodeSupport >> >> I want to avoid re-inventing any facilities which already exist. >> Except where they prevent us reaching the goals of: >> - sortable UTF8 strings >> - sortable UTF16 strings >> - equivalence testing of 2 UTF8 strings >> - equivalence testing of 2 UTF16 strings >> - round-tripping UTF8 strings through Smalltalk >> - roundtripping UTF16 strings through Smalltalk. >> As I understand it, we have limited Unicode support atm. >> >> Current state of play >> =============== >> ByteString gets converted to WideString when need is automagically >> detected. >> >> Is there anything else that currently exists? >> >> Definition of Terms >> ============== >> A quick definition of terms before I go any further: >> >> Standard terms from the Unicode standard >> =============================== >> a compatibility character : an additional encoding of a *normal* >> character, for compatibility and round-trip conversion purposes. For >> instance, a 1-byte encoding of a Latin character with a diacritic. >> >> Made-up terms >> ============ >> a convenience codepoint : a single codepoint which represents an item >> that is also encoded as a string of codepoints. >> >> (I tend to use the terms compatibility character and compatibility >> codepoint interchangably. The standard only refers to them as >> compatibility characters. However, the standard is determined to >> emphasise that characters are abstract and that codepoints are >> concrete. So I think it is often more useful and productive to think >> of compatibility or convenience codepoints). >> >> a composed character : a character made up of several codepoints >> >> Unicode encoding explained >> ===================== >> A convenience codepoint can therefore be thought of as a code point >> used for a character which also has a composed form. >> >> The way Unicode works is that sometimes you can encode a character in >> one byte, sometimes not. Sometimes you can encode it in two bytes, >> sometimes not. >> >> You can therefore have a long stream of ASCII which is single-byte >> Unicode. If there is an occasional Cyrillic or Greek character in the >> stream, it would be represented either by a compatibility character or >> by a multi-byte combination. >> >> Using compatibility characters can prevent proper sorting and >> equivalence testing. >> >> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >> and round-tripping probelms. Although avoiding them can *also* cause >> compatibility issues and round-tripping problems. >> >> Currently my thinking is: >> >> a Utf8String class >> an Ordered collection, with 1 byte characters as the modal element, >> but short arrays of wider strings where necessary >> a Utf16String class >> an Ordered collection, with 2 byte characters as the modal element, >> but short arrays of wider strings >> beginning with a 2-byte endianness indicator. >> >> Utf8Strings sometimes need to be sortable, and sometimes need to be >> compatible. >> >> So my thinking is that Utf8String will contain convenience codepoints, >> for round-tripping. And where there are multiple convenience >> codepoints for a character, that it standardises on one. >> >> And that there is a Utf8SortableString which uses *only* normal >> characters. >> >> We then need methods to convert between the two. >> >> aUtf8String asUtf8SortableString >> >> and >> >> aUtf8SortableString asUtf8String >> >> >> Sort orders are culture and context dependent - Sweden and Germany >> have different sort orders for the same diacritic-ed characters. Some >> countries have one order in general usage, and another for specific >> usages, such as phone directories (e.g. UK and France) >> >> Similarly for Utf16 : Utf16String and Utf16SortableString and >> conversion methods >> >> A list of sorted words would be a SortedCollection, and there could be >> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >> seOrder, ukOrder, etc >> >> along the lines of >> aListOfWords := SortedCollection sortBlock: deOrder >> >> If a word is either a Utf8SortableString, or a well-formed Utf8String, >> then we can perform equivalence testing on them trivially. >> >> To make sure a Utf8String is well formed, we would need to have a way >> of cleaning up any convenience codepoints which were valid, but which >> were for a character which has multiple equally-valid alternative >> convenience codepoints, and for which the string currently had the >> "wrong" convenience codepoint. (i.e for any character with valid >> alternative convenience codepoints, we would choose one to be in the >> well-formed Utf8String, and we would need a method for cleaning the >> alternative convenience codepoints out of the string, and replacing >> them with the chosen approved convenience codepoint. >> >> aUtf8String cleanUtf8String >> >> With WideString, a lot of the issues disappear - except >> round-tripping(although I'm sure I have seen something recently about >> 4-byte strings that also have an additional bit. Which would make >> some Unicode characters 5-bytes long.) >> >> >> (I'm starting to zone out now - if I've overlooked anything - obvious, >> subtle, or somewhere in between, please let me know) >> >> Cheers, >> Euan >> >> > >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

"Canonicalisation and sorting issues are hardly discussed. In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $Ã© if you know that it is possible to represent $Ã© as just $Ã© and as $e + $Â´ ?" This, for me, is one of the chief purposes of Unicode support. What you have it a convertor for "might contain compatibility codepoints" to "contains only composed sequences of codepoints, and no compatibility codepoints". As long as you're not using Strings where you should use Streams, it should be okay. And of course, for passing back to ISO Latin 1 or ASCII systems, you need to have a convertor to "contains only compatibility codepoints, and no composed sets of codepoints". As long as you can tell one type from the other, it's not a problem. Any string that mixes both can be converted in either direction by the same methods which I've just outlined. Once you have these, we can do this for all 1 byte characters. We can then expand this to have Classes and methods for character strings which contain the occasional character from other ISO character sets. Cheers, Euan On 6 December 2015 at 17:44, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >> On 05 Dec 2015, at 17:35, Todd Blanchard <tblanchard(a)mac.com> wrote: >> >> would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time. >> >> Read http://utf8everywhere.org/ >> >> I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site. > > Well, I read the page/document/site as well. It was very interesting indeed, thanks for sharing it. > > In some sense it made me reconsider my aversion against in-image utf-8 encoding, maybe it could have some value. Absolute storage is more efficient, some processing might also be more efficient, i/o conversions to/from utf-8 become a no-op. What I found nice is the suggestion that most structured parsing (XML, JSON, CSV, STON, ...) could actually ignore the encoding for a large part and just assume its ASCII, which would/could be nice for performance. Also the fact that a lot of strings are (or should be) treated as opaque makes a lot of sense. > > What I did not like is that much of argumentation is based on issue in the Windows world, take all that away and the document shrinks in half. I would have liked a bit more fundamental CS arguments. > > Canonicalisation and sorting issues are hardly discussed. > > In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $Ã© if you know that it is possible to represent $Ã© as just $Ã© and as $e + $Â´ ? > > Sven > >> Sent from the road >> >> On Dec 5, 2015, at 05:08, stepharo <stepharo(a)free.fr> wrote: >> >>> Hi EuanM >>> >>> Le 4/12/15 12:42, EuanM a Ã©crit : >>>> I'm currently groping my way to seeing how feature-complete our >>>> Unicode support is. I am doing this to establish what still needs to >>>> be done to provide full Unicode support. >>> >>> this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) >>> on the pharo github project feel free to add this one there. >>>> >>>> This seems to me to be an area where it would be best to write it >>>> once, and then have the same codebase incorporated into the Smalltalks >>>> that most share a common ancestry. >>>> >>>> I am keen to get: equality-testing for strings; sortability for >>>> strings which have ligatures and diacritic characters; and correct >>>> round-tripping of data. >>> Go! >>> My suggestion is >>> start small >>> make steady progress >>> write tests >>> commit often :) >>> >>> Stef >>> >>> What is the french phoneBook ordering because this is the first time I hear about it. >>>> >>>> Call to action: >>>> ========== >>>> >>>> If you have comments on these proposals - such as "but we already have >>>> that facility" or "the reason we do not have these facilities is >>>> because they are dog-slow" - please let me know them. >>>> >>>> If you would like to help out, please let me know. >>>> >>>> If you have Unicode experience and expertise, and would like to be, or >>>> would be willing to be, in the 'council of experts' for this project, >>>> please let me know. >>>> >>>> If you have comments or ideas on anything mentioned in this email >>>> >>>> In the first instance, the initiative's website will be: >>>> http://smalltalk.uk.to/unicode.html >>>> >>>> I have created a SqueakSource.com project called UnicodeSupport >>>> >>>> I want to avoid re-inventing any facilities which already exist. >>>> Except where they prevent us reaching the goals of: >>>> - sortable UTF8 strings >>>> - sortable UTF16 strings >>>> - equivalence testing of 2 UTF8 strings >>>> - equivalence testing of 2 UTF16 strings >>>> - round-tripping UTF8 strings through Smalltalk >>>> - roundtripping UTF16 strings through Smalltalk. >>>> As I understand it, we have limited Unicode support atm. >>>> >>>> Current state of play >>>> =============== >>>> ByteString gets converted to WideString when need is automagically detected. >>>> >>>> Is there anything else that currently exists? >>>> >>>> Definition of Terms >>>> ============== >>>> A quick definition of terms before I go any further: >>>> >>>> Standard terms from the Unicode standard >>>> =============================== >>>> a compatibility character : an additional encoding of a *normal* >>>> character, for compatibility and round-trip conversion purposes. For >>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>> >>>> Made-up terms >>>> ============ >>>> a convenience codepoint : a single codepoint which represents an item >>>> that is also encoded as a string of codepoints. >>>> >>>> (I tend to use the terms compatibility character and compatibility >>>> codepoint interchangably. The standard only refers to them as >>>> compatibility characters. However, the standard is determined to >>>> emphasise that characters are abstract and that codepoints are >>>> concrete. So I think it is often more useful and productive to think >>>> of compatibility or convenience codepoints). >>>> >>>> a composed character : a character made up of several codepoints >>>> >>>> Unicode encoding explained >>>> ===================== >>>> A convenience codepoint can therefore be thought of as a code point >>>> used for a character which also has a composed form. >>>> >>>> The way Unicode works is that sometimes you can encode a character in >>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>> sometimes not. >>>> >>>> You can therefore have a long stream of ASCII which is single-byte >>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>> stream, it would be represented either by a compatibility character or >>>> by a multi-byte combination. >>>> >>>> Using compatibility characters can prevent proper sorting and >>>> equivalence testing. >>>> >>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>> and round-tripping probelms. Although avoiding them can *also* cause >>>> compatibility issues and round-tripping problems. >>>> >>>> Currently my thinking is: >>>> >>>> a Utf8String class >>>> an Ordered collection, with 1 byte characters as the modal element, >>>> but short arrays of wider strings where necessary >>>> a Utf16String class >>>> an Ordered collection, with 2 byte characters as the modal element, >>>> but short arrays of wider strings >>>> beginning with a 2-byte endianness indicator. >>>> >>>> Utf8Strings sometimes need to be sortable, and sometimes need to be compatible. >>>> >>>> So my thinking is that Utf8String will contain convenience codepoints, >>>> for round-tripping. And where there are multiple convenience >>>> codepoints for a character, that it standardises on one. >>>> >>>> And that there is a Utf8SortableString which uses *only* normal characters. >>>> >>>> We then need methods to convert between the two. >>>> >>>> aUtf8String asUtf8SortableString >>>> >>>> and >>>> >>>> aUtf8SortableString asUtf8String >>>> >>>> >>>> Sort orders are culture and context dependent - Sweden and Germany >>>> have different sort orders for the same diacritic-ed characters. Some >>>> countries have one order in general usage, and another for specific >>>> usages, such as phone directories (e.g. UK and France) >>>> >>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>> conversion methods >>>> >>>> A list of sorted words would be a SortedCollection, and there could be >>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>> seOrder, ukOrder, etc >>>> >>>> along the lines of >>>> aListOfWords := SortedCollection sortBlock: deOrder >>>> >>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>> then we can perform equivalence testing on them trivially. >>>> >>>> To make sure a Utf8String is well formed, we would need to have a way >>>> of cleaning up any convenience codepoints which were valid, but which >>>> were for a character which has multiple equally-valid alternative >>>> convenience codepoints, and for which the string currently had the >>>> "wrong" convenience codepoint. (i.e for any character with valid >>>> alternative convenience codepoints, we would choose one to be in the >>>> well-formed Utf8String, and we would need a method for cleaning the >>>> alternative convenience codepoints out of the string, and replacing >>>> them with the chosen approved convenience codepoint. >>>> >>>> aUtf8String cleanUtf8String >>>> >>>> With WideString, a lot of the issues disappear - except >>>> round-tripping(although I'm sure I have seen something recently about >>>> 4-byte strings that also have an additional bit. Which would make >>>> some Unicode characters 5-bytes long.) >>>> >>>> >>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>> subtle, or somewhere in between, please let me know) >>>> >>>> Cheers, >>>> Euan >>>> >>>> >>> >>> > >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

Thanks for those pointers, Steph. I'll make sure they are on my reading list. (I have a limited weekly time-budget for Unicode work, but I expect this is a long-term project). I'll keep in touch with Steph, so any new facilities can be immediately useful to Pharo, and someone can guide them to a proper home in Pharo's Class hierarchy. For now, I've stuck stuff on my blog, http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… in an email here and at smalltalk.uk.to/unicode-utf.html On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: > Hi EuanM > > Le 4/12/15 12:42, EuanM a Ã©crit : >> >> I'm currently groping my way to seeing how feature-complete our >> Unicode support is. I am doing this to establish what still needs to >> be done to provide full Unicode support. > > > this is great. Thanks for pushing this. I wrote and collected some roadmap > (analyses on different topics) > on the pharo github project feel free to add this one there. >> >> >> This seems to me to be an area where it would be best to write it >> once, and then have the same codebase incorporated into the Smalltalks >> that most share a common ancestry. >> >> I am keen to get: equality-testing for strings; sortability for >> strings which have ligatures and diacritic characters; and correct >> round-tripping of data. > > Go! > My suggestion is > start small > make steady progress > write tests > commit often :) > > Stef > > What is the french phoneBook ordering because this is the first time I hear > about it. > >> >> Call to action: >> ========== >> >> If you have comments on these proposals - such as "but we already have >> that facility" or "the reason we do not have these facilities is >> because they are dog-slow" - please let me know them. >> >> If you would like to help out, please let me know. >> >> If you have Unicode experience and expertise, and would like to be, or >> would be willing to be, in the 'council of experts' for this project, >> please let me know. >> >> If you have comments or ideas on anything mentioned in this email >> >> In the first instance, the initiative's website will be: >> http://smalltalk.uk.to/unicode.html >> >> I have created a SqueakSource.com project called UnicodeSupport >> >> I want to avoid re-inventing any facilities which already exist. >> Except where they prevent us reaching the goals of: >> - sortable UTF8 strings >> - sortable UTF16 strings >> - equivalence testing of 2 UTF8 strings >> - equivalence testing of 2 UTF16 strings >> - round-tripping UTF8 strings through Smalltalk >> - roundtripping UTF16 strings through Smalltalk. >> As I understand it, we have limited Unicode support atm. >> >> Current state of play >> =============== >> ByteString gets converted to WideString when need is automagically >> detected. >> >> Is there anything else that currently exists? >> >> Definition of Terms >> ============== >> A quick definition of terms before I go any further: >> >> Standard terms from the Unicode standard >> =============================== >> a compatibility character : an additional encoding of a *normal* >> character, for compatibility and round-trip conversion purposes. For >> instance, a 1-byte encoding of a Latin character with a diacritic. >> >> Made-up terms >> ============ >> a convenience codepoint : a single codepoint which represents an item >> that is also encoded as a string of codepoints. >> >> (I tend to use the terms compatibility character and compatibility >> codepoint interchangably. The standard only refers to them as >> compatibility characters. However, the standard is determined to >> emphasise that characters are abstract and that codepoints are >> concrete. So I think it is often more useful and productive to think >> of compatibility or convenience codepoints). >> >> a composed character : a character made up of several codepoints >> >> Unicode encoding explained >> ===================== >> A convenience codepoint can therefore be thought of as a code point >> used for a character which also has a composed form. >> >> The way Unicode works is that sometimes you can encode a character in >> one byte, sometimes not. Sometimes you can encode it in two bytes, >> sometimes not. >> >> You can therefore have a long stream of ASCII which is single-byte >> Unicode. If there is an occasional Cyrillic or Greek character in the >> stream, it would be represented either by a compatibility character or >> by a multi-byte combination. >> >> Using compatibility characters can prevent proper sorting and >> equivalence testing. >> >> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >> and round-tripping probelms. Although avoiding them can *also* cause >> compatibility issues and round-tripping problems. >> >> Currently my thinking is: >> >> a Utf8String class >> an Ordered collection, with 1 byte characters as the modal element, >> but short arrays of wider strings where necessary >> a Utf16String class >> an Ordered collection, with 2 byte characters as the modal element, >> but short arrays of wider strings >> beginning with a 2-byte endianness indicator. >> >> Utf8Strings sometimes need to be sortable, and sometimes need to be >> compatible. >> >> So my thinking is that Utf8String will contain convenience codepoints, >> for round-tripping. And where there are multiple convenience >> codepoints for a character, that it standardises on one. >> >> And that there is a Utf8SortableString which uses *only* normal >> characters. >> >> We then need methods to convert between the two. >> >> aUtf8String asUtf8SortableString >> >> and >> >> aUtf8SortableString asUtf8String >> >> >> Sort orders are culture and context dependent - Sweden and Germany >> have different sort orders for the same diacritic-ed characters. Some >> countries have one order in general usage, and another for specific >> usages, such as phone directories (e.g. UK and France) >> >> Similarly for Utf16 : Utf16String and Utf16SortableString and >> conversion methods >> >> A list of sorted words would be a SortedCollection, and there could be >> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >> seOrder, ukOrder, etc >> >> along the lines of >> aListOfWords := SortedCollection sortBlock: deOrder >> >> If a word is either a Utf8SortableString, or a well-formed Utf8String, >> then we can perform equivalence testing on them trivially. >> >> To make sure a Utf8String is well formed, we would need to have a way >> of cleaning up any convenience codepoints which were valid, but which >> were for a character which has multiple equally-valid alternative >> convenience codepoints, and for which the string currently had the >> "wrong" convenience codepoint. (i.e for any character with valid >> alternative convenience codepoints, we would choose one to be in the >> well-formed Utf8String, and we would need a method for cleaning the >> alternative convenience codepoints out of the string, and replacing >> them with the chosen approved convenience codepoint. >> >> aUtf8String cleanUtf8String >> >> With WideString, a lot of the issues disappear - except >> round-tripping(although I'm sure I have seen something recently about >> 4-byte strings that also have an additional bit. Which would make >> some Unicode characters 5-bytes long.) >> >> >> (I'm starting to zone out now - if I've overlooked anything - obvious, >> subtle, or somewhere in between, please let me know) >> >> Cheers, >> Euan >> >> > >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-an… My current thinking, and understanding. ============================== 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes. 1) Smalltalk has long had multiple String classes. 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex. 3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1 4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII 5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both. b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1 6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8. 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8. 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8 9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character. 10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints". 11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters. 12) Some concrete examples: A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex Â£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: Â£ - A3 hex ISO-8859-1: Â£ - A3 hex Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla) 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display. 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently. 15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently. 16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order. 17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations. Some practical upshots ================== 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid. 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1. 3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1. 4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML). 5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints. Class + protocol proposals a Utf8CompatibilityString class. asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex. asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1 asAsciiString - ensures only characters 00hex - 7F hex are used. asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings. Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases. Custom sortBlocks will define the applicable sort order. We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks. This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. If anyone has better names for the classes, please let me know. If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know. n.b. I have had absolutely no experience of Ropes. My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing. Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in. Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990. So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10). The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure) Cheers, Euan On 5 December 2015 at 13:08, stepharo <stepharo(a)free.fr> wrote: > Hi EuanM > > Le 4/12/15 12:42, EuanM a Ã©crit : >> >> I'm currently groping my way to seeing how feature-complete our >> Unicode support is. I am doing this to establish what still needs to >> be done to provide full Unicode support. > > > this is great. Thanks for pushing this. I wrote and collected some roadmap > (analyses on different topics) > on the pharo github project feel free to add this one there. >> >> >> This seems to me to be an area where it would be best to write it >> once, and then have the same codebase incorporated into the Smalltalks >> that most share a common ancestry. >> >> I am keen to get: equality-testing for strings; sortability for >> strings which have ligatures and diacritic characters; and correct >> round-tripping of data. > > Go! > My suggestion is > start small > make steady progress > write tests > commit often :) > > Stef > > What is the french phoneBook ordering because this is the first time I hear > about it. > >> >> Call to action: >> ========== >> >> If you have comments on these proposals - such as "but we already have >> that facility" or "the reason we do not have these facilities is >> because they are dog-slow" - please let me know them. >> >> If you would like to help out, please let me know. >> >> If you have Unicode experience and expertise, and would like to be, or >> would be willing to be, in the 'council of experts' for this project, >> please let me know. >> >> If you have comments or ideas on anything mentioned in this email >> >> In the first instance, the initiative's website will be: >> http://smalltalk.uk.to/unicode.html >> >> I have created a SqueakSource.com project called UnicodeSupport >> >> I want to avoid re-inventing any facilities which already exist. >> Except where they prevent us reaching the goals of: >> - sortable UTF8 strings >> - sortable UTF16 strings >> - equivalence testing of 2 UTF8 strings >> - equivalence testing of 2 UTF16 strings >> - round-tripping UTF8 strings through Smalltalk >> - roundtripping UTF16 strings through Smalltalk. >> As I understand it, we have limited Unicode support atm. >> >> Current state of play >> =============== >> ByteString gets converted to WideString when need is automagically >> detected. >> >> Is there anything else that currently exists? >> >> Definition of Terms >> ============== >> A quick definition of terms before I go any further: >> >> Standard terms from the Unicode standard >> =============================== >> a compatibility character : an additional encoding of a *normal* >> character, for compatibility and round-trip conversion purposes. For >> instance, a 1-byte encoding of a Latin character with a diacritic. >> >> Made-up terms >> ============ >> a convenience codepoint : a single codepoint which represents an item >> that is also encoded as a string of codepoints. >> >> (I tend to use the terms compatibility character and compatibility >> codepoint interchangably. The standard only refers to them as >> compatibility characters. However, the standard is determined to >> emphasise that characters are abstract and that codepoints are >> concrete. So I think it is often more useful and productive to think >> of compatibility or convenience codepoints). >> >> a composed character : a character made up of several codepoints >> >> Unicode encoding explained >> ===================== >> A convenience codepoint can therefore be thought of as a code point >> used for a character which also has a composed form. >> >> The way Unicode works is that sometimes you can encode a character in >> one byte, sometimes not. Sometimes you can encode it in two bytes, >> sometimes not. >> >> You can therefore have a long stream of ASCII which is single-byte >> Unicode. If there is an occasional Cyrillic or Greek character in the >> stream, it would be represented either by a compatibility character or >> by a multi-byte combination. >> >> Using compatibility characters can prevent proper sorting and >> equivalence testing. >> >> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >> and round-tripping probelms. Although avoiding them can *also* cause >> compatibility issues and round-tripping problems. >> >> Currently my thinking is: >> >> a Utf8String class >> an Ordered collection, with 1 byte characters as the modal element, >> but short arrays of wider strings where necessary >> a Utf16String class >> an Ordered collection, with 2 byte characters as the modal element, >> but short arrays of wider strings >> beginning with a 2-byte endianness indicator. >> >> Utf8Strings sometimes need to be sortable, and sometimes need to be >> compatible. >> >> So my thinking is that Utf8String will contain convenience codepoints, >> for round-tripping. And where there are multiple convenience >> codepoints for a character, that it standardises on one. >> >> And that there is a Utf8SortableString which uses *only* normal >> characters. >> >> We then need methods to convert between the two. >> >> aUtf8String asUtf8SortableString >> >> and >> >> aUtf8SortableString asUtf8String >> >> >> Sort orders are culture and context dependent - Sweden and Germany >> have different sort orders for the same diacritic-ed characters. Some >> countries have one order in general usage, and another for specific >> usages, such as phone directories (e.g. UK and France) >> >> Similarly for Utf16 : Utf16String and Utf16SortableString and >> conversion methods >> >> A list of sorted words would be a SortedCollection, and there could be >> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >> seOrder, ukOrder, etc >> >> along the lines of >> aListOfWords := SortedCollection sortBlock: deOrder >> >> If a word is either a Utf8SortableString, or a well-formed Utf8String, >> then we can perform equivalence testing on them trivially. >> >> To make sure a Utf8String is well formed, we would need to have a way >> of cleaning up any convenience codepoints which were valid, but which >> were for a character which has multiple equally-valid alternative >> convenience codepoints, and for which the string currently had the >> "wrong" convenience codepoint. (i.e for any character with valid >> alternative convenience codepoints, we would choose one to be in the >> well-formed Utf8String, and we would need a method for cleaning the >> alternative convenience codepoints out of the string, and replacing >> them with the chosen approved convenience codepoint. >> >> aUtf8String cleanUtf8String >> >> With WideString, a lot of the issues disappear - except >> round-tripping(although I'm sure I have seen something recently about >> 4-byte strings that also have an additional bit. Which would make >> some Unicode characters 5-bytes long.) >> >> >> (I'm starting to zone out now - if I've overlooked anything - obvious, >> subtle, or somewhere in between, please let me know) >> >> Cheers, >> Euan >> >> > >

Dec. 7, 2015

Re: [Pharo-dev] Community Building [was: please name your thread]

by Tudor Girba

Hi, > On Dec 4, 2015, at 1:19 AM, EuanM <euanmee(a)gmail.com> wrote: > > Tudor, > > You say that we are talking about different mails. I hope so. Most of the talk in the original thread was about your mail about Oracle and Java. I still fail to see how this is relevant. > "We need to focus on building a great platform." > "acting is the only thing that will push us further.". > > I agree that action is what achieves things. > > To make sure the correct actions are taken, in a productive and > efficient way, it can be valuable to take stock and think, > periodically. I prefer to be thinking all the time, actually :). > It is important to carry out a variety of actions, not solely coding. You seem to imply that the only actions that are being undertaken or encouraged are code related. That is far from reality. However, we do favor actions. For example, when you offered a tutorial, you were welcome and you received direct feedback. This is the effect of action. > Too narrow a focus on keeping a small group of people coding as fast > as they can is not the fastest path to the end goal. It can easily > lead to improving the platform at a slower rate than other great > platforms are improving - a process of relative decay. > > I would like to help with community building efforts. But unless the > leaders of the community also help and prioritise such efforts, they > are unlikely to make any difference. > > There is no point gaining the interest of new people unless we have > mechanisms in place to make them welcome, and put them on the path to > becoming valuable contributors to the community. > > Small communities with low levels of resource must, to succeed, do > things more efficiently than large groups with lots of funding. > > Currently, we lack: > 1) debugged beginner's training materials. > 2) community processes for getting the interest of new people > 3) community processes for onboarding interested new people with the > lowest loss rate Please be concrete. What do you think is missing concretely? > 4) community processes for providing positive feedback and > encouragement to people taking these actions. We think this actually happens. For example, you received that feedback. > 5) community processes for stopping bit-rot in mature libraries and > facilities - including key facilities like SqueakSource, Smalltalkhub > and SqueakSource3 This requires actions. Btw, SmalltalkHub is maintained through the money that the Consortium and Inria are bringing in the game. This is possible due to actions that are far from simple coding. > 6) community processes to help focus resources on merging libraries > into a single best-of-breed best-of-breed mature libraries. We often > have two competing incomplete libraries for any given problem space, > e.g. Spec/Bloc, MuTalk / SMutant I do not see how this is a problem. Having competing solutions is a good thing. The way we deal with these problems is that we try to have people define projects and work with these for a longer time. For example, Bloc is a long term project started several years before and that now starts to come to fruition and will become a core piece in Pharo. GT was created as a separate project since more than 5 years until parts of it became ready to be integrated. And the list goes on. This does not mean that we do not want to have more consolidation. We do and we want people to pick an area and provide longer term support. > 7) community processes to document the community processes > 8) documented processes for bringing newcomers up to speed with "the > way things are done round hereâ Letâs be concrete on this point. Please list the things that you have needed. We can use that experience and produce something concrete that will help other people that will come in the future. > We need to take these issues seriously - they affect how fast things > get done around here much more than this amount of time away from > decoding Pharo-core does But, we are taking it seriously. > Closing down all discussions where these issues surface does not > communicate "We welcome any action that anyone would want to undertake > in this direction". Quite the reverse. I did not close any discussion. Only the ones about fuzzy topics that lead to nowhere in particular. > We need to do more than just Smalltalk the Smalltalk - we need to walk the walk. Thank you. That is what I was calling for. Cheers, Doru > Cheers, > Euan > > Here is the thread I am keeping alive: > On 30 November 2015 at 23:32, Tudor Girba <tudor(a)tudorgirba.com> wrote: >> In any case, please name you thread appropriately. And, we should remember that acting is the only thing that will push us further. > >>> On Dec 1, 2015, at 12:18 AM, EuanM <euanmee(a)gmail.com> wrote: >>> Welcoming is, as welcoming does. > >>> On 30 November 2015 at 11:20, Tudor Girba <tudor(a)tudorgirba.com> wrote: >>>> Increasing the community is certainly important, and we welcome any action that anyone would want to undertake in this direction. > >>>>> On Nov 30, 2015, at 4:44 AM, EuanM <euanmee(a)gmail.com> wrote: >>>>> We also need to concentrate on building our community. >>>>> >>>>> We build a better platform faster if we have more people. >>>>> >>>>> We build a more valuable platform if we have a wider range of valuable >>>>> use cases to target. >>>>> >>>>> Unless and until we hit a critical mass of people joining our >>>>> community, we *need* to spend some of our focus on community-building. >>>>> >>>>> Part of great is being able to build things to sufficient completeness >>>>> *and* keep them in working order over the long haul. This is easier >>>>> with more contributors. > >>>>>>> On Nov 27, 2015, at 10:05 PM, EuanM <euanmee(a)gmail.com> wrote: >>>>>>> We *should* wish to take advantage of [the rumoured Java] announcement. >>>>>>> >>>>>>> After all, think what difference having even 0.01% of the world's Java >>>>>>> coders moving to Smalltalk would make. How could we help that >>>>>>> happen? > > [Full thread at foot of mail] > > ==================== > > > > On 30 November 2015 at 23:32, Tudor Girba <tudor(a)tudorgirba.com> wrote: >> Hi, >> >> I think we are not talking about the same email. >> >> In any case, please name you thread appropriately. And, we should remember that acting is the only thing that will push us further. >> >> Cheers, >> Tudor >> >> >>> On Dec 1, 2015, at 12:18 AM, EuanM <euanmee(a)gmail.com> wrote: >>> >>> Tudor, the email that you replied to, saying the discussion should be >>> closed down, was specifically about the impacts on our community and >>> how to leverage them. >>> >>> Welcoming is, as welcoming does. >>> >>> On 30 November 2015 at 11:20, Tudor Girba <tudor(a)tudorgirba.com> wrote: >>>> Hi, >>>> >>>> Increasing the community is certainly important, and we welcome any action that anyone would want to undertake in this direction. >>>> >>>> However, talking about the future of Java does not fall in this category. >>>> >>>> Cheers, >>>> Doru >>>> >>>> >>>>> On Nov 30, 2015, at 4:44 AM, EuanM <euanmee(a)gmail.com> wrote: >>>>> >>>>> We also need to concentrate on building our community. >>>>> >>>>> We build a better platform faster if we have more people. >>>>> >>>>> We build a more valuable platform if we have a wider range of valuable >>>>> use cases to target. >>>>> >>>>> Unless and until we hit a critical mass of people joining our >>>>> community, we *need* to spend some of our focus on community-building. >>>>> >>>>> Part of great is being able to build things to sufficient completeness >>>>> *and* keep them in working order over the long haul. This is easier >>>>> with more contributors. >>>>> >>>>> On 27 November 2015 at 21:27, Tudor Girba <tudor(a)tudorgirba.com> wrote: >>>>>> Hello everyone, >>>>>> >>>>>> Please stop this thread on this mailing list. We need to focus on building a great platform. >>>>>> >>>>>> Cheers, >>>>>> Doru >>>>>> >>>>>> >>>>>>> On Nov 27, 2015, at 10:05 PM, EuanM <euanmee(a)gmail.com> wrote: >>>>>>> >>>>>>> First of all - is this true? Where can we read about it? >>>>>>> >>>>>>> I cannot find anything about this at >>>>>>> https://www.oracle.com/search/press >>>>>>> >>>>>>> ======================================= >>>>>>> >>>>>>> If Oracle did make this statement, then what people have said so far >>>>>>> is true. BUT... >>>>>>> >>>>>>> Java got about 40% of its initial momentum from IBM dumping VisualAge >>>>>>> and putting all their resources into Java. >>>>>>> >>>>>>> Oracle are targetting this move at IBM more than anyone else. >>>>>>> >>>>>>> IBM will start to think about how to migrate from Java - as Oracle are >>>>>>> telling them they will have to. (It's OUR bat and its OUR ball, and >>>>>>> no-one else can play with it. Not even the Java Community). And >>>>>>> IBM's coders do not pay for Java, Eclipse users do not pay for Java. I >>>>>>> expect the licence-fee income for JREs is small. >>>>>>> >>>>>>> Oracle are doing one of two things - announcing that Java is for sale >>>>>>> to device providers - phones (Google is the obvious buyer) or the >>>>>>> impending Internet of Things (which was what Java was designed for >>>>>>> originally) or announcing that no-one making an internet of things >>>>>>> offering should consider Java. >>>>>>> >>>>>>> Yes, things live on and on in a kind of zombie state. So yes, things >>>>>>> live on as long as their ecosystem does. And they gently wither and >>>>>>> their ecosystem withers is a long slow drawn out spiral. Which is why >>>>>>> we still have Cobol. >>>>>>> >>>>>>> People and organisations tend to move from one technology to another >>>>>>> in an incremental fashion. Swapping a little bit here, and a little >>>>>>> bit there. >>>>>>> >>>>>>> The new target platforms are ones which >>>>>>> 1) look like they have longevity, and >>>>>>> 2) have a migration pathway that provides incremental steps. >>>>>>> >>>>>>> Offering a compelling advantage is good - but only if the steps 1) >>>>>>> and 2) are catered to. >>>>>>> >>>>>>> IBM VisualAge Smalltalk is still robust, commercially available >>>>>>> software, and VisualStudio and Gemstone continue to represent >>>>>>> Smalltalk out to the big world of corporate development. >>>>>>> >>>>>>> So that's a start. >>>>>>> >>>>>>> Say only 5% of the Java world moves away from Java each year, as a >>>>>>> result of this announcement. >>>>>>> >>>>>>> We *should* wish to take advantage of this announcement. >>>>>>> >>>>>>> After all, think what difference having even 0.01% of the world's Java >>>>>>> coders moving to Smalltalk would make. How could we help that >>>>>>> happen? >>>>>>> >>>>>>> Think what it would be like to have thought-leaders like Kent Beck and >>>>>>> Ward Cunningham back in the Smalltalk fold. How could we help that >>>>>>> happen? >>>>>>> >>>>>>> Think what it would be like to get back all the universities who moved >>>>>>> from teaching OO concepts using Smalltalk into teaching them via Java. >>>>>>> We now know almost all the ones using Smalltalk as a teaching language >>>>>>> by name. Does anyone know even how many universities teach OO via >>>>>>> Java? What would it be like if 5% of those universities moved to >>>>>>> Smalltalk each year. How could we help that happen? >>>>>>> >>>>>>> Next - do we have any big brained thinkers who can see specific ways >>>>>>> we can improve interoperation between Java facilities and libraries >>>>>>> and the Smalltalks? For the next 12 months, we should work on Java >>>>>>> integration, rather than C++ integration. We should identify the >>>>>>> three best things for us to do in this regard, and make them polished >>>>>>> and compelling. Who is in a position to help that happen? >>>>>>> >>>>>>> The final way we can take advantage help the maximum number of people >>>>>>> find their way to us is to present a united community front to the >>>>>>> outside world. In the same way I am both a European and a Scot, we >>>>>>> need to be Smalltalkers *and*members of our individual >>>>>>> Smalltalk-platform communities. >>>>>>> >>>>>>> How can we help make that happen? >>>>>>> >>>>>>> This is not a silver bullet. It's going to cause a long-term trend in >>>>>>> events, not a sudden abrupt change. But it will have a real, if >>>>>>> gradual effect. (assuming that >>>>>>> >>>>>>> Equally, it is not something we should ignore. It is something we >>>>>>> should make use of. We need to put effort into raising our profile >>>>>>> over the next 6 months. >>>>>>> > -- www.tudorgirba.com "Live like you mean it."

Dec. 7, 2015

Jump to page: