December 2015 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [squeak-dev] Unicode Support

by Mark Bestley

On 06/12/2015 19:08, Sven Van Caekenberghe wrote: > BTW, does anyone know of any programming language that did go that way or has a library that directly implements 'storing all strings as utf-8' ? Java is UTF-16 Python3, Go and Swift are UTF-8 as I suspect are other new languages not based on .Net or the JVM Mark > >> On 06 Dec 2015, at 18:45, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> Well written, Todd. I agree, the loss of indexing might not be that big a problem in practice. The only way to find out it to try an experiment I guess. >> >> Sven >> >>> On 06 Dec 2015, at 17:37, Todd Blanchard <tblanchard(a)mac.com> wrote: >>> >>> (Resent because of bounce notification (email handling in osx is really beginning to annoy me). Sorry if its a dup) >>> >>> I used to worry a lot about strings being indexable. And then I eventually let go of that and realized that it isn't a particularly important property for them to have. >>> >>> I think you will find that UTF8 is generally the most convenient for a lot of things but its a bit like light in that you treat it alternately as a wave or particle depending on what you are trying to do. >>> >>> So goes strings - they can be treated alternately as streams or byte arrays (not character arrays - stop thinking in characters). In practice, this tends to not be a problem since a lot of the times when you want to replace a character or pick out the nth one you are doing something very computerish and the characters you are working with are the single byte (ASCII legacy) variety. You generally know when you can get away with that and when you can't. >>> >>> Otherwise you are most likely doing things that are best dealt with in a streaming paradigm. For most computation, you come to realize you don't generally care how many characters but how much space (bytes) you need to store your chunk of text. Collation is tricky and complicated in unicode in general but it isn't any worse in UTF8 than any other encoding. You are still going to scan each sortable item from front to back to determine its order, regardless. >>> >>> Most of the outside world has settled on UTF8 and any ASCII file is already UTF8 - which is why it ends up being so convenient. Most of our old text handling infrastructure can still handle UTF8 while it tends to choke on wider encodings. >>> >>> -Todd Blanchard >>> >>>> On Dec 6, 2015, at 07:23, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>>> >>>>> We do the same thing, but that doesn't mean it's a good idea to create a >>>>> new String-like class having its content encoded in UTF-8, because >>>>> UTF-8-encoded strings can't be modified like regular strings. While it >>>>> would be possible to implement all operations, such implementation would >>>>> become the next SortedCollection (bad performance due to misuse). >>> >> > > > -- Mark

Dec. 6, 2015

Re: [Pharo-dev] Unicode Support

by Max Leske

> On 06 Dec 2015, at 18:44, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> On 05 Dec 2015, at 17:35, Todd Blanchard <tblanchard(a)mac.com> wrote: >> >> would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time. >> >> Read http://utf8everywhere.org/ >> >> I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site. > > Well, I read the page/document/site as well. It was very interesting indeed, thanks for sharing it. > > In some sense it made me reconsider my aversion against in-image utf-8 encoding, maybe it could have some value. Absolute storage is more efficient, some processing might also be more efficient, i/o conversions to/from utf-8 become a no-op. What I found nice is the suggestion that most structured parsing (XML, JSON, CSV, STON, ...) could actually ignore the encoding for a large part and just assume its ASCII, which would/could be nice for performance. Also the fact that a lot of strings are (or should be) treated as opaque makes a lot of sense. > > What I did not like is that much of argumentation is based on issue in the Windows world, take all that away and the document shrinks in half. I would have liked a bit more fundamental CS arguments. > > Canonicalisation and sorting issues are hardly discussed. > > In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $Ã© if you know that it is possible to represent $Ã© as just $Ã© and as $e + $Â´ ? Thatâs what normalization is for: http://unicode.org/faq/normalization.html. It will generate the same codepoint for two strings where one contains the combining character and the other is a âsingle characterâ. > > Sven > >> Sent from the road >> >> On Dec 5, 2015, at 05:08, stepharo <stepharo(a)free.fr> wrote: >> >>> Hi EuanM >>> >>> Le 4/12/15 12:42, EuanM a Ã©crit : >>>> I'm currently groping my way to seeing how feature-complete our >>>> Unicode support is. I am doing this to establish what still needs to >>>> be done to provide full Unicode support. >>> >>> this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) >>> on the pharo github project feel free to add this one there. >>>> >>>> This seems to me to be an area where it would be best to write it >>>> once, and then have the same codebase incorporated into the Smalltalks >>>> that most share a common ancestry. >>>> >>>> I am keen to get: equality-testing for strings; sortability for >>>> strings which have ligatures and diacritic characters; and correct >>>> round-tripping of data. >>> Go! >>> My suggestion is >>> start small >>> make steady progress >>> write tests >>> commit often :) >>> >>> Stef >>> >>> What is the french phoneBook ordering because this is the first time I hear about it. >>>> >>>> Call to action: >>>> ========== >>>> >>>> If you have comments on these proposals - such as "but we already have >>>> that facility" or "the reason we do not have these facilities is >>>> because they are dog-slow" - please let me know them. >>>> >>>> If you would like to help out, please let me know. >>>> >>>> If you have Unicode experience and expertise, and would like to be, or >>>> would be willing to be, in the 'council of experts' for this project, >>>> please let me know. >>>> >>>> If you have comments or ideas on anything mentioned in this email >>>> >>>> In the first instance, the initiative's website will be: >>>> http://smalltalk.uk.to/unicode.html >>>> >>>> I have created a SqueakSource.com project called UnicodeSupport >>>> >>>> I want to avoid re-inventing any facilities which already exist. >>>> Except where they prevent us reaching the goals of: >>>> - sortable UTF8 strings >>>> - sortable UTF16 strings >>>> - equivalence testing of 2 UTF8 strings >>>> - equivalence testing of 2 UTF16 strings >>>> - round-tripping UTF8 strings through Smalltalk >>>> - roundtripping UTF16 strings through Smalltalk. >>>> As I understand it, we have limited Unicode support atm. >>>> >>>> Current state of play >>>> =============== >>>> ByteString gets converted to WideString when need is automagically detected. >>>> >>>> Is there anything else that currently exists? >>>> >>>> Definition of Terms >>>> ============== >>>> A quick definition of terms before I go any further: >>>> >>>> Standard terms from the Unicode standard >>>> =============================== >>>> a compatibility character : an additional encoding of a *normal* >>>> character, for compatibility and round-trip conversion purposes. For >>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>> >>>> Made-up terms >>>> ============ >>>> a convenience codepoint : a single codepoint which represents an item >>>> that is also encoded as a string of codepoints. >>>> >>>> (I tend to use the terms compatibility character and compatibility >>>> codepoint interchangably. The standard only refers to them as >>>> compatibility characters. However, the standard is determined to >>>> emphasise that characters are abstract and that codepoints are >>>> concrete. So I think it is often more useful and productive to think >>>> of compatibility or convenience codepoints). >>>> >>>> a composed character : a character made up of several codepoints >>>> >>>> Unicode encoding explained >>>> ===================== >>>> A convenience codepoint can therefore be thought of as a code point >>>> used for a character which also has a composed form. >>>> >>>> The way Unicode works is that sometimes you can encode a character in >>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>> sometimes not. >>>> >>>> You can therefore have a long stream of ASCII which is single-byte >>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>> stream, it would be represented either by a compatibility character or >>>> by a multi-byte combination. >>>> >>>> Using compatibility characters can prevent proper sorting and >>>> equivalence testing. >>>> >>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>> and round-tripping probelms. Although avoiding them can *also* cause >>>> compatibility issues and round-tripping problems. >>>> >>>> Currently my thinking is: >>>> >>>> a Utf8String class >>>> an Ordered collection, with 1 byte characters as the modal element, >>>> but short arrays of wider strings where necessary >>>> a Utf16String class >>>> an Ordered collection, with 2 byte characters as the modal element, >>>> but short arrays of wider strings >>>> beginning with a 2-byte endianness indicator. >>>> >>>> Utf8Strings sometimes need to be sortable, and sometimes need to be compatible. >>>> >>>> So my thinking is that Utf8String will contain convenience codepoints, >>>> for round-tripping. And where there are multiple convenience >>>> codepoints for a character, that it standardises on one. >>>> >>>> And that there is a Utf8SortableString which uses *only* normal characters. >>>> >>>> We then need methods to convert between the two. >>>> >>>> aUtf8String asUtf8SortableString >>>> >>>> and >>>> >>>> aUtf8SortableString asUtf8String >>>> >>>> >>>> Sort orders are culture and context dependent - Sweden and Germany >>>> have different sort orders for the same diacritic-ed characters. Some >>>> countries have one order in general usage, and another for specific >>>> usages, such as phone directories (e.g. UK and France) >>>> >>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>> conversion methods >>>> >>>> A list of sorted words would be a SortedCollection, and there could be >>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>> seOrder, ukOrder, etc >>>> >>>> along the lines of >>>> aListOfWords := SortedCollection sortBlock: deOrder >>>> >>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>> then we can perform equivalence testing on them trivially. >>>> >>>> To make sure a Utf8String is well formed, we would need to have a way >>>> of cleaning up any convenience codepoints which were valid, but which >>>> were for a character which has multiple equally-valid alternative >>>> convenience codepoints, and for which the string currently had the >>>> "wrong" convenience codepoint. (i.e for any character with valid >>>> alternative convenience codepoints, we would choose one to be in the >>>> well-formed Utf8String, and we would need a method for cleaning the >>>> alternative convenience codepoints out of the string, and replacing >>>> them with the chosen approved convenience codepoint. >>>> >>>> aUtf8String cleanUtf8String >>>> >>>> With WideString, a lot of the issues disappear - except >>>> round-tripping(although I'm sure I have seen something recently about >>>> 4-byte strings that also have an additional bit. Which would make >>>> some Unicode characters 5-bytes long.) >>>> >>>> >>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>> subtle, or somewhere in between, please let me know) >>>> >>>> Cheers, >>>> Euan >>>> >>>> >>> >>> > >

Dec. 6, 2015

Re: [Pharo-dev] [squeak-dev] Unicode Support

by Sven Van Caekenberghe

BTW, does anyone know of any programming language that did go that way or has a library that directly implements 'storing all strings as utf-8' ? > On 06 Dec 2015, at 18:45, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Well written, Todd. I agree, the loss of indexing might not be that big a problem in practice. The only way to find out it to try an experiment I guess. > > Sven > >> On 06 Dec 2015, at 17:37, Todd Blanchard <tblanchard(a)mac.com> wrote: >> >> (Resent because of bounce notification (email handling in osx is really beginning to annoy me). Sorry if its a dup) >> >> I used to worry a lot about strings being indexable. And then I eventually let go of that and realized that it isn't a particularly important property for them to have. >> >> I think you will find that UTF8 is generally the most convenient for a lot of things but its a bit like light in that you treat it alternately as a wave or particle depending on what you are trying to do. >> >> So goes strings - they can be treated alternately as streams or byte arrays (not character arrays - stop thinking in characters). In practice, this tends to not be a problem since a lot of the times when you want to replace a character or pick out the nth one you are doing something very computerish and the characters you are working with are the single byte (ASCII legacy) variety. You generally know when you can get away with that and when you can't. >> >> Otherwise you are most likely doing things that are best dealt with in a streaming paradigm. For most computation, you come to realize you don't generally care how many characters but how much space (bytes) you need to store your chunk of text. Collation is tricky and complicated in unicode in general but it isn't any worse in UTF8 than any other encoding. You are still going to scan each sortable item from front to back to determine its order, regardless. >> >> Most of the outside world has settled on UTF8 and any ASCII file is already UTF8 - which is why it ends up being so convenient. Most of our old text handling infrastructure can still handle UTF8 while it tends to choke on wider encodings. >> >> -Todd Blanchard >> >>> On Dec 6, 2015, at 07:23, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>> >>>> We do the same thing, but that doesn't mean it's a good idea to create a >>>> new String-like class having its content encoded in UTF-8, because >>>> UTF-8-encoded strings can't be modified like regular strings. While it >>>> would be possible to implement all operations, such implementation would >>>> become the next SortedCollection (bad performance due to misuse). >> >

Dec. 6, 2015

displayingProgress on demand

by HernÃ¡n Morales Durand

Hello everyone, Is there a reason why #collect:displayingProgress: was removed or never included? And as more general question, is there any pointer to automatically add progress bar to iterator messages? For example doing: SmalltalkImage setDisplayProgressTo: MyClass. SmalltalkImage setDisplayProgressTo: MyPackage. and then having all enumeration messages sent in MyClass or MyPackage with a progress bar collect: collect:as: collect:into: collect:thenDo: select: select:as: select:thenDo: select:thenCollect:as: ... HernÃ¡n

Dec. 6, 2015

Re: [Pharo-dev] [Help needed] final push for Rubric editor

by Alain Plantec

> On 06 Dec 2015, at 17:07, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > in: > > 17192 <https://pharo.fogbugz.com/f/cases/17192/UITheme-should-not-use-PluggableTex…> UITheme should not use PluggableTextMorph > > what should we use to replace PluggableTextMorph, RubPluggableTextMorph or RubScrolledTextMorph ? > I think RubPluggableTextMorph, because we need a way to adapt the callers model as a RubrikText model, but the comment of > RubPluggableTextMorph says : "Try to not use it â yes because it uses the change/update pattern. but you're right, RubPluggableTextMorph was added to ease the transition and to be used as a first step. Cheers Alain >

Dec. 6, 2015

Re: [Pharo-dev] [squeak-dev] Unicode Support

by Sven Van Caekenberghe

Well written, Todd. I agree, the loss of indexing might not be that big a problem in practice. The only way to find out it to try an experiment I guess. Sven > On 06 Dec 2015, at 17:37, Todd Blanchard <tblanchard(a)mac.com> wrote: > > (Resent because of bounce notification (email handling in osx is really beginning to annoy me). Sorry if its a dup) > > I used to worry a lot about strings being indexable. And then I eventually let go of that and realized that it isn't a particularly important property for them to have. > > I think you will find that UTF8 is generally the most convenient for a lot of things but its a bit like light in that you treat it alternately as a wave or particle depending on what you are trying to do. > > So goes strings - they can be treated alternately as streams or byte arrays (not character arrays - stop thinking in characters). In practice, this tends to not be a problem since a lot of the times when you want to replace a character or pick out the nth one you are doing something very computerish and the characters you are working with are the single byte (ASCII legacy) variety. You generally know when you can get away with that and when you can't. > > Otherwise you are most likely doing things that are best dealt with in a streaming paradigm. For most computation, you come to realize you don't generally care how many characters but how much space (bytes) you need to store your chunk of text. Collation is tricky and complicated in unicode in general but it isn't any worse in UTF8 than any other encoding. You are still going to scan each sortable item from front to back to determine its order, regardless. > > Most of the outside world has settled on UTF8 and any ASCII file is already UTF8 - which is why it ends up being so convenient. Most of our old text handling infrastructure can still handle UTF8 while it tends to choke on wider encodings. > > -Todd Blanchard > >> On Dec 6, 2015, at 07:23, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >> >>> We do the same thing, but that doesn't mean it's a good idea to create a >>> new String-like class having its content encoded in UTF-8, because >>> UTF-8-encoded strings can't be modified like regular strings. While it >>> would be possible to implement all operations, such implementation would >>> become the next SortedCollection (bad performance due to misuse). >

Dec. 6, 2015

Re: [Pharo-dev] Unicode Support

by Sven Van Caekenberghe

> On 05 Dec 2015, at 17:35, Todd Blanchard <tblanchard(a)mac.com> wrote: > > would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time. > > Read http://utf8everywhere.org/ > > I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site. Well, I read the page/document/site as well. It was very interesting indeed, thanks for sharing it. In some sense it made me reconsider my aversion against in-image utf-8 encoding, maybe it could have some value. Absolute storage is more efficient, some processing might also be more efficient, i/o conversions to/from utf-8 become a no-op. What I found nice is the suggestion that most structured parsing (XML, JSON, CSV, STON, ...) could actually ignore the encoding for a large part and just assume its ASCII, which would/could be nice for performance. Also the fact that a lot of strings are (or should be) treated as opaque makes a lot of sense. What I did not like is that much of argumentation is based on issue in the Windows world, take all that away and the document shrinks in half. I would have liked a bit more fundamental CS arguments. Canonicalisation and sorting issues are hardly discussed. In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $Ã© if you know that it is possible to represent $Ã© as just $Ã© and as $e + $Â´ ? Sven > Sent from the road > > On Dec 5, 2015, at 05:08, stepharo <stepharo(a)free.fr> wrote: > >> Hi EuanM >> >> Le 4/12/15 12:42, EuanM a Ã©crit : >>> I'm currently groping my way to seeing how feature-complete our >>> Unicode support is. I am doing this to establish what still needs to >>> be done to provide full Unicode support. >> >> this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) >> on the pharo github project feel free to add this one there. >>> >>> This seems to me to be an area where it would be best to write it >>> once, and then have the same codebase incorporated into the Smalltalks >>> that most share a common ancestry. >>> >>> I am keen to get: equality-testing for strings; sortability for >>> strings which have ligatures and diacritic characters; and correct >>> round-tripping of data. >> Go! >> My suggestion is >> start small >> make steady progress >> write tests >> commit often :) >> >> Stef >> >> What is the french phoneBook ordering because this is the first time I hear about it. >>> >>> Call to action: >>> ========== >>> >>> If you have comments on these proposals - such as "but we already have >>> that facility" or "the reason we do not have these facilities is >>> because they are dog-slow" - please let me know them. >>> >>> If you would like to help out, please let me know. >>> >>> If you have Unicode experience and expertise, and would like to be, or >>> would be willing to be, in the 'council of experts' for this project, >>> please let me know. >>> >>> If you have comments or ideas on anything mentioned in this email >>> >>> In the first instance, the initiative's website will be: >>> http://smalltalk.uk.to/unicode.html >>> >>> I have created a SqueakSource.com project called UnicodeSupport >>> >>> I want to avoid re-inventing any facilities which already exist. >>> Except where they prevent us reaching the goals of: >>> - sortable UTF8 strings >>> - sortable UTF16 strings >>> - equivalence testing of 2 UTF8 strings >>> - equivalence testing of 2 UTF16 strings >>> - round-tripping UTF8 strings through Smalltalk >>> - roundtripping UTF16 strings through Smalltalk. >>> As I understand it, we have limited Unicode support atm. >>> >>> Current state of play >>> =============== >>> ByteString gets converted to WideString when need is automagically detected. >>> >>> Is there anything else that currently exists? >>> >>> Definition of Terms >>> ============== >>> A quick definition of terms before I go any further: >>> >>> Standard terms from the Unicode standard >>> =============================== >>> a compatibility character : an additional encoding of a *normal* >>> character, for compatibility and round-trip conversion purposes. For >>> instance, a 1-byte encoding of a Latin character with a diacritic. >>> >>> Made-up terms >>> ============ >>> a convenience codepoint : a single codepoint which represents an item >>> that is also encoded as a string of codepoints. >>> >>> (I tend to use the terms compatibility character and compatibility >>> codepoint interchangably. The standard only refers to them as >>> compatibility characters. However, the standard is determined to >>> emphasise that characters are abstract and that codepoints are >>> concrete. So I think it is often more useful and productive to think >>> of compatibility or convenience codepoints). >>> >>> a composed character : a character made up of several codepoints >>> >>> Unicode encoding explained >>> ===================== >>> A convenience codepoint can therefore be thought of as a code point >>> used for a character which also has a composed form. >>> >>> The way Unicode works is that sometimes you can encode a character in >>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>> sometimes not. >>> >>> You can therefore have a long stream of ASCII which is single-byte >>> Unicode. If there is an occasional Cyrillic or Greek character in the >>> stream, it would be represented either by a compatibility character or >>> by a multi-byte combination. >>> >>> Using compatibility characters can prevent proper sorting and >>> equivalence testing. >>> >>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>> and round-tripping probelms. Although avoiding them can *also* cause >>> compatibility issues and round-tripping problems. >>> >>> Currently my thinking is: >>> >>> a Utf8String class >>> an Ordered collection, with 1 byte characters as the modal element, >>> but short arrays of wider strings where necessary >>> a Utf16String class >>> an Ordered collection, with 2 byte characters as the modal element, >>> but short arrays of wider strings >>> beginning with a 2-byte endianness indicator. >>> >>> Utf8Strings sometimes need to be sortable, and sometimes need to be compatible. >>> >>> So my thinking is that Utf8String will contain convenience codepoints, >>> for round-tripping. And where there are multiple convenience >>> codepoints for a character, that it standardises on one. >>> >>> And that there is a Utf8SortableString which uses *only* normal characters. >>> >>> We then need methods to convert between the two. >>> >>> aUtf8String asUtf8SortableString >>> >>> and >>> >>> aUtf8SortableString asUtf8String >>> >>> >>> Sort orders are culture and context dependent - Sweden and Germany >>> have different sort orders for the same diacritic-ed characters. Some >>> countries have one order in general usage, and another for specific >>> usages, such as phone directories (e.g. UK and France) >>> >>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>> conversion methods >>> >>> A list of sorted words would be a SortedCollection, and there could be >>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>> seOrder, ukOrder, etc >>> >>> along the lines of >>> aListOfWords := SortedCollection sortBlock: deOrder >>> >>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>> then we can perform equivalence testing on them trivially. >>> >>> To make sure a Utf8String is well formed, we would need to have a way >>> of cleaning up any convenience codepoints which were valid, but which >>> were for a character which has multiple equally-valid alternative >>> convenience codepoints, and for which the string currently had the >>> "wrong" convenience codepoint. (i.e for any character with valid >>> alternative convenience codepoints, we would choose one to be in the >>> well-formed Utf8String, and we would need a method for cleaning the >>> alternative convenience codepoints out of the string, and replacing >>> them with the chosen approved convenience codepoint. >>> >>> aUtf8String cleanUtf8String >>> >>> With WideString, a lot of the issues disappear - except >>> round-tripping(although I'm sure I have seen something recently about >>> 4-byte strings that also have an additional bit. Which would make >>> some Unicode characters 5-bytes long.) >>> >>> >>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>> subtle, or somewhere in between, please let me know) >>> >>> Cheers, >>> Euan >>> >>> >> >>

Dec. 6, 2015

Re: [Pharo-dev] [squeak-dev] Unicode Support

by Todd Blanchard

(Resent because of bounce notification (email handling in osx is really beginning to annoy me). Sorry if its a dup) I used to worry a lot about strings being indexable. And then I eventually let go of that and realized that it isn't a particularly important property for them to have. I think you will find that UTF8 is generally the most convenient for a lot of things but its a bit like light in that you treat it alternately as a wave or particle depending on what you are trying to do. So goes strings - they can be treated alternately as streams or byte arrays (not character arrays - stop thinking in characters). In practice, this tends to not be a problem since a lot of the times when you want to replace a character or pick out the nth one you are doing something very computerish and the characters you are working with are the single byte (ASCII legacy) variety. You generally know when you can get away with that and when you can't. Otherwise you are most likely doing things that are best dealt with in a streaming paradigm. For most computation, you come to realize you don't generally care how many characters but how much space (bytes) you need to store your chunk of text. Collation is tricky and complicated in unicode in general but it isn't any worse in UTF8 than any other encoding. You are still going to scan each sortable item from front to back to determine its order, regardless. Most of the outside world has settled on UTF8 and any ASCII file is already UTF8 - which is why it ends up being so convenient. Most of our old text handling infrastructure can still handle UTF8 while it tends to choke on wider encodings. -Todd Blanchard > On Dec 6, 2015, at 07:23, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: > >> We do the same thing, but that doesn't mean it's a good idea to create a >> new String-like class having its content encoded in UTF-8, because >> UTF-8-encoded strings can't be modified like regular strings. While it >> would be possible to implement all operations, such implementation would >> become the next SortedCollection (bad performance due to misuse).

Dec. 6, 2015

Re: [Pharo-dev] [Help needed] final push for Rubric editor

by Nicolai Hess

in: 17192 <https://pharo.fogbugz.com/f/cases/17192/UITheme-should-not-use-PluggableTex…> UITheme should not use PluggableTextMorph what should we use to replace PluggableTextMorph, RubPluggableTextMorph or RubScrolledTextMorph ? I think RubPluggableTextMorph, because we need a way to adapt the callers model as a RubrikText model, but the comment of RubPluggableTextMorph says : "Try to not use it " 2015-12-05 12:13 GMT+01:00 Marcus Denker <marcus.denker(a)inria.fr>: > Hi, > > We are using Rubric now nearly everywhere. But there are still some places > left > where the old PluggableTextMorph is used. > > I made a TODO with a list of what needs to be done: > > https://pharo.fogbugz.com/f/search/?sSearchFor=outline%3A15596 > > > Marcus >

Dec. 6, 2015

Re: [Pharo-dev] [squeak-dev] Unicode Support

by H. Hirzel

On 12/6/15, Levente Uzonyi <leves(a)caesar.elte.hu> wrote: > On Sat, 5 Dec 2015, Colin Putney wrote: > >> First, what's UTF-32? Second, we have the whole language tag thing that >> nobody else uses. > > In Squeak, Strings use UTF-32 encoding[1]. It's straightforward > to see for WideString, but ByteString is just a subset of WideString, so > it uses the same encoding. We also use language tags, but that's a > different story. > Language tags make it possible to work around the problems introduced by > the Han unification[2]. We shouldn't really use them for non-CJKV > languages. > >> >> Finally, UTF-8 is a great encoding that certain kinds of applications >> really ought to use. Web apps, in particular, benefit from using UTF-8 so >> the don't have to decode and then re-encode strings coming in from the >> network. In DabbleDB we used UTF-8 encoded string in the image, and just >> ignored the fact that they were displayed incorrectly by inspectors. >> Having a proper UTF-8 string class would be useful. > > We do the same thing, but that doesn't mean it's a good idea to create a > new String-like class having its content encoded in UTF-8, because > UTF-8-encoded strings can't be modified like regular strings. While it > would be possible to implement all operations, such implementation would > become the next SortedCollection (bad performance due to misuse). This is not the case if you go for ropes https://github.com/KenDickey/Cuis-Smalltalk-Ropes > > Levente > > [1] https://en.wikipedia.org/wiki/UTF-32 > [2] https://en.wikipedia.org/wiki/Han_unification > >> >> - Colin >> >> >>> On Dec 4, 2015, at 6:46 AM, Levente Uzonyi <leves(a)caesar.elte.hu> wrote: >>> >>> Why would you want to have strings with UTF-8 or UTF-16 encoding in the >>> image? >>> What's wrong with the current UTF-32 representation? >>> >>> Levente >>> >>>> On Fri, 4 Dec 2015, EuanM wrote: >>>> >>>> I'm currently groping my way to seeing how feature-complete our >>>> Unicode support is. I am doing this to establish what still needs to >>>> be done to provide full Unicode support. >>>> >>>> This seems to me to be an area where it would be best to write it >>>> once, and then have the same codebase incorporated into the Smalltalks >>>> that most share a common ancestry. >>>> >>>> I am keen to get: equality-testing for strings; sortability for >>>> strings which have ligatures and diacritic characters; and correct >>>> round-tripping of data. >>>> >>>> Call to action: >>>> ========== >>>> >>>> If you have comments on these proposals - such as "but we already have >>>> that facility" or "the reason we do not have these facilities is >>>> because they are dog-slow" - please let me know them. >>>> >>>> If you would like to help out, please let me know. >>>> >>>> If you have Unicode experience and expertise, and would like to be, or >>>> would be willing to be, in the 'council of experts' for this project, >>>> please let me know. >>>> >>>> If you have comments or ideas on anything mentioned in this email >>>> >>>> In the first instance, the initiative's website will be: >>>> http://smalltalk.uk.to/unicode.html >>>> >>>> I have created a SqueakSource.com project called UnicodeSupport >>>> >>>> I want to avoid re-inventing any facilities which already exist. >>>> Except where they prevent us reaching the goals of: >>>> - sortable UTF8 strings >>>> - sortable UTF16 strings >>>> - equivalence testing of 2 UTF8 strings >>>> - equivalence testing of 2 UTF16 strings >>>> - round-tripping UTF8 strings through Smalltalk >>>> - roundtripping UTF16 strings through Smalltalk. >>>> As I understand it, we have limited Unicode support atm. >>>> >>>> Current state of play >>>> =============== >>>> ByteString gets converted to WideString when need is automagically >>>> detected. >>>> >>>> Is there anything else that currently exists? >>>> >>>> Definition of Terms >>>> ============== >>>> A quick definition of terms before I go any further: >>>> >>>> Standard terms from the Unicode standard >>>> =============================== >>>> a compatibility character : an additional encoding of a *normal* >>>> character, for compatibility and round-trip conversion purposes. For >>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>> >>>> Made-up terms >>>> ============ >>>> a convenience codepoint : a single codepoint which represents an item >>>> that is also encoded as a string of codepoints. >>>> >>>> (I tend to use the terms compatibility character and compatibility >>>> codepoint interchangably. The standard only refers to them as >>>> compatibility characters. However, the standard is determined to >>>> emphasise that characters are abstract and that codepoints are >>>> concrete. So I think it is often more useful and productive to think >>>> of compatibility or convenience codepoints). >>>> >>>> a composed character : a character made up of several codepoints >>>> >>>> Unicode encoding explained >>>> ===================== >>>> A convenience codepoint can therefore be thought of as a code point >>>> used for a character which also has a composed form. >>>> >>>> The way Unicode works is that sometimes you can encode a character in >>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>> sometimes not. >>>> >>>> You can therefore have a long stream of ASCII which is single-byte >>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>> stream, it would be represented either by a compatibility character or >>>> by a multi-byte combination. >>>> >>>> Using compatibility characters can prevent proper sorting and >>>> equivalence testing. >>>> >>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>> and round-tripping probelms. Although avoiding them can *also* cause >>>> compatibility issues and round-tripping problems. >>>> >>>> Currently my thinking is: >>>> >>>> a Utf8String class >>>> an Ordered collection, with 1 byte characters as the modal element, >>>> but short arrays of wider strings where necessary >>>> a Utf16String class >>>> an Ordered collection, with 2 byte characters as the modal element, >>>> but short arrays of wider strings >>>> beginning with a 2-byte endianness indicator. >>>> >>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>> compatible. >>>> >>>> So my thinking is that Utf8String will contain convenience codepoints, >>>> for round-tripping. And where there are multiple convenience >>>> codepoints for a character, that it standardises on one. >>>> >>>> And that there is a Utf8SortableString which uses *only* normal >>>> characters. >>>> >>>> We then need methods to convert between the two. >>>> >>>> aUtf8String asUtf8SortableString >>>> >>>> and >>>> >>>> aUtf8SortableString asUtf8String >>>> >>>> >>>> Sort orders are culture and context dependent - Sweden and Germany >>>> have different sort orders for the same diacritic-ed characters. Some >>>> countries have one order in general usage, and another for specific >>>> usages, such as phone directories (e.g. UK and France) >>>> >>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>> conversion methods >>>> >>>> A list of sorted words would be a SortedCollection, and there could be >>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>> seOrder, ukOrder, etc >>>> >>>> along the lines of >>>> aListOfWords := SortedCollection sortBlock: deOrder >>>> >>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>> then we can perform equivalence testing on them trivially. >>>> >>>> To make sure a Utf8String is well formed, we would need to have a way >>>> of cleaning up any convenience codepoints which were valid, but which >>>> were for a character which has multiple equally-valid alternative >>>> convenience codepoints, and for which the string currently had the >>>> "wrong" convenience codepoint. (i.e for any character with valid >>>> alternative convenience codepoints, we would choose one to be in the >>>> well-formed Utf8String, and we would need a method for cleaning the >>>> alternative convenience codepoints out of the string, and replacing >>>> them with the chosen approved convenience codepoint. >>>> >>>> aUtf8String cleanUtf8String >>>> >>>> With WideString, a lot of the issues disappear - except >>>> round-tripping(although I'm sure I have seen something recently about >>>> 4-byte strings that also have an additional bit. Which would make >>>> some Unicode characters 5-bytes long.) >>>> >>>> >>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>> subtle, or somewhere in between, please let me know) >>>> >>>> Cheers, >>>> Euan >>>> >>>> >>> >> >> > >

Dec. 6, 2015