- Pharo-dev - lists.pharo.org

Fwd: [squeak-dev] Unicode Support
by Ben Coman Dec. 7, 2015

Dec. 7, 2015

Sorry to double post, I missed the cross-post. On Fri, Dec 4, 2015 at 7:42 PM, EuanM <euanmee(a)gmail.com> wrote: > I'm currently groping my way to seeing how feature-complete our > Unicode support is. I am doing this to establish what still needs to > be done to provide full Unicode support. > > This seems to me to be an area where it would be best to write it > once, and then have the same codebase incorporated into the Smalltalks > that most share a common ancestry. > > I am keen to get: equality-testing for strings; sortability for > strings which have ligatures and diacritic characters; and correct > round-tripping of data. > > Call to action: > ========== > > If you have comments on these proposals - such as "but we already have > that facility" or "the reason we do not have these facilities is > because they are dog-slow" - please let me know them. > > If you would like to help out, please let me know. > > If you have Unicode experience and expertise, and would like to be, or > would be willing to be, in the 'council of experts' for this project, > please let me know. > > If you have comments or ideas on anything mentioned in this email > > In the first instance, the initiative's website will be: > http://smalltalk.uk.to/unicode.html > > I have created a SqueakSource.com project called UnicodeSupport > > I want to avoid re-inventing any facilities which already exist. > Except where they prevent us reaching the goals of: > - sortable UTF8 strings > - sortable UTF16 strings > - equivalence testing of 2 UTF8 strings > - equivalence testing of 2 UTF16 strings > - round-tripping UTF8 strings through Smalltalk > - roundtripping UTF16 strings through Smalltalk. > As I understand it, we have limited Unicode support atm. > > Current state of play > =============== > ByteString gets converted to WideString when need is automagically detected. > > Is there anything else that currently exists? > > Definition of Terms > ============== > A quick definition of terms before I go any further: > > Standard terms from the Unicode standard > =============================== > a compatibility character : an additional encoding of a *normal* > character, for compatibility and round-trip conversion purposes. For > instance, a 1-byte encoding of a Latin character with a diacritic. > > Made-up terms > ============ > a convenience codepoint : a single codepoint which represents an item > that is also encoded as a string of codepoints. > > (I tend to use the terms compatibility character and compatibility > codepoint interchangably. The standard only refers to them as > compatibility characters. However, the standard is determined to > emphasise that characters are abstract and that codepoints are > concrete. So I think it is often more useful and productive to think > of compatibility or convenience codepoints). > > a composed character : a character made up of several codepoints > > Unicode encoding explained > ===================== > A convenience codepoint can therefore be thought of as a code point > used for a character which also has a composed form. > > The way Unicode works is that sometimes you can encode a character in > one byte, sometimes not. Sometimes you can encode it in two bytes, > sometimes not. > > You can therefore have a long stream of ASCII which is single-byte > Unicode. If there is an occasional Cyrillic or Greek character in the > stream, it would be represented either by a compatibility character or > by a multi-byte combination. > > Using compatibility characters can prevent proper sorting and > equivalence testing. > > Using "pure" Unicode, ie. "normal encodings", can cause compatibility > and round-tripping probelms. Although avoiding them can *also* cause > compatibility issues and round-tripping problems. > > Currently my thinking is: > > a Utf8String class > an Ordered collection, with 1 byte characters as the modal element, > but short arrays of wider strings where necessary > a Utf16String class > an Ordered collection, with 2 byte characters as the modal element, > but short arrays of wider strings > beginning with a 2-byte endianness indicator. > > Utf8Strings sometimes need to be sortable, and sometimes need to be compatible. > > So my thinking is that Utf8String will contain convenience codepoints, > for round-tripping. And where there are multiple convenience > codepoints for a character, that it standardises on one. > > And that there is a Utf8SortableString which uses *only* normal characters. > > We then need methods to convert between the two. > > aUtf8String asUtf8SortableString > > and > > aUtf8SortableString asUtf8String > > > Sort orders are culture and context dependent - Sweden and Germany > have different sort orders for the same diacritic-ed characters. Some > countries have one order in general usage, and another for specific > usages, such as phone directories (e.g. UK and France) > > Similarly for Utf16 : Utf16String and Utf16SortableString and > conversion methods > > A list of sorted words would be a SortedCollection, and there could be > pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, > seOrder, ukOrder, etc > > along the lines of > aListOfWords := SortedCollection sortBlock: deOrder > > If a word is either a Utf8SortableString, or a well-formed Utf8String, > then we can perform equivalence testing on them trivially. > > To make sure a Utf8String is well formed, we would need to have a way > of cleaning up any convenience codepoints which were valid, but which > were for a character which has multiple equally-valid alternative > convenience codepoints, and for which the string currently had the > "wrong" convenience codepoint. (i.e for any character with valid > alternative convenience codepoints, we would choose one to be in the > well-formed Utf8String, and we would need a method for cleaning the > alternative convenience codepoints out of the string, and replacing > them with the chosen approved convenience codepoint. > > aUtf8String cleanUtf8String > > With WideString, a lot of the issues disappear - except > round-tripping(although I'm sure I have seen something recently about > 4-byte strings that also have an additional bit. Which would make > some Unicode characters 5-bytes long.) > > > (I'm starting to zone out now - if I've overlooked anything - obvious, > subtle, or somewhere in between, please let me know) > > Cheers, > Euan > Good initiative. Here is some info I've bookmarked over time... http://www.joelonsoftware.com/articles/Unicode.html >> The Single Most Important Fact About Encodings - It does not make sense to have a string without knowing what encoding it uses. You can no longer stick your head in the sand and pretend that "plain" text is ASCII. There Ain't No Such Thing As Plain Text. If you have a string, in memory, in a file, or in an email message, you have to know what encoding it is in or you cannot interpret it or display it to users correctly. http://kunststube.net/encoding/ >> So what does it mean for a language to natively support or not support Unicode? It basically refers to whether the language assumes that one character equals one byte or not. >> What does it mean for a language to support Unicode then? Javascript for example supports Unicode. In fact, any string in Javascript is UTF-16 encoded. In fact, it's the only thing Javascript deals with. You cannot have a string in Javascript that is not UTF-16 encoded. Javascript worships Unicode to the extent that there's no facility to deal with any other encoding in the core language. >> Other languages are simply encoding-aware. Internally they store strings in a particular encoding, http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/ >> 1. I am Unicode, thy character set. Thou shalt have no other character sets before me. >> 4. Thou shalt not refer to Unicode as a two-byte character set. >> 6. Thou shalt count and index Unicode characters, not UTF-16 code points. >> 7. Thou shalt use UTF-8 as the preferred encoding wherever possible. https://xkcd.com/1137/ >> ruomuh https://developer.apple.com/library/prerelease/ios/documentation/Swift/Conc… >> Strings and Characters in Swift http://oleb.net/blog/2014/07/swift-strings/ Interesting article, though I don't yet grok it all, I thought it worth sharing... >> SwiftÊ¼s string implemenation makes working with Unicode easier and significantly less error-prone. >> Strings in Swift are represented by the String type. A String is a collection of Charactervalues. A Swift Character represents one perceived character (what a person thinks of as a single character, called a grapheme). Since Unicode often uses two or more code points(called a grapheme cluster) to form one perceived character, this implies that a Charactercan be composed of multiple Unicode scalar values if they form a single grapheme cluster. (Unicode scalar is the term for any Unicode code point except surrogate pair characters, which are used to encode UTF-16.) >> This change has the potential to prevent many common errors when dealing with string lengths or substrings. It is a huge difference to most other Unicode-aware string libraries (including NSString) where the building blocks of a string are usually UTF-16 code units or single Unicode scalars. cheers -ben

1 0

Re: [Pharo-dev] [Unicode] collation sequences (Re: [squeak-dev] Unicode Support)
by EuanM Dec. 7, 2015

Dec. 7, 2015

Hannes, The Unicode standard provide compatibility codepoints for compatibility purposes and prefers all characters to be represented composed form - as that way they are comparable and sortable. (Some composed characters have *more than one* compatibility codepoint. The canonical example is the composed character #(0041 030a) which can be represented by EITHER the compatibility codepoint #(00c5) "Latin Capital Letter A with Ring" above OR by #(212b) "Angstrom sign" ) On 7 December 2015 at 08:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: > On 12/7/15, EuanM <euanmee(a)gmail.com> wrote: >> My current thinking for collation sequences: >> >> All strings being collated have had all compatibility codepoints >> expanded into composed sequences. > > What does the Unicode manual suggest? (www.unicode.org reference?) > > > >> >> Strings containing composed sequences and UTF >> -8 strings containing multi-byte characters have these represented by >> a very short ordered collection in place of the single Byte of a >> ByteString. >> >> When we sort characters, words or phrases of strings that contain >> zero compatibility codepoints, we simply pull a pre-defined sortBlock >> out of a Dictionary of pre-defined sortBlocks >> >> >> aDictionaryOfSorts at: ukPhoneBook put: aSortBlock >> or >> aDictionaryOfSorts at: ukPhoneBook put: '[aString representing the >> code of a sortBlock]' . >> >> ASortedCollectionOfUtf8Strings sortBlock: aDictionaryOfSorts at: >> ukPhoneBook >> >> - or some actual working code! :-) >> > > Yes, focusing on this is a real need. > > >> >> On 6 December 2015 at 15:14, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>> P.S. The 30-bit value for each character in Squeak/Pharo (if necessary >>> together with an additional language tag) is a potentially very >>> capable infrastructure. Not really used much at the moment. >>> >>> The challenge is to to make _existing_ Unicode-know-how defined >>> elsewhere (e.g. www.unicode.org) available in Squeak/Pharo/Cuis. >>> >>> Most simple cases would be to start with collation sequences in >>> Italian, French, German, Spanish, Portugese. Later on more complex >>> cases like Arabic. >>> >>> --HH >>> >>> On 12/6/15, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>>> Hi Euan, >>>> >>>> On 12/4/15, EuanM <euanmee(a)gmail.com> wrote: >>>>> I'm currently groping my way to seeing how feature-complete our >>>>> Unicode support is. I am doing this to establish what still needs to >>>>> be done to provide full Unicode support. >>>>> >>>>> This seems to me to be an area where it would be best to write it >>>>> once, and then have the same codebase incorporated into the Smalltalks >>>>> that most share a common ancestry. >>>>> >>>>> I am keen to get: equality-testing for strings; sortability for >>>>> strings which have ligatures and diacritic characters; and correct >>>>> round-tripping of data. >>>> >>>> These goals call for a package with SUnit tests which you then can >>>> run on all platforms. This will be a tool to evalutate platforms for >>>> the level of Unicode support. >>>> As mentioned in the thread I would focus on UTF8 only as far as >>>> external files are concerned. >>>> I.E. the test package writes a sample UFT8 file and then reads it to >>>> do the various tests. >>>> I have started doing this for Squeak and Cuis some time ago with a few >>>> tests. >>>> >>>> I am interested in sortability. Round-tripping is fine if you go for >>>> UTF8. >>>> Important of course is which languages you think the package should >>>> work. Some of them are easy, some not. >>>> >>>> This afternoon I did some updates on the Squeak wiki >>>> http://wiki.squeak.org/squeak/recent >>>> >>>> --Hannes >>>> >>>>> >>>>> Call to action: >>>>> ========== >>>>> >>>>> If you have comments on these proposals - such as "but we already have >>>>> that facility" or "the reason we do not have these facilities is >>>>> because they are dog-slow" - please let me know them. >>>>> >>>>> If you would like to help out, please let me know. >>>>> >>>>> If you have Unicode experience and expertise, and would like to be, or >>>>> would be willing to be, in the 'council of experts' for this project, >>>>> please let me know. >>>>> >>>>> If you have comments or ideas on anything mentioned in this email >>>>> >>>>> In the first instance, the initiative's website will be: >>>>> http://smalltalk.uk.to/unicode.html >>>>> >>>>> I have created a SqueakSource.com project called UnicodeSupport >>>>> >>>>> I want to avoid re-inventing any facilities which already exist. >>>>> Except where they prevent us reaching the goals of: >>>>> - sortable UTF8 strings >>>>> - sortable UTF16 strings >>>>> - equivalence testing of 2 UTF8 strings >>>>> - equivalence testing of 2 UTF16 strings >>>>> - round-tripping UTF8 strings through Smalltalk >>>>> - roundtripping UTF16 strings through Smalltalk. >>>>> As I understand it, we have limited Unicode support atm. >>>>> >>>>> Current state of play >>>>> =============== >>>>> ByteString gets converted to WideString when need is automagically >>>>> detected. >>>>> >>>>> Is there anything else that currently exists? >>>>> >>>>> Definition of Terms >>>>> ============== >>>>> A quick definition of terms before I go any further: >>>>> >>>>> Standard terms from the Unicode standard >>>>> =============================== >>>>> a compatibility character : an additional encoding of a *normal* >>>>> character, for compatibility and round-trip conversion purposes. For >>>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>>> >>>>> Made-up terms >>>>> ============ >>>>> a convenience codepoint : a single codepoint which represents an item >>>>> that is also encoded as a string of codepoints. >>>>> >>>>> (I tend to use the terms compatibility character and compatibility >>>>> codepoint interchangably. The standard only refers to them as >>>>> compatibility characters. However, the standard is determined to >>>>> emphasise that characters are abstract and that codepoints are >>>>> concrete. So I think it is often more useful and productive to think >>>>> of compatibility or convenience codepoints). >>>>> >>>>> a composed character : a character made up of several codepoints >>>>> >>>>> Unicode encoding explained >>>>> ===================== >>>>> A convenience codepoint can therefore be thought of as a code point >>>>> used for a character which also has a composed form. >>>>> >>>>> The way Unicode works is that sometimes you can encode a character in >>>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>>> sometimes not. >>>>> >>>>> You can therefore have a long stream of ASCII which is single-byte >>>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>>> stream, it would be represented either by a compatibility character or >>>>> by a multi-byte combination. >>>>> >>>>> Using compatibility characters can prevent proper sorting and >>>>> equivalence testing. >>>>> >>>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>>> and round-tripping probelms. Although avoiding them can *also* cause >>>>> compatibility issues and round-tripping problems. >>>>> >>>>> Currently my thinking is: >>>>> >>>>> a Utf8String class >>>>> an Ordered collection, with 1 byte characters as the modal element, >>>>> but short arrays of wider strings where necessary >>>>> a Utf16String class >>>>> an Ordered collection, with 2 byte characters as the modal element, >>>>> but short arrays of wider strings >>>>> beginning with a 2-byte endianness indicator. >>>>> >>>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>>> compatible. >>>>> >>>>> So my thinking is that Utf8String will contain convenience codepoints, >>>>> for round-tripping. And where there are multiple convenience >>>>> codepoints for a character, that it standardises on one. >>>>> >>>>> And that there is a Utf8SortableString which uses *only* normal >>>>> characters. >>>>> >>>>> We then need methods to convert between the two. >>>>> >>>>> aUtf8String asUtf8SortableString >>>>> >>>>> and >>>>> >>>>> aUtf8SortableString asUtf8String >>>>> >>>>> >>>>> Sort orders are culture and context dependent - Sweden and Germany >>>>> have different sort orders for the same diacritic-ed characters. Some >>>>> countries have one order in general usage, and another for specific >>>>> usages, such as phone directories (e.g. UK and France) >>>>> >>>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>>> conversion methods >>>>> >>>>> A list of sorted words would be a SortedCollection, and there could be >>>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>>> seOrder, ukOrder, etc >>>>> >>>>> along the lines of >>>>> aListOfWords := SortedCollection sortBlock: deOrder >>>>> >>>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>>> then we can perform equivalence testing on them trivially. >>>>> >>>>> To make sure a Utf8String is well formed, we would need to have a way >>>>> of cleaning up any convenience codepoints which were valid, but which >>>>> were for a character which has multiple equally-valid alternative >>>>> convenience codepoints, and for which the string currently had the >>>>> "wrong" convenience codepoint. (i.e for any character with valid >>>>> alternative convenience codepoints, we would choose one to be in the >>>>> well-formed Utf8String, and we would need a method for cleaning the >>>>> alternative convenience codepoints out of the string, and replacing >>>>> them with the chosen approved convenience codepoint. >>>>> >>>>> aUtf8String cleanUtf8String >>>>> >>>>> With WideString, a lot of the issues disappear - except >>>>> round-tripping(although I'm sure I have seen something recently about >>>>> 4-byte strings that also have an additional bit. Which would make >>>>> some Unicode characters 5-bytes long.) >>>>> >>>>> >>>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>>> subtle, or somewhere in between, please let me know) >>>>> >>>>> Cheers, >>>>> Euan >>>>> >>>>> >>>> >>

2 1

Re: [Pharo-dev] [Unicode] Class Unicode --> data from unicode.org
by EuanM Dec. 7, 2015

Dec. 7, 2015

Gnu-Smalltalk also has the classes UnicodeCharacter UnicodeString which I'm having a good look at. On 7 December 2015 at 08:52, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: > there is a class 'Unicode' both in Squeak and Pharo. > > http://wiki.squeak.org/squeak/6225 > > It surely needs attention as it loads the data from www.unicode.org to > initialize the Unicode knowhow of Squeak / Pharo.

4 4

Re: [Pharo-dev] [Unicode] Update of page 'Multilingual Squeak'
by EuanM Dec. 7, 2015

Dec. 7, 2015

Hi Hannes, It's looking like we should be able to conflate a bunch of existing facilities and code. The various Smalltalks seem to have approached Unicode like the old story of the blind men describing an elephant. There are lots of components when are usefully described and dealt with, and if we can just put them all together it'll will be less new work, and we should get a more portable outcome, and a more robust solution. On 7 December 2015 at 08:21, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: > I updated the page > > 'Multilingual Squeak' > http://wiki.squeak.org/squeak/919 > > and some related pages. > > > Many pages have not been touched for 10 years. > I realized that a lot of infrastructure exists but seemingly people > are not aware of it or do not know how to use it. > > > In particular collation sequences are missing though there are > LanguageEnvironments. Pharo being a fork has most of it. It would be > an issue to check how much it differs and fold back or forth necessary > changes. The infrastructure as such seems to be capable. > > --Hannes

1 0

[pharo-project/pharo-core]
by GitHub Dec. 7, 2015

Dec. 7, 2015

Branch: refs/tags/40626 Home: https://github.com/pharo-project/pharo-core

1 0

[pharo-project/pharo-core] 606269: 40626
by GitHub Dec. 7, 2015

Dec. 7, 2015

Branch: refs/heads/4.0 Home: https://github.com/pharo-project/pharo-core Commit: 60626987f9c7151f82a7ad72ab5022fd2f588899 https://github.com/pharo-project/pharo-core/commit/60626987f9c7151f82a7ad72… Author: Jenkins Build Server <board(a)pharo-project.org> Date: 2015-12-07 (Mon, 07 Dec 2015) Changed paths: M Morphic-Base.package/extension/Morph/instance/addHalo_.st M Morphic-Core.package/Morph.class/instance/-- all --/addHalo_.st A MorphicTests.package/MorphTest.class/instance/-- all --/testHaloIsDisable.st A MorphicTests.package/MorphTest.class/instance/test/testHaloIsDisable.st M OpalCompiler-Core.package/IRBytecodeGenerator.class/instance/-- all --/label_.st M OpalCompiler-Core.package/IRBytecodeGenerator.class/instance/-- all --/pushClosureCopyNumCopiedValues_numArgs_to_.st M OpalCompiler-Core.package/IRBytecodeGenerator.class/instance/-- all --/pushNClosureTemps_.st M OpalCompiler-Core.package/IRBytecodeGenerator.class/instance/instructions/label_.st M OpalCompiler-Core.package/IRBytecodeGenerator.class/instance/instructions/pushClosureCopyNumCopiedValues_numArgs_to_.st M OpalCompiler-Core.package/IRBytecodeGenerator.class/instance/instructions/pushNClosureTemps_.st A OpalCompiler-Core.package/IRClosureStackCount.class/README.md A OpalCompiler-Core.package/IRClosureStackCount.class/class/-- all --/newOn_.st A OpalCompiler-Core.package/IRClosureStackCount.class/class/instance creation/newOn_.st A OpalCompiler-Core.package/IRClosureStackCount.class/definition.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/-- all --/initialize.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/-- all --/length.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/-- all --/numMethodTempVars.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/-- all --/numMethodTempVars_.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/accessing/length.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/accessing/numMethodTempVars.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/accessing/numMethodTempVars_.st A OpalCompiler-Core.package/IRClosureStackCount.class/instance/initialization/initialize.st A OpalCompiler-Core.package/IRStackCount.class/class/-- all --/newOn_.st A OpalCompiler-Core.package/IRStackCount.class/class/instance creation/newOn_.st M OpalCompiler-Core.package/IRStackCount.class/instance/-- all --/linkTo_.st M OpalCompiler-Core.package/IRStackCount.class/instance/results/linkTo_.st M ScriptLoader40.package/ScriptLoader.class/instance/-- all --/commentForCurrentUpdate.st R ScriptLoader40.package/ScriptLoader.class/instance/-- all --/script625.st A ScriptLoader40.package/ScriptLoader.class/instance/-- all --/script626.st R ScriptLoader40.package/ScriptLoader.class/instance/-- all --/update40625.st A ScriptLoader40.package/ScriptLoader.class/instance/-- all --/update40626.st R ScriptLoader40.package/ScriptLoader.class/instance/pharo - scripts/script625.st A ScriptLoader40.package/ScriptLoader.class/instance/pharo - scripts/script626.st R ScriptLoader40.package/ScriptLoader.class/instance/pharo - updates/update40625.st A ScriptLoader40.package/ScriptLoader.class/instance/pharo - updates/update40626.st M ScriptLoader40.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st Log Message: ----------- 40626 17057 BackPort Pharo4: 13854 frameSize calculated wrongly for #lineSegmentsDo: https://pharo.fogbugz.com/f/cases/17057 17047 Backport pharo4: 17040 Morph HalosEnabled variable is not use. https://pharo.fogbugz.com/f/cases/17047 http://files.pharo.org/image/40/40626.zip

1 0

Re: [Pharo-dev] [squeak-dev] Unicode Support
by H. Hirzel Dec. 7, 2015

Dec. 7, 2015

On 12/6/15, Levente Uzonyi <leves(a)caesar.elte.hu> wrote: > On Sat, 5 Dec 2015, Colin Putney wrote: > >> First, what's UTF-32? Second, we have the whole language tag thing that >> nobody else uses. > > In Squeak, Strings use UTF-32 encoding[1]. It's straightforward > to see for WideString, but ByteString is just a subset of WideString, so > it uses the same encoding. We also use language tags, but that's a > different story. > Language tags make it possible to work around the problems introduced by > the Han unification[2]. We shouldn't really use them for non-CJKV > languages. > >> >> Finally, UTF-8 is a great encoding that certain kinds of applications >> really ought to use. Web apps, in particular, benefit from using UTF-8 so >> the don't have to decode and then re-encode strings coming in from the >> network. In DabbleDB we used UTF-8 encoded string in the image, and just >> ignored the fact that they were displayed incorrectly by inspectors. >> Having a proper UTF-8 string class would be useful. > > We do the same thing, but that doesn't mean it's a good idea to create a > new String-like class having its content encoded in UTF-8, because > UTF-8-encoded strings can't be modified like regular strings. While it > would be possible to implement all operations, such implementation would > become the next SortedCollection (bad performance due to misuse). This is not the case if you go for ropes https://github.com/KenDickey/Cuis-Smalltalk-Ropes > > Levente > > [1] https://en.wikipedia.org/wiki/UTF-32 > [2] https://en.wikipedia.org/wiki/Han_unification > >> >> - Colin >> >> >>> On Dec 4, 2015, at 6:46 AM, Levente Uzonyi <leves(a)caesar.elte.hu> wrote: >>> >>> Why would you want to have strings with UTF-8 or UTF-16 encoding in the >>> image? >>> What's wrong with the current UTF-32 representation? >>> >>> Levente >>> >>>> On Fri, 4 Dec 2015, EuanM wrote: >>>> >>>> I'm currently groping my way to seeing how feature-complete our >>>> Unicode support is. I am doing this to establish what still needs to >>>> be done to provide full Unicode support. >>>> >>>> This seems to me to be an area where it would be best to write it >>>> once, and then have the same codebase incorporated into the Smalltalks >>>> that most share a common ancestry. >>>> >>>> I am keen to get: equality-testing for strings; sortability for >>>> strings which have ligatures and diacritic characters; and correct >>>> round-tripping of data. >>>> >>>> Call to action: >>>> ========== >>>> >>>> If you have comments on these proposals - such as "but we already have >>>> that facility" or "the reason we do not have these facilities is >>>> because they are dog-slow" - please let me know them. >>>> >>>> If you would like to help out, please let me know. >>>> >>>> If you have Unicode experience and expertise, and would like to be, or >>>> would be willing to be, in the 'council of experts' for this project, >>>> please let me know. >>>> >>>> If you have comments or ideas on anything mentioned in this email >>>> >>>> In the first instance, the initiative's website will be: >>>> http://smalltalk.uk.to/unicode.html >>>> >>>> I have created a SqueakSource.com project called UnicodeSupport >>>> >>>> I want to avoid re-inventing any facilities which already exist. >>>> Except where they prevent us reaching the goals of: >>>> - sortable UTF8 strings >>>> - sortable UTF16 strings >>>> - equivalence testing of 2 UTF8 strings >>>> - equivalence testing of 2 UTF16 strings >>>> - round-tripping UTF8 strings through Smalltalk >>>> - roundtripping UTF16 strings through Smalltalk. >>>> As I understand it, we have limited Unicode support atm. >>>> >>>> Current state of play >>>> =============== >>>> ByteString gets converted to WideString when need is automagically >>>> detected. >>>> >>>> Is there anything else that currently exists? >>>> >>>> Definition of Terms >>>> ============== >>>> A quick definition of terms before I go any further: >>>> >>>> Standard terms from the Unicode standard >>>> =============================== >>>> a compatibility character : an additional encoding of a *normal* >>>> character, for compatibility and round-trip conversion purposes. For >>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>> >>>> Made-up terms >>>> ============ >>>> a convenience codepoint : a single codepoint which represents an item >>>> that is also encoded as a string of codepoints. >>>> >>>> (I tend to use the terms compatibility character and compatibility >>>> codepoint interchangably. The standard only refers to them as >>>> compatibility characters. However, the standard is determined to >>>> emphasise that characters are abstract and that codepoints are >>>> concrete. So I think it is often more useful and productive to think >>>> of compatibility or convenience codepoints). >>>> >>>> a composed character : a character made up of several codepoints >>>> >>>> Unicode encoding explained >>>> ===================== >>>> A convenience codepoint can therefore be thought of as a code point >>>> used for a character which also has a composed form. >>>> >>>> The way Unicode works is that sometimes you can encode a character in >>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>> sometimes not. >>>> >>>> You can therefore have a long stream of ASCII which is single-byte >>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>> stream, it would be represented either by a compatibility character or >>>> by a multi-byte combination. >>>> >>>> Using compatibility characters can prevent proper sorting and >>>> equivalence testing. >>>> >>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>> and round-tripping probelms. Although avoiding them can *also* cause >>>> compatibility issues and round-tripping problems. >>>> >>>> Currently my thinking is: >>>> >>>> a Utf8String class >>>> an Ordered collection, with 1 byte characters as the modal element, >>>> but short arrays of wider strings where necessary >>>> a Utf16String class >>>> an Ordered collection, with 2 byte characters as the modal element, >>>> but short arrays of wider strings >>>> beginning with a 2-byte endianness indicator. >>>> >>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>> compatible. >>>> >>>> So my thinking is that Utf8String will contain convenience codepoints, >>>> for round-tripping. And where there are multiple convenience >>>> codepoints for a character, that it standardises on one. >>>> >>>> And that there is a Utf8SortableString which uses *only* normal >>>> characters. >>>> >>>> We then need methods to convert between the two. >>>> >>>> aUtf8String asUtf8SortableString >>>> >>>> and >>>> >>>> aUtf8SortableString asUtf8String >>>> >>>> >>>> Sort orders are culture and context dependent - Sweden and Germany >>>> have different sort orders for the same diacritic-ed characters. Some >>>> countries have one order in general usage, and another for specific >>>> usages, such as phone directories (e.g. UK and France) >>>> >>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>> conversion methods >>>> >>>> A list of sorted words would be a SortedCollection, and there could be >>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>> seOrder, ukOrder, etc >>>> >>>> along the lines of >>>> aListOfWords := SortedCollection sortBlock: deOrder >>>> >>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>> then we can perform equivalence testing on them trivially. >>>> >>>> To make sure a Utf8String is well formed, we would need to have a way >>>> of cleaning up any convenience codepoints which were valid, but which >>>> were for a character which has multiple equally-valid alternative >>>> convenience codepoints, and for which the string currently had the >>>> "wrong" convenience codepoint. (i.e for any character with valid >>>> alternative convenience codepoints, we would choose one to be in the >>>> well-formed Utf8String, and we would need a method for cleaning the >>>> alternative convenience codepoints out of the string, and replacing >>>> them with the chosen approved convenience codepoint. >>>> >>>> aUtf8String cleanUtf8String >>>> >>>> With WideString, a lot of the issues disappear - except >>>> round-tripping(although I'm sure I have seen something recently about >>>> 4-byte strings that also have an additional bit. Which would make >>>> some Unicode characters 5-bytes long.) >>>> >>>> >>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>> subtle, or somewhere in between, please let me know) >>>> >>>> Cheers, >>>> Euan >>>> >>>> >>> >> >> > >

6 7

Community Building [was: please name your thread]
by EuanM Dec. 7, 2015

Dec. 7, 2015

Tudor, You say that we are talking about different mails. I hope so. "We need to focus on building a great platform." "acting is the only thing that will push us further.". I agree that action is what achieves things. To make sure the correct actions are taken, in a productive and efficient way, it can be valuable to take stock and think, periodically. It is important to carry out a variety of actions, not solely coding. Too narrow a focus on keeping a small group of people coding as fast as they can is not the fastest path to the end goal. It can easily lead to improving the platform at a slower rate than other great platforms are improving - a process of relative decay. I would like to help with community building efforts. But unless the leaders of the community also help and prioritise such efforts, they are unlikely to make any difference. There is no point gaining the interest of new people unless we have mechanisms in place to make them welcome, and put them on the path to becoming valuable contributors to the community. Small communities with low levels of resource must, to succeed, do things more efficiently than large groups with lots of funding. Currently, we lack: 1) debugged beginner's training materials. 2) community processes for getting the interest of new people 3) community processes for onboarding interested new people with the lowest loss rate 4) community processes for providing positive feedback and encouragement to people taking these actions. 5) community processes for stopping bit-rot in mature libraries and facilities - including key facilities like SqueakSource, Smalltalkhub and SqueakSource3 6) community processes to help focus resources on merging libraries into a single best-of-breed best-of-breed mature libraries. We often have two competing incomplete libraries for any given problem space, e.g. Spec/Bloc, MuTalk / SMutant 7) community processes to document the community processes 8) documented processes for bringing newcomers up to speed with "the way things are done round here" We need to take these issues seriously - they affect how fast things get done around here much more than this amount of time away from decoding Pharo-core does Closing down all discussions where these issues surface does not communicate "We welcome any action that anyone would want to undertake in this direction". Quite the reverse. We need to do more than just Smalltalk the Smalltalk - we need to walk the walk. Cheers, Euan Here is the thread I am keeping alive: On 30 November 2015 at 23:32, Tudor Girba <tudor(a)tudorgirba.com> wrote: > In any case, please name you thread appropriately. And, we should remember that acting is the only thing that will push us further. >> On Dec 1, 2015, at 12:18 AM, EuanM <euanmee(a)gmail.com> wrote: >> Welcoming is, as welcoming does. >> On 30 November 2015 at 11:20, Tudor Girba <tudor(a)tudorgirba.com> wrote: >>> Increasing the community is certainly important, and we welcome any action that anyone would want to undertake in this direction. >>>> On Nov 30, 2015, at 4:44 AM, EuanM <euanmee(a)gmail.com> wrote: >>>> We also need to concentrate on building our community. >>>> >>>> We build a better platform faster if we have more people. >>>> >>>> We build a more valuable platform if we have a wider range of valuable >>>> use cases to target. >>>> >>>> Unless and until we hit a critical mass of people joining our >>>> community, we *need* to spend some of our focus on community-building. >>>> >>>> Part of great is being able to build things to sufficient completeness >>>> *and* keep them in working order over the long haul. This is easier >>>> with more contributors. >>>>>> On Nov 27, 2015, at 10:05 PM, EuanM <euanmee(a)gmail.com> wrote: >>>>>> We *should* wish to take advantage of [the rumoured Java] announcement. >>>>>> >>>>>> After all, think what difference having even 0.01% of the world's Java >>>>>> coders moving to Smalltalk would make. How could we help that >>>>>> happen? [Full thread at foot of mail] ==================== On 30 November 2015 at 23:32, Tudor Girba <tudor(a)tudorgirba.com> wrote: > Hi, > > I think we are not talking about the same email. > > In any case, please name you thread appropriately. And, we should remember that acting is the only thing that will push us further. > > Cheers, > Tudor > > >> On Dec 1, 2015, at 12:18 AM, EuanM <euanmee(a)gmail.com> wrote: >> >> Tudor, the email that you replied to, saying the discussion should be >> closed down, was specifically about the impacts on our community and >> how to leverage them. >> >> Welcoming is, as welcoming does. >> >> On 30 November 2015 at 11:20, Tudor Girba <tudor(a)tudorgirba.com> wrote: >>> Hi, >>> >>> Increasing the community is certainly important, and we welcome any action that anyone would want to undertake in this direction. >>> >>> However, talking about the future of Java does not fall in this category. >>> >>> Cheers, >>> Doru >>> >>> >>>> On Nov 30, 2015, at 4:44 AM, EuanM <euanmee(a)gmail.com> wrote: >>>> >>>> We also need to concentrate on building our community. >>>> >>>> We build a better platform faster if we have more people. >>>> >>>> We build a more valuable platform if we have a wider range of valuable >>>> use cases to target. >>>> >>>> Unless and until we hit a critical mass of people joining our >>>> community, we *need* to spend some of our focus on community-building. >>>> >>>> Part of great is being able to build things to sufficient completeness >>>> *and* keep them in working order over the long haul. This is easier >>>> with more contributors. >>>> >>>> On 27 November 2015 at 21:27, Tudor Girba <tudor(a)tudorgirba.com> wrote: >>>>> Hello everyone, >>>>> >>>>> Please stop this thread on this mailing list. We need to focus on building a great platform. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>>>> On Nov 27, 2015, at 10:05 PM, EuanM <euanmee(a)gmail.com> wrote: >>>>>> >>>>>> First of all - is this true? Where can we read about it? >>>>>> >>>>>> I cannot find anything about this at >>>>>> https://www.oracle.com/search/press >>>>>> >>>>>> ======================================= >>>>>> >>>>>> If Oracle did make this statement, then what people have said so far >>>>>> is true. BUT... >>>>>> >>>>>> Java got about 40% of its initial momentum from IBM dumping VisualAge >>>>>> and putting all their resources into Java. >>>>>> >>>>>> Oracle are targetting this move at IBM more than anyone else. >>>>>> >>>>>> IBM will start to think about how to migrate from Java - as Oracle are >>>>>> telling them they will have to. (It's OUR bat and its OUR ball, and >>>>>> no-one else can play with it. Not even the Java Community). And >>>>>> IBM's coders do not pay for Java, Eclipse users do not pay for Java. I >>>>>> expect the licence-fee income for JREs is small. >>>>>> >>>>>> Oracle are doing one of two things - announcing that Java is for sale >>>>>> to device providers - phones (Google is the obvious buyer) or the >>>>>> impending Internet of Things (which was what Java was designed for >>>>>> originally) or announcing that no-one making an internet of things >>>>>> offering should consider Java. >>>>>> >>>>>> Yes, things live on and on in a kind of zombie state. So yes, things >>>>>> live on as long as their ecosystem does. And they gently wither and >>>>>> their ecosystem withers is a long slow drawn out spiral. Which is why >>>>>> we still have Cobol. >>>>>> >>>>>> People and organisations tend to move from one technology to another >>>>>> in an incremental fashion. Swapping a little bit here, and a little >>>>>> bit there. >>>>>> >>>>>> The new target platforms are ones which >>>>>> 1) look like they have longevity, and >>>>>> 2) have a migration pathway that provides incremental steps. >>>>>> >>>>>> Offering a compelling advantage is good - but only if the steps 1) >>>>>> and 2) are catered to. >>>>>> >>>>>> IBM VisualAge Smalltalk is still robust, commercially available >>>>>> software, and VisualStudio and Gemstone continue to represent >>>>>> Smalltalk out to the big world of corporate development. >>>>>> >>>>>> So that's a start. >>>>>> >>>>>> Say only 5% of the Java world moves away from Java each year, as a >>>>>> result of this announcement. >>>>>> >>>>>> We *should* wish to take advantage of this announcement. >>>>>> >>>>>> After all, think what difference having even 0.01% of the world's Java >>>>>> coders moving to Smalltalk would make. How could we help that >>>>>> happen? >>>>>> >>>>>> Think what it would be like to have thought-leaders like Kent Beck and >>>>>> Ward Cunningham back in the Smalltalk fold. How could we help that >>>>>> happen? >>>>>> >>>>>> Think what it would be like to get back all the universities who moved >>>>>> from teaching OO concepts using Smalltalk into teaching them via Java. >>>>>> We now know almost all the ones using Smalltalk as a teaching language >>>>>> by name. Does anyone know even how many universities teach OO via >>>>>> Java? What would it be like if 5% of those universities moved to >>>>>> Smalltalk each year. How could we help that happen? >>>>>> >>>>>> Next - do we have any big brained thinkers who can see specific ways >>>>>> we can improve interoperation between Java facilities and libraries >>>>>> and the Smalltalks? For the next 12 months, we should work on Java >>>>>> integration, rather than C++ integration. We should identify the >>>>>> three best things for us to do in this regard, and make them polished >>>>>> and compelling. Who is in a position to help that happen? >>>>>> >>>>>> The final way we can take advantage help the maximum number of people >>>>>> find their way to us is to present a united community front to the >>>>>> outside world. In the same way I am both a European and a Scot, we >>>>>> need to be Smalltalkers *and*members of our individual >>>>>> Smalltalk-platform communities. >>>>>> >>>>>> How can we help make that happen? >>>>>> >>>>>> This is not a silver bullet. It's going to cause a long-term trend in >>>>>> events, not a sudden abrupt change. But it will have a real, if >>>>>> gradual effect. (assuming that >>>>>> >>>>>> Equally, it is not something we should ignore. It is something we >>>>>> should make use of. We need to put effort into raising our profile >>>>>> over the next 6 months. >>>>>>

2 1

model vs interactionModel in rubric
by Andrei Chis Dec. 6, 2015

Dec. 6, 2015

Hi, RubScrolledTextModel fails sometimes if the model does not have an interaction model set. For example in a few places there is a check that verifies if the model understands a message and then the message is send to the interaction model. evaluate: aStream andDo: aBlock .... (self model respondsTo: #doItReceiver) "check on the model" ifTrue: [ rcvr := self model doItReceiver. ctxt := self model doItContext] And then doItReceiver is implemented like: doItReceiver ^ interactionModel ifNotNil: [ :im | im doItReceiver ] "message send to the interaction model" Also in some cases a condition is checked on the model and then the actual result is obtained from the interaction model: (self model systemNavigation allImplementorsOf: aSelector) isEmpty ifTrue: [ self internalCallToBrowse: aSelector ] ifFalse: [ self model interactionModel systemNavigation browseAllImplementorsOf: aSelector ] I'll say these are bugs. Also should a RubScrolledTextModel always have an interaction model? Cheers, Andrei

2 1

[OT] Microsoft JavaScript engine goes open source
by Torsten Bergmann Dec. 6, 2015

Dec. 6, 2015

https://blogs.windows.com/msedgedev/2015/12/05/open-source-chakra-core/

1 0