December 2015 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [Unicode] collation sequences (Re: [squeak-dev] Unicode Support)

by Henrik Johansen

> On 07 Dec 2015, at 12:18 , EuanM <euanmee(a)gmail.com> wrote: > > Hannes, > > The Unicode standard provide compatibility codepoints for > compatibility purposes and prefers all characters to be represented > composed form - as that way they are comparable and sortable. > > (Some composed characters have *more than one* compatibility codepoint. > > The canonical example is the composed character #(0041 030a) which can > be represented by EITHER the compatibility codepoint #(00c5) "Latin > Capital Letter A with Ring" above OR by #(212b) "Angstrom sign" ) > > On 7 December 2015 at 08:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >> On 12/7/15, EuanM <euanmee(a)gmail.com> wrote: >>> My current thinking for collation sequences: >>> >>> All strings being collated have had all compatibility codepoints >>> expanded into composed sequences. >> >> What does the Unicode manual suggest? (www.unicode.org reference?) >> While nice, it is by no means required to keep/convert the strings to Normalizion form D, as long as the operation returns the same results as if they were (http://unicode.org/reports/tr10/#Main_Algorithm) "Step 1. Produce a normalized form of each input string, applying S1.1. S1.1 <http://unicode.org/reports/tr10/#S1.1> Use the Unicode canonical algorithm to decompose characters according to the canonical mappings. That is, put the string into Normalization Form D (see [UAX15 <http://unicode.org/reports/tr10/#UAX15>]). Conformant implementations may skip this step in certain circumstances, as long as they get the same results. For techniques that may be useful in such an approach, seeSection 6.5, Avoiding Normalization <http://unicode.org/reports/tr10/#Avoiding_Normalization>." So it's a tradeoff between whether you complicate other string operations that create/modify strings to maintain normalized data, or take the hit when collating. Cheers, Henry

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by Henrik Johansen

> On 07 Dec 2015, at 1:05 , EuanM <euanmee(a)gmail.com> wrote: > > Hi Henry, > > To be honest, at some point I'm going to long for the for the much > more succinct semantics of healthcare systems and sports scoring and > administration systems again. :-) > > codepoints are any of *either* > - the representation of a component of an abstract character, *or* > eg. "A" #(0041) as a component of > - the sole representation of the whole of an abstract character *or* of > - a representation of an abstract character provided for backwards > compatibility which is more properly represented by a series of > codepoints representing a composed character > > e.g. > > The "A" #(0041) as a codepoint can be: > the sole representation of the whole of an abstract character "A" #(0041) > > The representation of a component of the composed (i.e. preferred) > version of the abstract character â« #(0041 030a) > > â« (#00C5) represents one valid compatibility form of the abstract > character â« which is most properly represented by #(0041 030a). > > â« (#212b) also represents one valid compatibility form of the abstract > character â« which is most properly represented by #(0041 030a). > > With any luck, this satisfies both our semantic understandings of the > concept of "codepoint" > > Would you agree with that? > > In Unicode, codepoints are *NOT* an abstract numerical representation > of a text character. > > At least not as we generally understand the term "text character" from > our experience of non-Unicode character mappings. I agree, they are numerical representations of what Unicode refers to as characters. > > codepoints represent "*encoded characters*" No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding. > and "a *text element* ... > is represented by a sequence of one or more codepoints". (And the > term "text element" is deliberately left undefined in the Unicode > standard) > > Individual codepoints are very often *not* the encoded form of an > abstract character that we are interested in. Unless we are > communicating to or from another system (Which in some cases is the > Smalltalk ByteString class) > > i.e. in other words > > *Some* individual codepoints *may* be a representation of a specific > *abstract character*, but only in special cases. > > The general case in Unicode is that Unicode defines (a) > representation(s) of a Unicode *abstract character*. > > The Unicode standard representation of an abstract character is a > composed sequence of codepoints, where in some cases that sequence is > as short as 1 codepoint. > > In other cases, Unicode has a compatibility alias of a single > codepoint which is *also* a representation of an abstract character > > There are some cases where an abstract character can be represented by > more than one single-codepoint compatibility codepoint. > > Cheers, > Euan I agree you have a good grasp of the distinction between an abstract character (characters and character sequences which should be treated equivalent wrt, equality / sorting / display, etc.) and a character (which each have a code point assigned). That is besides the point both Sven and I tried to get through, which is the difference between a code point and the encoded form(s) of said code point. When you write: "and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included" I's quite clear you confuse the two. 0xFEFF is the codepoint of the character used as bom. When you state that it can be written ffef (I assume you meant FFFE), you are again confusing the code point and its encoded value (an encoded value which only occurs in UTF16/32, no less). When this distinction is clear, it might be easier to see that value in that Strings are kept as Unicode code points arrays, and converted to encoded forms when entering/exiting the system. Cheers, Henry

Dec. 7, 2015

Re: [Pharo-dev] Help needed: old issues issue tracker entries

by Esteban Lorenzano

> On 07 Dec 2015, at 13:42, Peter Uhnak <i.uhnak(a)gmail.com> wrote: > >> Can *YOU* help to get it to 637? > > Without activity/reply for some time and no clear action --- maybe close > as stale?: > > * https://pharo.fogbugz.com/f/cases/10047/Regex-not-independant-of-platform > * https://pharo.fogbugz.com/f/cases/13479/ValueHolder-and-NewValueHolder-do-n… > * https://pharo.fogbugz.com/f/cases/14500/Manual-class-replacement-unwanted-o… > * https://pharo.fogbugz.com/f/cases/15098/Default-button-size > > How do I push VM change?: > > * https://pharo.fogbugz.com/f/cases/14759/Linux-X11-VM-default-encoding-shoul… Iâlll review and accept it itâs been there from some time but I remember I didnât merge at the time for some reasonâ¦ now I just forget what was (shame on me, I didnât take a note :( )â¦ so I will re-review it :) Esteban > > -- > Peter >

Dec. 7, 2015

Re: [Pharo-dev] Help needed: old issues issue tracker entries

by Peter Uhnak

> Can *YOU* help to get it to 637? Without activity/reply for some time and no clear action --- maybe close as stale?: * https://pharo.fogbugz.com/f/cases/10047/Regex-not-independant-of-platform * https://pharo.fogbugz.com/f/cases/13479/ValueHolder-and-NewValueHolder-do-n… * https://pharo.fogbugz.com/f/cases/14500/Manual-class-replacement-unwanted-o… * https://pharo.fogbugz.com/f/cases/15098/Default-button-size How do I push VM change?: * https://pharo.fogbugz.com/f/cases/14759/Linux-X11-VM-default-encoding-shoul… -- Peter

Dec. 7, 2015

Re: [Pharo-dev] [squeak-dev] Re: [Unicode] Class Unicode --> data from unicode.org

by EuanM

Thanks for the heads up - I'll be sure o stay out of their code and only work with Gnu0-Smalltalk to the extent that we can polyfilla over them. I'll contact Paolo, although his last statement pointed to: http://smalltalk.gnu.org/faq/32 "Does copyright restrict the use of GNU Smalltalk? By Paolo Bonzini - Posted on June 20th, 2007 It does, but in general you don't need to worry much about the licensing of your Smalltalk programs. The class library is licensed under the GNU LGPL. Thus, it permits using the library together with non-free programs. The virtual machine and the bindings to external libraries are licensed under the GNU GPL, with a special exception that allows Smalltalk programs to be linked with the functions exported by the virtual machine and said bindings." and "whatever happens Smalltalk programs are clear and only covered by the GPL." (http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-January/123790.…) The LGPL states "You may convey a Combined Work [i.e. a work containing both LGPL'd source code and non-LGPL's source code] under ***terms of your choice*** that, taken together, effectively do not restrict modification of the portions of the [LGPL'd code] contained in the Combined Work". If we use LGPL'd Smalltalk from Gnu-Smalltalk, we may not close the LGPL'd portion of the code. Was anyone planning on closing the Unicode parts of Cuis/Pharo/Squeak? It *is* an issue with Dolphin's .exe facility, which is one of my own personal use-cases. But for Pharo/Squeak/Cuis it seems to be a non-issue. If anyone has concrete evidence to the contrary, please raise it now. (IANAL but I can read contracts and do risk management of legal issues). If anyone continues to have long-term concerns, let me know. I can get some qualified legal opinions. Tbh, very little of my work is done in the USA's legal jurisdiction. My own risk is low. On 7 December 2015 at 11:31, Frank Shearar <frank.shearar(a)gmail.com> wrote: > Please take a look at the issues around licencing, where GNU Smalltalk > is concerned: http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-January/123761.… > > I'd suggest talking to Paolo Bonzini before continuing. > > frank > > On 7 December 2015 at 11:22, EuanM <euanmee(a)gmail.com> wrote: >> Gnu-Smalltalk also has the classes >> >> UnicodeCharacter >> UnicodeString >> >> which I'm having a good look at. >> >> On 7 December 2015 at 08:52, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>> there is a class 'Unicode' both in Squeak and Pharo. >>> >>> http://wiki.squeak.org/squeak/6225 >>> >>> It surely needs attention as it loads the data from www.unicode.org to >>> initialize the Unicode knowhow of Squeak / Pharo. >> >

Dec. 7, 2015

Re: [Pharo-dev] Unicode Support

by EuanM

Hi Henry, To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-) codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character e.g. The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041) The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a) â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a). â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a). With any luck, this satisfies both our semantic understandings of the concept of "codepoint" Would you agree with that? In Unicode, codepoints are *NOT* an abstract numerical representation of a text character. At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings. codepoints represent "*encoded characters*" and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard) Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class) i.e. in other words *Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases. The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*. The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint. In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint. Cheers, Euan On 7 December 2015 at 11:11, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: > >> On 07 Dec 2015, at 11:51 , EuanM <euanmee(a)gmail.com> wrote: >> >> And indeed, in principle. >> >> On 7 December 2015 at 10:51, EuanM <euanmee(a)gmail.com> wrote: >>> Verifying assumptions is the key reason why you should documents like >>> this out for review. >>> >>> Sven - >>> >>> I'm confident I understand the use of UTF-8 in principal. > > I can only second Sven's sentiment that you need to better differentiate code points (an abstract numerical representation of a character, where a set of such mappings > define a charset, such as Unicode), and character encoding forms. (which are how code points are represented in bytes by a defined process such as UTF-8, UTF-16 etc). > > I know you'll probably think I'm arguing semantics again, but these are *important* semantics ;) > > Cheers, > Henry

Dec. 7, 2015

Re: [Pharo-dev] [Unicode] Class Unicode --> data from unicode.org

by Damien Cassou

EuanM <euanmee(a)gmail.com> writes: > Gnu-Smalltalk also has the classes > > UnicodeCharacter > UnicodeString > > which I'm having a good look at. pay attention to license issues. AFAIK, Gnu-Smalltalk is under the GPL whereas Pharo is MIT. You may not even get inspired by GPL code to write MIT code. But I'm not a lawyer, I may be wrong. -- Damien Cassou http://damiencassou.seasidehosting.st "Success is the ability to go from one failure to another without losing enthusiasm." --Winston Churchill

Dec. 7, 2015

Quick debugging failed #assert:equals tests (Differ in comment)

by Torsten Bergmann

Hi the #assert:equals method is often used in Test cases: we have 2491 senders in recent Pharo5.0 Latest update: #50487. Anytime I have to debug a failing test using it I find it hard to compare the expected and actual value when it is a string in the debugger panes. Pharo tools come to the rescue because we already have a differ inside the image. I would therefore propose to add the following comment into TAssertable>>#assert:equals: assert: actual equals: expected " DiffModel new showOptions: true; leftText: actual; rightText: expected; contextClass: nil; openWithSpec " ^ self assert: expected = actual description: [self comparingStringBetween: actual and: expected] With this additional comment one can quickly open the diff tool and check the differences in the expected vs. actual string. Directly by evaluating the commented expression in the debugger. What do you think? Should we add it? Are there any other solutions to quicly find the differences? Thanks T.

Dec. 7, 2015

Re: [Pharo-dev] [Pharo-users] Help needed: old issues issue tracker entries

by Marcus Denker

The over 600 number looks scary sometimes.. but just by doing one after the other, we it is actually doable. The proof is that we closed 1587 already for Pharo5â¦ and for Pharo all versions we are at 14295 issues âdoneâ. 638 looks small compared to that. Marcus > On 07 Dec 2015, at 12:30, Tudor Girba <tudor(a)tudorgirba.com> wrote: > > Thanks a lot, Marcus for this encouragements! > > Doru > > >> On Dec 7, 2015, at 6:15 AM, Marcus Denker <marcus.denker(a)inria.fr> wrote: >> >> >>> On 05 Dec 2015, at 19:31, Marcus Denker <marcus.denker(a)inria.fr> wrote: >>> >>> >>>> On 30 Nov 2015, at 13:21, marcus.denker(a)inria.fr wrote: >>>> >>>> Hi, >>>> >>>> The issue tracker has 658 open issues. This number is more or less constant. >>>> >>> >>> down to 642! >> >> Down to 638. Can *YOU* help to get it to 637? >> >> Marcus > > -- > www.tudorgirba.com > > "We cannot reach the flow of things unless we let go." > > > >

Dec. 7, 2015

Re: [Pharo-dev] [squeak-dev] Re: [Unicode] Class Unicode --> data from unicode.org

by Tudor Girba

Indeed, it is important to remember that GNU Smalltalk has an incompatible license with that of Pharo. Doru > On Dec 7, 2015, at 6:31 AM, Frank Shearar <frank.shearar(a)gmail.com> wrote: > > Please take a look at the issues around licencing, where GNU Smalltalk > is concerned: http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-January/123761.… > > I'd suggest talking to Paolo Bonzini before continuing. > > frank > > On 7 December 2015 at 11:22, EuanM <euanmee(a)gmail.com> wrote: >> Gnu-Smalltalk also has the classes >> >> UnicodeCharacter >> UnicodeString >> >> which I'm having a good look at. >> >> On 7 December 2015 at 08:52, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>> there is a class 'Unicode' both in Squeak and Pharo. >>> >>> http://wiki.squeak.org/squeak/6225 >>> >>> It surely needs attention as it loads the data from www.unicode.org to >>> initialize the Unicode knowhow of Squeak / Pharo. >> > -- www.tudorgirba.com "In a world where everything is moving ever faster, one might have better chances to win by moving slower."

Dec. 7, 2015