- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] #sum:, #detectSum:, #sumNumbers: (Max Leske)
by Richard Uttner Dec. 22, 2015

Dec. 22, 2015

2015-12-21 23:48 GMT+01:00 Richard Uttner <richard.uttner(a)gmail.com>: > This is my first contribution to the DevList and I hope you forgive me > that it sounds more like "I see something we dont need" than the other way > round, but there will be more positive posts of mine concering the Unicode > debate (which I will postpone to the New Year in order to be able to > respond quickly). > From a developer's point of view that is dealing with more than one > environment (3 Smalltalk environments including Pharo, where I am a newby) > and a handful of other languages, I dont like to reuse interfaces that are > hard to remember and understand though the task done by them is so easy > that I can write it down in one simple #inject:into-line 10 times faster > than looking it up in the implementors browser. Furthermore, something like > #inject:into: will work forever and never be deprecated! Fullstop. > Next thing that is easily overlooked is the semantics. I am not that good > in mathematics, but I would suppose that summing up does by definition not > need a neutral element - only an implementation that can be written down in > 2 lines needs it. A mathematically correct implementation of summing up > anything (including nil ... see reason below) that implements #+ would be: > > Collection:>>#sum > | undef result | > undef := result := Object new. > self do: [:each | result := result == undef ifTrue: [each] ifFalse: > [result + each]]. > "(Sorry for the incomplete mail .... Gmail in Browser apparently has a keyboard shortcut for sending the mail if caret key is pressed along with spaces ...weird ... :() Continuing with missing return statement for above:" ^result (Why I am using "Object new" instead of nil: I would not recommend implementing #+ in nil, but not prohibit it in a general interface, if somebody needs it in an own extension. Thus relying on nil would spoil this theoretical possibility unnecessarily.) The reasons for writing it down like above are: - no sideeffects with possibile overflows (think of a collection containing one element near maxval that is added twice like in the other proposed solutions, and then subtracted finally) - no need to implement #- in special classes that are no numbers but want to use #sum - no need for a neutral element Main disadvantage of my proposed implementation is the fact that we hardly can distinguish a valid result from an invalid one if we dont use nil. What I think is the main reason for all argueing around this interface is simply: Though it sounds reasonable to define something like #sum for Collection at first sight, in fact its semantics is too blurry - we need to pass the desired result in order to reasonably handle empty collections. For most practicable application cases, this can be done instead by simply injecting 0. So please avoid so many new methods that developers have to study in order to understand their respective limit conditions (and primitives are even worse - how will coercing be done if the result gets large? Will it fail on Fraction elements? ...?) So my personal summary is: No implementation of #sum - variants in Collection due to vague semantics and no real advantage against directly using #inject:into. Best regards, Richard > > 2015-12-21 18:00 GMT+01:00 <pharo-dev-request(a)lists.pharo.org>: > >> Send Pharo-dev mailing list submissions to >> pharo-dev(a)lists.pharo.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://lists.pharo.org/mailman/listinfo/pharo-dev_lists.pharo.org >> or, via email, send a message with subject or body 'help' to >> pharo-dev-request(a)lists.pharo.org >> >> You can reach the person managing the list at >> pharo-dev-owner(a)lists.pharo.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Pharo-dev digest..." >> >> >> Today's Topics: >> >> 1. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Dimitris Chloupis) >> 2. Re: #sum:, #detectSum:, #sumNumbers: (Max Leske) >> 3. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Mariano Martinez Peck) >> 4. Re: #sum:, #detectSum:, #sumNumbers: (Max Leske) >> 5. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Dimitris Chloupis) >> 6. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Thierry Goubier) >> 7. Re: #sum:, #detectSum:, #sumNumbers: (Tudor Girba) >> 8. [pharo-project/pharo-core] (GitHub) >> 9. [pharo-project/pharo-core] e8216b: 50508 (GitHub) >> 10. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Denis Kudriashov) >> 11. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Ben Coman) >> 12. Re: [squeak-dev] [Unicode] Summary (Re: Unicode Support // e >> acute example --> decomposition in Pharo?) (Sven Van Caekenberghe) >> 13. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Thierry Goubier) >> 14. Re: [squeak-dev] [Unicode] Summary (Re: Unicode Support // e >> acute example --> decomposition in Pharo?) (Sven Van Caekenberghe) >> 15. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Robert Withers) >> 16. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (phil(a)highoctane.be) >> 17. Re: [ANN] Multiple Desktop support for Pharo 5 (H. Hirzel) >> 18. No refactoring for equal/hash generation (Denis Kudriashov) >> 19. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Denis Kudriashov) >> 20. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Mariano Martinez Peck) >> 21. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored >> Development Effort (Robert Withers) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sun, 20 Dec 2015 17:17:42 +0000 >> From: Dimitris Chloupis <kilon.alios(a)gmail.com> >> To: Pharo Development List <pharo-dev(a)lists.pharo.org> >> Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium >> Sponsored Development Effort >> Message-ID: >> <CAN88a2F3CNMTZRbdHttwmHzzsxKgs5qZUj= >> uBE4UAM4tAVZcrA(a)mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> how the communication is happening via Fuel , does it save to a fuel file >> or it does it through memory some way ? >> >> On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis <lewis(a)mail.msen.com> >> wrote: >> >> > On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote: >> > > On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: >> > > > Fuel I assume enters here the equation as a data exchange format , >> the >> > > > problem I have with fuel is that its not backward compatible which >> for >> > me >> > > >> > >> > Fuel works well in this case. Version compatibility is not an issue, >> > because >> > when you fork an image/VM the two cooperating images are essentially >> > identical. >> > >> > > Once you have a set of OS processes running Pharo how they talk to >> each >> > other >> > > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, >> > JSONRPC, >> > > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. >> > >> > Any of those would work, but Fuel is a good way to enable all kinds of >> > objects to be copied from one image to another through a stream. In the >> > case of RemoteTask, all of the results of a remote computation can be >> put >> > into a single result object (an array or a dictionary or whatever), and >> > that result can be passed back to the parent image by reading the single >> > object. So - fork a copy of the image, have it do some work, and read >> the >> > result object from a stream. You know you are done when you have read >> > exactly one object. >> > >> > Dave >> > >> > >>

2 1

SecureSession protocol version 1.0 ( was Re: [Vm-dev] Re: SecureSession frame types)
by Robert Withers Dec. 22, 2015

Dec. 22, 2015

I updated with all the feedback and here is a version 1.0 pdf specification. I appreciate any comments or suggestions you may have for this. https://www.dropbox.com/s/ywu9pjxrfvg1hys/FrameTypes.pdf?dl=0 Thank you, Robert On 12/20/2015 10:51 AM, Robert Withers wrote: > More, sorry. I left out the msgSize in the FEC spec. Let me reorder > the tagging, fit the 3 bytes of normal msgSpec into 4bytes and thusly... > > > ---Default non-FEC msgSpec + header + payload layout: > > - 8bytes (X) messageSpecification... > - > - (32bits...) > - 4bits tagging > - 6bits multicastSymbol > - 6bits messageVersion > - 2bits sanguinity > - 6bits headerType "Implies Y header size. NOTE: except for FEC > encodings" > - 8bits unused > - (32bits...) > - 4bytes messageSize (X+Y+Z) (Z bytes = (messageSize - headerSize(Y) - > 8bytes msgSpec (X = 8)) > - > - Ybytes message header > https://www.dropbox.com/s/ywu9pjxrfvg1hys/FrameTypes.pdf?dl=0 > - > - Zbytes message payload > > > > Special FEC coded 64bit <msgSpec + header> + payload layout, for 64bit > alignment: > > - 8bytes (X) message specification... > - > - (32bits...) > - 4bits tagging > - 6bits multicastSymbol > - 6bits messageVersion > - 2bits sanguinity > - 6bits FEC message type "NOTE: this means different layout" > - 2bits rsMode > - 6bits partial blockCount > - (32bits...) > - 4bits blockCount "NOTE: for 1MB encoded/982KB data - > blockCount * blockCodeBytes = Z" > - 20bits messageSize (X+Y+Z) (Z bytes = (messageSize - headerSize(Y) - > 8bytes msgSpec (X = 8)) "NOTE: this can specify 1MB data" > - 8bits primitivePolynomial spec (good for our current rsModes) > - > - Y = 0 > - > - Zbytes-sized payload > > Ok, that's what it is I think, this proposal. > > robert > > > > On 12/20/2015 10:33 AM, Robert Withers wrote: >> ---Default non-FEC msgSpec + header + payload layout: >> >> - 6bits multicastSymbol >> - 2bits sanguinity >> - 6bits messageVersion >> - 6bits headerType "NOTE: except for FEC encodings" >> - 4bytes messageSize >> - Xbytes header >> - >> - (messageSized - headerSize - 7specBytes) Bytes-sized payload >> >> >> Special FEC coded 64bit <msgSpec + header> + payload layout, for >> 64bit alignment: >> >> - (32bits...) >> - 6bits multicastSymbol >> - 2bits sanguinity >> - 6bits messageVersion >> - 6bits FEC message type "NOTE: this means different layout" >> - 2bits rsMode >> - (32bits...) >> - 10bits blockCount "NOTE: for 1MB encoded/982KB data" >> - 20bits messageSize "NOTE: this can specify 1MB data plus >> - 8bits primitivePolynomial spec (good for our current rsModes) >> - 4bits tagging >> - >> - (messageSized - headerSize - 4specBytes) OR (blockCount * rsMode's >> blockCodeBytes) Bytes-sized payload > -- . .. ... ^,^ robert Go Panthers!

1 1

Ephemerons
by Esteban A. Maringolo Dec. 21, 2015

Dec. 21, 2015

Hi All, Elliot ;-) Do we have Ephemerons in Pharo? Will we have them? I'm migrating from VW some code that uses it, and if we're not going to have them I'd like to delete the code or package it separately and hopefully it will work if Ephemerons are added later to Pharo. Regards! Esteban A. Maringolo

2 3

Re: [Pharo-dev] #sum:, #detectSum:, #sumNumbers: (Max Leske)
by Richard Uttner Dec. 21, 2015

Dec. 21, 2015

This is my first contribution to the DevList and I hope you forgive me that it sounds more like "I see something we dont need" than the other way round, but there will be more positive posts of mine concering the Unicode debate (which I will postpone to the New Year in order to be able to respond quickly). >From a developer's point of view that is dealing with more than one environment (3 Smalltalk environments including Pharo, where I am a newby) and a handful of other languages, I dont like to reuse interfaces that are hard to remember and understand though the task done by them is so easy that I can write it down in one simple #inject:into-line 10 times faster than looking it up in the implementors browser. Furthermore, something like #inject:into: will work forever and never be deprecated! Fullstop. Next thing that is easily overlooked is the semantics. I am not that good in mathematics, but I would suppose that summing up does by definition not need a neutral element - only an implementation that can be written down in 2 lines needs it. A mathematically correct implementation of summing up anything (including nil ... see reason below) that implements #+ would be: Collection:>>#sum | undef result | undef := result := Object new. self do: [:each | result := result == undef ifTrue: [each] ifFalse: [result + each]]. 2015-12-21 18:00 GMT+01:00 <pharo-dev-request(a)lists.pharo.org>: > Send Pharo-dev mailing list submissions to > pharo-dev(a)lists.pharo.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.pharo.org/mailman/listinfo/pharo-dev_lists.pharo.org > or, via email, send a message with subject or body 'help' to > pharo-dev-request(a)lists.pharo.org > > You can reach the person managing the list at > pharo-dev-owner(a)lists.pharo.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pharo-dev digest..." > > > Today's Topics: > > 1. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Dimitris Chloupis) > 2. Re: #sum:, #detectSum:, #sumNumbers: (Max Leske) > 3. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Mariano Martinez Peck) > 4. Re: #sum:, #detectSum:, #sumNumbers: (Max Leske) > 5. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Dimitris Chloupis) > 6. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Thierry Goubier) > 7. Re: #sum:, #detectSum:, #sumNumbers: (Tudor Girba) > 8. [pharo-project/pharo-core] (GitHub) > 9. [pharo-project/pharo-core] e8216b: 50508 (GitHub) > 10. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Denis Kudriashov) > 11. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Ben Coman) > 12. Re: [squeak-dev] [Unicode] Summary (Re: Unicode Support // e > acute example --> decomposition in Pharo?) (Sven Van Caekenberghe) > 13. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Thierry Goubier) > 14. Re: [squeak-dev] [Unicode] Summary (Re: Unicode Support // e > acute example --> decomposition in Pharo?) (Sven Van Caekenberghe) > 15. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Robert Withers) > 16. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (phil(a)highoctane.be) > 17. Re: [ANN] Multiple Desktop support for Pharo 5 (H. Hirzel) > 18. No refactoring for equal/hash generation (Denis Kudriashov) > 19. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Denis Kudriashov) > 20. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Mariano Martinez Peck) > 21. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Robert Withers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 20 Dec 2015 17:17:42 +0000 > From: Dimitris Chloupis <kilon.alios(a)gmail.com> > To: Pharo Development List <pharo-dev(a)lists.pharo.org> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAN88a2F3CNMTZRbdHttwmHzzsxKgs5qZUj= > uBE4UAM4tAVZcrA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > how the communication is happening via Fuel , does it save to a fuel file > or it does it through memory some way ? > > On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis <lewis(a)mail.msen.com> > wrote: > > > On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote: > > > On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: > > > > Fuel I assume enters here the equation as a data exchange format , > the > > > > problem I have with fuel is that its not backward compatible which > for > > me > > > > > > > Fuel works well in this case. Version compatibility is not an issue, > > because > > when you fork an image/VM the two cooperating images are essentially > > identical. > > > > > Once you have a set of OS processes running Pharo how they talk to each > > other > > > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, > > JSONRPC, > > > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. > > > > Any of those would work, but Fuel is a good way to enable all kinds of > > objects to be copied from one image to another through a stream. In the > > case of RemoteTask, all of the results of a remote computation can be put > > into a single result object (an array or a dictionary or whatever), and > > that result can be passed back to the parent image by reading the single > > object. So - fork a copy of the image, have it do some work, and read the > > result object from a stream. You know you are done when you have read > > exactly one object. > > > > Dave > > > > >

2 1

Re: [Pharo-dev] [Vm-dev] VM Maker: VMMaker.oscog-eem.1609.mcz
by Eliot Miranda Dec. 21, 2015

Dec. 21, 2015

Hi All, the real Cog 64-bit Spur x64 VM just evaluated 3+4 correctly on Mac OS X: â Early days yet. This is a debug VM. The optimised VM does not yet display the prompt. But we can be confident that a 64-bit JIT for Spur will be available some time in January. â On Thu, Dec 17, 2015 at 10:42 AM, <commits(a)source.squeak.org> wrote: > > Eliot Miranda uploaded a new version of VMMaker to project VM Maker: > http://source.squeak.org/VMMaker/VMMaker.oscog-eem.1609.mcz > > ==================== Summary ==================== > > Name: VMMaker.oscog-eem.1609 > Author: eem > Time: 17 December 2015, 10:41:49.192 am > UUID: 771cdc25-5d27-4818-8943-2ba5c0e31791 > Ancestors: VMMaker.oscog-eem.1608 > > Cogit: Fix the (arguably bogus) register save/restore code for safe > trampolines, including the ceScheduleScavenge call. > > =============== Diff against VMMaker.oscog-eem.1608 =============== > > Item was added: > + ----- Method: CogARMCompiler>>genSaveRegForCCall (in category 'abi') > ----- > + genSaveRegForCCall > + "Save the general purpose registers for a call into the C run-time > from a trampoline." > + "Save none, because the ARM ABI only defines callee saved > registers, no caller-saved regs." > + "cogit gen: STMFD operand: 16r7F"! > > Item was removed: > - ----- Method: CogARMCompiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - "Save none, because the ARM ABI only defines callee saved > registers, no caller-saved regs." > - "cogit gen: STMFD operand: 16r7F"! > > Item was added: > + ----- Method: CogARMCompiler>>genSaveRegsForCCall (in category 'abi') > ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a trampoline call." > + "Save none, because the ARM ABI only defines callee saved > registers, no caller-saved regs." > + "cogit gen: STMFD operand: 16r7F" > + self flag: 'this will change with Sista when we hope to be able to > allocate arbitrary registers'! > > Item was removed: > - ----- Method: CogAbstractInstruction>>genSaveRegisters (in category > 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - self subclassResponsibility! > > Item was changed: > ----- Method: CogIA32Compiler>>genRestoreRegsExcept: (in category 'abi') > ----- > genRestoreRegsExcept: abstractReg > | realReg | > realReg := self concreteRegister: abstractReg. > self assert: (EDI > EAX and: [EDI - EAX + 1 = 6]). > EAX to: EDI do: > [:reg| > + (reg between: ESP and: EBP) ifFalse: > + [realReg = reg > + ifTrue: [cogit AddCq: 4 R: ESP] > + ifFalse: [cogit PopR: reg]]]. > - realReg = reg ifTrue: [cogit AddCq: 4 R: ESP] ifFalse: > [cogit PopR: reg]]. > ^0! > > Item was removed: > - ----- Method: CogIA32Compiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - > - self assert: (EDI > EAX and: [EDI - EAX + 1 = 8]). > - EDI to: EAX by: -1 do: > - [:reg| > - (reg between: ESP and: EBP) ifFalse: > - [cogit PushR: reg]]. > - ^0! > > Item was added: > + ----- Method: CogIA32Compiler>>genSaveRegsForCCall (in category 'abi') > ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a call into the C run-time > from a trampoline." > + > + self assert: (EDI > EAX and: [EDI - EAX + 1 = 8]). > + EDI to: EAX by: -1 do: > + [:reg| > + (reg between: ESP and: EBP) ifFalse: > + [cogit PushR: reg]]. > + ^0! > > Item was removed: > - ----- Method: CogMIPSELCompiler>>genSaveRegisters (in category 'abi') > ----- > - genSaveRegisters > - "This method is poorly named. Is this for a Smalltalk -> C call or > C -> Smalltalk call? > - If the former we don't need to do anything because all of the > abstract registers are > - allocated to C preserved registers." > - self flag: #bogus.! > > Item was added: > + ----- Method: CogMIPSELCompiler>>genSaveRegsForCCall (in category 'abi') > ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a call into the C run-time > from a trampoline. > + We don't need to do anything because all of the abstract > registers are > + allocated to C preserved registers." > + self flag: 'this will change with Sista when we hope to be able to > allocate arbitrary registers'! > > Item was changed: > ----- Method: CogX64Compiler>>genRestoreRegs (in category 'abi') ----- > genRestoreRegs > "Restore the general purpose registers for a trampoline call. > c.f. genSaveRegisters" > + RAX to: R15 do: > + [:reg| > + (reg between: RSP and: RBP) ifFalse: > + [cogit PopR: reg]]. > - cogit > - PopR: RAX; > - PopR: RBX; > - PopR: RCX; > - PopR: RDX; > - PopR: RSI; > - PopR: RDI; > - PopR: R8; > - PopR: R9; > - PopR: R10; > - PopR: R11; > - PopR: R12; > - PopR: R13; > - PopR: R14; > - PopR: R15. > ^0! > > Item was changed: > ----- Method: CogX64Compiler>>genRestoreRegsExcept: (in category 'abi') > ----- > genRestoreRegsExcept: abstractReg > | realReg | > realReg := self concreteRegister: abstractReg. > self assert: (R15 > RAX and: [R15 - RAX + 1 = 16]). > RAX to: R15 do: > [:reg| > + (reg between: RSP and: RBP) ifFalse: > + [realReg = reg > + ifTrue: [cogit AddCq: 8 R: RSP] > + ifFalse: [cogit PopR: reg]]]. > - realReg = reg ifTrue: [cogit AddCq: 4 R: RSP] ifFalse: > [cogit PopR: reg]]. > ^0! > > Item was removed: > - ----- Method: CogX64Compiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - > - self assert: (R15 > RAX and: [R15 - RAX + 1 = 16]). > - R15 to: RAX by: -1 do: [:reg| cogit PushR: reg]. > - ^0! > > Item was added: > + ----- Method: CogX64Compiler>>genSaveRegsForCCall (in category 'abi') > ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a trampoline call." > + > + self assert: (R15 > RAX and: [R15 - RAX + 1 = 16]). > + R15 to: RAX by: -1 do: > + [:reg| > + (reg between: RSP and: RBP) ifFalse: > + [cogit PushR: reg]]. > + ^0! > > Item was changed: > ----- Method: > Cogit>>compileCallFor:numArgs:arg:arg:arg:arg:resultReg:saveRegs: (in > category 'initialization') ----- > compileCallFor: aRoutine numArgs: numArgs arg: regOrConst0 arg: > regOrConst1 arg: regOrConst2 arg: regOrConst3 resultReg: resultRegOrNone > saveRegs: saveRegs > "Generate a call to aRoutine with up to 4 arguments. If > resultRegOrNone is not > NoReg assign the C result to resultRegOrNone. If saveRegs, save > all registers. > Hack: a negative arg value indicates an abstract register, a > non-negative value > indicates a constant." > <var: #aRoutine type: #'void *'> > <inline: false> > cStackAlignment > objectMemory wordSize ifTrue: > [backEnd > genAlignCStackSavingRegisters: saveRegs > numArgs: numArgs > wordAlignment: cStackAlignment / objectMemory > wordSize]. > saveRegs ifTrue: > + [backEnd genSaveRegsForCCall]. > - [backEnd genSaveRegisters]. > backEnd genMarshallNArgs: numArgs arg: regOrConst0 arg: > regOrConst1 arg: regOrConst2 arg: regOrConst3. > self CallFullRT: (self cCode: [aRoutine asUnsignedInteger] > inSmalltalk: [self > simulatedTrampolineFor: aRoutine]). > resultRegOrNone ~= NoReg ifTrue: > [backEnd genWriteCResultIntoReg: resultRegOrNone]. > saveRegs ifTrue: > [numArgs > 0 ifTrue: > [backEnd genRemoveNArgsFromStack: numArgs]. > resultRegOrNone ~= NoReg > ifTrue: [backEnd genRestoreRegsExcept: > resultRegOrNone] > ifFalse: [backEnd genRestoreRegs]]! > > -- _,,,^..^,,,_ best, Eliot

8 9

in Togo teaching Pharo with Stef
by Esteban Lorenzano Dec. 21, 2015

Dec. 21, 2015

Hi guys, just letting you know that Iâm not working until January 4th because Iâm in Togo, helping Stef to teach Pharoâ¦ so the spur problems will need to wait a bit to be solved :) cheers, Esteban

3 2

[ANN] Multiple Desktop support for Pharo 5
by Torsten Bergmann Dec. 21, 2015

Dec. 21, 2015

Julien Delplanque provided this week a goodie to switch between "desktops" - but his initial solution was more or less hiding windows and not really switching between real Pharo worlds/desktops. I gave him some tips what could be done on the pharo-user list. Havent heard from him afterwards. Now I was able to spend a few hours on this topic myself and implemented a full multiple desktop solution myself. This works in Pharo 5 only (currently) and requires latest VM (at least on Windows) from files.pharo.org to get the keyboard shortcuts right. To try: Gofer new smalltalkhubUser: 'TorstenBergmann' project: 'DesktopManager'; configuration; loadDevelopment. The goodie has some nice features like keyboard navigation, world menu integration and even a custom spotter with preview of the desktops. Quick start: =========== - evaluate the above expression in a Pharo 5 image - check the world menu "Desktop" - you can press CTRL + D and then CTRL + A (= Desktop Add) to add a new desktop - you can press CTRL + D and then CTRL + D (= Desktop Desktop) to open the overview - you can press CTRL + D and then CTRL + P (= Desktop Previous) to navigate to the previous desktop - you can press CTRL + D and then CTRL + N (= Desktop Next) to navigate to the next desktop The code is hopefully a good example on how to build a custom spotter, shortcuts, inspector extensions, ... Additionally all this is described in a new article including screenshots and a guide on how to use this new goodie: https://medium.com/@astares/multiple-desktops-for-pharo-5cbc46f3179f Actually the article took more time to write than coding but I hope it helps explaining Pharo and why it is a power tool. Project is on http://www.smalltalkhub.com/#!/~TorstenBergmann/DesktopManager Article and code is still subject to change. Have fun T.

4 6

No refactoring for equal/hash generation
by Denis Kudriashov Dec. 21, 2015

Dec. 21, 2015

Hi. There is no menu item in Nautilus to generate equal/hash methods for class. But there is RBGenerateEqualHashRefactoring class which can do that. And I remember that in past we had such feature from the browser. Should we restore it? I can submit issue for this.

1 0

[Unicode] Summary (Re: Unicode Support // e acute example --> decomposition in Pharo?)
by H. Hirzel Dec. 21, 2015

Dec. 21, 2015

Hello Sven Thank you for your report about about your experimental, proof of concept, prototype project, that aims to improve Unicode support. Please include me in the loop. Below is is my attempt at summarizing the Unicode discussion of the last weeks. Corrections /comments / additions are welcome. Kind regards Hannes 1) There is a need for improved Unicode support implemented _within_ the image , probably as a library. 1a) This follows the example of the the Twitter CLDR library (i.e. re-implementation of ICU components for Ruby). https://github.com/twitter/twitter-cldr-rb Other languages/libraries have similar approaches - dotNet, https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeIn…) - Python https://docs.python.org/3/howto/unicode.html - Go http://blog.golang.org/strings - Swift, https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… - Perl http://blog.golang.org/strings 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This is because of security and portability reasons (Eliot Miranda) and because of the Smalltalk approach that wants to expose basic algorithms in Smalltalk code. In addition the 16bit based ICU library does not fit well with the Squeak/Pharo UTF32 model. 2) The Unicode infrastructure (21(32) bit wide Characters as immediate objects, use of UTF-32 internally, indexable strings, UTF8 for outside communication, support of code converters) is a very valuable foundation which makes algorithms more straightforward at the expense of a more memory usage. It not used to its full potential at all currently though a lot of hard work has been done. 3) The Unicode algorithms are mostly table / database driven. This means that dictionary lookup is a prominent part of the algorithms. The essential building block for this is that the Unicode character database UCD (http://www.unicode.org/ucd/) is made available _within_ the image with the full content as needed by the target languages / scripts one wants to deal with. The process of loading the UCD should be made configurable. 3a) a lot of people are interested in the Latin script (and scripts of similar complexity) only. 3b) The UCD data in XML form http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with and without the CJK characters. 4) The next step is to implement normalization (http://www.unicode.org/reports/tr15/#Norm_Forms) Glad to read that you have reached results here with the test data: http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. 5) Pharo offers nice inspectors to view dictionaries and ordered collections (table view, drill down) which facilitates the development to table driven algorithms. The data structures and algorithm are do not depend on a particular dialect though and may be ported to Squeak or Cuis. 6) After having implemented normalization, comparison may be implemented. This needs CLDR access (collation, Unicode Common Locale Data Repository, http://cldr.unicode.org/ ). 7) An architecture has the following subsystems 7a) Basic character handling (21(32)bit characters in indexable strings, point 2) 7b) Runtime access to the Unicode Character Database (point 3) 7c) Converters 7d) Normalization (point 4) 7e) CLDR access (point 6) 8) The implementation should be driven by the current needs. An attainable next goal is to release 8a) a StringBuilder utility class for easier construction of test strings i.e. instead of > normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > String). do normalizer composeString: (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') and construct some test cases with it which illustrate some basic Unicode issues. 8b) identity testing for major languages (e.g. French, German, Spanish) and scripts of similar complexity. I 8c) to provide some more documentation of past and concurrent efforts. Note: This summary has only covered string manipulation, not rendering on the screen which is a different issue. On 12/16/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Hannes, > > My detailed comments/answers below, after quoting 2 of your emails: > >> On 10 Dec 2015, at 22:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >> >> Hello Sven >> >> On 12/9/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >>> The simplest example in a common language is (the French letter Ã©) is >>> >>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] >>> >>> which can also be written as >>> >>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT >>> [U+0301] >>> >>> The former being a composed normal form, the latter a decomposed normal >>> form. (And yes, it is even much more complicated than that, it goes on >>> for >>> 1000s of pages). >>> >>> In the above example, the concept of character/string is indeed fuzzy. >>> >>> HTH, >>> >>> Sven >> >> Thanks for this example. I have created a wiki page with it >> >> I wonder what the Pharo equivalent is of the following Squeak expression >> >> $Ã© asString asDecomposedUnicode >> >> Regards >> >> Hannes > > You also wrote: > >> The text below shows how to deal with the Unicode e acute example >> brought up by Sven in terms of comparing strings. Currently Pharo and >> Cuis do not do Normalization of strings. Limited support is in Squeak. >> It will be shown how NFD normalization may be implemented. >> >> >> Swift programming language >> ----------------------------------------- >> >> How does the Swift programming language [1] deal with Unicode strings? >> >> // "Voulez-vous un cafÃ©?" using LATIN SMALL LETTER E WITH ACUTE >> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" >> >> // "Voulez-vous un café?" using LATIN SMALL LETTER E and >> COMBINING ACUTE ACCENT >> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" >> >> if eAcuteQuestion == combinedEAcuteQuestion { >> print("These two strings are considered equal") >> } >> // prints "These two strings are considered equal" >> >> The equality operator uses the NFD (Normalization Form Decomposed)[2] >> form for the comparison appyling a method >> #decomposedStringWithCanonicalMapping[3] >> >> >> Squeak / Pharo >> ----------------------- >> >> Comparison without NFD [3] >> >> >> "Voulez-vous un cafÃ©?" >> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >> asString, '?'. >> >> >> eAcuteQuestion = combinedEAcuteQuestion >> false >> >> eAcuteQuestion == combinedEAcuteQuestion >> false >> >> The result is false. A Unicode conformant application however should >> return *true*. >> >> Reason for this is that Squeak / Pharo strings are not put into NFD >> before testing for equality = >> >> >> Squeak Unicode strings may be tested for Unicode conformant equality >> by converting them to NFD before testing. >> >> >> >> Squeak using NFD >> >> asDecomposedUnicode[4] transforms a string into NFD for cases where a >> Unicode code point if decomposed, is decomposed only to two code >> points [5]. This is so because when initializing [6] the Unicode >> Character Database in Squeak this is a limitation imposed by the code >> which reads UnicodeData.txt [7][8]. This is not a necessary >> limitation. The code may be rewritten at the price of a more complex >> implementation of #asDecomposedUnicode. >> >> "Voulez-vous un cafÃ©?" >> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >> asString, '?'. >> >> >> eAcuteQuestion asDecomposedUnicode = >> combinedEAcuteQuestion asDecomposedUnicode >> >> true >> >> >> >> Conclusion >> ------------------ >> >> Implementing a method like #decomposedStringWithCanonicalMapping >> (swift) which puts a string into NFD (Normalization Form D) is an >> important building block towards better Unicode compliance. A Squeak >> proposal is given by [4]. It needs to be reviewed.extended. >> >> It should probably be extended for cases where there are more than >> two code points in the decomposed form (3 or more?) >> >> The implementing of NFD comparison gives us an equality test for a >> comparatively small effort for simple cases covering a large number of >> use cases (Languages using the Latin script). >> >> The algorithm is table driven by the UCD [8]. From this follows an >> simple but important fact for conformant implementations need runtime >> access to information from the Unicode Character Database [UCD][9]. >> >> >> [1] >> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… >> [2] http://www.unicode.org/glossary/#normalization_form_d >> [3] >> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Found… >> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 >> [5] http://www.unicode.org/glossary/#code_point >> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 >> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt >> [8] Unicode Character Database documentation >> http://unicode.org/reports/tr44/ >> [9] http://www.unicode.org/reports/tr23/ > > > Today, we have a Unicode and CombinedCharacter class in Pharo, and there is > different but similar Unicode code in Squeak. These are too simple (even > though they might work, partially). > > The scope of the original threads is way too wide: a new string type, > normalisation, collation, being cross dialect, mixing all kinds of character > and encoding definitions. All interesting, but not much will come out of it. > But the point that we cannot leave proper text string handling to an outside > library is indeed key. > > That is why a couple of people in the Pharo community (myself included) > started an experimental, proof of concept, prototype project, that aims to > improve Unicode support. We will announce it to a wider public when we feel > we have something to show for. The goal is in the first place to understand > and implement the fundamental algorithms, starting with the 4 forms of > Normalisation. But we're working on collation/sorting too. > > This work is of course being done for/in Pharo, using some of the facilities > only available there. It probably won't be difficult to port, but we can't > be bothered with probability right now. > > What we started with is loading UCD data and making it available as a nice > objects (30.000 of them). > > So now you can do things like > > $Ã© unicodeCharacterData. > > => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" > > $Ã© unicodeCharacterData uppercase asCharacter. > > => "$Ã" > > $Ã© unicodeCharacterData decompositionMapping. > > => "#(101 769)" > > There is also a cool GT Inspector view: > > > > Next we started implementing a normaliser. It was rather easy to get support > for simpler languages going. The next code snippets use explicit code > arrays, because copying decomposed diacritics to my mail client does not > work (they get automatically composed), in a Pharo Workspace this does work > nicely with plain strings. The higher numbers are the diacritics. > > (normalizer decomposeString: 'les Ã©lÃ¨ves FranÃ§ais') collect: #codePoint as: > Array. > > => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 > 807 97 105 115)" > > (normalizer decomposeString: 'DÃ¼sseldorf KÃ¶nigsallee') collect: #codePoint > as: Array. > > => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 > 115 97 108 108 101 101)" > > normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 > 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). > > => "'les Ã©lÃ¨ves FranÃ§ais'" > > normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > String). > > => "'DÃ¼sseldorf KÃ¶nigsallee'" > > However, the real algorithm following the official specification (and other > elements of Unicode that interact with it) is way more complicated (think > about all those special languages/scripts out there). We're focused on > understanding/implementing that now. > > Next, unit tests were added (of course). As well as a test that uses > http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > 75.000 individual test cases to check conformance to the official Unicode > Normalization specification. > > Right now (with super cool hangul / jamo code by Henrik), we hit the > following stats: > > #testNFC 16998/18593 (91.42%) > #testNFD 16797/18593 (90.34%) > #testNFKC 13321/18593 (71.65%) > #testNFKD 16564/18593 (89.09%) > > Way better than the naive implementations, but not yet there. > > We are also experimenting and thinking a lot about how to best implement all > this, trying out different models/ideas/apis/representations. > > It will move slowly, but you will hear from us again in the coming > weeks/months. > > Sven > > PS: Pharo developers with a good understanding of this subject area that > want to help, let me know and we'll put you in the loop. Hacking and > specification reading are required ;-) > >

2 1

Re: [Pharo-dev] [squeak-dev] [Unicode] Summary (Re: Unicode Support // e acute example --> decomposition in Pharo?)
by Sven Van Caekenberghe Dec. 21, 2015

Dec. 21, 2015

Andres, There are no plans at all to drop any of the existing character encodings from Pharo. UTF-16 LE & BE will remain part of the standard image, as are all single byte encodings. No need to worry. Sven > On 19 Dec 2015, at 03:04, Andres Valloud <avalloud(a)smalltalk.comcastbiz.net> wrote: > > So a lot of Windows APIs require UTF-16. What's up with UTF-8 being the only choice mentioned for external communication? > > Unicode string encodings like UTF-* and strings of "characters" (that is, sequences of Unicode code points) should be clearly distinguished. Do you really mean "UTF-32", or do you mean "UCS-4"? Even those two are not exactly the same. > > On 12/18/15 5:47 , H. Hirzel wrote: >> Hello Sven >> >> Thank you for your report about about your experimental, proof of >> concept, prototype project, that aims to improve Unicode support. >> Please include me in the loop. >> >> Below is is my attempt at summarizing the Unicode discussion of the last weeks. >> Corrections /comments / additions are welcome. >> >> Kind regards >> >> Hannes >> >> >> 1) There is a need for improved Unicode support implemented _within_ >> the image , probably as a library. >> >> 1a) This follows the example of the the Twitter CLDR library (i.e. >> re-implementation of ICU components for Ruby). >> https://github.com/twitter/twitter-cldr-rb >> >> Other languages/libraries have similar approaches >> - dotNet, https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeIn…) >> - Python https://docs.python.org/3/howto/unicode.html >> - Go http://blog.golang.org/strings >> - Swift, https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… >> - Perl http://blog.golang.org/strings >> >> 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This >> is because of security and portability reasons (Eliot Miranda) and >> because of the Smalltalk approach that wants to expose basic >> algorithms in Smalltalk code. In addition the 16bit based ICU library >> does not fit well with the Squeak/Pharo UTF32 model. >> >> 2) The Unicode infrastructure (21(32) bit wide Characters as immediate >> objects, use of UTF-32 internally, indexable strings, UTF8 for outside >> communication, support of code converters) is a very valuable >> foundation which makes algorithms more straightforward at the expense >> of a more memory usage. It not used to its full potential at all >> currently though a lot of hard work has been done. >> >> 3) The Unicode algorithms are mostly table / database driven. This >> means that dictionary lookup is a prominent part of the algorithms. >> The essential building block for this is that the Unicode character >> database UCD (http://www.unicode.org/ucd/) is made available >> _within_ the image with the full content as needed by the target >> languages / scripts one wants to deal with. The process of loading the >> UCD should be made configurable. >> >> 3a) a lot of people are interested in the Latin script (and scripts of >> similar complexity) only. >> 3b) The UCD data in XML form >> http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with >> and without the CJK characters. >> >> 4) The next step is to implement normalization >> (http://www.unicode.org/reports/tr15/#Norm_Forms) Glad to read that >> you have reached results here with the test data: >> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. >> >> 5) Pharo offers nice inspectors to view dictionaries and ordered >> collections (table view, drill down) which facilitates the development >> to table driven algorithms. The data structures and algorithm are do >> not depend on a particular dialect though and may be ported to Squeak >> or Cuis. >> >> 6) After having implemented normalization, comparison may be >> implemented. This needs CLDR access (collation, Unicode Common Locale >> Data Repository, http://cldr.unicode.org/ ). >> >> >> 7) An architecture has the following subsystems >> >> 7a) Basic character handling (21(32)bit characters in indexable >> strings, point 2) >> 7b) Runtime access to the Unicode Character Database (point 3) >> 7c) Converters >> 7d) Normalization (point 4) >> 7e) CLDR access (point 6) >> >> >> 8) The implementation should be driven by the current needs. >> >> An attainable next goal is to release >> >> 8a) a StringBuilder utility class for easier construction of test strings >> i.e. instead of >> >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: >>> String). >> >> do >> normalizer composeString: >> (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') >> >> and construct some test cases with it which illustrate some basic >> Unicode issues. >> >> 8b) identity testing for major languages (e.g. French, German, >> Spanish) and scripts of similar complexity. I >> >> 8c) to provide some more documentation of past and concurrent efforts. >> >> Note: This summary has only covered string manipulation, not rendering >> on the screen which is a different issue. >> >> >> On 12/16/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> Hi Hannes, >>> >>> My detailed comments/answers below, after quoting 2 of your emails: >>> >>>> On 10 Dec 2015, at 22:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>>> >>>> Hello Sven >>>> >>>> On 12/9/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>> >>>>> The simplest example in a common language is (the French letter Ã©) is >>>>> >>>>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] >>>>> >>>>> which can also be written as >>>>> >>>>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT >>>>> [U+0301] >>>>> >>>>> The former being a composed normal form, the latter a decomposed normal >>>>> form. (And yes, it is even much more complicated than that, it goes on >>>>> for >>>>> 1000s of pages). >>>>> >>>>> In the above example, the concept of character/string is indeed fuzzy. >>>>> >>>>> HTH, >>>>> >>>>> Sven >>>> >>>> Thanks for this example. I have created a wiki page with it >>>> >>>> I wonder what the Pharo equivalent is of the following Squeak expression >>>> >>>> $Ã© asString asDecomposedUnicode >>>> >>>> Regards >>>> >>>> Hannes >>> >>> You also wrote: >>> >>>> The text below shows how to deal with the Unicode e acute example >>>> brought up by Sven in terms of comparing strings. Currently Pharo and >>>> Cuis do not do Normalization of strings. Limited support is in Squeak. >>>> It will be shown how NFD normalization may be implemented. >>>> >>>> >>>> Swift programming language >>>> ----------------------------------------- >>>> >>>> How does the Swift programming language [1] deal with Unicode strings? >>>> >>>> // "Voulez-vous un cafÃ©?" using LATIN SMALL LETTER E WITH ACUTE >>>> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" >>>> >>>> // "Voulez-vous un café?" using LATIN SMALL LETTER E and >>>> COMBINING ACUTE ACCENT >>>> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" >>>> >>>> if eAcuteQuestion == combinedEAcuteQuestion { >>>> print("These two strings are considered equal") >>>> } >>>> // prints "These two strings are considered equal" >>>> >>>> The equality operator uses the NFD (Normalization Form Decomposed)[2] >>>> form for the comparison appyling a method >>>> #decomposedStringWithCanonicalMapping[3] >>>> >>>> >>>> Squeak / Pharo >>>> ----------------------- >>>> >>>> Comparison without NFD [3] >>>> >>>> >>>> "Voulez-vous un cafÃ©?" >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >>>> asString, '?'. >>>> >>>> >>>> eAcuteQuestion = combinedEAcuteQuestion >>>> false >>>> >>>> eAcuteQuestion == combinedEAcuteQuestion >>>> false >>>> >>>> The result is false. A Unicode conformant application however should >>>> return *true*. >>>> >>>> Reason for this is that Squeak / Pharo strings are not put into NFD >>>> before testing for equality = >>>> >>>> >>>> Squeak Unicode strings may be tested for Unicode conformant equality >>>> by converting them to NFD before testing. >>>> >>>> >>>> >>>> Squeak using NFD >>>> >>>> asDecomposedUnicode[4] transforms a string into NFD for cases where a >>>> Unicode code point if decomposed, is decomposed only to two code >>>> points [5]. This is so because when initializing [6] the Unicode >>>> Character Database in Squeak this is a limitation imposed by the code >>>> which reads UnicodeData.txt [7][8]. This is not a necessary >>>> limitation. The code may be rewritten at the price of a more complex >>>> implementation of #asDecomposedUnicode. >>>> >>>> "Voulez-vous un cafÃ©?" >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >>>> asString, '?'. >>>> >>>> >>>> eAcuteQuestion asDecomposedUnicode = >>>> combinedEAcuteQuestion asDecomposedUnicode >>>> >>>> true >>>> >>>> >>>> >>>> Conclusion >>>> ------------------ >>>> >>>> Implementing a method like #decomposedStringWithCanonicalMapping >>>> (swift) which puts a string into NFD (Normalization Form D) is an >>>> important building block towards better Unicode compliance. A Squeak >>>> proposal is given by [4]. It needs to be reviewed.extended. >>>> >>>> It should probably be extended for cases where there are more than >>>> two code points in the decomposed form (3 or more?) >>>> >>>> The implementing of NFD comparison gives us an equality test for a >>>> comparatively small effort for simple cases covering a large number of >>>> use cases (Languages using the Latin script). >>>> >>>> The algorithm is table driven by the UCD [8]. From this follows an >>>> simple but important fact for conformant implementations need runtime >>>> access to information from the Unicode Character Database [UCD][9]. >>>> >>>> >>>> [1] >>>> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… >>>> [2] http://www.unicode.org/glossary/#normalization_form_d >>>> [3] >>>> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Found… >>>> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 >>>> [5] http://www.unicode.org/glossary/#code_point >>>> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 >>>> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt >>>> [8] Unicode Character Database documentation >>>> http://unicode.org/reports/tr44/ >>>> [9] http://www.unicode.org/reports/tr23/ >>> >>> >>> Today, we have a Unicode and CombinedCharacter class in Pharo, and there is >>> different but similar Unicode code in Squeak. These are too simple (even >>> though they might work, partially). >>> >>> The scope of the original threads is way too wide: a new string type, >>> normalisation, collation, being cross dialect, mixing all kinds of character >>> and encoding definitions. All interesting, but not much will come out of it. >>> But the point that we cannot leave proper text string handling to an outside >>> library is indeed key. >>> >>> That is why a couple of people in the Pharo community (myself included) >>> started an experimental, proof of concept, prototype project, that aims to >>> improve Unicode support. We will announce it to a wider public when we feel >>> we have something to show for. The goal is in the first place to understand >>> and implement the fundamental algorithms, starting with the 4 forms of >>> Normalisation. But we're working on collation/sorting too. >>> >>> This work is of course being done for/in Pharo, using some of the facilities >>> only available there. It probably won't be difficult to port, but we can't >>> be bothered with probability right now. >>> >>> What we started with is loading UCD data and making it available as a nice >>> objects (30.000 of them). >>> >>> So now you can do things like >>> >>> $Ã© unicodeCharacterData. >>> >>> => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" >>> >>> $Ã© unicodeCharacterData uppercase asCharacter. >>> >>> => "$Ã" >>> >>> $Ã© unicodeCharacterData decompositionMapping. >>> >>> => "#(101 769)" >>> >>> There is also a cool GT Inspector view: >>> >>> >>> >>> Next we started implementing a normaliser. It was rather easy to get support >>> for simpler languages going. The next code snippets use explicit code >>> arrays, because copying decomposed diacritics to my mail client does not >>> work (they get automatically composed), in a Pharo Workspace this does work >>> nicely with plain strings. The higher numbers are the diacritics. >>> >>> (normalizer decomposeString: 'les Ã©lÃ¨ves FranÃ§ais') collect: #codePoint as: >>> Array. >>> >>> => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 >>> 807 97 105 115)" >>> >>> (normalizer decomposeString: 'DÃ¼sseldorf KÃ¶nigsallee') collect: #codePoint >>> as: Array. >>> >>> => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 >>> 115 97 108 108 101 101)" >>> >>> normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 >>> 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). >>> >>> => "'les Ã©lÃ¨ves FranÃ§ais'" >>> >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: >>> String). >>> >>> => "'DÃ¼sseldorf KÃ¶nigsallee'" >>> >>> However, the real algorithm following the official specification (and other >>> elements of Unicode that interact with it) is way more complicated (think >>> about all those special languages/scripts out there). We're focused on >>> understanding/implementing that now. >>> >>> Next, unit tests were added (of course). As well as a test that uses >>> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about >>> 75.000 individual test cases to check conformance to the official Unicode >>> Normalization specification. >>> >>> Right now (with super cool hangul / jamo code by Henrik), we hit the >>> following stats: >>> >>> #testNFC 16998/18593 (91.42%) >>> #testNFD 16797/18593 (90.34%) >>> #testNFKC 13321/18593 (71.65%) >>> #testNFKD 16564/18593 (89.09%) >>> >>> Way better than the naive implementations, but not yet there. >>> >>> We are also experimenting and thinking a lot about how to best implement all >>> this, trying out different models/ideas/apis/representations. >>> >>> It will move slowly, but you will hear from us again in the coming >>> weeks/months. >>> >>> Sven >>> >>> PS: Pharo developers with a good understanding of this subject area that >>> want to help, let me know and we'll put you in the loop. Hacking and >>> specification reading are required ;-) >>> >>> >> >> . >> > On 19 Dec 2015, at 03:04, Andres Valloud <avalloud(a)smalltalk.comcastbiz.net> wrote: > > So a lot of Windows APIs require UTF-16. What's up with UTF-8 being the only choice mentioned for external communication? > > Unicode string encodings like UTF-* and strings of "characters" (that is, sequences of Unicode code points) should be clearly distinguished. Do you really mean "UTF-32", or do you mean "UCS-4"? Even those two are not exactly the same. > > On 12/18/15 5:47 , H. Hirzel wrote: >> Hello Sven >> >> Thank you for your report about about your experimental, proof of >> concept, prototype project, that aims to improve Unicode support. >> Please include me in the loop. >> >> Below is is my attempt at summarizing the Unicode discussion of the last weeks. >> Corrections /comments / additions are welcome. >> >> Kind regards >> >> Hannes >> >> >> 1) There is a need for improved Unicode support implemented _within_ >> the image , probably as a library. >> >> 1a) This follows the example of the the Twitter CLDR library (i.e. >> re-implementation of ICU components for Ruby). >> https://github.com/twitter/twitter-cldr-rb >> >> Other languages/libraries have similar approaches >> - dotNet, https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeIn…) >> - Python https://docs.python.org/3/howto/unicode.html >> - Go http://blog.golang.org/strings >> - Swift, https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… >> - Perl http://blog.golang.org/strings >> >> 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This >> is because of security and portability reasons (Eliot Miranda) and >> because of the Smalltalk approach that wants to expose basic >> algorithms in Smalltalk code. In addition the 16bit based ICU library >> does not fit well with the Squeak/Pharo UTF32 model. >> >> 2) The Unicode infrastructure (21(32) bit wide Characters as immediate >> objects, use of UTF-32 internally, indexable strings, UTF8 for outside >> communication, support of code converters) is a very valuable >> foundation which makes algorithms more straightforward at the expense >> of a more memory usage. It not used to its full potential at all >> currently though a lot of hard work has been done. >> >> 3) The Unicode algorithms are mostly table / database driven. This >> means that dictionary lookup is a prominent part of the algorithms. >> The essential building block for this is that the Unicode character >> database UCD (http://www.unicode.org/ucd/) is made available >> _within_ the image with the full content as needed by the target >> languages / scripts one wants to deal with. The process of loading the >> UCD should be made configurable. >> >> 3a) a lot of people are interested in the Latin script (and scripts of >> similar complexity) only. >> 3b) The UCD data in XML form >> http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with >> and without the CJK characters. >> >> 4) The next step is to implement normalization >> (http://www.unicode.org/reports/tr15/#Norm_Forms) Glad to read that >> you have reached results here with the test data: >> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. >> >> 5) Pharo offers nice inspectors to view dictionaries and ordered >> collections (table view, drill down) which facilitates the development >> to table driven algorithms. The data structures and algorithm are do >> not depend on a particular dialect though and may be ported to Squeak >> or Cuis. >> >> 6) After having implemented normalization, comparison may be >> implemented. This needs CLDR access (collation, Unicode Common Locale >> Data Repository, http://cldr.unicode.org/ ). >> >> >> 7) An architecture has the following subsystems >> >> 7a) Basic character handling (21(32)bit characters in indexable >> strings, point 2) >> 7b) Runtime access to the Unicode Character Database (point 3) >> 7c) Converters >> 7d) Normalization (point 4) >> 7e) CLDR access (point 6) >> >> >> 8) The implementation should be driven by the current needs. >> >> An attainable next goal is to release >> >> 8a) a StringBuilder utility class for easier construction of test strings >> i.e. instead of >> >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: >>> String). >> >> do >> normalizer composeString: >> (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') >> >> and construct some test cases with it which illustrate some basic >> Unicode issues. >> >> 8b) identity testing for major languages (e.g. French, German, >> Spanish) and scripts of similar complexity. I >> >> 8c) to provide some more documentation of past and concurrent efforts. >> >> Note: This summary has only covered string manipulation, not rendering >> on the screen which is a different issue. >> >> >> On 12/16/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> Hi Hannes, >>> >>> My detailed comments/answers below, after quoting 2 of your emails: >>> >>>> On 10 Dec 2015, at 22:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>>> >>>> Hello Sven >>>> >>>> On 12/9/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>> >>>>> The simplest example in a common language is (the French letter Ã©) is >>>>> >>>>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] >>>>> >>>>> which can also be written as >>>>> >>>>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT >>>>> [U+0301] >>>>> >>>>> The former being a composed normal form, the latter a decomposed normal >>>>> form. (And yes, it is even much more complicated than that, it goes on >>>>> for >>>>> 1000s of pages). >>>>> >>>>> In the above example, the concept of character/string is indeed fuzzy. >>>>> >>>>> HTH, >>>>> >>>>> Sven >>>> >>>> Thanks for this example. I have created a wiki page with it >>>> >>>> I wonder what the Pharo equivalent is of the following Squeak expression >>>> >>>> $Ã© asString asDecomposedUnicode >>>> >>>> Regards >>>> >>>> Hannes >>> >>> You also wrote: >>> >>>> The text below shows how to deal with the Unicode e acute example >>>> brought up by Sven in terms of comparing strings. Currently Pharo and >>>> Cuis do not do Normalization of strings. Limited support is in Squeak. >>>> It will be shown how NFD normalization may be implemented. >>>> >>>> >>>> Swift programming language >>>> ----------------------------------------- >>>> >>>> How does the Swift programming language [1] deal with Unicode strings? >>>> >>>> // "Voulez-vous un cafÃ©?" using LATIN SMALL LETTER E WITH ACUTE >>>> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" >>>> >>>> // "Voulez-vous un café?" using LATIN SMALL LETTER E and >>>> COMBINING ACUTE ACCENT >>>> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" >>>> >>>> if eAcuteQuestion == combinedEAcuteQuestion { >>>> print("These two strings are considered equal") >>>> } >>>> // prints "These two strings are considered equal" >>>> >>>> The equality operator uses the NFD (Normalization Form Decomposed)[2] >>>> form for the comparison appyling a method >>>> #decomposedStringWithCanonicalMapping[3] >>>> >>>> >>>> Squeak / Pharo >>>> ----------------------- >>>> >>>> Comparison without NFD [3] >>>> >>>> >>>> "Voulez-vous un cafÃ©?" >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >>>> asString, '?'. >>>> >>>> >>>> eAcuteQuestion = combinedEAcuteQuestion >>>> false >>>> >>>> eAcuteQuestion == combinedEAcuteQuestion >>>> false >>>> >>>> The result is false. A Unicode conformant application however should >>>> return *true*. >>>> >>>> Reason for this is that Squeak / Pharo strings are not put into NFD >>>> before testing for equality = >>>> >>>> >>>> Squeak Unicode strings may be tested for Unicode conformant equality >>>> by converting them to NFD before testing. >>>> >>>> >>>> >>>> Squeak using NFD >>>> >>>> asDecomposedUnicode[4] transforms a string into NFD for cases where a >>>> Unicode code point if decomposed, is decomposed only to two code >>>> points [5]. This is so because when initializing [6] the Unicode >>>> Character Database in Squeak this is a limitation imposed by the code >>>> which reads UnicodeData.txt [7][8]. This is not a necessary >>>> limitation. The code may be rewritten at the price of a more complex >>>> implementation of #asDecomposedUnicode. >>>> >>>> "Voulez-vous un cafÃ©?" >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >>>> asString, '?'. >>>> >>>> >>>> eAcuteQuestion asDecomposedUnicode = >>>> combinedEAcuteQuestion asDecomposedUnicode >>>> >>>> true >>>> >>>> >>>> >>>> Conclusion >>>> ------------------ >>>> >>>> Implementing a method like #decomposedStringWithCanonicalMapping >>>> (swift) which puts a string into NFD (Normalization Form D) is an >>>> important building block towards better Unicode compliance. A Squeak >>>> proposal is given by [4]. It needs to be reviewed.extended. >>>> >>>> It should probably be extended for cases where there are more than >>>> two code points in the decomposed form (3 or more?) >>>> >>>> The implementing of NFD comparison gives us an equality test for a >>>> comparatively small effort for simple cases covering a large number of >>>> use cases (Languages using the Latin script). >>>> >>>> The algorithm is table driven by the UCD [8]. From this follows an >>>> simple but important fact for conformant implementations need runtime >>>> access to information from the Unicode Character Database [UCD][9]. >>>> >>>> >>>> [1] >>>> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… >>>> [2] http://www.unicode.org/glossary/#normalization_form_d >>>> [3] >>>> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Found… >>>> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 >>>> [5] http://www.unicode.org/glossary/#code_point >>>> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 >>>> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt >>>> [8] Unicode Character Database documentation >>>> http://unicode.org/reports/tr44/ >>>> [9] http://www.unicode.org/reports/tr23/ >>> >>> >>> Today, we have a Unicode and CombinedCharacter class in Pharo, and there is >>> different but similar Unicode code in Squeak. These are too simple (even >>> though they might work, partially). >>> >>> The scope of the original threads is way too wide: a new string type, >>> normalisation, collation, being cross dialect, mixing all kinds of character >>> and encoding definitions. All interesting, but not much will come out of it. >>> But the point that we cannot leave proper text string handling to an outside >>> library is indeed key. >>> >>> That is why a couple of people in the Pharo community (myself included) >>> started an experimental, proof of concept, prototype project, that aims to >>> improve Unicode support. We will announce it to a wider public when we feel >>> we have something to show for. The goal is in the first place to understand >>> and implement the fundamental algorithms, starting with the 4 forms of >>> Normalisation. But we're working on collation/sorting too. >>> >>> This work is of course being done for/in Pharo, using some of the facilities >>> only available there. It probably won't be difficult to port, but we can't >>> be bothered with probability right now. >>> >>> What we started with is loading UCD data and making it available as a nice >>> objects (30.000 of them). >>> >>> So now you can do things like >>> >>> $Ã© unicodeCharacterData. >>> >>> => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" >>> >>> $Ã© unicodeCharacterData uppercase asCharacter. >>> >>> => "$Ã" >>> >>> $Ã© unicodeCharacterData decompositionMapping. >>> >>> => "#(101 769)" >>> >>> There is also a cool GT Inspector view: >>> >>> >>> >>> Next we started implementing a normaliser. It was rather easy to get support >>> for simpler languages going. The next code snippets use explicit code >>> arrays, because copying decomposed diacritics to my mail client does not >>> work (they get automatically composed), in a Pharo Workspace this does work >>> nicely with plain strings. The higher numbers are the diacritics. >>> >>> (normalizer decomposeString: 'les Ã©lÃ¨ves FranÃ§ais') collect: #codePoint as: >>> Array. >>> >>> => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 >>> 807 97 105 115)" >>> >>> (normalizer decomposeString: 'DÃ¼sseldorf KÃ¶nigsallee') collect: #codePoint >>> as: Array. >>> >>> => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 >>> 115 97 108 108 101 101)" >>> >>> normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 >>> 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). >>> >>> => "'les Ã©lÃ¨ves FranÃ§ais'" >>> >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: >>> String). >>> >>> => "'DÃ¼sseldorf KÃ¶nigsallee'" >>> >>> However, the real algorithm following the official specification (and other >>> elements of Unicode that interact with it) is way more complicated (think >>> about all those special languages/scripts out there). We're focused on >>> understanding/implementing that now. >>> >>> Next, unit tests were added (of course). As well as a test that uses >>> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about >>> 75.000 individual test cases to check conformance to the official Unicode >>> Normalization specification. >>> >>> Right now (with super cool hangul / jamo code by Henrik), we hit the >>> following stats: >>> >>> #testNFC 16998/18593 (91.42%) >>> #testNFD 16797/18593 (90.34%) >>> #testNFKC 13321/18593 (71.65%) >>> #testNFKD 16564/18593 (89.09%) >>> >>> Way better than the naive implementations, but not yet there. >>> >>> We are also experimenting and thinking a lot about how to best implement all >>> this, trying out different models/ideas/apis/representations. >>> >>> It will move slowly, but you will hear from us again in the coming >>> weeks/months. >>> >>> Sven >>> >>> PS: Pharo developers with a good understanding of this subject area that >>> want to help, let me know and we'll put you in the loop. Hacking and >>> specification reading are required ;-) >>> >>> >> >> . >>

1 0