- Pharo-users - lists.pharo.org

Re: [Pharo-users] Class name with diacritic character and Pharo

by Sven Van Caekenberghe

https://github.com/pharo-project/pharo/issues/2395 > On 27 Jan 2019, at 17:03, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi Dominique, > >> On 27 Jan 2019, at 11:40, Dominique Dartois <dom(a)dartois.org> wrote: >> >> Hello all. >> If a use french diacritic character in a class name, the code runs but I canât fileout the package nor save it with Monticello. >> For example, the C cedilla in the class name drive me to an âZnInvalidUTF8:Illegal byte for utf-8 encoding' when filing out. >> >> Is it a bug or a feature? >> Thank you >> >> <image.png>--- >> Dominique Dartois > > Thanks for reporting this. This is most definitely a bug, I can confirm its occurrence. > > I'm CC pharo-dev as this is quite important. This will be a long mail. > > > This is one manifestation of a problem that has been present for quite a while. > > I'll start by describing what I did, what went well and where/how this fails, some generic points, and two conceptual solutions (that need further verification). > > Like you, I created a new subclass: > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage' > > With comment: > > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > > And two methods (in the 'test' protocol): > > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve' > > euro > ^ 'â¬' > > I added the euro sign (because that is encoded in UTF-8 with 3 bytes, not 2 like Ã§). > Like you said, the system can cope with such class and method names and seems to function fine. > > Looking at the .changes file, the correct source code was appended: > > ----SNAPSHOT----2019-01-26T23:36:18.548555+01:00 work.image priorSource: 339848! > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage'! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:25' prior: 0! > I am ClasseFranÃ§aise.! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:26'! > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve'! ! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:27' prior: 33898360! > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > ! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:27'! > euro > ^ 'â¬'! ! > > > Doing a file out (or otherwise saving the source code) fails. The reason is an incorrect manipulation of this source file while looking for what is called the method preamble, in SourcFileArray>>#getPreambleFrom:at: position > > An programmatic way to invoke the same error is by doing > > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) timeStamp. > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) author. > > Both fail with the same error. > > > The source code of methods is (currently) stored in a .sources or .changes file. CompiledMethods know their source pointer, an offset in one of these files. Right before the place where the source starts is a preamble that contains some meta information (including the author and timestamp). To access that preamble, the source code pointer is moved backwards to the beginning of the preamble (which begins and ends with a !). > > > The current approach fails in the presence of non-ASCII characters. More specifically because of a mixup between the concept of byte position and character position when using UTF-8, a variable length encoding (both the .changes and the .sources are UTF-8 encoded). > > For example, consider > > 'Ã partir de 10 â¬' size. "16" > 'Ã partir de 10 â¬' utf8Encoded size. "19" > > So although the string contains 16 characters, it is encoded as 19 bytes, Ã using 2 bytes and â¬ using 3 bytes. In general, moving backwards or forwards in UTF-8 encoded bytes cannot be done without understanding UTF-8 itself. > > ZnUTF8Encoder can do both (moving forward is #nextFromStream: while moving backwards is #backOnStream:). However, ZnUTF8Encoder is also strict: it will signal an error when forced to operate in between encoded characters, which is what happens here. > > It is thus not possible to move to arbitrary bytes positions and assume/hope to always arrive on the correct character boundaries and it is also wrong to take the difference between two byte positions as the count of characters present (since their encoding is of variable length). > > SourcFileArray>>#getPreambleFrom:at: is doing both of these wrong (but gets away with it in 99.99% of all cases since very few people name their classes like that). > > There are two solutions: operate mostly on the byte level or operate correctly on the character level. Here are two conceptual solutions (you must execute either solution 1 or 2, not both), with two different inputs. > > > src := '!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 83" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 83. "at start of euro, the methods source string" > str upToEnd. > > str position: (83 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Same code, different input (and starting position): > > > src := '!ABC!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 87" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 87. "at start of euro, the methods source string" > str upToEnd. > > str position: (87 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Solution 1 (at the byte level) works because $! (ASCII code 33) is also encoded as such in UTF-8 and this byte pattern cannot be part of a 2, 3 or 4 byte UTF-8 encoded character. The final byte segment must then be given to the proper encoder to turn it into characters. > > Solution 2 (at the character level) works because it essentially counts characters while moving backwards correctly (it moves back 2 steps each time because #next moves 1 step forward, while #peek would not help). > > Note that in Solution 1 the moving position is held in an external variable, while in Solution 2 it is held inside the stream. > > > Implementing either solution requires more internal access of the streams (#wrappedStream), so SourcFileArray>>#getPreambleFrom:at: should at least be moved to SourceFile, IMHO. > > Obviously this is a sensitive/critical area to touch. > > We will also need a regression unit test. > > On a higher design level, I think that CompiledMethod>>#timeStamp and CompiledMethod>>#author should be cached at the instance level (a unix timestamp and a symbol would cost very little). > > Sven > > > > > >

Jan. 29, 2019

Re: [Pharo-users] Fwd: startup error in windows 10

by Ben Coman

To provide the best chance for someone to help, can you provide: * base Image version * vm version * systeminfo.exe (stripped of personally identifying data) * with dates added to hotfixes from ( https://www.tenforums.com/tutorials/5472-view-windows-update-history-window… ) * video info from (wmic path win32_VideoController get /all /format:htable >> c:\temp\graphics.html) cheers -ben On Tue, 29 Jan 2019 at 00:28, Hilaire <hilaire(a)drgeo.eu> wrote: > > Hi, > > Any idea? It seems related to an error calling Cairo library. > > Hilaire > > > > -------- Message transfÃ©rÃ© -------- > Sujet : startup error in windows 10 > Date : Sun, 27 Jan 2019 23:53:36 -0700 (MST) > De : d_koticha [via Dr. Geo Forum] <ml+s996172n4025532h46(a)n3.nabble.com> > Pour : Hilaire Fernandes <hfern(a)free.fr> > > > I Installed Dr Geo and used it few times before November 2018. > after updating windows 10 in January 2019 the app doesn't display drawing area. it loads but shows a red screen with some code and a cross. screen shot is attached. > error.jpg

Jan. 28, 2019

Fwd: startup error in windows 10

by Hilaire

Hi, Any idea? It seems related to an error calling Cairo library. Hilaire -------- Message transfÃ©rÃ© -------- SujetÂ : startup error in windows 10 DateÂ : Sun, 27 Jan 2019 23:53:36 -0700 (MST) DeÂ : d_koticha [via Dr. Geo Forum] <ml+s996172n4025532h46(a)n3.nabble.com> PourÂ : Hilaire Fernandes <hfern(a)free.fr> I Installed Dr Geo and used it few times before November 2018. after updating windows 10 in January 2019 the app doesn't display drawing area. it loads but shows a red screen with some code and a cross. screen shot is attached. error.jpg <http://forum.drgeo.eu/file/n4025532/error.jpg> ------------------------------------------------------------------------ If you reply to this email, your message will be added to the discussion below: http://forum.drgeo.eu/startup-error-in-windows-10-tp4025532.html To start a new topic under Dr. Geo users, email ml+s996172n4023913h57(a)n3.nabble.com To unsubscribe from Dr. Geo Forum, click here <http://forum.drgeo.eu/template/NamlServlet.jtp?macro=unsubscribe_by_code&no…>. NAML <http://forum.drgeo.eu/template/NamlServlet.jtp?macro=macro_viewer&id=instan…>

Jan. 28, 2019

Re: [Pharo-users] What's wrong?

by Ben Coman

Its not something that I had a clue about. thanks for the update. cheers -ben On Mon, 28 Jan 2019 at 16:40, eftomi <tomaz.turk(a)ef.uni-lj.si> wrote: > OK, I solved the problem with a complete wipe out of Pharo from the system > and fresh install. Pharo stored the run-time information in a separate > image > folder even if I launched the vm and image without PharoLauncher, directly > from the console. > > Fresh install approach solved the other problem, too > ( > http://forum.world.st/PrimitiveFailed-primitive-createDirectory-in-WindowsS…) > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html > >

Jan. 28, 2019

Re: [Pharo-users] What's wrong?

by eftomi

OK, I solved the problem with a complete wipe out of Pharo from the system and fresh install. Pharo stored the run-time information in a separate image folder even if I launched the vm and image without PharoLauncher, directly from the console. Fresh install approach solved the other problem, too (http://forum.world.st/PrimitiveFailed-primitive-createDirectory-in-WindowsS…) -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Jan. 28, 2019

Re: [Pharo-users] Class name with diacritic character and Pharo

by Sven Van Caekenberghe

Yer, author and classname are both part of the preamble, so both are capable of breaking it. Same problem, same solution. > On 28 Jan 2019, at 05:10, Benoit St-Jean <bstjean(a)yahoo.com> wrote: > > Sorry, my toothache meds are kicking in! lol > > Correction to my last post: > > I mean *fileouts" don't work when author name has diacritic French characters! > > Obviously, couldn't test filing in! > > > On 2019-01-27 11:03, Sven Van Caekenberghe wrote: >> Hi Dominique, >> >>> On 27 Jan 2019, at 11:40, Dominique Dartois <dom(a)dartois.org> wrote: >>> >>> Hello all. >>> If a use french diacritic character in a class name, the code runs but I canât fileout the package nor save it with Monticello. >>> For example, the C cedilla in the class name drive me to an âZnInvalidUTF8:Illegal byte for utf-8 encoding' when filing out. >>> >>> Is it a bug or a feature? >>> Thank you >>> >>> <image.png>--- >>> Dominique Dartois >> Thanks for reporting this. This is most definitely a bug, I can confirm its occurrence. >> >> I'm CC pharo-dev as this is quite important. This will be a long mail. >> >> >> This is one manifestation of a problem that has been present for quite a while. >> >> I'll start by describing what I did, what went well and where/how this fails, some generic points, and two conceptual solutions (that need further verification). >> >> Like you, I created a new subclass: >> >> Object subclass: #ClasseFranÃ§aise >> instanceVariableNames: '' >> classVariableNames: '' >> package: '_UnpackagedPackage' >> >> With comment: >> >> I am ClasseFranÃ§aise. >> >> Try: >> >> ClasseFranÃ§aise new Ã©lÃ¨ve. >> ClasseFranÃ§aise new euro. >> >> And two methods (in the 'test' protocol): >> >> Ã©lÃ¨ve >> ^ 'Ã©lÃ¨ve' >> >> euro >> ^ 'â¬' >> >> I added the euro sign (because that is encoded in UTF-8 with 3 bytes, not 2 like Ã§). >> Like you said, the system can cope with such class and method names and seems to function fine. >> >> Looking at the .changes file, the correct source code was appended: >> >> ----SNAPSHOT----2019-01-26T23:36:18.548555+01:00 work.image priorSource: 339848! >> >> Object subclass: #ClasseFranÃ§aise >> instanceVariableNames: '' >> classVariableNames: '' >> package: '_UnpackagedPackage'! >> !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:25' prior: 0! >> I am ClasseFranÃ§aise.! >> !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:26'! >> Ã©lÃ¨ve >> ^ 'Ã©lÃ¨ve'! ! >> !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:27' prior: 33898360! >> I am ClasseFranÃ§aise. >> >> Try: >> >> ClasseFranÃ§aise new Ã©lÃ¨ve. >> ClasseFranÃ§aise new euro. >> ! >> !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:27'! >> euro >> ^ 'â¬'! ! >> >> >> Doing a file out (or otherwise saving the source code) fails. The reason is an incorrect manipulation of this source file while looking for what is called the method preamble, in SourcFileArray>>#getPreambleFrom:at: position >> >> An programmatic way to invoke the same error is by doing >> >> (ClasseFranÃ§aise>>#Ã©lÃ¨ve) timeStamp. >> (ClasseFranÃ§aise>>#Ã©lÃ¨ve) author. >> >> Both fail with the same error. >> >> >> The source code of methods is (currently) stored in a .sources or .changes file. CompiledMethods know their source pointer, an offset in one of these files. Right before the place where the source starts is a preamble that contains some meta information (including the author and timestamp). To access that preamble, the source code pointer is moved backwards to the beginning of the preamble (which begins and ends with a !). >> >> >> The current approach fails in the presence of non-ASCII characters. More specifically because of a mixup between the concept of byte position and character position when using UTF-8, a variable length encoding (both the .changes and the .sources are UTF-8 encoded). >> >> For example, consider >> >> 'Ã partir de 10 â¬' size. "16" >> 'Ã partir de 10 â¬' utf8Encoded size. "19" >> >> So although the string contains 16 characters, it is encoded as 19 bytes, Ã using 2 bytes and â¬ using 3 bytes. In general, moving backwards or forwards in UTF-8 encoded bytes cannot be done without understanding UTF-8 itself. >> >> ZnUTF8Encoder can do both (moving forward is #nextFromStream: while moving backwards is #backOnStream:). However, ZnUTF8Encoder is also strict: it will signal an error when forced to operate in between encoded characters, which is what happens here. >> >> It is thus not possible to move to arbitrary bytes positions and assume/hope to always arrive on the correct character boundaries and it is also wrong to take the difference between two byte positions as the count of characters present (since their encoding is of variable length). >> >> SourcFileArray>>#getPreambleFrom:at: is doing both of these wrong (but gets away with it in 99.99% of all cases since very few people name their classes like that). >> >> There are two solutions: operate mostly on the byte level or operate correctly on the character level. Here are two conceptual solutions (you must execute either solution 1 or 2, not both), with two different inputs. >> >> >> src := '!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! >> euro >> ^ ''â¬''! !'. >> >> "startPosition := 83" >> >> str := ZnCharacterReadStream on: (src utf8Encoded readStream). >> str position: 83. "at start of euro, the methods source string" >> str upToEnd. >> >> str position: (83 - 3). "before ! before euro" >> >> "find the previous ! before position" >> position := str position. >> binary := str wrappedStream. >> encoder := str encoder. >> >> "solution 1" >> [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. >> position. >> encoder decodeBytes: (binary next: 80 - position). >> >> "solution 2" >> count := 0. >> [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ >> encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. >> str position. >> str next: count. >> >> >> Same code, different input (and starting position): >> >> >> src := '!ABC!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! >> euro >> ^ ''â¬''! !'. >> >> "startPosition := 87" >> >> str := ZnCharacterReadStream on: (src utf8Encoded readStream). >> str position: 87. "at start of euro, the methods source string" >> str upToEnd. >> >> str position: (87 - 3). "before ! before euro" >> >> "find the previous ! before position" >> position := str position. >> binary := str wrappedStream. >> encoder := str encoder. >> >> "solution 1" >> [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. >> position. >> encoder decodeBytes: (binary next: 80 - position). >> >> "solution 2" >> count := 0. >> [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ >> encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. >> str position. >> str next: count. >> >> >> Solution 1 (at the byte level) works because $! (ASCII code 33) is also encoded as such in UTF-8 and this byte pattern cannot be part of a 2, 3 or 4 byte UTF-8 encoded character. The final byte segment must then be given to the proper encoder to turn it into characters. >> >> Solution 2 (at the character level) works because it essentially counts characters while moving backwards correctly (it moves back 2 steps each time because #next moves 1 step forward, while #peek would not help). >> >> Note that in Solution 1 the moving position is held in an external variable, while in Solution 2 it is held inside the stream. >> >> >> Implementing either solution requires more internal access of the streams (#wrappedStream), so SourcFileArray>>#getPreambleFrom:at: should at least be moved to SourceFile, IMHO. >> >> Obviously this is a sensitive/critical area to touch. >> >> We will also need a regression unit test. >> >> On a higher design level, I think that CompiledMethod>>#timeStamp and CompiledMethod>>#author should be cached at the instance level (a unix timestamp and a symbol would cost very little). >> >> Sven >> >> >> >> >> >> >> > -- > ----------------- > BenoÃ®t St-Jean > Yahoo! Messenger: bstjean > Twitter: @BenLeChialeux > Pinterest: benoitstjean > Instagram: Chef_Benito > IRC: lamneth > Blogue: endormitoire.wordpress.com > "A standpoint is an intellectual horizon of radius zero". (A. Einstein) >

Jan. 28, 2019

Re: [Pharo-users] Class name with diacritic character and Pharo

by Benoit St-Jean

Sorry, my toothache meds are kicking in! lol Correction to my last post: I mean *fileouts" don't work when author name has diacritic French characters! Obviously, couldn't test filing in! On 2019-01-27 11:03, Sven Van Caekenberghe wrote: > Hi Dominique, > >> On 27 Jan 2019, at 11:40, Dominique Dartois <dom(a)dartois.org> wrote: >> >> Hello all. >> If a use french diacritic character in a class name, the code runs but I canât fileout the package nor save it with Monticello. >> For example, the C cedilla in the class name drive me to an âZnInvalidUTF8:Illegal byte for utf-8 encoding' when filing out. >> >> Is it a bug or a feature? >> Thank you >> >> <image.png>--- >> Dominique Dartois > Thanks for reporting this. This is most definitely a bug, I can confirm its occurrence. > > I'm CC pharo-dev as this is quite important. This will be a long mail. > > > This is one manifestation of a problem that has been present for quite a while. > > I'll start by describing what I did, what went well and where/how this fails, some generic points, and two conceptual solutions (that need further verification). > > Like you, I created a new subclass: > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage' > > With comment: > > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > > And two methods (in the 'test' protocol): > > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve' > > euro > ^ 'â¬' > > I added the euro sign (because that is encoded in UTF-8 with 3 bytes, not 2 like Ã§). > Like you said, the system can cope with such class and method names and seems to function fine. > > Looking at the .changes file, the correct source code was appended: > > ----SNAPSHOT----2019-01-26T23:36:18.548555+01:00 work.image priorSource: 339848! > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage'! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:25' prior: 0! > I am ClasseFranÃ§aise.! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:26'! > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve'! ! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:27' prior: 33898360! > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > ! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:27'! > euro > ^ 'â¬'! ! > > > Doing a file out (or otherwise saving the source code) fails. The reason is an incorrect manipulation of this source file while looking for what is called the method preamble, in SourcFileArray>>#getPreambleFrom:at: position > > An programmatic way to invoke the same error is by doing > > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) timeStamp. > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) author. > > Both fail with the same error. > > > The source code of methods is (currently) stored in a .sources or .changes file. CompiledMethods know their source pointer, an offset in one of these files. Right before the place where the source starts is a preamble that contains some meta information (including the author and timestamp). To access that preamble, the source code pointer is moved backwards to the beginning of the preamble (which begins and ends with a !). > > > The current approach fails in the presence of non-ASCII characters. More specifically because of a mixup between the concept of byte position and character position when using UTF-8, a variable length encoding (both the .changes and the .sources are UTF-8 encoded). > > For example, consider > > 'Ã partir de 10 â¬' size. "16" > 'Ã partir de 10 â¬' utf8Encoded size. "19" > > So although the string contains 16 characters, it is encoded as 19 bytes, Ã using 2 bytes and â¬ using 3 bytes. In general, moving backwards or forwards in UTF-8 encoded bytes cannot be done without understanding UTF-8 itself. > > ZnUTF8Encoder can do both (moving forward is #nextFromStream: while moving backwards is #backOnStream:). However, ZnUTF8Encoder is also strict: it will signal an error when forced to operate in between encoded characters, which is what happens here. > > It is thus not possible to move to arbitrary bytes positions and assume/hope to always arrive on the correct character boundaries and it is also wrong to take the difference between two byte positions as the count of characters present (since their encoding is of variable length). > > SourcFileArray>>#getPreambleFrom:at: is doing both of these wrong (but gets away with it in 99.99% of all cases since very few people name their classes like that). > > There are two solutions: operate mostly on the byte level or operate correctly on the character level. Here are two conceptual solutions (you must execute either solution 1 or 2, not both), with two different inputs. > > > src := '!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 83" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 83. "at start of euro, the methods source string" > str upToEnd. > > str position: (83 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Same code, different input (and starting position): > > > src := '!ABC!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 87" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 87. "at start of euro, the methods source string" > str upToEnd. > > str position: (87 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Solution 1 (at the byte level) works because $! (ASCII code 33) is also encoded as such in UTF-8 and this byte pattern cannot be part of a 2, 3 or 4 byte UTF-8 encoded character. The final byte segment must then be given to the proper encoder to turn it into characters. > > Solution 2 (at the character level) works because it essentially counts characters while moving backwards correctly (it moves back 2 steps each time because #next moves 1 step forward, while #peek would not help). > > Note that in Solution 1 the moving position is held in an external variable, while in Solution 2 it is held inside the stream. > > > Implementing either solution requires more internal access of the streams (#wrappedStream), so SourcFileArray>>#getPreambleFrom:at: should at least be moved to SourceFile, IMHO. > > Obviously this is a sensitive/critical area to touch. > > We will also need a regression unit test. > > On a higher design level, I think that CompiledMethod>>#timeStamp and CompiledMethod>>#author should be cached at the instance level (a unix timestamp and a symbol would cost very little). > > Sven > > > > > > > -- ----------------- BenoÃ®t St-Jean Yahoo! Messenger: bstjean Twitter: @BenLeChialeux Pinterest: benoitstjean Instagram: Chef_Benito IRC: lamneth Blogue: endormitoire.wordpress.com "A standpoint is an intellectual horizon of radius zero". (A. Einstein)

Jan. 28, 2019

Re: [Pharo-users] Class name with diacritic character and Pharo

by Benoit St-Jean

While we're at it, a similar problem arises when the author name (in my case BenoÃ®tStJean) contains a French diacritic. Just tested it with Pharo 7 (64 bit on Windows 10)... Fileout works fine.Â But filing in crashes! On 2019-01-27 11:03, Sven Van Caekenberghe wrote: > Hi Dominique, > >> On 27 Jan 2019, at 11:40, Dominique Dartois <dom(a)dartois.org> wrote: >> >> Hello all. >> If a use french diacritic character in a class name, the code runs but I canât fileout the package nor save it with Monticello. >> For example, the C cedilla in the class name drive me to an âZnInvalidUTF8:Illegal byte for utf-8 encoding' when filing out. >> >> Is it a bug or a feature? >> Thank you >> >> <image.png>--- >> Dominique Dartois > Thanks for reporting this. This is most definitely a bug, I can confirm its occurrence. > > I'm CC pharo-dev as this is quite important. This will be a long mail. > > > This is one manifestation of a problem that has been present for quite a while. > > I'll start by describing what I did, what went well and where/how this fails, some generic points, and two conceptual solutions (that need further verification). > > Like you, I created a new subclass: > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage' > > With comment: > > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > > And two methods (in the 'test' protocol): > > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve' > > euro > ^ 'â¬' > > I added the euro sign (because that is encoded in UTF-8 with 3 bytes, not 2 like Ã§). > Like you said, the system can cope with such class and method names and seems to function fine. > > Looking at the .changes file, the correct source code was appended: > > ----SNAPSHOT----2019-01-26T23:36:18.548555+01:00 work.image priorSource: 339848! > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage'! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:25' prior: 0! > I am ClasseFranÃ§aise.! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:26'! > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve'! ! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:27' prior: 33898360! > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > ! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:27'! > euro > ^ 'â¬'! ! > > > Doing a file out (or otherwise saving the source code) fails. The reason is an incorrect manipulation of this source file while looking for what is called the method preamble, in SourcFileArray>>#getPreambleFrom:at: position > > An programmatic way to invoke the same error is by doing > > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) timeStamp. > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) author. > > Both fail with the same error. > > > The source code of methods is (currently) stored in a .sources or .changes file. CompiledMethods know their source pointer, an offset in one of these files. Right before the place where the source starts is a preamble that contains some meta information (including the author and timestamp). To access that preamble, the source code pointer is moved backwards to the beginning of the preamble (which begins and ends with a !). > > > The current approach fails in the presence of non-ASCII characters. More specifically because of a mixup between the concept of byte position and character position when using UTF-8, a variable length encoding (both the .changes and the .sources are UTF-8 encoded). > > For example, consider > > 'Ã partir de 10 â¬' size. "16" > 'Ã partir de 10 â¬' utf8Encoded size. "19" > > So although the string contains 16 characters, it is encoded as 19 bytes, Ã using 2 bytes and â¬ using 3 bytes. In general, moving backwards or forwards in UTF-8 encoded bytes cannot be done without understanding UTF-8 itself. > > ZnUTF8Encoder can do both (moving forward is #nextFromStream: while moving backwards is #backOnStream:). However, ZnUTF8Encoder is also strict: it will signal an error when forced to operate in between encoded characters, which is what happens here. > > It is thus not possible to move to arbitrary bytes positions and assume/hope to always arrive on the correct character boundaries and it is also wrong to take the difference between two byte positions as the count of characters present (since their encoding is of variable length). > > SourcFileArray>>#getPreambleFrom:at: is doing both of these wrong (but gets away with it in 99.99% of all cases since very few people name their classes like that). > > There are two solutions: operate mostly on the byte level or operate correctly on the character level. Here are two conceptual solutions (you must execute either solution 1 or 2, not both), with two different inputs. > > > src := '!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 83" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 83. "at start of euro, the methods source string" > str upToEnd. > > str position: (83 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Same code, different input (and starting position): > > > src := '!ABC!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 87" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 87. "at start of euro, the methods source string" > str upToEnd. > > str position: (87 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Solution 1 (at the byte level) works because $! (ASCII code 33) is also encoded as such in UTF-8 and this byte pattern cannot be part of a 2, 3 or 4 byte UTF-8 encoded character. The final byte segment must then be given to the proper encoder to turn it into characters. > > Solution 2 (at the character level) works because it essentially counts characters while moving backwards correctly (it moves back 2 steps each time because #next moves 1 step forward, while #peek would not help). > > Note that in Solution 1 the moving position is held in an external variable, while in Solution 2 it is held inside the stream. > > > Implementing either solution requires more internal access of the streams (#wrappedStream), so SourcFileArray>>#getPreambleFrom:at: should at least be moved to SourceFile, IMHO. > > Obviously this is a sensitive/critical area to touch. > > We will also need a regression unit test. > > On a higher design level, I think that CompiledMethod>>#timeStamp and CompiledMethod>>#author should be cached at the instance level (a unix timestamp and a symbol would cost very little). > > Sven > > > > > > > -- ----------------- BenoÃ®t St-Jean Yahoo! Messenger: bstjean Twitter: @BenLeChialeux Pinterest: benoitstjean Instagram: Chef_Benito IRC: lamneth Blogue: endormitoire.wordpress.com "A standpoint is an intellectual horizon of radius zero". (A. Einstein)

Jan. 28, 2019

Re: [Pharo-users] Class name with diacritic character and Pharo

by Dominique Dartois

Hi Sven. Thank you for the time spend for your reply. I have tried the same code in Pharo 6.1 (21.0) instead of 7.1.0 and I had NO problem. It seems the class implementation for ZnUTF8Encoder is different. Thanks again. --- Dominique > Le 27 janvier 2019 Ã 17:03, Sven Van Caekenberghe <sven(a)stfx.eu> a Ã©critÂ : > > > Hi Dominique, > > > On 27 Jan 2019, at 11:40, Dominique Dartois <dom(a)dartois.org> wrote: > > > > Hello all. > > If a use french diacritic character in a class name, the code runs but I canât fileout the package nor save it with Monticello. > > For example, the C cedilla in the class name drive me to an âZnInvalidUTF8:Illegal byte for utf-8 encoding' when filing out. > > > > Is it a bug or a feature? > > Thank you > > > > <image.png>--- > > Dominique Dartois > > Thanks for reporting this. This is most definitely a bug, I can confirm its occurrence. > > I'm CC pharo-dev as this is quite important. This will be a long mail. > > > This is one manifestation of a problem that has been present for quite a while. > > I'll start by describing what I did, what went well and where/how this fails, some generic points, and two conceptual solutions (that need further verification). > > Like you, I created a new subclass: > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage' > > With comment: > > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > > And two methods (in the 'test' protocol): > > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve' > > euro > ^ 'â¬' > > I added the euro sign (because that is encoded in UTF-8 with 3 bytes, not 2 like Ã§). > Like you said, the system can cope with such class and method names and seems to function fine. > > Looking at the .changes file, the correct source code was appended: > > ----SNAPSHOT----2019-01-26T23:36:18.548555+01:00 work.image priorSource: 339848! > > Object subclass: #ClasseFranÃ§aise > instanceVariableNames: '' > classVariableNames: '' > package: '_UnpackagedPackage'! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:25' prior: 0! > I am ClasseFranÃ§aise.! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:26'! > Ã©lÃ¨ve > ^ 'Ã©lÃ¨ve'! ! > !ClasseFranÃ§aise commentStamp: 'SvenVanCaekenberghe 1/27/2019 12:27' prior: 33898360! > I am ClasseFranÃ§aise. > > Try: > > ClasseFranÃ§aise new Ã©lÃ¨ve. > ClasseFranÃ§aise new euro. > ! > !ClasseFranÃ§aise methodsFor: 'test' stamp: 'SvenVanCaekenberghe 1/27/2019 12:27'! > euro > ^ 'â¬'! ! > > > Doing a file out (or otherwise saving the source code) fails. The reason is an incorrect manipulation of this source file while looking for what is called the method preamble, in SourcFileArray>>#getPreambleFrom:at: position > > An programmatic way to invoke the same error is by doing > > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) timeStamp. > (ClasseFranÃ§aise>>#Ã©lÃ¨ve) author. > > Both fail with the same error. > > > The source code of methods is (currently) stored in a .sources or .changes file. CompiledMethods know their source pointer, an offset in one of these files. Right before the place where the source starts is a preamble that contains some meta information (including the author and timestamp). To access that preamble, the source code pointer is moved backwards to the beginning of the preamble (which begins and ends with a !). > > > The current approach fails in the presence of non-ASCII characters. More specifically because of a mixup between the concept of byte position and character position when using UTF-8, a variable length encoding (both the .changes and the .sources are UTF-8 encoded). > > For example, consider > > 'Ã partir de 10 â¬' size. "16" > 'Ã partir de 10 â¬' utf8Encoded size. "19" > > So although the string contains 16 characters, it is encoded as 19 bytes, Ã using 2 bytes and â¬ using 3 bytes. In general, moving backwards or forwards in UTF-8 encoded bytes cannot be done without understanding UTF-8 itself. > > ZnUTF8Encoder can do both (moving forward is #nextFromStream: while moving backwards is #backOnStream:). However, ZnUTF8Encoder is also strict: it will signal an error when forced to operate in between encoded characters, which is what happens here. > > It is thus not possible to move to arbitrary bytes positions and assume/hope to always arrive on the correct character boundaries and it is also wrong to take the difference between two byte positions as the count of characters present (since their encoding is of variable length). > > SourcFileArray>>#getPreambleFrom:at: is doing both of these wrong (but gets away with it in 99.99% of all cases since very few people name their classes like that). > > There are two solutions: operate mostly on the byte level or operate correctly on the character level. Here are two conceptual solutions (you must execute either solution 1 or 2, not both), with two different inputs. > > > src := '!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 83" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 83. "at start of euro, the methods source string" > str upToEnd. > > str position: (83 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Same code, different input (and starting position): > > > src := '!ABC!ClasseFranÃ§aise methodsFor: ''test'' stamp: ''SvenVanCaekenberghe 1/27/2019 12:27''! > euro > ^ ''â¬''! !'. > > "startPosition := 87" > > str := ZnCharacterReadStream on: (src utf8Encoded readStream). > str position: 87. "at start of euro, the methods source string" > str upToEnd. > > str position: (87 - 3). "before ! before euro" > > "find the previous ! before position" > position := str position. > binary := str wrappedStream. > encoder := str encoder. > > "solution 1" > [ position >= 0 and: [ binary position: position. binary next ~= 33 ] ] whileTrue: [ position := position - 1 ]. > position. > encoder decodeBytes: (binary next: 80 - position). > > "solution 2" > count := 0. > [ str position >= 0 and: [ str next ~= $! ] ] whileTrue: [ > encoder backOnStream: binary; backOnStream: binary. count := count + 1 ]. > str position. > str next: count. > > > Solution 1 (at the byte level) works because $! (ASCII code 33) is also encoded as such in UTF-8 and this byte pattern cannot be part of a 2, 3 or 4 byte UTF-8 encoded character. The final byte segment must then be given to the proper encoder to turn it into characters. > > Solution 2 (at the character level) works because it essentially counts characters while moving backwards correctly (it moves back 2 steps each time because #next moves 1 step forward, while #peek would not help). > > Note that in Solution 1 the moving position is held in an external variable, while in Solution 2 it is held inside the stream. > > > Implementing either solution requires more internal access of the streams (#wrappedStream), so SourcFileArray>>#getPreambleFrom:at: should at least be moved to SourceFile, IMHO. > > Obviously this is a sensitive/critical area to touch. > > We will also need a regression unit test. > > On a higher design level, I think that CompiledMethod>>#timeStamp and CompiledMethod>>#author should be cached at the instance level (a unix timestamp and a symbol would cost very little). > > Sven > > > > > > >

Jan. 27, 2019

What's wrong?

by eftomi

Every time that I start Pharo 7.0.1 32-bit image (it's name is "Pharo 7.0 - 32bit" and it resides in the directory of the same name), a new directory is created under images directory, with the name "Pharo 7.0 - 64bit (development version)-01". Within it, Pharo is using directories "ombu-sessions", "play-cache" to store its run-time information. I'm using PharoLauncher version: 1.5.1. The situation is the same if I do this with a fresh 7.0.1 image. System reporter shows: Image ----- C:\Users\eftomi.MISKOTI\EFTOMI\PHARO\images\Pharo 7.0 - 32bit\Pharo 7.0 - 32bit.image Pharo7.0.1 Build information: Pharo-7.0.1+build.143.sha.eca26da119bccd95e463c7717a44b814453df4e8 (32 Bit) Unnamed Virtual Machine --------------- C:\Users\eftomi.MISKOTI\EFTOMI\PHARO\vms\70-x86\Pharo.exe CoInterpreter VMMaker.oscog-eem.2504 uuid: a00b0fad-c04c-47a6-8a11-5dbff110ac11 Jan 5 2019 StackToRegisterMappingCogit VMMaker.oscog-eem.2504 uuid: a00b0fad-c04c-47a6-8a11-5dbff110ac11 Jan 5 2019 VM: 201901051900 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Sat Jan 5 20:00:11 2019 CommitHash: 7a3c6b6 Plugins: 201901051900 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Win32 built on Jan 5 2019 20:12:30 GMT Compiler: 7.4.0 VMMaker versionString VM: 201901051900 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Sat Jan 5 20:00:11 2019 CommitHash: 7a3c6b6 Plugins: 201901051900 https://github.com/OpenSmalltalk/opensmalltalk-vm.git CoInterpreter VMMaker.oscog-eem.2504 uuid: a00b0fad-c04c-47a6-8a11-5dbff110ac11 Jan 5 2019 StackToRegisterMappingCogit VMMaker.oscog-eem.2504 uuid: a00b0fad-c04c-47a6-8a11-5dbff110ac11 Jan 5 2019 -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Jan. 27, 2019