May 2009 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Relicensing effort

by Gabriel Cotelli

Hi, I will start on re-writting this methods: Number>>reciprocal Fraction class>>one Fraction>>reciprocal Float class>>one Float>>reciprocal Integer class>>one All of this are almost trivial to re-write so I think it's a good start point. My plan is to first write a test case covering this, delete the old method (if present) and re-implement it. I signed the License Agrement on Friday and sent an scanned copy to Stephane by e-mail (probably he still doesn't update the list). Thanks, Gabriel On Sat, May 23, 2009 at 3:58 PM, Adrian Lienhard <adi(a)netstyle.ch> wrote: > Hi Gabriel, > > On May 23, 2009, at 19:55 , Gabriel Cotelli wrote: > > > Hi! > > > > There is somebody coordinating the re-write of non re-licensed > > methods? > > As I started working on it last week, I can coordinate. > > > I probably could re-write some of this methods but don't want to > > duplicate > > the effort :) > > Great! > > > Should I take the Pharo-Core image or the Pharo-Dev for this? > > The core image. > > I already fixed more than half of the methods. What is left can be > seen here: > > http://code.google.com/p/pharo/wiki/ResultofRelicensing. > > I suggest that whoever starts working on a set of methods announces it > on this mailing list. Like this we don't get a conflict. The easiest > thing is to produce a change set and then attach it to the issue: > http://code.google.com/p/pharo/issues/detail?id=842 > > Thanks, > Adrian > > BTW, I don't see you on the list of OK committers. Did you sign the > agreement? Else your changes will reappear on the list of non-clean > methods! ;) > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] Relicensing effort

by Adrian Lienhard

Hi Gabriel, On May 23, 2009, at 19:55 , Gabriel Cotelli wrote: > Hi! > > There is somebody coordinating the re-write of non re-licensed > methods? As I started working on it last week, I can coordinate. > I probably could re-write some of this methods but don't want to > duplicate > the effort :) Great! > Should I take the Pharo-Core image or the Pharo-Dev for this? The core image. I already fixed more than half of the methods. What is left can be seen here: http://code.google.com/p/pharo/wiki/ResultofRelicensing. I suggest that whoever starts working on a set of methods announces it on this mailing list. Like this we don't get a conflict. The easiest thing is to produce a change set and then attach it to the issue: http://code.google.com/p/pharo/issues/detail?id=842 Thanks, Adrian BTW, I don't see you on the list of OK committers. Did you sign the agreement? Else your changes will reappear on the list of non-clean methods! ;)

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by Adrian Lienhard

Wow, great analysis, Nicolas! I was trying to find the cause for several hours now. Your third track exactly matches my findings. For example in Object>>#doesNotUnderstand: prior to the condensing, the source contained a non-ASCII character (UTF8 encoded as the two bytes: 192 160). This gets correctly transferred during the condensing into the new changes file. When you don't save the image (and hence have the standard stream without UTF8 encoder) what you see in the source is the character Ã (this is 192). That is, we suddenly have two characters, 192 and 160 where before there was just one. If you load a package, MC will compare methods and think this is a change. When loading the method from the MC file, the source is UTF8 encoded, producing a unicode character 160. When storing this source to the file (still without the encoder), it will just directly put 160 there. At this point we have lost the leading byte 192. Next time we start or save the image and have the right encoder again, it will choke because 160 is an invalid first byte in UTF8. I think it's save to fix the invalid methods by overriding their source. So we don't have to backtrack to version 10297. Thanks, Adrian On May 23, 2009, at 19:57 , Nicolas Cellier wrote: > I confirm the scenario: > 1) update10298 condenseChanges that let (SourceFiles at: 2) class = > StandardFileStream > This is the seed of further problems, because further changes will > be encoded in latin1 (or MacRoman I don't really wnt to know) > 2) update10302 changes the methods with non ASCII characters > 3) Stef save the image after update10304, that does reopen > (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the > apple. > > If you save the image just after the condenseChanges, no problem > because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes > have gotten into it, and reopened UTF-8 before any changes got into > it. > We must track undue usage of StandardFileStream such as > #condenseChanges. > > 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >> What happened exactly is very hard to trace because these FileStream >> are a can of worms... >> Here are some of my perigrinations: >> >> FIRST POSSIBLE TRACK: >> >> All methods were changed in 10305. >> Monticello snapshot/source.st is not UTF-8. >> If the file is opened UTF-8, then we get decompiledCode, I don't >> know why yet... >> But the changes still go into the change log in correct UTF-8 form, >> so >> that's just another bug, but not the real source of the problem. >> For getting some worms out of the can just browse inst var defs of >> converter in MultiByteFileStream: >> The accessor #converter initialize converter with TextConverter >> defaultSystemConverter which depends on LanguageEnvironment. >> That is a Latin1TextConverter in my latin image. >> Unless #reset is called first, in which case it will initialize >> with a >> UTF8TextConverter. >> Yes, but open: fileName forWrite: writeMode, does the job too with a >> UTF8TextConverter. >> You still follow? me neither. >> A better behaved is #setConverterForCode that should let non UTF-8 >> .mcz work in UTF-8 environment, but not sure if called where >> required... >> I think Yoshiki changes are necessary only for writing source code >> with character code > 255. >> This was not the case of incriminated methods. >> >> SECOND POSSIBLE TRACK: >> >> Everything going to the change log pass thru the MultiByteFileStream, >> so how did non UTF-8 characters went in? >> I tried to follow two other clues: >> 1) There are senders of #primWrite:from:startingAt:count: not >> redefined in MultiByteFileStream... >> for example, using #next:putAll:startingAt: will bypass the >> converter. >> 2) using nextPutAll: with a ByteArray argument also does bypass the >> converter (See MultiByteFileStream>>#nextPutAll:) >> I did not find the senders (you really believe senders of nextPutAll: >> can be analyzed?). >> I tried to instrument code with Notification, but I'm unable to >> reproduce the problem, so that was vain... >> >> THIRD POSSIBLE TRACK: >> >> http://gforge.inria.fr/frs/download.php/22283/ >> Pharo0.1Core-10304cl.zip >> has the invalid UTF-8 problem, just before 10305 changes that >> introduced decompiled code... >> So we might attack the problem with another code snippet: >> >> (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: >> #SourceFiles))... >> >> Hmm, I might have a better clue now. >> The problem might possibly come from the condenseChanges in >> update10298. >> What happen in a condenseChanges? >> Changes are copied to this file: >> >> f := FileStream fileNamed: 'ST80.temp'. >> >> So far, so good, because the concreteStream is a MultiByteFileStream. >> >> But the end finishes with: >> >> SourceFiles >> at: 2 >> put: (StandardFileStream oldFileNamed: oldChanges name) >> >> Waouh, no MultiByteFileStream here, so no more UTF-8. >> But hey, that would be the inverse problem: reading UTF-8 text with >> latin1 reader: I can't get an error doing this, only some strange >> sequence of characters... (The UTF-8 encoding)... >> Unless incriminated methods are further changed in #script376 or any >> other method... In which case they are written in latin1 in the >> changeLog... >> Hmm... That could be the case eventually. We must restart update >> process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip >> >> One thing is sure, at next returnFromSnapshot, FileDirectory >> class>>startup will reopen changes UTF-8. >> So saving the image will reopen UTF-8... >> >> But wait... Maybe we get enough pieces of the puzzle: >> Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane >> applied >> several updates before snapshoting the image. So if Kernel and >> System-Support are changed between 10298 and 10304, then we get the >> explanation: >> - condense changes put all in the .changes in UTF-8 but reopen the >> changes in latin1 >> - further updates up to 10304 write changes in latin1 >> - image snapshot reopen changes in UTF-8 and thus we get further >> invalid UTF-8... >> >> That's easy to reproduce. Stef, can you confirm? >> >> That also explain why I did not get the problem at home: I update >> early and always save my image after. >> After that we still have to detect and clean while Monticello sources >> are interpreted UTF-8 when they should not (FIRST TRACK) , and >> eventually make source code go UTF-8 in Monticello, so that non latin >> programmers can use their favourite language eventually... >> >> Nicolas >> >> 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >>> No problem I never interpreted it like that. >>> Me too I want a system that is working >>> >>> Adrian I will publish a fix for DNU now >>> and I will try later to check the fixes proposed by yoshiki >>> >>> stef >>> >>> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >>> >>>> Actually, the fix is even simpler: if you find a method that raises >>>> "invalid utf8 input detected", just browse to it with a class >>>> browser, >>>> and re-accept it :). >>>> >>>> With my previous mail, I was not implying that someone should fix >>>> it >>>> for me, I was merely asking for what could a quick solution be, >>>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>>> discussing it. >>>> >>>> Cheers, >>>> Doru >>>> >>>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I attached here a DNU implementation I took from an older image. >>>>> After filing this one in, I can debug DNU problems. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> <Object-doesNotUnderstand.st> >>>>> >>>>> >>>>> >>>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>>> >>>>>> I did the following >>>>>> >>>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>>> invalid.... >>>>>> >>>>>> Now when I take another method >>>>>> >>>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>>> >>>>>> I will reread carefully the mails of nicolas to try to >>>>>> understand, >>>>>> I do not know if the fixes of yoh >>>>>> >>>>>> http://bugs.squeak.org/view.php?id=5996 >>>>>> is related. >>>>>> >>>>>> Nicolas >>>>>> >>>>>>>> {Object>>#doesNotUnderstand:. >>>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>>> Utilities class>>#changeStampPerSe. >>>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>>> collect: >>>>>>>> [:c | c charCode]] >>>>>> >>>>>> I cannot get that code running it break before with me. >>>>>> >>>>>> Stef >>>>>> >>>>>> _______________________________________________ >>>>>> Pharo-project mailing list >>>>>> Pharo-project(a)lists.gforge.inria.fr >>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>>> >>>>> -- >>>>> www.tudorgirba.com >>>>> >>>>> "Not knowing how to do something is not an argument for how it >>>>> cannot be done." >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>> project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Problem solving efficiency grows with the abstractness level of >>>> problem understanding." >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

May 23, 2009

[Pharo-project] windowsesque theme tweaks

by Steve Wirts

Gary here's some tweaks I talked about (progressbar and tabbar fixed). After the attached is filedin, evaluate UIThemeXP setup then to see the tweaks try (progress bar) SystemProgressMorph example and (tab bar) UITheme exampleGroups also an odd thing that may hint at a bug, is the progress bar moves faster while you move the cursor. thanks. sw

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by StÃ©phane Ducasse

On May 23, 2009, at 7:57 PM, Nicolas Cellier wrote: > I confirm the scenario: > 1) update10298 condenseChanges that let (SourceFiles at: 2) class = > StandardFileStream > This is the seed of further problems, because further changes will > be encoded in latin1 (or MacRoman I don't really wnt to know) > 2) update10302 changes the methods with non ASCII characters > 3) Stef save the image after update10304, that does reopen > (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the > apple. > > If you save the image just after the condenseChanges, no problem > because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes > have gotten into it, and reopened UTF-8 before any changes got into > it. > We must track undue usage of StandardFileStream such as > #condenseChanges. Ok now we cannot really rollback the changes and I fixed the methods that were leading to invalid UTF. But it means that we should check the StandardFileStream usage. I"m doing some experiences with umejava code Stef > > > 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >> What happened exactly is very hard to trace because these FileStream >> are a can of worms... >> Here are some of my perigrinations: >> >> FIRST POSSIBLE TRACK: >> >> All methods were changed in 10305. >> Monticello snapshot/source.st is not UTF-8. >> If the file is opened UTF-8, then we get decompiledCode, I don't >> know why yet... >> But the changes still go into the change log in correct UTF-8 form, >> so >> that's just another bug, but not the real source of the problem. >> For getting some worms out of the can just browse inst var defs of >> converter in MultiByteFileStream: >> The accessor #converter initialize converter with TextConverter >> defaultSystemConverter which depends on LanguageEnvironment. >> That is a Latin1TextConverter in my latin image. >> Unless #reset is called first, in which case it will initialize >> with a >> UTF8TextConverter. >> Yes, but open: fileName forWrite: writeMode, does the job too with a >> UTF8TextConverter. >> You still follow? me neither. >> A better behaved is #setConverterForCode that should let non UTF-8 >> .mcz work in UTF-8 environment, but not sure if called where >> required... >> I think Yoshiki changes are necessary only for writing source code >> with character code > 255. >> This was not the case of incriminated methods. >> >> SECOND POSSIBLE TRACK: >> >> Everything going to the change log pass thru the MultiByteFileStream, >> so how did non UTF-8 characters went in? >> I tried to follow two other clues: >> 1) There are senders of #primWrite:from:startingAt:count: not >> redefined in MultiByteFileStream... >> for example, using #next:putAll:startingAt: will bypass the >> converter. >> 2) using nextPutAll: with a ByteArray argument also does bypass the >> converter (See MultiByteFileStream>>#nextPutAll:) >> I did not find the senders (you really believe senders of nextPutAll: >> can be analyzed?). >> I tried to instrument code with Notification, but I'm unable to >> reproduce the problem, so that was vain... >> >> THIRD POSSIBLE TRACK: >> >> http://gforge.inria.fr/frs/download.php/22283/ >> Pharo0.1Core-10304cl.zip >> has the invalid UTF-8 problem, just before 10305 changes that >> introduced decompiled code... >> So we might attack the problem with another code snippet: >> >> (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: >> #SourceFiles))... >> >> Hmm, I might have a better clue now. >> The problem might possibly come from the condenseChanges in >> update10298. >> What happen in a condenseChanges? >> Changes are copied to this file: >> >> f := FileStream fileNamed: 'ST80.temp'. >> >> So far, so good, because the concreteStream is a MultiByteFileStream. >> >> But the end finishes with: >> >> SourceFiles >> at: 2 >> put: (StandardFileStream oldFileNamed: oldChanges name) >> >> Waouh, no MultiByteFileStream here, so no more UTF-8. >> But hey, that would be the inverse problem: reading UTF-8 text with >> latin1 reader: I can't get an error doing this, only some strange >> sequence of characters... (The UTF-8 encoding)... >> Unless incriminated methods are further changed in #script376 or any >> other method... In which case they are written in latin1 in the >> changeLog... >> Hmm... That could be the case eventually. We must restart update >> process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip >> >> One thing is sure, at next returnFromSnapshot, FileDirectory >> class>>startup will reopen changes UTF-8. >> So saving the image will reopen UTF-8... >> >> But wait... Maybe we get enough pieces of the puzzle: >> Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane >> applied >> several updates before snapshoting the image. So if Kernel and >> System-Support are changed between 10298 and 10304, then we get the >> explanation: >> - condense changes put all in the .changes in UTF-8 but reopen the >> changes in latin1 >> - further updates up to 10304 write changes in latin1 >> - image snapshot reopen changes in UTF-8 and thus we get further >> invalid UTF-8... >> >> That's easy to reproduce. Stef, can you confirm? >> >> That also explain why I did not get the problem at home: I update >> early and always save my image after. >> After that we still have to detect and clean while Monticello sources >> are interpreted UTF-8 when they should not (FIRST TRACK) , and >> eventually make source code go UTF-8 in Monticello, so that non latin >> programmers can use their favourite language eventually... >> >> Nicolas >> >> 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >>> No problem I never interpreted it like that. >>> Me too I want a system that is working >>> >>> Adrian I will publish a fix for DNU now >>> and I will try later to check the fixes proposed by yoshiki >>> >>> stef >>> >>> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >>> >>>> Actually, the fix is even simpler: if you find a method that raises >>>> "invalid utf8 input detected", just browse to it with a class >>>> browser, >>>> and re-accept it :). >>>> >>>> With my previous mail, I was not implying that someone should fix >>>> it >>>> for me, I was merely asking for what could a quick solution be, >>>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>>> discussing it. >>>> >>>> Cheers, >>>> Doru >>>> >>>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I attached here a DNU implementation I took from an older image. >>>>> After filing this one in, I can debug DNU problems. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> <Object-doesNotUnderstand.st> >>>>> >>>>> >>>>> >>>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>>> >>>>>> I did the following >>>>>> >>>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>>> invalid.... >>>>>> >>>>>> Now when I take another method >>>>>> >>>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>>> >>>>>> I will reread carefully the mails of nicolas to try to >>>>>> understand, >>>>>> I do not know if the fixes of yoh >>>>>> >>>>>> http://bugs.squeak.org/view.php?id=5996 >>>>>> is related. >>>>>> >>>>>> Nicolas >>>>>> >>>>>>>> {Object>>#doesNotUnderstand:. >>>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>>> Utilities class>>#changeStampPerSe. >>>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>>> collect: >>>>>>>> [:c | c charCode]] >>>>>> >>>>>> I cannot get that code running it break before with me. >>>>>> >>>>>> Stef >>>>>> >>>>>> _______________________________________________ >>>>>> Pharo-project mailing list >>>>>> Pharo-project(a)lists.gforge.inria.fr >>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>>> >>>>> -- >>>>> www.tudorgirba.com >>>>> >>>>> "Not knowing how to do something is not an argument for how it >>>>> cannot be done." >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>> project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Problem solving efficiency grows with the abstractness level of >>>> problem understanding." >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by Nicolas Cellier

I confirm the scenario: 1) update10298 condenseChanges that let (SourceFiles at: 2) class = StandardFileStream This is the seed of further problems, because further changes will be encoded in latin1 (or MacRoman I don't really wnt to know) 2) update10302 changes the methods with non ASCII characters 3) Stef save the image after update10304, that does reopen (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the apple. If you save the image just after the condenseChanges, no problem because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes have gotten into it, and reopened UTF-8 before any changes got into it. We must track undue usage of StandardFileStream such as #condenseChanges. 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > What happened exactly is very hard to trace because these FileStream > are a can of worms... > Here are some of my perigrinations: > > FIRST POSSIBLE TRACK: > > All methods were changed in 10305. > Monticello snapshot/source.st is not UTF-8. > If the file is opened UTF-8, then we get decompiledCode, I don't know why yet... > But the changes still go into the change log in correct UTF-8 form, so > that's just another bug, but not the real source of the problem. > For getting some worms out of the can just browse inst var defs of > converter in MultiByteFileStream: > The accessor #converter initialize converter with TextConverter > defaultSystemConverter which depends on LanguageEnvironment. > That is a Latin1TextConverter in my latin image. > Unless #reset is called first, in which case it will initialize with a > UTF8TextConverter. > Yes, but open: fileName forWrite: writeMode, does the job too with a > UTF8TextConverter. > You still follow? me neither. > A better behaved is #setConverterForCode that should let non UTF-8 > .mcz work in UTF-8 environment, but not sure if called where > required... > I think Yoshiki changes are necessary only for writing source code > with character code > 255. > This was not the case of incriminated methods. > > SECOND POSSIBLE TRACK: > > Everything going to the change log pass thru the MultiByteFileStream, > so how did non UTF-8 characters went in? > I tried to follow two other clues: > 1) There are senders of #primWrite:from:startingAt:count: not > redefined in MultiByteFileStream... > for example, using #next:putAll:startingAt: will bypass the converter. > 2) using nextPutAll: with a ByteArray argument also does bypass the > converter (See MultiByteFileStream>>#nextPutAll:) > I did not find the senders (you really believe senders of nextPutAll: > can be analyzed?). > I tried to instrument code with Notification, but I'm unable to > reproduce the problem, so that was vain... > > THIRD POSSIBLE TRACK: > > http://gforge.inria.fr/frs/download.php/22283/Pharo0.1Core-10304cl.zip > has the invalid UTF-8 problem, just before 10305 changes that > introduced decompiled code... > So we might attack the problem with another code snippet: > > (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: > #SourceFiles))... > > Hmm, I might have a better clue now. > The problem might possibly come from the condenseChanges in update10298. > What happen in a condenseChanges? > Changes are copied to this file: > > f := FileStream fileNamed: 'ST80.temp'. > > So far, so good, because the concreteStream is a MultiByteFileStream. > > But the end finishes with: > > SourceFiles > at: 2 > put: (StandardFileStream oldFileNamed: oldChanges name) > > Waouh, no MultiByteFileStream here, so no more UTF-8. > But hey, that would be the inverse problem: reading UTF-8 text with > latin1 reader: I can't get an error doing this, only some strange > sequence of characters... (The UTF-8 encoding)... > Unless incriminated methods are further changed in #script376 or any > other method... In which case they are written in latin1 in the > changeLog... > Hmm... That could be the case eventually. We must restart update > process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip > > One thing is sure, at next returnFromSnapshot, FileDirectory > class>>startup will reopen changes UTF-8. > So saving the image will reopen UTF-8... > > But wait... Maybe we get enough pieces of the puzzle: > Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane applied > several updates before snapshoting the image. So if Kernel and > System-Support are changed between 10298 and 10304, then we get the > explanation: > - condense changes put all in the .changes in UTF-8 but reopen the > changes in latin1 > - further updates up to 10304 write changes in latin1 > - image snapshot reopen changes in UTF-8 and thus we get further > invalid UTF-8... > > That's easy to reproduce. Stef, can you confirm? > > That also explain why I did not get the problem at home: I update > early and always save my image after. > After that we still have to detect and clean while Monticello sources > are interpreted UTF-8 when they should not (FIRST TRACK) , and > eventually make source code go UTF-8 in Monticello, so that non latin > programmers can use their favourite language eventually... > > Nicolas > > 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >> No problem I never interpreted it like that. >> Me too I want a system that is working >> >> Adrian I will publish a fix for DNU now >> and I will try later to check the fixes proposed by yoshiki >> >> stef >> >> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >> >>> Actually, the fix is even simpler: if you find a method that raises >>> "invalid utf8 input detected", just browse to it with a class browser, >>> and re-accept it :). >>> >>> With my previous mail, I was not implying that someone should fix it >>> for me, I was merely asking for what could a quick solution be, >>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>> discussing it. >>> >>> Cheers, >>> Doru >>> >>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>> >>>> Hi, >>>> >>>> I attached here a DNU implementation I took from an older image. >>>> After filing this one in, I can debug DNU problems. >>>> >>>> Cheers, >>>> Doru >>>> >>>> <Object-doesNotUnderstand.st> >>>> >>>> >>>> >>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>> >>>>> I did the following >>>>> >>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>> invalid.... >>>>> >>>>> Now when I take another method >>>>> >>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>> >>>>> I will reread carefully the mails of nicolas to try to understand, >>>>> I do not know if the fixes of yoh >>>>> >>>>> http://bugs.squeak.org/view.php?id=5996 >>>>> is related. >>>>> >>>>> Nicolas >>>>> >>>>>>> {Object>>#doesNotUnderstand:. >>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>> Utilities class>>#changeStampPerSe. >>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>> collect: >>>>>>> [:c | c charCode]] >>>>> >>>>> I cannot get that code running it break before with me. >>>>> >>>>> Stef >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Not knowing how to do something is not an argument for how it >>>> cannot be done." >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >>> -- >>> www.tudorgirba.com >>> >>> "Problem solving efficiency grows with the abstractness level of >>> problem understanding." >>> >>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >

May 23, 2009

[Pharo-project] Relicensing effort

by Gabriel Cotelli

Hi! There is somebody coordinating the re-write of non re-licensed methods? I probably could re-write some of this methods but don't want to duplicate the effort :) Should I take the Pharo-Core image or the Pharo-Dev for this? Thanks Gabriel

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by StÃ©phane Ducasse

HI nicolas I was reading the changes of yoshiki I will integrate but indeed this is not for our case. My reply below... I tried to follow :) > What happened exactly is very hard to trace because these FileStream > are a can of worms... > Here are some of my perigrinations: > > FIRST POSSIBLE TRACK: > > All methods were changed in 10305. > Monticello snapshot/source.st is not UTF-8. > If the file is opened UTF-8, then we get decompiledCode, I don't > know why yet... > But the changes still go into the change log in correct UTF-8 form, so > that's just another bug, but not the real source of the problem. > For getting some worms out of the can just browse inst var defs of > converter in MultiByteFileStream: > The accessor #converter initialize converter with TextConverter > defaultSystemConverter which depends on LanguageEnvironment. > That is a Latin1TextConverter in my latin image. > Unless #reset is called first, in which case it will initialize with a > UTF8TextConverter. > Yes, but open: fileName forWrite: writeMode, does the job too with a > UTF8TextConverter. > You still follow? me neither. > A better behaved is #setConverterForCode that should let non UTF-8 > .mcz work in UTF-8 environment, but not sure if called where > required... > I think Yoshiki changes are necessary only for writing source code > with character code > 255. > This was not the case of incriminated methods. > > SECOND POSSIBLE TRACK: > > Everything going to the change log pass thru the MultiByteFileStream, > so how did non UTF-8 characters went in? > I tried to follow two other clues: > 1) There are senders of #primWrite:from:startingAt:count: not > redefined in MultiByteFileStream... > for example, using #next:putAll:startingAt: will bypass the > converter. > 2) using nextPutAll: with a ByteArray argument also does bypass the > converter (See MultiByteFileStream>>#nextPutAll:) > I did not find the senders (you really believe senders of nextPutAll: > can be analyzed?). > I tried to instrument code with Notification, but I'm unable to > reproduce the problem, so that was vain... > > THIRD POSSIBLE TRACK: > > http://gforge.inria.fr/frs/download.php/22283/Pharo0.1Core-10304cl.zip > has the invalid UTF-8 problem, just before 10305 changes that > introduced decompiled code... > So we might attack the problem with another code snippet: > > (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: > #SourceFiles))... > > Hmm, I might have a better clue now. > The problem might possibly come from the condenseChanges in > update10298. > What happen in a condenseChanges? > Changes are copied to this file: > > f := FileStream fileNamed: 'ST80.temp'. > > So far, so good, because the concreteStream is a MultiByteFileStream. > > But the end finishes with: > > SourceFiles > at: 2 > put: (StandardFileStream oldFileNamed: oldChanges name) > > Waouh, no MultiByteFileStream here, so no more UTF-8. > But hey, that would be the inverse problem: reading UTF-8 text with > latin1 reader: I can't get an error doing this, only some strange > sequence of characters... (The UTF-8 encoding)... > Unless incriminated methods are further changed in #script376 or any > other method... In which case they are written in latin1 in the > changeLog... > Hmm... That could be the case eventually. We must restart update > process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip > > One thing is sure, at next returnFromSnapshot, FileDirectory > class>>startup will reopen changes UTF-8. > So saving the image will reopen UTF-8... > > But wait... Maybe we get enough pieces of the puzzle: > Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane applied > several updates before snapshoting the image. So if Kernel and > System-Support are changed between 10298 and 10304, then we get the > explanation: > - condense changes put all in the .changes in UTF-8 but reopen the > changes in latin1 > - further updates up to 10304 write changes in latin1 > - image snapshot reopen changes in UTF-8 and thus we get further > invalid UTF-8... > > That's easy to reproduce. Stef, can you confirm? how do you want me to confirm? That I redo the image. What we can do is change the update method to block the update at a certain number. > That also explain why I did not get the problem at home: I update > early and always save my image after. > After that we still have to detect and clean while Monticello sources > are interpreted UTF-8 when they should not (FIRST TRACK) , and > eventually make source code go UTF-8 in Monticello, so that non latin > programmers can use their favourite language eventually... > > Nicolas > > 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >> No problem I never interpreted it like that. >> Me too I want a system that is working >> >> Adrian I will publish a fix for DNU now >> and I will try later to check the fixes proposed by yoshiki >> >> stef >> >> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >> >>> Actually, the fix is even simpler: if you find a method that raises >>> "invalid utf8 input detected", just browse to it with a class >>> browser, >>> and re-accept it :). >>> >>> With my previous mail, I was not implying that someone should fix it >>> for me, I was merely asking for what could a quick solution be, >>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>> discussing it. >>> >>> Cheers, >>> Doru >>> >>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>> >>>> Hi, >>>> >>>> I attached here a DNU implementation I took from an older image. >>>> After filing this one in, I can debug DNU problems. >>>> >>>> Cheers, >>>> Doru >>>> >>>> <Object-doesNotUnderstand.st> >>>> >>>> >>>> >>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>> >>>>> I did the following >>>>> >>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>> invalid.... >>>>> >>>>> Now when I take another method >>>>> >>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>> >>>>> I will reread carefully the mails of nicolas to try to understand, >>>>> I do not know if the fixes of yoh >>>>> >>>>> http://bugs.squeak.org/view.php?id=5996 >>>>> is related. >>>>> >>>>> Nicolas >>>>> >>>>>>> {Object>>#doesNotUnderstand:. >>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>> Utilities class>>#changeStampPerSe. >>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>> collect: >>>>>>> [:c | c charCode]] >>>>> >>>>> I cannot get that code running it break before with me. >>>>> >>>>> Stef >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>> project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Not knowing how to do something is not an argument for how it >>>> cannot be done." >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >>> -- >>> www.tudorgirba.com >>> >>> "Problem solving efficiency grows with the abstractness level of >>> problem understanding." >>> >>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by Nicolas Cellier

What happened exactly is very hard to trace because these FileStream are a can of worms... Here are some of my perigrinations: FIRST POSSIBLE TRACK: All methods were changed in 10305. Monticello snapshot/source.st is not UTF-8. If the file is opened UTF-8, then we get decompiledCode, I don't know why yet... But the changes still go into the change log in correct UTF-8 form, so that's just another bug, but not the real source of the problem. For getting some worms out of the can just browse inst var defs of converter in MultiByteFileStream: The accessor #converter initialize converter with TextConverter defaultSystemConverter which depends on LanguageEnvironment. That is a Latin1TextConverter in my latin image. Unless #reset is called first, in which case it will initialize with a UTF8TextConverter. Yes, but open: fileName forWrite: writeMode, does the job too with a UTF8TextConverter. You still follow? me neither. A better behaved is #setConverterForCode that should let non UTF-8 .mcz work in UTF-8 environment, but not sure if called where required... I think Yoshiki changes are necessary only for writing source code with character code > 255. This was not the case of incriminated methods. SECOND POSSIBLE TRACK: Everything going to the change log pass thru the MultiByteFileStream, so how did non UTF-8 characters went in? I tried to follow two other clues: 1) There are senders of #primWrite:from:startingAt:count: not redefined in MultiByteFileStream... for example, using #next:putAll:startingAt: will bypass the converter. 2) using nextPutAll: with a ByteArray argument also does bypass the converter (See MultiByteFileStream>>#nextPutAll:) I did not find the senders (you really believe senders of nextPutAll: can be analyzed?). I tried to instrument code with Notification, but I'm unable to reproduce the problem, so that was vain... THIRD POSSIBLE TRACK: http://gforge.inria.fr/frs/download.php/22283/Pharo0.1Core-10304cl.zip has the invalid UTF-8 problem, just before 10305 changes that introduced decompiled code... So we might attack the problem with another code snippet: (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: #SourceFiles))... Hmm, I might have a better clue now. The problem might possibly come from the condenseChanges in update10298. What happen in a condenseChanges? Changes are copied to this file: f := FileStream fileNamed: 'ST80.temp'. So far, so good, because the concreteStream is a MultiByteFileStream. But the end finishes with: SourceFiles at: 2 put: (StandardFileStream oldFileNamed: oldChanges name) Waouh, no MultiByteFileStream here, so no more UTF-8. But hey, that would be the inverse problem: reading UTF-8 text with latin1 reader: I can't get an error doing this, only some strange sequence of characters... (The UTF-8 encoding)... Unless incriminated methods are further changed in #script376 or any other method... In which case they are written in latin1 in the changeLog... Hmm... That could be the case eventually. We must restart update process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip One thing is sure, at next returnFromSnapshot, FileDirectory class>>startup will reopen changes UTF-8. So saving the image will reopen UTF-8... But wait... Maybe we get enough pieces of the puzzle: Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane applied several updates before snapshoting the image. So if Kernel and System-Support are changed between 10298 and 10304, then we get the explanation: - condense changes put all in the .changes in UTF-8 but reopen the changes in latin1 - further updates up to 10304 write changes in latin1 - image snapshot reopen changes in UTF-8 and thus we get further invalid UTF-8... That's easy to reproduce. Stef, can you confirm? That also explain why I did not get the problem at home: I update early and always save my image after. After that we still have to detect and clean while Monticello sources are interpreted UTF-8 when they should not (FIRST TRACK) , and eventually make source code go UTF-8 in Monticello, so that non latin programmers can use their favourite language eventually... Nicolas 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: > No problem I never interpreted it like that. > Me too I want a system that is working > > Adrian I will publish a fix for DNU now > and I will try later to check the fixes proposed by yoshiki > > stef > > On May 23, 2009, at 1:29 PM, Tudor Girba wrote: > >> Actually, the fix is even simpler: if you find a method that raises >> "invalid utf8 input detected", just browse to it with a class browser, >> and re-accept it :). >> >> With my previous mail, I was not implying that someone should fix it >> for me, I was merely asking for what could a quick solution be, >> because I was a bit lost (scared) :). Now, I am happy. Thanks for >> discussing it. >> >> Cheers, >> Doru >> >> On 23 May 2009, at 13:07, Tudor Girba wrote: >> >>> Hi, >>> >>> I attached here a DNU implementation I took from an older image. >>> After filing this one in, I can debug DNU problems. >>> >>> Cheers, >>> Doru >>> >>> <Object-doesNotUnderstand.st> >>> >>> >>> >>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>> >>>> I did the following >>>> >>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>> invalid.... >>>> >>>> Now when I take another method >>>> >>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>> >>>> I will reread carefully the mails of nicolas to try to understand, >>>> I do not know if the fixes of yoh >>>> >>>> http://bugs.squeak.org/view.php?id=5996 >>>> is related. >>>> >>>> Nicolas >>>> >>>>>> {Object>>#doesNotUnderstand:. >>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>> Utilities class>>#changeStampPerSe. >>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>> collect: >>>>>> [:c | c charCode]] >>>> >>>> I cannot get that code running it break before with me. >>>> >>>> Stef >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >>> -- >>> www.tudorgirba.com >>> >>> "Not knowing how to do something is not an argument for how it >>> cannot be done." >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> -- >> www.tudorgirba.com >> >> "Problem solving efficiency grows with the abstractness level of >> problem understanding." >> >> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] Alien FFI & COM

by StÃ©phane Ducasse

just load Alien from the alien squeaksource project On May 23, 2009, at 12:58 PM, Carlos Crosetti wrote: > Fernando, plese can you send the note you mention again? Thanks, > Carlos > > -----Mensaje original----- > De: pharo-project-bounces(a)lists.gforge.inria.fr > [mailto:pharo-project-bounces@lists.gforge.inria.fr]En nombre de > Fernando olivero > Enviado el: SÃ¡bado, 23 de Mayo de 2009 06:56 a.m. > Para: Pharo-project(a)lists.gforge.inria.fr > Asunto: Re: [Pharo-project] Alien FFI & COM > > > Carlos, plese let me know if you have problems loading the AlienFFI > package. > > Look at the email i sent a couple of days ago, in there i explained > the status and how to load. > > I'm using alien to reify C libraries and be able to call its behavior > in smalltalk, doing that is fairly easy with AlienFFI and it works > fast. > > Once you get it installed and test the examples i could help you > more. > > Saludos, > Fernando > > On May 23, 2009, at 11:40 AM, StÃ©phane Ducasse wrote: > >> did you succeed to load well alien. >> Because fernando and me where cleaning it but I did not check what >> fernando published. >> I know that this is much cleaner (we remove newspeak bytecode). >> >> Now we need people to shake it. >> >> >> On May 22, 2009, at 10:55 PM, Carlos Crosetti wrote: >> >>> looking for pointers to a sample Alien FFi code calling a COM based >>> DLL.... >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > > -- > Internal Virus Database is out-of-date. > Checked by AVG. > Version: 7.5.524 / Virus Database: 270.12.11/2089 - Release Date: > 30/04/2009 > 05:53 p.m. > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009