- Pharo-dev - lists.pharo.org

Re: [Pharo-project] Relicensing effort

by Gabriel Cotelli

Ok, I will check that first... The last core image is 10304cl? On Sat, May 23, 2009 at 4:40 PM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr > wrote: > Excellent! > Gabriel what I suggest you to do is to check if the previous author is > not a person that signed the agreement (Squeak + pharo) > Often what we see is that some method got changes for a simple fix in > comment or formatting. > > Stef > > On May 23, 2009, at 9:34 PM, Gabriel Cotelli wrote: > > > > > Hi, > > I will start on re-writting this methods: > > > > Number>>reciprocal > > Fraction class>>one > > Fraction>>reciprocal > > Float class>>one > > Float>>reciprocal > > Integer class>>one > > > > All of this are almost trivial to re-write so I think it's a good > > start point. My plan is to first write a test case covering this, > > delete the old method (if present) and re-implement it. > > > > I signed the License Agrement on Friday and sent an scanned copy to > > Stephane by e-mail (probably he still doesn't update the list). > > > > Thanks, > > Gabriel > > > > On Sat, May 23, 2009 at 3:58 PM, Adrian Lienhard <adi(a)netstyle.ch> > > wrote: > > Hi Gabriel, > > > > On May 23, 2009, at 19:55 , Gabriel Cotelli wrote: > > > > > Hi! > > > > > > There is somebody coordinating the re-write of non re-licensed > > > methods? > > > > As I started working on it last week, I can coordinate. > > > > > I probably could re-write some of this methods but don't want to > > > duplicate > > > the effort :) > > > > Great! > > > > > Should I take the Pharo-Core image or the Pharo-Dev for this? > > > > The core image. > > > > I already fixed more than half of the methods. What is left can be > > seen here: > > > > http://code.google.com/p/pharo/wiki/ResultofRelicensing. > > > > I suggest that whoever starts working on a set of methods announces it > > on this mailing list. Like this we don't get a conflict. The easiest > > thing is to produce a change set and then attach it to the issue: > http://code.google.com/p/pharo/issues/detail?id=842 > > > > Thanks, > > Adrian > > > > BTW, I don't see you on the list of OK committers. Did you sign the > > agreement? Else your changes will reappear on the list of non-clean > > methods! ;) > > > > _______________________________________________ > > Pharo-project mailing list > > Pharo-project(a)lists.gforge.inria.fr > > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > > _______________________________________________ > > Pharo-project mailing list > > Pharo-project(a)lists.gforge.inria.fr > > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] windowsesque theme tweaks

by StÃ©phane Ducasse

gary I would like to include that in pharo (may be later we can package it better and more modular but for now, this is important not to lose it). So let me know. Stef On May 23, 2009, at 8:44 PM, Steve Wirts wrote: > Gary here's some tweaks I talked about (progressbar and tabbar fixed). > After the attached is filedin, evaluate > > UIThemeXP setup > > then to see the tweaks try (progress bar) > > SystemProgressMorph example > > and (tab bar) > > UITheme exampleGroups > > also an odd thing that may hint at a bug, is > the progress bar moves faster while you move > the cursor. > > thanks. > sw > > <windowsesque.st>_______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

May 23, 2009

Re: [Pharo-project] Relicensing effort

by StÃ©phane Ducasse

Excellent! Gabriel what I suggest you to do is to check if the previous author is not a person that signed the agreement (Squeak + pharo) Often what we see is that some method got changes for a simple fix in comment or formatting. Stef On May 23, 2009, at 9:34 PM, Gabriel Cotelli wrote: > > Hi, > I will start on re-writting this methods: > > Number>>reciprocal > Fraction class>>one > Fraction>>reciprocal > Float class>>one > Float>>reciprocal > Integer class>>one > > All of this are almost trivial to re-write so I think it's a good > start point. My plan is to first write a test case covering this, > delete the old method (if present) and re-implement it. > > I signed the License Agrement on Friday and sent an scanned copy to > Stephane by e-mail (probably he still doesn't update the list). > > Thanks, > Gabriel > > On Sat, May 23, 2009 at 3:58 PM, Adrian Lienhard <adi(a)netstyle.ch> > wrote: > Hi Gabriel, > > On May 23, 2009, at 19:55 , Gabriel Cotelli wrote: > > > Hi! > > > > There is somebody coordinating the re-write of non re-licensed > > methods? > > As I started working on it last week, I can coordinate. > > > I probably could re-write some of this methods but don't want to > > duplicate > > the effort :) > > Great! > > > Should I take the Pharo-Core image or the Pharo-Dev for this? > > The core image. > > I already fixed more than half of the methods. What is left can be > seen here: > > http://code.google.com/p/pharo/wiki/ResultofRelicensing. > > I suggest that whoever starts working on a set of methods announces it > on this mailing list. Like this we don't get a conflict. The easiest > thing is to produce a change set and then attach it to the issue: http://code.google.com/p/pharo/issues/detail?id=842 > > Thanks, > Adrian > > BTW, I don't see you on the list of OK committers. Did you sign the > agreement? Else your changes will reappear on the list of non-clean > methods! ;) > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by StÃ©phane Ducasse

Excellent! Thanks guys. I'm preparing a lectures for torino and I will experiment with umejava mcz fixes. Stef On May 23, 2009, at 8:49 PM, Adrian Lienhard wrote: > Wow, great analysis, Nicolas! > > I was trying to find the cause for several hours now. Your third track > exactly matches my findings. > > For example in Object>>#doesNotUnderstand: prior to the condensing, > the source contained a non-ASCII character (UTF8 encoded as the two > bytes: 192 160). This gets correctly transferred during the condensing > into the new changes file. When you don't save the image (and hence > have the standard stream without UTF8 encoder) what you see in the > source is the character Ã (this is 192). That is, we suddenly have two > characters, 192 and 160 where before there was just one. If you load a > package, MC will compare methods and think this is a change. When > loading the method from the MC file, the source is UTF8 encoded, > producing a unicode character 160. When storing this source to the > file (still without the encoder), it will just directly put 160 there. > At this point we have lost the leading byte 192. Next time we start > or save the image and have the right encoder again, it will choke > because 160 is an invalid first byte in UTF8. > > I think it's save to fix the invalid methods by overriding their > source. So we don't have to backtrack to version 10297. > > Thanks, > Adrian > > > On May 23, 2009, at 19:57 , Nicolas Cellier wrote: > >> I confirm the scenario: >> 1) update10298 condenseChanges that let (SourceFiles at: 2) class = >> StandardFileStream >> This is the seed of further problems, because further changes will >> be encoded in latin1 (or MacRoman I don't really wnt to know) >> 2) update10302 changes the methods with non ASCII characters >> 3) Stef save the image after update10304, that does reopen >> (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the >> apple. >> >> If you save the image just after the condenseChanges, no problem >> because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes >> have gotten into it, and reopened UTF-8 before any changes got into >> it. >> We must track undue usage of StandardFileStream such as >> #condenseChanges. >> >> 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >>> What happened exactly is very hard to trace because these FileStream >>> are a can of worms... >>> Here are some of my perigrinations: >>> >>> FIRST POSSIBLE TRACK: >>> >>> All methods were changed in 10305. >>> Monticello snapshot/source.st is not UTF-8. >>> If the file is opened UTF-8, then we get decompiledCode, I don't >>> know why yet... >>> But the changes still go into the change log in correct UTF-8 form, >>> so >>> that's just another bug, but not the real source of the problem. >>> For getting some worms out of the can just browse inst var defs of >>> converter in MultiByteFileStream: >>> The accessor #converter initialize converter with TextConverter >>> defaultSystemConverter which depends on LanguageEnvironment. >>> That is a Latin1TextConverter in my latin image. >>> Unless #reset is called first, in which case it will initialize >>> with a >>> UTF8TextConverter. >>> Yes, but open: fileName forWrite: writeMode, does the job too with a >>> UTF8TextConverter. >>> You still follow? me neither. >>> A better behaved is #setConverterForCode that should let non UTF-8 >>> .mcz work in UTF-8 environment, but not sure if called where >>> required... >>> I think Yoshiki changes are necessary only for writing source code >>> with character code > 255. >>> This was not the case of incriminated methods. >>> >>> SECOND POSSIBLE TRACK: >>> >>> Everything going to the change log pass thru the >>> MultiByteFileStream, >>> so how did non UTF-8 characters went in? >>> I tried to follow two other clues: >>> 1) There are senders of #primWrite:from:startingAt:count: not >>> redefined in MultiByteFileStream... >>> for example, using #next:putAll:startingAt: will bypass the >>> converter. >>> 2) using nextPutAll: with a ByteArray argument also does bypass the >>> converter (See MultiByteFileStream>>#nextPutAll:) >>> I did not find the senders (you really believe senders of >>> nextPutAll: >>> can be analyzed?). >>> I tried to instrument code with Notification, but I'm unable to >>> reproduce the problem, so that was vain... >>> >>> THIRD POSSIBLE TRACK: >>> >>> http://gforge.inria.fr/frs/download.php/22283/ >>> Pharo0.1Core-10304cl.zip >>> has the invalid UTF-8 problem, just before 10305 changes that >>> introduced decompiled code... >>> So we might attack the problem with another code snippet: >>> >>> (SystemNavigation default browseAllCallsOn: (Smalltalk >>> associationAt: >>> #SourceFiles))... >>> >>> Hmm, I might have a better clue now. >>> The problem might possibly come from the condenseChanges in >>> update10298. >>> What happen in a condenseChanges? >>> Changes are copied to this file: >>> >>> f := FileStream fileNamed: 'ST80.temp'. >>> >>> So far, so good, because the concreteStream is a >>> MultiByteFileStream. >>> >>> But the end finishes with: >>> >>> SourceFiles >>> at: 2 >>> put: (StandardFileStream oldFileNamed: oldChanges name) >>> >>> Waouh, no MultiByteFileStream here, so no more UTF-8. >>> But hey, that would be the inverse problem: reading UTF-8 text with >>> latin1 reader: I can't get an error doing this, only some strange >>> sequence of characters... (The UTF-8 encoding)... >>> Unless incriminated methods are further changed in #script376 or any >>> other method... In which case they are written in latin1 in the >>> changeLog... >>> Hmm... That could be the case eventually. We must restart update >>> process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip >>> >>> One thing is sure, at next returnFromSnapshot, FileDirectory >>> class>>startup will reopen changes UTF-8. >>> So saving the image will reopen UTF-8... >>> >>> But wait... Maybe we get enough pieces of the puzzle: >>> Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane >>> applied >>> several updates before snapshoting the image. So if Kernel and >>> System-Support are changed between 10298 and 10304, then we get the >>> explanation: >>> - condense changes put all in the .changes in UTF-8 but reopen the >>> changes in latin1 >>> - further updates up to 10304 write changes in latin1 >>> - image snapshot reopen changes in UTF-8 and thus we get further >>> invalid UTF-8... >>> >>> That's easy to reproduce. Stef, can you confirm? >>> >>> That also explain why I did not get the problem at home: I update >>> early and always save my image after. >>> After that we still have to detect and clean while Monticello >>> sources >>> are interpreted UTF-8 when they should not (FIRST TRACK) , and >>> eventually make source code go UTF-8 in Monticello, so that non >>> latin >>> programmers can use their favourite language eventually... >>> >>> Nicolas >>> >>> 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >>>> No problem I never interpreted it like that. >>>> Me too I want a system that is working >>>> >>>> Adrian I will publish a fix for DNU now >>>> and I will try later to check the fixes proposed by yoshiki >>>> >>>> stef >>>> >>>> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >>>> >>>>> Actually, the fix is even simpler: if you find a method that >>>>> raises >>>>> "invalid utf8 input detected", just browse to it with a class >>>>> browser, >>>>> and re-accept it :). >>>>> >>>>> With my previous mail, I was not implying that someone should fix >>>>> it >>>>> for me, I was merely asking for what could a quick solution be, >>>>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>>>> discussing it. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I attached here a DNU implementation I took from an older image. >>>>>> After filing this one in, I can debug DNU problems. >>>>>> >>>>>> Cheers, >>>>>> Doru >>>>>> >>>>>> <Object-doesNotUnderstand.st> >>>>>> >>>>>> >>>>>> >>>>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>>>> >>>>>>> I did the following >>>>>>> >>>>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>>>> invalid.... >>>>>>> >>>>>>> Now when I take another method >>>>>>> >>>>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>>>> >>>>>>> I will reread carefully the mails of nicolas to try to >>>>>>> understand, >>>>>>> I do not know if the fixes of yoh >>>>>>> >>>>>>> http://bugs.squeak.org/view.php?id=5996 >>>>>>> is related. >>>>>>> >>>>>>> Nicolas >>>>>>> >>>>>>>>> {Object>>#doesNotUnderstand:. >>>>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>>>> Utilities class>>#changeStampPerSe. >>>>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>>>> collect: >>>>>>>>> [:c | c charCode]] >>>>>>> >>>>>>> I cannot get that code running it break before with me. >>>>>>> >>>>>>> Stef >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Pharo-project mailing list >>>>>>> Pharo-project(a)lists.gforge.inria.fr >>>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>>>> >>>>>> -- >>>>>> www.tudorgirba.com >>>>>> >>>>>> "Not knowing how to do something is not an argument for how it >>>>>> cannot be done." >>>>>> >>>>>> _______________________________________________ >>>>>> Pharo-project mailing list >>>>>> Pharo-project(a)lists.gforge.inria.fr >>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>>> project >>>>> >>>>> -- >>>>> www.tudorgirba.com >>>>> >>>>> "Problem solving efficiency grows with the abstractness level of >>>>> problem understanding." >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>> project >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] Relicensing effort

by Gabriel Cotelli

Hi, I will start on re-writting this methods: Number>>reciprocal Fraction class>>one Fraction>>reciprocal Float class>>one Float>>reciprocal Integer class>>one All of this are almost trivial to re-write so I think it's a good start point. My plan is to first write a test case covering this, delete the old method (if present) and re-implement it. I signed the License Agrement on Friday and sent an scanned copy to Stephane by e-mail (probably he still doesn't update the list). Thanks, Gabriel On Sat, May 23, 2009 at 3:58 PM, Adrian Lienhard <adi(a)netstyle.ch> wrote: > Hi Gabriel, > > On May 23, 2009, at 19:55 , Gabriel Cotelli wrote: > > > Hi! > > > > There is somebody coordinating the re-write of non re-licensed > > methods? > > As I started working on it last week, I can coordinate. > > > I probably could re-write some of this methods but don't want to > > duplicate > > the effort :) > > Great! > > > Should I take the Pharo-Core image or the Pharo-Dev for this? > > The core image. > > I already fixed more than half of the methods. What is left can be > seen here: > > http://code.google.com/p/pharo/wiki/ResultofRelicensing. > > I suggest that whoever starts working on a set of methods announces it > on this mailing list. Like this we don't get a conflict. The easiest > thing is to produce a change set and then attach it to the issue: > http://code.google.com/p/pharo/issues/detail?id=842 > > Thanks, > Adrian > > BTW, I don't see you on the list of OK committers. Did you sign the > agreement? Else your changes will reappear on the list of non-clean > methods! ;) > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] Relicensing effort

by Adrian Lienhard

Hi Gabriel, On May 23, 2009, at 19:55 , Gabriel Cotelli wrote: > Hi! > > There is somebody coordinating the re-write of non re-licensed > methods? As I started working on it last week, I can coordinate. > I probably could re-write some of this methods but don't want to > duplicate > the effort :) Great! > Should I take the Pharo-Core image or the Pharo-Dev for this? The core image. I already fixed more than half of the methods. What is left can be seen here: http://code.google.com/p/pharo/wiki/ResultofRelicensing. I suggest that whoever starts working on a set of methods announces it on this mailing list. Like this we don't get a conflict. The easiest thing is to produce a change set and then attach it to the issue: http://code.google.com/p/pharo/issues/detail?id=842 Thanks, Adrian BTW, I don't see you on the list of OK committers. Did you sign the agreement? Else your changes will reappear on the list of non-clean methods! ;)

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by Adrian Lienhard

Wow, great analysis, Nicolas! I was trying to find the cause for several hours now. Your third track exactly matches my findings. For example in Object>>#doesNotUnderstand: prior to the condensing, the source contained a non-ASCII character (UTF8 encoded as the two bytes: 192 160). This gets correctly transferred during the condensing into the new changes file. When you don't save the image (and hence have the standard stream without UTF8 encoder) what you see in the source is the character Ã (this is 192). That is, we suddenly have two characters, 192 and 160 where before there was just one. If you load a package, MC will compare methods and think this is a change. When loading the method from the MC file, the source is UTF8 encoded, producing a unicode character 160. When storing this source to the file (still without the encoder), it will just directly put 160 there. At this point we have lost the leading byte 192. Next time we start or save the image and have the right encoder again, it will choke because 160 is an invalid first byte in UTF8. I think it's save to fix the invalid methods by overriding their source. So we don't have to backtrack to version 10297. Thanks, Adrian On May 23, 2009, at 19:57 , Nicolas Cellier wrote: > I confirm the scenario: > 1) update10298 condenseChanges that let (SourceFiles at: 2) class = > StandardFileStream > This is the seed of further problems, because further changes will > be encoded in latin1 (or MacRoman I don't really wnt to know) > 2) update10302 changes the methods with non ASCII characters > 3) Stef save the image after update10304, that does reopen > (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the > apple. > > If you save the image just after the condenseChanges, no problem > because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes > have gotten into it, and reopened UTF-8 before any changes got into > it. > We must track undue usage of StandardFileStream such as > #condenseChanges. > > 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >> What happened exactly is very hard to trace because these FileStream >> are a can of worms... >> Here are some of my perigrinations: >> >> FIRST POSSIBLE TRACK: >> >> All methods were changed in 10305. >> Monticello snapshot/source.st is not UTF-8. >> If the file is opened UTF-8, then we get decompiledCode, I don't >> know why yet... >> But the changes still go into the change log in correct UTF-8 form, >> so >> that's just another bug, but not the real source of the problem. >> For getting some worms out of the can just browse inst var defs of >> converter in MultiByteFileStream: >> The accessor #converter initialize converter with TextConverter >> defaultSystemConverter which depends on LanguageEnvironment. >> That is a Latin1TextConverter in my latin image. >> Unless #reset is called first, in which case it will initialize >> with a >> UTF8TextConverter. >> Yes, but open: fileName forWrite: writeMode, does the job too with a >> UTF8TextConverter. >> You still follow? me neither. >> A better behaved is #setConverterForCode that should let non UTF-8 >> .mcz work in UTF-8 environment, but not sure if called where >> required... >> I think Yoshiki changes are necessary only for writing source code >> with character code > 255. >> This was not the case of incriminated methods. >> >> SECOND POSSIBLE TRACK: >> >> Everything going to the change log pass thru the MultiByteFileStream, >> so how did non UTF-8 characters went in? >> I tried to follow two other clues: >> 1) There are senders of #primWrite:from:startingAt:count: not >> redefined in MultiByteFileStream... >> for example, using #next:putAll:startingAt: will bypass the >> converter. >> 2) using nextPutAll: with a ByteArray argument also does bypass the >> converter (See MultiByteFileStream>>#nextPutAll:) >> I did not find the senders (you really believe senders of nextPutAll: >> can be analyzed?). >> I tried to instrument code with Notification, but I'm unable to >> reproduce the problem, so that was vain... >> >> THIRD POSSIBLE TRACK: >> >> http://gforge.inria.fr/frs/download.php/22283/ >> Pharo0.1Core-10304cl.zip >> has the invalid UTF-8 problem, just before 10305 changes that >> introduced decompiled code... >> So we might attack the problem with another code snippet: >> >> (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: >> #SourceFiles))... >> >> Hmm, I might have a better clue now. >> The problem might possibly come from the condenseChanges in >> update10298. >> What happen in a condenseChanges? >> Changes are copied to this file: >> >> f := FileStream fileNamed: 'ST80.temp'. >> >> So far, so good, because the concreteStream is a MultiByteFileStream. >> >> But the end finishes with: >> >> SourceFiles >> at: 2 >> put: (StandardFileStream oldFileNamed: oldChanges name) >> >> Waouh, no MultiByteFileStream here, so no more UTF-8. >> But hey, that would be the inverse problem: reading UTF-8 text with >> latin1 reader: I can't get an error doing this, only some strange >> sequence of characters... (The UTF-8 encoding)... >> Unless incriminated methods are further changed in #script376 or any >> other method... In which case they are written in latin1 in the >> changeLog... >> Hmm... That could be the case eventually. We must restart update >> process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip >> >> One thing is sure, at next returnFromSnapshot, FileDirectory >> class>>startup will reopen changes UTF-8. >> So saving the image will reopen UTF-8... >> >> But wait... Maybe we get enough pieces of the puzzle: >> Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane >> applied >> several updates before snapshoting the image. So if Kernel and >> System-Support are changed between 10298 and 10304, then we get the >> explanation: >> - condense changes put all in the .changes in UTF-8 but reopen the >> changes in latin1 >> - further updates up to 10304 write changes in latin1 >> - image snapshot reopen changes in UTF-8 and thus we get further >> invalid UTF-8... >> >> That's easy to reproduce. Stef, can you confirm? >> >> That also explain why I did not get the problem at home: I update >> early and always save my image after. >> After that we still have to detect and clean while Monticello sources >> are interpreted UTF-8 when they should not (FIRST TRACK) , and >> eventually make source code go UTF-8 in Monticello, so that non latin >> programmers can use their favourite language eventually... >> >> Nicolas >> >> 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >>> No problem I never interpreted it like that. >>> Me too I want a system that is working >>> >>> Adrian I will publish a fix for DNU now >>> and I will try later to check the fixes proposed by yoshiki >>> >>> stef >>> >>> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >>> >>>> Actually, the fix is even simpler: if you find a method that raises >>>> "invalid utf8 input detected", just browse to it with a class >>>> browser, >>>> and re-accept it :). >>>> >>>> With my previous mail, I was not implying that someone should fix >>>> it >>>> for me, I was merely asking for what could a quick solution be, >>>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>>> discussing it. >>>> >>>> Cheers, >>>> Doru >>>> >>>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I attached here a DNU implementation I took from an older image. >>>>> After filing this one in, I can debug DNU problems. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> <Object-doesNotUnderstand.st> >>>>> >>>>> >>>>> >>>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>>> >>>>>> I did the following >>>>>> >>>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>>> invalid.... >>>>>> >>>>>> Now when I take another method >>>>>> >>>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>>> >>>>>> I will reread carefully the mails of nicolas to try to >>>>>> understand, >>>>>> I do not know if the fixes of yoh >>>>>> >>>>>> http://bugs.squeak.org/view.php?id=5996 >>>>>> is related. >>>>>> >>>>>> Nicolas >>>>>> >>>>>>>> {Object>>#doesNotUnderstand:. >>>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>>> Utilities class>>#changeStampPerSe. >>>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>>> collect: >>>>>>>> [:c | c charCode]] >>>>>> >>>>>> I cannot get that code running it break before with me. >>>>>> >>>>>> Stef >>>>>> >>>>>> _______________________________________________ >>>>>> Pharo-project mailing list >>>>>> Pharo-project(a)lists.gforge.inria.fr >>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>>> >>>>> -- >>>>> www.tudorgirba.com >>>>> >>>>> "Not knowing how to do something is not an argument for how it >>>>> cannot be done." >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>> project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Problem solving efficiency grows with the abstractness level of >>>> problem understanding." >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

May 23, 2009

[Pharo-project] windowsesque theme tweaks

by Steve Wirts

Gary here's some tweaks I talked about (progressbar and tabbar fixed). After the attached is filedin, evaluate UIThemeXP setup then to see the tweaks try (progress bar) SystemProgressMorph example and (tab bar) UITheme exampleGroups also an odd thing that may hint at a bug, is the progress bar moves faster while you move the cursor. thanks. sw

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by StÃ©phane Ducasse

On May 23, 2009, at 7:57 PM, Nicolas Cellier wrote: > I confirm the scenario: > 1) update10298 condenseChanges that let (SourceFiles at: 2) class = > StandardFileStream > This is the seed of further problems, because further changes will > be encoded in latin1 (or MacRoman I don't really wnt to know) > 2) update10302 changes the methods with non ASCII characters > 3) Stef save the image after update10304, that does reopen > (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the > apple. > > If you save the image just after the condenseChanges, no problem > because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes > have gotten into it, and reopened UTF-8 before any changes got into > it. > We must track undue usage of StandardFileStream such as > #condenseChanges. Ok now we cannot really rollback the changes and I fixed the methods that were leading to invalid UTF. But it means that we should check the StandardFileStream usage. I"m doing some experiences with umejava code Stef > > > 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >> What happened exactly is very hard to trace because these FileStream >> are a can of worms... >> Here are some of my perigrinations: >> >> FIRST POSSIBLE TRACK: >> >> All methods were changed in 10305. >> Monticello snapshot/source.st is not UTF-8. >> If the file is opened UTF-8, then we get decompiledCode, I don't >> know why yet... >> But the changes still go into the change log in correct UTF-8 form, >> so >> that's just another bug, but not the real source of the problem. >> For getting some worms out of the can just browse inst var defs of >> converter in MultiByteFileStream: >> The accessor #converter initialize converter with TextConverter >> defaultSystemConverter which depends on LanguageEnvironment. >> That is a Latin1TextConverter in my latin image. >> Unless #reset is called first, in which case it will initialize >> with a >> UTF8TextConverter. >> Yes, but open: fileName forWrite: writeMode, does the job too with a >> UTF8TextConverter. >> You still follow? me neither. >> A better behaved is #setConverterForCode that should let non UTF-8 >> .mcz work in UTF-8 environment, but not sure if called where >> required... >> I think Yoshiki changes are necessary only for writing source code >> with character code > 255. >> This was not the case of incriminated methods. >> >> SECOND POSSIBLE TRACK: >> >> Everything going to the change log pass thru the MultiByteFileStream, >> so how did non UTF-8 characters went in? >> I tried to follow two other clues: >> 1) There are senders of #primWrite:from:startingAt:count: not >> redefined in MultiByteFileStream... >> for example, using #next:putAll:startingAt: will bypass the >> converter. >> 2) using nextPutAll: with a ByteArray argument also does bypass the >> converter (See MultiByteFileStream>>#nextPutAll:) >> I did not find the senders (you really believe senders of nextPutAll: >> can be analyzed?). >> I tried to instrument code with Notification, but I'm unable to >> reproduce the problem, so that was vain... >> >> THIRD POSSIBLE TRACK: >> >> http://gforge.inria.fr/frs/download.php/22283/ >> Pharo0.1Core-10304cl.zip >> has the invalid UTF-8 problem, just before 10305 changes that >> introduced decompiled code... >> So we might attack the problem with another code snippet: >> >> (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: >> #SourceFiles))... >> >> Hmm, I might have a better clue now. >> The problem might possibly come from the condenseChanges in >> update10298. >> What happen in a condenseChanges? >> Changes are copied to this file: >> >> f := FileStream fileNamed: 'ST80.temp'. >> >> So far, so good, because the concreteStream is a MultiByteFileStream. >> >> But the end finishes with: >> >> SourceFiles >> at: 2 >> put: (StandardFileStream oldFileNamed: oldChanges name) >> >> Waouh, no MultiByteFileStream here, so no more UTF-8. >> But hey, that would be the inverse problem: reading UTF-8 text with >> latin1 reader: I can't get an error doing this, only some strange >> sequence of characters... (The UTF-8 encoding)... >> Unless incriminated methods are further changed in #script376 or any >> other method... In which case they are written in latin1 in the >> changeLog... >> Hmm... That could be the case eventually. We must restart update >> process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip >> >> One thing is sure, at next returnFromSnapshot, FileDirectory >> class>>startup will reopen changes UTF-8. >> So saving the image will reopen UTF-8... >> >> But wait... Maybe we get enough pieces of the puzzle: >> Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane >> applied >> several updates before snapshoting the image. So if Kernel and >> System-Support are changed between 10298 and 10304, then we get the >> explanation: >> - condense changes put all in the .changes in UTF-8 but reopen the >> changes in latin1 >> - further updates up to 10304 write changes in latin1 >> - image snapshot reopen changes in UTF-8 and thus we get further >> invalid UTF-8... >> >> That's easy to reproduce. Stef, can you confirm? >> >> That also explain why I did not get the problem at home: I update >> early and always save my image after. >> After that we still have to detect and clean while Monticello sources >> are interpreted UTF-8 when they should not (FIRST TRACK) , and >> eventually make source code go UTF-8 in Monticello, so that non latin >> programmers can use their favourite language eventually... >> >> Nicolas >> >> 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >>> No problem I never interpreted it like that. >>> Me too I want a system that is working >>> >>> Adrian I will publish a fix for DNU now >>> and I will try later to check the fixes proposed by yoshiki >>> >>> stef >>> >>> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >>> >>>> Actually, the fix is even simpler: if you find a method that raises >>>> "invalid utf8 input detected", just browse to it with a class >>>> browser, >>>> and re-accept it :). >>>> >>>> With my previous mail, I was not implying that someone should fix >>>> it >>>> for me, I was merely asking for what could a quick solution be, >>>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>>> discussing it. >>>> >>>> Cheers, >>>> Doru >>>> >>>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I attached here a DNU implementation I took from an older image. >>>>> After filing this one in, I can debug DNU problems. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> <Object-doesNotUnderstand.st> >>>>> >>>>> >>>>> >>>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>>> >>>>>> I did the following >>>>>> >>>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>>> invalid.... >>>>>> >>>>>> Now when I take another method >>>>>> >>>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>>> >>>>>> I will reread carefully the mails of nicolas to try to >>>>>> understand, >>>>>> I do not know if the fixes of yoh >>>>>> >>>>>> http://bugs.squeak.org/view.php?id=5996 >>>>>> is related. >>>>>> >>>>>> Nicolas >>>>>> >>>>>>>> {Object>>#doesNotUnderstand:. >>>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>>> Utilities class>>#changeStampPerSe. >>>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>>> collect: >>>>>>>> [:c | c charCode]] >>>>>> >>>>>> I cannot get that code running it break before with me. >>>>>> >>>>>> Stef >>>>>> >>>>>> _______________________________________________ >>>>>> Pharo-project mailing list >>>>>> Pharo-project(a)lists.gforge.inria.fr >>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>>> >>>>> -- >>>>> www.tudorgirba.com >>>>> >>>>> "Not knowing how to do something is not an argument for how it >>>>> cannot be done." >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo- >>>>> project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Problem solving efficiency grows with the abstractness level of >>>> problem understanding." >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

May 23, 2009

Re: [Pharo-project] invalid utf8 input detected

by Nicolas Cellier

I confirm the scenario: 1) update10298 condenseChanges that let (SourceFiles at: 2) class = StandardFileStream This is the seed of further problems, because further changes will be encoded in latin1 (or MacRoman I don't really wnt to know) 2) update10302 changes the methods with non ASCII characters 3) Stef save the image after update10304, that does reopen (SourceFiles at: 2) in UTF-8, but that's too late, the worm is in the apple. If you save the image just after the condenseChanges, no problem because (SourceFiles at: 2) is opened in Latin1 AFTER all the changes have gotten into it, and reopened UTF-8 before any changes got into it. We must track undue usage of StandardFileStream such as #condenseChanges. 2009/5/23 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > What happened exactly is very hard to trace because these FileStream > are a can of worms... > Here are some of my perigrinations: > > FIRST POSSIBLE TRACK: > > All methods were changed in 10305. > Monticello snapshot/source.st is not UTF-8. > If the file is opened UTF-8, then we get decompiledCode, I don't know why yet... > But the changes still go into the change log in correct UTF-8 form, so > that's just another bug, but not the real source of the problem. > For getting some worms out of the can just browse inst var defs of > converter in MultiByteFileStream: > The accessor #converter initialize converter with TextConverter > defaultSystemConverter which depends on LanguageEnvironment. > That is a Latin1TextConverter in my latin image. > Unless #reset is called first, in which case it will initialize with a > UTF8TextConverter. > Yes, but open: fileName forWrite: writeMode, does the job too with a > UTF8TextConverter. > You still follow? me neither. > A better behaved is #setConverterForCode that should let non UTF-8 > .mcz work in UTF-8 environment, but not sure if called where > required... > I think Yoshiki changes are necessary only for writing source code > with character code > 255. > This was not the case of incriminated methods. > > SECOND POSSIBLE TRACK: > > Everything going to the change log pass thru the MultiByteFileStream, > so how did non UTF-8 characters went in? > I tried to follow two other clues: > 1) There are senders of #primWrite:from:startingAt:count: not > redefined in MultiByteFileStream... > for example, using #next:putAll:startingAt: will bypass the converter. > 2) using nextPutAll: with a ByteArray argument also does bypass the > converter (See MultiByteFileStream>>#nextPutAll:) > I did not find the senders (you really believe senders of nextPutAll: > can be analyzed?). > I tried to instrument code with Notification, but I'm unable to > reproduce the problem, so that was vain... > > THIRD POSSIBLE TRACK: > > http://gforge.inria.fr/frs/download.php/22283/Pharo0.1Core-10304cl.zip > has the invalid UTF-8 problem, just before 10305 changes that > introduced decompiled code... > So we might attack the problem with another code snippet: > > (SystemNavigation default browseAllCallsOn: (Smalltalk associationAt: > #SourceFiles))... > > Hmm, I might have a better clue now. > The problem might possibly come from the condenseChanges in update10298. > What happen in a condenseChanges? > Changes are copied to this file: > > f := FileStream fileNamed: 'ST80.temp'. > > So far, so good, because the concreteStream is a MultiByteFileStream. > > But the end finishes with: > > SourceFiles > at: 2 > put: (StandardFileStream oldFileNamed: oldChanges name) > > Waouh, no MultiByteFileStream here, so no more UTF-8. > But hey, that would be the inverse problem: reading UTF-8 text with > latin1 reader: I can't get an error doing this, only some strange > sequence of characters... (The UTF-8 encoding)... > Unless incriminated methods are further changed in #script376 or any > other method... In which case they are written in latin1 in the > changeLog... > Hmm... That could be the case eventually. We must restart update > process from http://gforge.inria.fr/frs/download.php/22167/Pharo0.1Core-10296cl-2.zip > > One thing is sure, at next returnFromSnapshot, FileDirectory > class>>startup will reopen changes UTF-8. > So saving the image will reopen UTF-8... > > But wait... Maybe we get enough pieces of the puzzle: > Analyzing the Pharo0.1Core-10304cl.changes tells that Stephane applied > several updates before snapshoting the image. So if Kernel and > System-Support are changed between 10298 and 10304, then we get the > explanation: > - condense changes put all in the .changes in UTF-8 but reopen the > changes in latin1 > - further updates up to 10304 write changes in latin1 > - image snapshot reopen changes in UTF-8 and thus we get further > invalid UTF-8... > > That's easy to reproduce. Stef, can you confirm? > > That also explain why I did not get the problem at home: I update > early and always save my image after. > After that we still have to detect and clean while Monticello sources > are interpreted UTF-8 when they should not (FIRST TRACK) , and > eventually make source code go UTF-8 in Monticello, so that non latin > programmers can use their favourite language eventually... > > Nicolas > > 2009/5/23 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >> No problem I never interpreted it like that. >> Me too I want a system that is working >> >> Adrian I will publish a fix for DNU now >> and I will try later to check the fixes proposed by yoshiki >> >> stef >> >> On May 23, 2009, at 1:29 PM, Tudor Girba wrote: >> >>> Actually, the fix is even simpler: if you find a method that raises >>> "invalid utf8 input detected", just browse to it with a class browser, >>> and re-accept it :). >>> >>> With my previous mail, I was not implying that someone should fix it >>> for me, I was merely asking for what could a quick solution be, >>> because I was a bit lost (scared) :). Now, I am happy. Thanks for >>> discussing it. >>> >>> Cheers, >>> Doru >>> >>> On 23 May 2009, at 13:07, Tudor Girba wrote: >>> >>>> Hi, >>>> >>>> I attached here a DNU implementation I took from an older image. >>>> After filing this one in, I can debug DNU problems. >>>> >>>> Cheers, >>>> Doru >>>> >>>> <Object-doesNotUnderstand.st> >>>> >>>> >>>> >>>> On 23 May 2009, at 13:04, StÃ©phane Ducasse wrote: >>>> >>>>> I did the following >>>>> >>>>> (Object>>#doesNotUNderstand) getSourceFromFile and I get an >>>>> invalid.... >>>>> >>>>> Now when I take another method >>>>> >>>>> (BalloonFontTest>>#testDefaultFont) I do not get problem. >>>>> >>>>> I will reread carefully the mails of nicolas to try to understand, >>>>> I do not know if the fixes of yoh >>>>> >>>>> http://bugs.squeak.org/view.php?id=5996 >>>>> is related. >>>>> >>>>> Nicolas >>>>> >>>>>>> {Object>>#doesNotUnderstand:. >>>>>>> SystemNavigation>>#browseMethodsWhoseNamesContain:. >>>>>>> Utilities class>>#changeStampPerSe. >>>>>>> Utilities class>>#methodsWithInitials:} collect: [:e | (e >>>>>>> getSourceFromFile select: [:s | s charCode > 127]) asArray >>>>>>> collect: >>>>>>> [:c | c charCode]] >>>>> >>>>> I cannot get that code running it break before with me. >>>>> >>>>> Stef >>>>> >>>>> _______________________________________________ >>>>> Pharo-project mailing list >>>>> Pharo-project(a)lists.gforge.inria.fr >>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>>> -- >>>> www.tudorgirba.com >>>> >>>> "Not knowing how to do something is not an argument for how it >>>> cannot be done." >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >>> -- >>> www.tudorgirba.com >>> >>> "Problem solving efficiency grows with the abstractness level of >>> problem understanding." >>> >>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >

May 23, 2009