March 2010 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] squeakToUTF-8 and related?

by Henrik Johansen

On Mar 29, 2010, at 11:52 43AM, Nicolas Cellier wrote: > 2010/3/29 Henrik Johansen <henrik.s.johansen(a)veloxit.no>: >> >> On Mar 29, 2010, at 11:16 30AM, Nicolas Cellier wrote: >> >>> I presume that under the idiom "latin1" you refer to code page 1252 >>> rather than iso8859-L1, right ? >>> >>> Nicolas >> Good question :) >> What IS the presumed internal encoding of Bytestrings in Squeak? >> That's the one I meant, I merely assumed it was latin1 seeing as how the text converter refers to it as such. >> Personally I thought it was iso8859-L1, seeing as the bytestring to unicode conversion does a simple shift of chars > 127 to the 0080 - 00FF range. >> >> Cheers, >> Henry >> > > From what I understood, CP1252 is Microsoft "latin1" and use codes 128 to 159. > ISO8859-L1 match fisrt 256 codes of unicode latin-1 and has codes 128 > to 159 unused. > You know, when Microsoft "uses" a standard, it's always a better standard ;) > > I have nothing against CP1252, it's an optimization which avoid > wasting 32 cheap codes. > But I'm not sure about various compatibility issues in/with the > external world... > > Squeak clearly uses CP1252. > For Pharo, there might be a mix of the two since Sophie-like > refactorings. Surely what John was refering to. > > Nicolas Ummm... All the utf8-converters in squeak use Unicode value:, which maps directly from charCode 128->255 to Unicode value 128->255. Unicode value 128->255 IS iso8859-L1, so if squeak uses CP1252 as internal format, all the converters in Squeak are wrong. Cheers, Henry

March 29, 2010

Re: [Pharo-project] Pharo 1.0 fixing tests

by StÃ©phane Ducasse

For now >> I think that theses tests do not show bugs but probably badly designed tests. >> So why don't you move them to expected failures and we release 1.0? > > Then there would be a failure when these tests don't fail, which happens every second run. So we would need to remove them. But if somebody has some cycles to figure out why they fail we could probably fix them. We should evaluate if we have the time and if not then psuh 1.0 out of the door. Stef

March 29, 2010

Re: [Pharo-project] squeakToUTF-8 and related?

by Nicolas Cellier

2010/3/29 Henrik Johansen <henrik.s.johansen(a)veloxit.no>: > > On Mar 29, 2010, at 11:16 30AM, Nicolas Cellier wrote: > >> I presume that under the idiom "latin1" you refer to code page 1252 >> rather than iso8859-L1, right ? >> >> Nicolas > Good question :) > What IS the presumed internal encoding of Bytestrings in Squeak? > That's the one I meant, I merely assumed it was latin1 seeing as how the text converter refers to it as such. > Personally I thought it was iso8859-L1, seeing as the bytestring to unicode conversion does a simple shift of chars > 127 to the 0080 - 00FF range. > > Cheers, > Henry > >From what I understood, CP1252 is Microsoft "latin1" and use codes 128 to 159. ISO8859-L1 match fisrt 256 codes of unicode latin-1 and has codes 128 to 159 unused. You know, when Microsoft "uses" a standard, it's always a better standard ;) I have nothing against CP1252, it's an optimization which avoid wasting 32 cheap codes. But I'm not sure about various compatibility issues in/with the external world... Squeak clearly uses CP1252. For Pharo, there might be a mix of the two since Sophie-like refactorings. Surely what John was refering to. Nicolas > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

March 29, 2010

Re: [Pharo-project] squeakToUTF-8 and related?

by Henrik Johansen

On Mar 29, 2010, at 11:16 30AM, Nicolas Cellier wrote: > I presume that under the idiom "latin1" you refer to code page 1252 > rather than iso8859-L1, right ? > > Nicolas Good question :) What IS the presumed internal encoding of Bytestrings in Squeak? That's the one I meant, I merely assumed it was latin1 seeing as how the text converter refers to it as such. Personally I thought it was iso8859-L1, seeing as the bytestring to unicode conversion does a simple shift of chars > 127 to the 0080 - 00FF range. Cheers, Henry

March 29, 2010

Re: [Pharo-project] The Pharo Linux Vm [WAS] Squeak VM+FT2Plugin, all Pharo 1.0 rc3 tests green

by laurent laffont

2010/3/29 Mariano Martinez Peck <marianopeck(a)gmail.com> > > >> >> >>> - In the README.TXT we also have to put links for help: squeakvm site, >>> posts like Laurent or Adrain >>> - Gunification >>> - It needs to have a name of PharoXXX. Not squeak. This is VERY confusing >>> for newcomers. >>> >> >> I agree we need to explain better the distinction between the VM and the >> image. I'm not sure about renaming squeak-vm -> pharo-vm. It's not a fork. I >> tend to think it will bring higher confusion. >> > > I don't agree. Image you are a new comer. For you, Pharo is Pharo. You > probaly have no idea what Squeak is. So...you go to Pharo website, and > download the image, and the VM. Imagine when you go to download the VM and > it says "SqueakVM". I would say "WTF?????" "WTF is squeak?" > I am not saying forking the VM. Just to rename it. Actually, the windows VM > has that name: " > http://gforge.inria.fr/frs/download.php/26654/PharoVM-Win32-3.11.8.zip" > And the same with exupery: " > http://gforge.inria.fr/frs/download.php/24736/pharo-vm-0.15.2f-linux.zip" > > OK. So this is one more argument to have a git repository to snapshot VM sources + patches for Pharo. Then the VM build process should produce binaries named pharo / pharovm / pharo.sh for all platforms. I don't know if it's easy, I'll try to take a look. > Maybe you can open a git repository on INRIA servers so we can keep track >> of patches + README included in the VM for Pharo. I've used Javier and >> Levente patches and these are not included in squeak-vm svn. I sent a mail >> last week on vm-dev but no answers. >> This way builders can do: >> git clone git://......squeak-vm >> cd ..squeak-vm../build >> ... configure && make && make install. >> For packagers it would be cool. >> Developpers can have their own branches and submit patches more easily, >> add more visibility. >> With git-svn we can synchronize from official Squeak-VM svn (until they >> switched to git or another DVCS :) >> >> What do you think ? >> >> > I like the idea....but what about the sync with the public/official SVN > repo ? how we will be able to merge ? or the GIT repo will be only for > patches ? > You can have a git branch synchronized with squeak-vm subversion (see http://progit.org/book/ch8-1.html) a git branch for pharo-vm. Then push regularly changes from squeak-vm branch to pharo-vm branch. Maybe there's a git expert reading this list ? (I'm not one). Laurent Laffont > > Cheers > > Mariano > > > >> Laurent Laffont >> >> >>> Cheers >>> >>> Mariano >>> >>> >>> >>> >>>> >>>> >>>>> Ok.....but not only plugins. For exaple, if I don't want UTF-8 ? >>>>> >>>>> >>>> errr... maybe it's time to ask to the vm-dev list :) >>>> >>>> >>>>> >>>>> >>>>> ufff.....what an idiot...thanks! did you already discovered I am a >>>>> newbie here ? :) >>>>> >>>> >>>> >>>> Me too, but a little less every day :) >>>> >>>> Laurent Laffont >>>> >>>> _______________________________________________ >>>> Pharo-project mailing list >>>> Pharo-project(a)lists.gforge.inria.fr >>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>>> >>> >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> Pharo-project(a)lists.gforge.inria.fr >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

March 29, 2010

Re: [Pharo-project] The Pharo Linux Vm [WAS] Squeak VM+FT2Plugin, all Pharo 1.0 rc3 tests green

by Henrik Johansen

On Mar 29, 2010, at 9:29 45AM, Mariano Martinez Peck wrote: > > Yes! The windows VM has already that. The MacVM, I sent an email few months ago to explain how to do it. > If you want, I can upload a Squeak 4.2.2beta1U.app but with the Pharo icons and named Pharo 4.2.2beta1U.app > Then we link from the website to that VM. What do you think ? > > I know there are newer MacOS VMs, but they are still in beta and for the Pharo 1.0 I think the best VM is the 4.2.2.beta1 > > What do you think ? > > Cheers > > Mariano Please use 4.2.3.beta1, as it includes the bitblt fixes which affects StrikeFont rendering. So if someone switches to Bitmap DejaVu fonts, they don't have to be annoyed their text looks strange when dragging windows :) Cheers, Henry

March 29, 2010

Re: [Pharo-project] squeakToUTF-8 and related?

by Nicolas Cellier

I presume that under the idiom "latin1" you refer to code page 1252 rather than iso8859-L1, right ? Nicolas 2010/3/29 Henrik Johansen <henrik.s.johansen(a)veloxit.no>: > > On Mar 28, 2010, at 4:36 13PM, StÃ©phane Ducasse wrote: > >> Hi >> >> I'm trying to remember the situation with the internal representation of string in pharo/squeak >> to revise http://book.seaside.st/book/in-action/serving-files/character-encodings/sea… >> >> I saw that in pharo we have this NonASCIIMap. I do not remember what have been done in pharo. >> Argh memory leaks.... Nicolas do you remember the situation? > NonASCIIMap is used for quickly determining whether the string with no character codes > 127 (ie only ascii characters). > It's very useful for doing primitive accellerated isAsciiString, which in the case of ascii-compatible encodings (utf8, latin1, macroman, etc.) would mean no conversion is required for it to be the "appropriate" internal bytestring format. > It's used f.ex. in the nextChunk code, > Strangely it is also used in Â FileStream writeSourceCodeFrom: baseName: isSt: , for some reason we there use a MacRoman if stream contents isAscii, which really makes no sense, but whatever. > > John pointed out some converters were lying, I'm not entirely sure that's true anymore, what IS certain though, is the external code format used is inconsistent, depending on from where/how you save/load it. > It really should be cleaned up to always store in utf8, and possibly also latin1 if possible. > All this should be cleared up to always try reading as UTF8, then raising an InvalidUTF8 error which can be handled by telling it to use a different converter and restart. > Possibly chosen from a menu when dropping a file on image, or choosing an alternative automatically if we know the possible other encodings a file could have been saved as, Â not sure how to best do it for scripts given as parameters when launching the vm > > On the font rendering side, I agree with Nicolas it's too complicated doing font rendering in-image, FT is an ok compromise though. > As for the bitmap strikefont rendering, what is really needed is a way to specify the charset it represents, and mappings from the internal string encodings to its glyphs. > F.ex., Bitmap DejaVu is really latin15, so it will currently render some ByteString characters incorrectly, as well as render some Unicode chars it really has glyphs for as ?. (such as the euro sign) > > Which all really has nothing to do with your initial question :) > The internal representation of strings really hasn't changed since it was written, with the exception that leadingChar for WideStrings are now zero. > As far as I can tell, that means the interal storage format of widestrings is now equivalent to utf32, not sure what Byte Order it uses though, or if that is even consistent across platforms. :) > > The point about using WaKomEncoded, and passing all strings going into/out of the image through an encoder is still valid. > > Cheers, > Henry > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

March 29, 2010

Re: [Pharo-project] squeakToUTF-8 and related?

by Henrik Johansen

On Mar 28, 2010, at 4:36 13PM, StÃ©phane Ducasse wrote: > Hi > > I'm trying to remember the situation with the internal representation of string in pharo/squeak > to revise http://book.seaside.st/book/in-action/serving-files/character-encodings/sea… > > I saw that in pharo we have this NonASCIIMap. I do not remember what have been done in pharo. > Argh memory leaks.... Nicolas do you remember the situation? NonASCIIMap is used for quickly determining whether the string with no character codes > 127 (ie only ascii characters). It's very useful for doing primitive accellerated isAsciiString, which in the case of ascii-compatible encodings (utf8, latin1, macroman, etc.) would mean no conversion is required for it to be the "appropriate" internal bytestring format. It's used f.ex. in the nextChunk code, Strangely it is also used in FileStream writeSourceCodeFrom: baseName: isSt: , for some reason we there use a MacRoman if stream contents isAscii, which really makes no sense, but whatever. John pointed out some converters were lying, I'm not entirely sure that's true anymore, what IS certain though, is the external code format used is inconsistent, depending on from where/how you save/load it. It really should be cleaned up to always store in utf8, and possibly also latin1 if possible. All this should be cleared up to always try reading as UTF8, then raising an InvalidUTF8 error which can be handled by telling it to use a different converter and restart. Possibly chosen from a menu when dropping a file on image, or choosing an alternative automatically if we know the possible other encodings a file could have been saved as, not sure how to best do it for scripts given as parameters when launching the vm On the font rendering side, I agree with Nicolas it's too complicated doing font rendering in-image, FT is an ok compromise though. As for the bitmap strikefont rendering, what is really needed is a way to specify the charset it represents, and mappings from the internal string encodings to its glyphs. F.ex., Bitmap DejaVu is really latin15, so it will currently render some ByteString characters incorrectly, as well as render some Unicode chars it really has glyphs for as ?. (such as the euro sign) Which all really has nothing to do with your initial question :) The internal representation of strings really hasn't changed since it was written, with the exception that leadingChar for WideStrings are now zero. As far as I can tell, that means the interal storage format of widestrings is now equivalent to utf32, not sure what Byte Order it uses though, or if that is even consistent across platforms. :) The point about using WaKomEncoded, and passing all strings going into/out of the image through an encoder is still valid. Cheers, Henry

March 29, 2010

[Pharo-project] [update] #10516

by Marcus Denker

10516 ----- Issue 2215: testFontFullName recursively deletes all files in the image directory -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

March 29, 2010

Re: [Pharo-project] squeakToUTF-8 and related?

by StÃ©phane Ducasse

Thanks john. >>> You should ask Sophie team, their knowledge certainly is far more >>> advanced than mine. >> >> The problem is that most of them disappeared after the java rewrite announce. > > Ok, well it's not as if we disappeared from the planet... > > The UTF8 work came out of work done on the OLPC/eToys, check with Bert > > You need to carefully check that some of the UTF8 encoders/decoder methods, or classes actually DO make UTF8 > One of them lied and did a UTF8ToMacRoman or was that UTF8ToLatin1 conversion when it claimed a UTF8ToUnicode32. Argh! So we should really find a way to rescue what was done in sophie. > In Sophie because we wrote it's own storage subsystem we could ensure that all textual data going to storage was UTF8, and > all text coming in was UTF8->WideStrings. Also all references to external resources were made into URI that were UTF8 and http encoded safe. > Mind we did have a tiny disagreements at times about unicode characters and encoding in URLs between Safari and FireFox, never was quite > sure who had the bug. > > -- > =========================================================================== > John M. McIntosh <johnmci(a)smalltalkconsulting.com> Twitter: squeaker68882 > Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com > =========================================================================== > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

March 29, 2010