- Pharo-users - lists.pharo.org

Re: [Pharo-users] Ridiculous we are

by Alain Rastoul

Le 25/09/2014 00:06, Sven Van Caekenberghe a Ã©crit : > Alain, > The character encoding situation in Pharo is pretty good actually. The only problem is that there is some old school code left that encodes strings into strings, but today you can easily write much better and conceptually correct code. > > You could have a look at this draft chapter of the upcoming 'Enterprise Pharo' book that I am currently writing: > > http://stfx.eu/EnterprisePharo/Zinc-Encoding-Meta/ > > Concerning file system paths, FilePathEncoder and FilePluginPrimitives already do the right thing. > > Now, your idea about using UTF-8 to represent internal Strings is something that has been discussed before and in many other languages as well. The short answer is that due to it being variable length, the inefficiency is (probably) just too high. Simple indexed access becomes a problem, let alone more complex string manipulations. I am not saying that it cannot be done, I think it is just not worth the trouble. The current solution in Pharo with ByteString and WideString is quite nice (check the chapter I mentioned before). > > Sven > Very interesting ! It seems that most of what I was saying is already here :) I was not saying that Pharo should use utf8 (I mentionned utf8 because it is a standard, but I find the variable length encoding very weird), I was rather talking of using WideString in UTF 16 or 32 and that's done. I saw asWideString but didn't know about automatic convertion or codepoint selector and internal wide string support. Does it means that Pharo Greek users (for example) use WideString for Strings without having to specify it or make explicit convertions (except of course when dealing with bytes if they want to) ? If yes, very good, job is almost done :) (personnally I would also deprecate ByteString, and get rid of it, just my opinion). Thanks for the link, another good chapter . Regards, Alain

Sept. 24, 2014

Re: [Pharo-users] Ridiculous we are

by Sven Van Caekenberghe

Alain, On 24 Sep 2014, at 23:00, Alain Rastoul <alf.mmm.cat(a)gmail.com> wrote: > Le 24/09/2014 19:09, Benjamin Pollack a Ã©crit : > >> If Pharo used > ByteArrays to represent paths, with convenience methods for working with >> UTF-8 (since I do agree that's the most likely thing for a user/dev to >> want), then you'd be able to work with all files no matter what, *and* >> have a convenient way of doing so for the common case. > Hi Ben, > I strongly disagree with you on this point: using byte arrays (or byte strings) is a pain in an international context. > The OS knows about its encoding: locale for unix, code page for windows. > Windows code pages depends on country, for english windows 1252 (similar to iso-8859-1), for other european countries, other variations of 8859-xx... (welcome to ISO soup), same for unix. > > Java uses UTF8 strings and dotNet uses UTF16 strings (don't know for Python) where chars are not bytes and they are not used as byte arrays but as Character arrays. > Both do conversions from OS character set encoding to internal encoding for strings (paths and whatever). > > There is already an UTF8 and UTF16 encoding support in Pharo, but the > standard String class uses bytes, and lot of files, directories and > system methods use ByteString class and that is the problem here. > UTF8 encoding in Pharo encodes to a variable lenght ByteString, which is not the same as an (hypothetical) Utf8String where all (variable length) chars would be utf8 encoded. > Using a new UTF8 or UTF16 string class could be a major rework, > but taking a decision about about internal string encoding is needed. > As Sven says, there is no emergency and you have a workaround, but > perhaps using the existing WideString encoded as UTF16 (or UTF32?) in > some well defined classes/methods could be a good start for this rework? > IMHO the workaround of using utf8 encoded byte strings is not a good way to deal with this problem and should not be granted as "the solution". The character encoding situation in Pharo is pretty good actually. The only problem is that there is some old school code left that encodes strings into strings, but today you can easily write much better and conceptually correct code. You could have a look at this draft chapter of the upcoming 'Enterprise Pharo' book that I am currently writing: http://stfx.eu/EnterprisePharo/Zinc-Encoding-Meta/ Concerning file system paths, FilePathEncoder and FilePluginPrimitives already do the right thing. Now, your idea about using UTF-8 to represent internal Strings is something that has been discussed before and in many other languages as well. The short answer is that due to it being variable length, the inefficiency is (probably) just too high. Simple indexed access becomes a problem, let alone more complex string manipulations. I am not saying that it cannot be done, I think it is just not worth the trouble. The current solution in Pharo with ByteString and WideString is quite nice (check the chapter I mentioned before). Sven

Sept. 24, 2014

Re: [Pharo-users] Ridiculous we are

by Alain Rastoul

Le 24/09/2014 19:09, Benjamin Pollack a Ã©crit : > If Pharo used > ByteArrays to represent paths, with convenience methods for working with > UTF-8 (since I do agree that's the most likely thing for a user/dev to > want), then you'd be able to work with all files no matter what, *and* > have a convenient way of doing so for the common case. Hi Ben, I strongly disagree with you on this point: using byte arrays (or byte strings) is a pain in an international context. The OS knows about its encoding: locale for unix, code page for windows. Windows code pages depends on country, for english windows 1252 (similar to iso-8859-1), for other european countries, other variations of 8859-xx... (welcome to ISO soup), same for unix. Java uses UTF8 strings and dotNet uses UTF16 strings (don't know for Python) where chars are not bytes and they are not used as byte arrays but as Character arrays. Both do conversions from OS character set encoding to internal encoding for strings (paths and whatever). There is already an UTF8 and UTF16 encoding support in Pharo, but the standard String class uses bytes, and lot of files, directories and system methods use ByteString class and that is the problem here. UTF8 encoding in Pharo encodes to a variable lenght ByteString, which is not the same as an (hypothetical) Utf8String where all (variable length) chars would be utf8 encoded. Using a new UTF8 or UTF16 string class could be a major rework, but taking a decision about about internal string encoding is needed. As Sven says, there is no emergency and you have a workaround, but perhaps using the existing WideString encoded as UTF16 (or UTF32?) in some well defined classes/methods could be a good start for this rework? IMHO the workaround of using utf8 encoded byte strings is not a good way to deal with this problem and should not be granted as "the solution".

Sept. 24, 2014

Update of AgileVisualization.com

by Alexandre Bergel

Dear All, AgileVisualization.com , the book about Roassal, has been updated with a new chapter. The HTML versions of the chapter are also online. Agile Visualization is written using Pillar and Skeleton. Thanks Damien Cassou and Yuriy Tymchuk (Uko) for these wonderful frameworks. I am also looking for contributors. I know that some of you guys have done wonderful things with Roassal. Sharing your knowledge with the rest of us would be fantastic. You may even get paid to write a chapter :-) Get in touch with me for more info. Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Sept. 24, 2014

Re: [Pharo-users] Ridiculous we are

by Sven Van Caekenberghe

On 24 Sep 2014, at 19:09, Benjamin Pollack <benjamin(a)bitquabit.com> wrote: > On Wed, 24 Sep 2014 13:03:57 -0400, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >> >> Did you read the actual conversation in the issue ? >> >> https://pharo.fogbugz.com/f/cases/14054/Issue-with-path-with-accented-chara… >> >> It has been renamed and there is a fix (as a change set, not as a slice, yet). Basically, there was a primitive call into a plugin that failed to do encoding. >> > > No, I apologize; I missed the bug link. Thanks for reposting it. > >> Now regarding the issues you raised. Pharo does not do Unicode canonicalisation or any of that other fancy stuff (like categorisation, proper ordering and so on). This is another orthogonal and way more general issue. >> >> Regarding the pathnames encoding: if the OS itself does not know it, how can we ? > > That's actually the argument *against* using UTF-8 as the standard Pharo way to represent filenames--at least on Unix systems. If Pharo used ByteArrays to represent paths, with convenience methods for working with UTF-8 (since I do agree that's the most likely thing for a user/dev to want), then you'd be able to work with all files no matter what, *and* have a convenient way of doing so for the common case. > > This is an old discussion, and I do see both sides of it. In terms of SCMs, Mercurial and Git both just say "it's a collection of bytes", whereas Subversion says "it's Unicode code points." This has some uncomfortable implications for both systems when working on multiple platforms. Benjamin, I think I understand the concern / situation that you describe. But I fail to see how not-interpreting it and interpreting it in different encodings can work in practice, especially since your point seems to be that there is no meta information that gives a definitive answer. I would guess that other languages, say Java or Python, have some approach to handle this problem ? Also, since we are living with the current approach without much problems, I think the issue is not terribly pressing. Sven

Sept. 24, 2014

Re: [Pharo-users] Ring package, when to use it?

by Marcus Denker

On 24 Sep 2014, at 18:35, Juraj Kubelka <juraj.kubelka(a)gmail.com> wrote: > Thank you Marcus for the explanation. > > So now I understand that if I want to analyse existing packages/class/methods/etc in the image, Ring is not a kind of interest. > > But as I think about it, if someone uses Ring as a base to analyse environment, then it could be useful to use the same analysis tool for any source, e.g. not loaded packages. Am I right or are the some limitations? > > That is the ideaâ¦ Ring Definition and normal classes/methods share a common API and can be used interchangeably. Marcus

Sept. 24, 2014

Re: [Pharo-users] Ridiculous we are

by Benjamin Pollack

On Wed, 24 Sep 2014 13:03:57 -0400, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Did you read the actual conversation in the issue ? > > https://pharo.fogbugz.com/f/cases/14054/Issue-with-path-with-accented-chara… > > It has been renamed and there is a fix (as a change set, not as a slice, > yet). Basically, there was a primitive call into a plugin that failed to > do encoding. > No, I apologize; I missed the bug link. Thanks for reposting it. > Now regarding the issues you raised. Pharo does not do Unicode > canonicalisation or any of that other fancy stuff (like categorisation, > proper ordering and so on). This is another orthogonal and way more > general issue. > > Regarding the pathnames encoding: if the OS itself does not know it, how > can we ? That's actually the argument *against* using UTF-8 as the standard Pharo way to represent filenames--at least on Unix systems. If Pharo used ByteArrays to represent paths, with convenience methods for working with UTF-8 (since I do agree that's the most likely thing for a user/dev to want), then you'd be able to work with all files no matter what, *and* have a convenient way of doing so for the common case. This is an old discussion, and I do see both sides of it. In terms of SCMs, Mercurial and Git both just say "it's a collection of bytes", whereas Subversion says "it's Unicode code points." This has some uncomfortable implications for both systems when working on multiple platforms. --Benjamin

Sept. 24, 2014

Re: [Pharo-users] Ridiculous we are

by Sven Van Caekenberghe

On 24 Sep 2014, at 18:48, Benjamin Pollack <benjamin(a)bitquabit.com> wrote: > On Tue, 23 Sep 2014 08:51:54 -0400, Hilaire <hilaire(a)drgeo.eu> wrote: > >> Le 23/09/2014 14:09, Damien Cassou a Ã©crit : >>> I recently read documents about utf-8 encoding. In all of them, the >>> author says that pathnames should be kept as is because you never know >>> which encoding the filesystem uses. So, a filename should probably be >>> a bytearray. >> >> >> yes, but a #Ã© should be encoded in two bytes. > > As noted in my previous message, "Ã©" could be represented as either one or two Unicode code points, and these in turn could validly be either two or three bytes in UTF-8. My gut says that $Ã© should be U+00E9, because otherwise you should have to use two Characters ($e and $Â´), but you could legitimately argue otherwise as well, and at any rate, #Ã© could definitely be either. This is likely the core of the issue you're hitting. Did you read the actual conversation in the issue ? https://pharo.fogbugz.com/f/cases/14054/Issue-with-path-with-accented-chara… It has been renamed and there is a fix (as a change set, not as a slice, yet). Basically, there was a primitive call into a plugin that failed to do encoding. Now regarding the issues you raised. Pharo does not do Unicode canonicalisation or any of that other fancy stuff (like categorisation, proper ordering and so on). This is another orthogonal and way more general issue. Regarding the pathnames encoding: if the OS itself does not know it, how can we ? I think that the current approach (assuming UTF-8) makes (the most) sense for a system that runs on multiple platforms. Sven

Sept. 24, 2014

Re: [Pharo-users] [ANN] I stop maintaining the Ubuntu packages

by volkert＠nivoba.de

Sad news. I switched from OSX to Ubuntu and i also plan to switch from Win8 to Ubuntu. I really hope "Pharo" will find a new maintainer for the most popular Linux Distribution. Thank you for your work ... BW, Volkert Am 24.09.2014 um 09:41 schrieb Damien Cassou: > Dear all, > > I've recently switched my Linux distribution from Ubuntu to NixOS. > This means I can no longer maintain the Ubuntu packages and their PPA > (https://launchpad.net/~pharo/) > > These packages need a maintainer! > > The good side of it is that it is not a lot of work as I did all the > automating infrastructure: > https://github.com/pharo-project/pharo-ubuntu. Basically, the > maintainer will have to launch a few shell scripts regularly (once a > month for example). I will obviously be available to help the new > maintainer. > > The community needs *you*. > > Good news is: I packaged the Pharo VM for the Nix package manager > which can be installed on many Unix (e.g., Linux, MacOS X, FreeBSD). > And NixOS will have the Pharo VM package in its next release in > October! > -- www.nivoba.de "The more complex an object, the larger the investment in learning to use it, and the greater the resistance to abandon it.", NW

Sept. 24, 2014

Re: [Pharo-users] Packaging Pharo for many distributions at once

by Damien Cassou

On Wed, Sep 24, 2014 at 12:47 PM, Thierry Goubier <thierry.goubier(a)gmail.com> wrote: > All non-native package managers have that issue. nix included -- Damien Cassou http://damiencassou.seasidehosting.st "Success is the ability to go from one failure to another without losing enthusiasm." Winston Churchill

Sept. 24, 2014