July 2018 - Pharo-dev - lists.pharo.org

IndexedSlots in combination with StatefulTraits

by Torsten Bergmann

I guess I found a bug in accessing Slots using #slots in combination with IndexedSlots and traits. To reproduce use latest Pharo 7 (Build 1126) First create a class with a slot, note that the slot needs to be : Object subclass: #ClassA slots: { IndexedSlot named: #upper } classVariables: { } package: 'Slot-Bugs' As it is the first slot it internally receives an index 1 as you can check with "ClassA slots first". Now define a stateful trait with another indexed slot: Trait named: #StatefulTrait uses: {} slots: { IndexedSlot named: #slotFromTrait } category: 'Slot-Bugs' And create a new subclass using the new Trait ClassA subclass: #ClassB uses: StatefulTrait instanceVariableNames: '' classVariableNames: '' package: 'Slot-Bugs' 1. when you evaluate "ClassA slots" it returns {#FirstSlot => Slot} which is correct 2. when you evaluate "StatefulTrait slots" it returns {#slotFromTrait => Slot} which is correct 3. when you evaluate "ClassB allSlots" returns "an OrderedCollection(#upper => InstanceVariableSlot #slotFromTrait => InstanceVariableSlot)" which is OK but 4. when you evaluate "ClassB slots" it returns {#slotFromTrait => InstanceVariableSlot} which is NOT correct as class B does not define the slot, it is defined in the trait So I think 4. is wrong and reasons is the implementation of #slot: slots "I remove the slots comming from a traitComposition" ^ super slots reject:[ :e | self traitComposition slots includes:e ] If you debug you will notice that super slots returns our #slotFromTrait with an index of 2 while self traitComposition slots returns our #slotFromTrait with an index of 1 while which is the same #slotFromTrait but a different index - therefore it does not get removed. Dont know what is the best way to fix this without any side effects. Commenst and help appreciated. Thanks T.

July 17, 2018

Re: [Pharo-dev] ZnUnicodeComposingReadStream?

by Alistair Grant

Hi Sven & Max, On Fri, Jul 13, 2018 at 07:59:32PM +0200, Sven Van Caekenberghe wrote: > Alistair, are you aware of the following (article/codebase) ? > > https://medium.com/concerning-pharo/an-implementation-of-unicode-normalizat… > > Due to the size of the full DB it is doubtful it would become a standard part of Pharo though. > > Sven Thanks again for the link, it has helped my (still limited) understanding of Unicode. The reason I started looking in to this was that my file attribute modification tests failed on Mac OS. The problem is that Mac OS requires file names to be decomposed UTF8, and my plugin wasn't doing the conversion. Following the general principle of trying to keep the VM minimal and do as much as possible in the image, I had hoped I could do the UTF8 (de)composition in the image. But it turns out that Mac OS doesn't follow the standard rules, so programs really need to use the native file name encoding routines on Mac OS. So that's the path I'll be following in this instance. I still really appreciate the link, and will be exploring the Unicode package more. On Fri, Jul 13, 2018 at 10:50:36PM +0200, Max Leske wrote: > Hi Alistair, > > *nix systems usually come with the iconv[1] command line program that > implements the normalization and denormalization algorithms, or Uconv 2, a > library that does the same thing. These algorithms include a lot of black magic > and I recommend to not make your hands dirty with them. With the FFI interface > Pharo has today it shouldn't be too hard to call out to Uconv (although I'm not > saying it's trivial; I've written a VM plugin that we use a work to interface > with Uconv and you do have to know how encodings and iconv work) or execute > iconv directly. > > I can send you a copy of the plugin code if you want, actually, I may put it on > github if there's interest. > > Cheers, > Max > > [1] https://linux.die.net/man/1/iconv > [2] https://en.wikipedia.org/wiki/Uconv > [3] http://site.icu-project.org/ > > On Sat, Jul 14, 2018 at 08:20:23AM +0200, Max Leske wrote: > I realize I got things mixed up a bit: Uconv is a program akin to Iconv. What > we interface with is libicu. > > Max The VM already uses libiconv for the encoding translation on linux. As far as I know, the routines haven't been exposed directly to the image (although I haven't looked carefully). I'd be interested in looking at your plugin - I'm still working through the current FilePlugin behaviour, but I think it would be useful to have these routines available directly from the image for debugging, etc. Thanks again, Alistair

July 16, 2018

Off-topic: The Value Shift in Commons-based Peer Production

by Eliot Miranda

The Value Shift in Commons-based Peer Production, or how open source communities can fairly create and share wealth while protecting themselves from extractive exploitation https://www.boell.de/sites/default/files/value_in_the_commons_economy.pdf _,,,^..^,,,_ (phone)

July 15, 2018

[Pharo 7.0-dev] Build #1126: 22261-Improving-method-comments-and-adding-examples

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1126 was: SUCCESS. The Pull Request #1636 was integrated: "22261-Improving-method-comments-and-adding-examples" Pull request url: https://github.com/pharo-project/pharo/pull/1636 Issue Url: https://pharo.fogbugz.com/f/cases/22261 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

July 15, 2018

[Pharo 7.0-dev] Build #1125: 22260-Small-Cleanup-ECompletion

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1125 was: SUCCESS. The Pull Request #1633 was integrated: "22260-Small-Cleanup-ECompletion" Pull request url: https://github.com/pharo-project/pharo/pull/1633 Issue Url: https://pharo.fogbugz.com/f/cases/22260 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

July 14, 2018

[Pharo 7.0-dev] Build #1124: 22267-option-for-inform-in-window

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1124 was: SUCCESS. The Pull Request #1637 was integrated: "22267-option-for-inform-in-window" Pull request url: https://github.com/pharo-project/pharo/pull/1637 Issue Url: https://pharo.fogbugz.com/f/cases/22267 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

July 14, 2018

Re: [Pharo-dev] Problem with zinc 2.9.2

by Max Leske

Hi Nicolas, The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file. Thanks for the pointer though. Cheers, Max [1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271 On 12 Jul 2018, at 23:19, Nicolas Cellier wrote: > Isn't there a recent PR on opensmalltalk that address just this? > > Le jeu. 12 juil. 2018 Ã 23:16, Max Leske <maxleske(a)gmail.com> a > Ã©crit : > >> Hi Norbert, >> >> I was able to reproduce the problem and then identify the culprit, >> although what I don't yet understand is how this is related to the >> changes >> in Zinc. >> >> Anyway, the problem is that the file stream isn't properly flushed in >> ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the >> last >> statement fixes your problem. Apparently, StandardFileStream / >> MultiByteFileStream perform a simple close() on the file which, >> according >> to the man page on close(), does *not* guarantee that the contents >> have >> all been written to file. In this case a flush is necessary because >> the >> entire file is immediately read again. >> >> Here's a smaller test case for you to play with Sven: >> >> ZnClient new >> url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; >> downloadTo: '/tmp/foobar.zip'. >> bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | >> s contents ]. >> Transcript open; show: bytes size; cr. >> 5 seconds asDelay wait. >> Transcript show: ('/tmp/foobar.zip' asFileReference >> binaryReadStreamDo: [ :s | s contents ]) size. >> >> The output in the Transcript should be: >> >> 315392 >> 318420 >> >> Cheers, >> Max >> >> On 12 Jul 2018, at 8:17, Norbert Hartl wrote: >> >> Am 12.07.2018 um 08:05 schrieb Max Leske maxleske(a)gmail.com: >> >> On 11 Jul 2018, at 22:44, Norbert Hartl wrote: >> >> Hi Max, >> >> I constructed a case that fails exactly like I experience it. Could >> you >> try? Just unpack the attachment on a linux, set PHARO_VM env to your >> executable and execute build.sh >> >> I will. Might take a couple of days though. >> >> No problem. Iâm happy if you find time. >> >> Norbert >> >> Max >> >> Norbert >> >> Am 10.07.2018 um 09:17 schrieb Max Leske maxleske(a)gmail.com: >> >> On 10 Jul 2018, at 8:48, Norbert Hartl wrote: >> >> Max, >> >> thanks for taking the effort. >> >> No worries. >> >> Am 10.07.2018 um 08:37 schrieb Max Leske maxleske(a)gmail.com: >> >> I did my best to reproduce the issue but didn't have any luck. I >> really >> need access to a Linux VM to debug this. So I'm praying that Apple >> fixes >> the access restrictions to kernel extensions in their next beta... >> >> BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there >> is >> indeed a chance that something goes wrong during download of the zip >> archive and that something may be tied to a difference in the >> versions of >> Zinc (although I still think that's unlikely). >> >> Yes there is potential but I donât see it. I take a fresh 6.1 image >> and >> load my project into. Iâm not sure but I think zinc 2.9.2 is loaded >> rather >> early in that process. So I wonder why it does not go wrong in the >> first >> phase. And also not if I load the test group within the first phase. >> It must be either the second Metacello invocation or the stopping, >> copying >> and starting of the image. >> I try to isolate this case more and provide a script that goes wrong >> on my >> machine. But it will take some time because I needed to stop trying >> to >> solve this as I wasted nearly two days on that already. >> >> Let me know once you have something and I'll try to help out. >> >> Norbert >> >> Max >> >> On 9 Jul 2018, at 19:43, Norbert Hartl wrote: >> >> Am 09.07.2018 um 19:10 schrieb Max Leske maxleske(a)gmail.com: >> >> I checked the Parasol Archive and it does not appear to be in Zip64 >> format >> (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive >> can >> read the Parasol zip). So my next guess is that there's either a >> problem >> with Metacello or Pharo in the way that ZipArchive is used (e.g. >> endianness >> problem or non-binary stream data). It would therefore be helpful to >> find >> out what happens in ZipArchive>>readFrom:, i.e. what kind of stream >> is >> passed / opened, is it binary, does the file exist and is it still >> the >> correct file. >> >> I couldnât see anything obvious. The file in the debug message >> exists, it >> is a readable zip file. The way Metacello uses it it is a >> StandardFileStream. It switches it to binary in the code. >> But the only difference between a working and non-working build is >> the >> upgrade to zinc 2.9.2. >> >> Norbert >> >> I'd debug it myself but I can't run VirtualBox at the moment because >> I'm >> on the macOS beta and it won't start... >> >> Max >> >> On 9 Jul 2018, at 18:31, Norbert Hartl wrote: >> >> Hi Max, >> >> Am 09.07.2018 um 18:18 schrieb Max Leske maxleske(a)gmail.com: >> >> Hi Norbert, >> >> This is a bit of a guess, but it's possible that the archive that is >> downloaded from github is in Zip64 format and that the libraries for >> extracting Zip64 are missing on your Linux. That would of course >> contradict >> the experience that the same operation appears to work in 6.1. >> >> to be honest I donât know what Zip64 format is. I thought the Zip >> classes >> are pure smalltalk for unpacking. >> >> Try extracting the archive manually on your Linux machine with the >> same >> method that Metacello uses (I assume, Metacello uses the ZipPlugin, >> which >> will probably use the system libraries). >> >> I have no ZipPlugin as library in any of my vms. >> >> But there are zips downloaded and unpacked. I start the image the >> first >> time loading all the code of my project. Then it is saved, copied to >> a new >> name and reopened to load the tests and then it fails. I did try to >> load >> the tests in the first run and then it works. >> >> Iâm running out of ideas >> >> thanks, >> >> Norbert >> >> Cheers, >> Max >> >> On 9 Jul 2018, at 17:14, Norbert Hartl wrote: >> >> With the help of Esteban I got one step further. If I do >> >> MCWorkingCopy >> managersForClass: ZnFileSystemUtils >> do: [ :each | each ancestry initialize ] >> >> before loading my project it updates Zinc-FileSystem as well. Sadly >> it >> still does not work for me because I get >> >> Loading baseline of BaselineOfMobilityMap... >> ...RETRY->BaselineOfParasol >> ...RETRY->BaselineOfParasol[31mError: can't find EOCD position >> >> and I donât know why. zip file is downloaded from github and >> present but >> it fails and only on my jenkins on linux. On my Mac this works. >> >> I try a few things but then Iâm back on pharo6.1 for the time being >> :( >> >> Norbert >> >> Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert(a)hartl.name: >> >> Really? I thought the same but then I didnât believe it works like >> that. >> >> Anyway this would be very easy to solve. We just need to ask Sven if >> he is >> fine with doing an empty .16 version for Zinc-FileSystem and does an >> in-place version reset of 2.9.2 or a new 2.9.3. Iâm not fully >> convinced >> that will solve it but the cost wonât be very high. >> >> Norbert >> >> Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot >> cyril.ferlicot(a)gmail.com: >> >> Need to investigate more. There are two .15 versions but there is 1 >> year >> in between (if you didnât notice TheIntegratior.15 is from 2017). >> Just want >> to have more context information because I can only see that this is >> strange but lacking insight. >> >> Iâm trying to figure out why it does not update Zinc-FileSystem. No >> matter >> what I do I cannot get Metacello to load the newer package. That >> would fix >> the issue because Iâm loading 2.9.2 which should have >> Zinc-FileSystem-SVC.15 and not stay on the one included in the image. >> >> I think this is important for everyone that has a product based on >> 6.1 and >> that want to migrate someday to pharo7. This way it is impossible to >> do it >> step by step. >> >> If there is a package .15 in the image and a package .15 in the repo, >> I >> think Metacello will not update because it rely on the numbers to >> know when >> to update. If it find a .15 and currently have a .15 I think >> Metacello will >> think they are at the same version. >> >> Norbert >> >> -- >> Cyril Ferlicot >> https://ferlicot.fr >> >>

July 14, 2018

Re: [Pharo-dev] ZnUnicodeComposingReadStream?

by Max Leske

I realize I got things mixed up a bit: Uconv is a program akin to Iconv. What we interface with is libicu. Max On 13 Jul 2018, at 22:50, Max Leske wrote: > Hi Alistair, > > *nix systems usually come with the iconv[1] command line program that > implements the normalization and denormalization algorithms, or Uconv > [2] (from ICU [3]), a library that does the same thing. These > algorithms include a lot of black magic and I recommend to not make > your hands dirty with them. With the FFI interface Pharo has today it > shouldn't be too hard to call out to Uconv (although I'm not saying > it's trivial; I've written a VM plugin that we use a work to interface > with Uconv and you do have to know how encodings and iconv work) or > execute iconv directly. > > I can send you a copy of the plugin code if you want, actually, I may > put it on github if there's interest. > > Cheers, > Max > > [1] https://linux.die.net/man/1/iconv > [2] https://en.wikipedia.org/wiki/Uconv > [3] http://site.icu-project.org/ > > > On 13 Jul 2018, at 20:22, Alistair Grant wrote: > >> Hi Sven, >> >> Thanks very much for your quick reply... >> >> On Fri, 13 Jul 2018 at 19:59, Sven Van Caekenberghe <sven(a)stfx.eu> >> wrote: >>> >>> Alistair, are you aware of the following (article/codebase) ? >>> >>> https://medium.com/concerning-pharo/an-implementation-of-unicode-normalizat… >>> >>> Due to the size of the full DB it is doubtful it would become a >>> standard part of Pharo though. >>> >>> Sven >> >> I hadn't seen this. I'll read it next (although I think it will take >> me longer than 17 minutes :-)). >> >> But a quick, partial answer is that I was planning on only supporting >> the composition and decomposition tables that are already included in >> the main image as part of CombinedChar (see the Composition and >> Decomposition class variables). >> >> Thanks again, >> Alistair >> >> >>>> On 13 Jul 2018, at 19:46, Alistair Grant <akgrant0710(a)gmail.com> >>>> wrote: >>>> >>>> Hi Sven & Everyone, >>>> >>>> I need to convert an UTF8 encoded decomposed stream (Mac OS file >>>> names) in to a composed string, e.g.: >>>> >>>> string: 'test-Ã¤Ã¶Ã¼-aÌoÌuÌ' >>>> code points: #(116 101 115 116 45 228 246 252 45 97 776 111 776 117 >>>> 776) >>>> utf8 encoding: #[116 101 115 116 45 195 164 195 182 195 188 45 97 >>>> 204 >>>> 136 111 204 136 117 204 136] >>>> >>>> In the above string, the first group of 3 accented characters are >>>> the >>>> same as the second group of 3, but are encoded differently - code >>>> points (228 246 252) vs (97 776 111 776 117 776). >>>> >>>> Reading the utf8 encoded stream should result in: >>>> >>>> string: 'test-Ã¤Ã¶Ã¼-Ã¤Ã¶Ã¼' >>>> code points: #(116 101 115 116 45 228 246 252 45 228 246 252) >>>> utf8 encoding: #[116 101 115 116 45 195 164 195 182 195 188 45 195 >>>> 164 >>>> 195 182 195 188] >>>> >>>> My current thought is to write a ZnUnicodeComposingReadStream which >>>> would wrap a ZnCharacterReadStream and return the composed >>>> characters. >>>> >>>> What do you think? >>>> >>>> Thanks! >>>> Alistair >>>> >>> >>>

July 14, 2018

Re: [Pharo-dev] ZnUnicodeComposingReadStream?

by Max Leske

Hi Alistair, *nix systems usually come with the iconv[1] command line program that implements the normalization and denormalization algorithms, or Uconv [2] (from ICU [3]), a library that does the same thing. These algorithms include a lot of black magic and I recommend to not make your hands dirty with them. With the FFI interface Pharo has today it shouldn't be too hard to call out to Uconv (although I'm not saying it's trivial; I've written a VM plugin that we use a work to interface with Uconv and you do have to know how encodings and iconv work) or execute iconv directly. I can send you a copy of the plugin code if you want, actually, I may put it on github if there's interest. Cheers, Max [1] https://linux.die.net/man/1/iconv [2] https://en.wikipedia.org/wiki/Uconv [3] http://site.icu-project.org/ On 13 Jul 2018, at 20:22, Alistair Grant wrote: > Hi Sven, > > Thanks very much for your quick reply... > > On Fri, 13 Jul 2018 at 19:59, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: >> >> Alistair, are you aware of the following (article/codebase) ? >> >> https://medium.com/concerning-pharo/an-implementation-of-unicode-normalizat… >> >> Due to the size of the full DB it is doubtful it would become a >> standard part of Pharo though. >> >> Sven > > I hadn't seen this. I'll read it next (although I think it will take > me longer than 17 minutes :-)). > > But a quick, partial answer is that I was planning on only supporting > the composition and decomposition tables that are already included in > the main image as part of CombinedChar (see the Composition and > Decomposition class variables). > > Thanks again, > Alistair > > >>> On 13 Jul 2018, at 19:46, Alistair Grant <akgrant0710(a)gmail.com> >>> wrote: >>> >>> Hi Sven & Everyone, >>> >>> I need to convert an UTF8 encoded decomposed stream (Mac OS file >>> names) in to a composed string, e.g.: >>> >>> string: 'test-Ã¤Ã¶Ã¼-aÌoÌuÌ' >>> code points: #(116 101 115 116 45 228 246 252 45 97 776 111 776 117 >>> 776) >>> utf8 encoding: #[116 101 115 116 45 195 164 195 182 195 188 45 97 >>> 204 >>> 136 111 204 136 117 204 136] >>> >>> In the above string, the first group of 3 accented characters are >>> the >>> same as the second group of 3, but are encoded differently - code >>> points (228 246 252) vs (97 776 111 776 117 776). >>> >>> Reading the utf8 encoded stream should result in: >>> >>> string: 'test-Ã¤Ã¶Ã¼-Ã¤Ã¶Ã¼' >>> code points: #(116 101 115 116 45 228 246 252 45 228 246 252) >>> utf8 encoding: #[116 101 115 116 45 195 164 195 182 195 188 45 195 >>> 164 >>> 195 182 195 188] >>> >>> My current thought is to write a ZnUnicodeComposingReadStream which >>> would wrap a ZnCharacterReadStream and return the composed >>> characters. >>> >>> What do you think? >>> >>> Thanks! >>> Alistair >>> >> >>

July 13, 2018

Re: [Pharo-dev] ZnUnicodeComposingReadStream?

by Alistair Grant

Hi Sven, Thanks very much for your quick reply... On Fri, 13 Jul 2018 at 19:59, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Alistair, are you aware of the following (article/codebase) ? > > https://medium.com/concerning-pharo/an-implementation-of-unicode-normalizat… > > Due to the size of the full DB it is doubtful it would become a standard part of Pharo though. > > Sven I hadn't seen this. I'll read it next (although I think it will take me longer than 17 minutes :-)). But a quick, partial answer is that I was planning on only supporting the composition and decomposition tables that are already included in the main image as part of CombinedChar (see the Composition and Decomposition class variables). Thanks again, Alistair > > On 13 Jul 2018, at 19:46, Alistair Grant <akgrant0710(a)gmail.com> wrote: > > > > Hi Sven & Everyone, > > > > I need to convert an UTF8 encoded decomposed stream (Mac OS file > > names) in to a composed string, e.g.: > > > > string: 'test-Ã¤Ã¶Ã¼-aÌoÌuÌ' > > code points: #(116 101 115 116 45 228 246 252 45 97 776 111 776 117 776) > > utf8 encoding: #[116 101 115 116 45 195 164 195 182 195 188 45 97 204 > > 136 111 204 136 117 204 136] > > > > In the above string, the first group of 3 accented characters are the > > same as the second group of 3, but are encoded differently - code > > points (228 246 252) vs (97 776 111 776 117 776). > > > > Reading the utf8 encoded stream should result in: > > > > string: 'test-Ã¤Ã¶Ã¼-Ã¤Ã¶Ã¼' > > code points: #(116 101 115 116 45 228 246 252 45 228 246 252) > > utf8 encoding: #[116 101 115 116 45 195 164 195 182 195 188 45 195 164 > > 195 182 195 188] > > > > My current thought is to write a ZnUnicodeComposingReadStream which > > would wrap a ZnCharacterReadStream and return the composed characters. > > > > What do you think? > > > > Thanks! > > Alistair > > > >

July 13, 2018