March 2018 - Pharo-users - lists.pharo.org

Issues with recent Pharos reading files

by Andrew P. Black

Iâm having trouble reading files with the most recent âstableâ versions of Pharo. Today, I downloaded Pharo (32-bit), loaded my code into the image using git and Iceberg, and half of my tests failed because Pharo claimed that the files that are on my disk are in a directory that does not exist. If I go back to an older image, everything works fine. Iâm seeing a similar thing on Travis-CI: the build under Pharo 6 works fine, and all the tests are green, but the build under Pharo 7 fails: > Loaded -> SmalltalkCI-Coverage-Core-fn.5 --- filetree:///home/travis/smalltalkCI-master/repository [:] --- cache > SmalltalkCIPharo class>>deprecationWarning (TestCoverage is Undeclared) > SmalltalkCIPharo7 class>>stdout (ZnCrPortableWriteStream is Undeclared) > Loaded -> SmalltalkCI-Pharo-Core-fn.62 --- filetree:///home/travis/smalltalkCI-master/repository [:] --- cache > ...finished baselineMessageNotUnderstood: receiver of "on:" is nil > UndefinedObject(Object)>>doesNotUnderstand: #on: > SmalltalkCIPharo7 class>>stdout > SmalltalkCIPharo7 class(SmalltalkCI class)>>travisFold:id:block: > SmalltalkCIPharo7 class(SmalltalkCI class)>>stage:id:block: > SmalltalkCIPharo7(SmalltalkCI)>>reportImageInfo > SmalltalkCIPharo7(SmalltalkCI)>>basicLoad > [ super basicLoad ] in [ [ super basicLoad ] > on: (Smalltalk at: #MCMergeOrLoadWarning) > do: [ :ex | ex resume: true ] ] in SmalltalkCIPharo7(SmalltalkCIPharo)>>basicLoad in Block: [ super basicLoad ] (The logs are here: https://travis-ci.org/apblack/GraceInPharo. Look at build #83. Iâm not sure if they are public â I think that they are!) Whatâs weird is that my co-worker, who I was introducing to Pharo, was able to download a Pharo distribution and install and run the same git commit with no problems. So I decided that maybe my launcher was out of date. I just downloaded and installed a new version of the launcher. When I use it to create a new image, it is unable to launch it, because it canât find the VM. (The VM package was there, but with a âno entryâ sign across it.) I eventually patch up the launcher by hand, and load my code into a new image. Half of my tests fail once again. There are two issues. First, file streams are now ZnCharacterReadStreams instead of MultiByteFileStreams. Ok, thatâs a change that I had been expecting. But ZnCharacterReadStreams donât understand readStream, whereas MultiByteFileStreams did (and returned self). I can code around that. The second issue is one that I canât code around. Iâm getting a "directory does not existâ error from DiskStore>>#basicEntry:path:nodesDo:, even though the sending method, FileSystemStore>>#directoryAt:ifAbsent:nodesDo: has just correctly verified that the directory does exist. Looking at the offending method, I think that the problem is that Primitives lookupEntryIn: encodedPathString index: index. always returns #badDirectoryPath. Since this method contains a call to a primitive, followed by ^ #badDirectoryPath if the primitive fails, Iâm guessing that either I donât have the primitive installed, or I have an old version of the primitive. How do I fix this? More interesting in some ways: if I download a new copy of the launcher, and then use the launcher to install a new copy of the development image, shouldnât all of the necessary files be installed along with it. >From the fact that this isnât working on Travis either, Iâm guessing that there might be a problem with the distribution. Andrew

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Thierry Goubier

2018-03-20 13:20 GMT+01:00 Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no>: > EstebanLM wrote >>> On 20 Mar 2018, at 12:56, Henrik Sperre Johansen < > >> henrik.s.johansen@ > >> > wrote: >>> >>> This is great! >>> And, +1 on the need for some more convenience constructors... >>> >>> Word of warning (which would be nice to mention in the guide); using >>> writeStreamDo: and manually constructed buffered writers will not #flush >>> the >>> buffered stream automatically on close, so it's crucial this is done >>> manually. >> >> wellâ¦ in fact it should flush it, isnât? >> >>> > > The binary filestream flushes to disk on each write operation iirc (or, at > least when it is being closed), but it has no way to know it is wrapped by a > buffered stream which also needs to be flushed to actually write the > remaining buffer contents to file. Could a binary filestream that has flush requirements (and close requirements as well) be made aware of the presence of the buffered stream above it to ensure a close / flush is done correctly? Could structures where you have multiple buffered streams concurrently opened on a single binary file stream considered as not exactly correct? Thierry > > Cheers, > Henry > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Henrik Sperre Johansen

EstebanLM wrote >> On 20 Mar 2018, at 12:56, Henrik Sperre Johansen < > henrik.s.johansen@ > > wrote: >> >> This is great! >> And, +1 on the need for some more convenience constructors... >> >> Word of warning (which would be nice to mention in the guide); using >> writeStreamDo: and manually constructed buffered writers will not #flush >> the >> buffered stream automatically on close, so it's crucial this is done >> manually. > > wellâ¦ in fact it should flush it, isnât? > >> The binary filestream flushes to disk on each write operation iirc (or, at least when it is being closed), but it has no way to know it is wrapped by a buffered stream which also needs to be flushed to actually write the remaining buffer contents to file. Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Esteban Lorenzano

> On 20 Mar 2018, at 12:56, Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > This is great! > And, +1 on the need for some more convenience constructors... > > Word of warning (which would be nice to mention in the guide); using > writeStreamDo: and manually constructed buffered writers will not #flush the > buffered stream automatically on close, so it's crucial this is done > manually. wellâ¦ in fact it should flush it, isnât? > > charGen := Random seed: 16543986234. > output := String new: 1024*1024*50. > 1 to: output size do: [ :ix | output at: ix put: ((charGen nextInt: 256) - > 1) asCharacter ]. > > [(File named: 'utf16test.txt') writeStreamDo: [:ws | |encStream| > encStream := ZnCharacterWriteStream > on: (ZnBufferedWriteStream on: ws) > encoding: 'utf16'. > encStream nextPutAll: output > ]] timeToRun. > > FileSize: 102 336 KB. > > [(File named: 'utf16test2.txt') writeStreamDo: [:ws | |encStream| > encStream := ZnCharacterWriteStream > on: (ZnBufferedWriteStream on: ws) > encoding: 'utf16'. > encStream nextPutAll: output;flush > ]] timeToRun. > > FileSize: 102 400 KB. > > It's about 4 times slower than the corresponding > > [FileStream forceNewFileNamed: 'utf16test2.txt' do: [:ws | > ws converter: UTF16TextConverter new. > ws nextPutAll: output > ]] timeToRun > > but it's 100MB of UTF16, and 4 secs vs 1, so probably not relevant to real > use cases . is always relevant. I wonder why would be 4x to the 2nd one. Esteban > > Another addition to the guide valuable to those of use still stuck in the > stone age, would be code equivalents to FileStream methods: > forceNewFileNamed:do: > newFileNamed:do: > oldFileNamed:do: > > Cheers, > Henry > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Henrik Sperre Johansen

This is great! And, +1 on the need for some more convenience constructors... Word of warning (which would be nice to mention in the guide); using writeStreamDo: and manually constructed buffered writers will not #flush the buffered stream automatically on close, so it's crucial this is done manually. charGen := Random seed: 16543986234. output := String new: 1024*1024*50. 1 to: output size do: [ :ix | output at: ix put: ((charGen nextInt: 256) - 1) asCharacter ]. [(File named: 'utf16test.txt') writeStreamDo: [:ws | |encStream| encStream := ZnCharacterWriteStream on: (ZnBufferedWriteStream on: ws) encoding: 'utf16'. encStream nextPutAll: output ]] timeToRun. FileSize: 102 336 KB. [(File named: 'utf16test2.txt') writeStreamDo: [:ws | |encStream| encStream := ZnCharacterWriteStream on: (ZnBufferedWriteStream on: ws) encoding: 'utf16'. encStream nextPutAll: output;flush ]] timeToRun. FileSize: 102 400 KB. It's about 4 times slower than the corresponding [FileStream forceNewFileNamed: 'utf16test2.txt' do: [:ws | ws converter: UTF16TextConverter new. ws nextPutAll: output ]] timeToRun but it's 100MB of UTF16, and 4 secs vs 1, so probably not relevant to real use cases . Another addition to the guide valuable to those of use still stuck in the stone age, would be code equivalents to FileStream methods: forceNewFileNamed:do: newFileNamed:do: oldFileNamed:do: Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Sven Van Caekenberghe

I would very much prefer to take things slowly. First we have to make the whole migration work, we are not yet there, there are lots of important things to fix. If there are serious performance regressions, then yes we have to fix them. But improving things or adding features should be done when everything is stable again. > On 20 Mar 2018, at 10:31, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > 2018-03-20 10:15 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > We can cover this case by extra variable maxBufferSize. > > Operations like upToEnd will relocate maximum buffer. > #next: related methods can be also improved by it too. We will need extra method: > > ZnBufferedReadStream>>nextBuffer: requiredSize > newBufferSize := requiredSize min: maxBufferSize max: minBufferSize. > buffer size = newBufferSize ifFalse: [ > buffer := self collectionSpecies new: size]. > limit := stream readInto: buffer startingAt: 1 count: buffer size. > position := 1 > > But it will require another variable minBufferSize. > > By default it would be equal to given size in method #sizeBuffer:. And special users will specify concrete value. FileHandler will use file size for this. > It will also improve buffer size for small files where we do not need big buffer. > > 2018-03-20 9:57 GMT+01:00 Guillermo Polito <guillermopolito(a)gmail.com>: > > > On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > > > > Guillermo Polito wrote > >> I've done also some profiling and it does not look like we've lost in > >> performance either (reading and decoding a 35MB file): > > > > Your 1st (new API) example: "0:00:00:01.535" > > Your 2nd (old API) example:"0:00:00:01.732" > > *** but *** > > Your 1st (new API) example using the same consuming selector as example #1 > > (#upToEnd): "0:00:00:07.816" or 4.5x slower > > Yes and no ;-) > > Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do basically the same as example 1 using #next: > > The Zn #upToEnd implementation does not use size knowledge (as streams are infinite). > > As a side note: using #size and #position[:] on variable encoded streams like UTF8 is *very* dangerous and should be avoided. Counting/indexing bytes in the underlying stream is *not* the same as counting/indexing decoded characters. It can be the source of subtle bugs. > > Sean, > > Think that the Buffered stream can be reused in the context of sockets too. And there size has no meaning. > > Basically the cost in there is on growing and copying the internal buffer used to read. This buffer for the 35M file ends up being ~35M, so it has to grow and copy several times... > Using #next: will preallocate the good size for the internal buffer. > > Now, I'd say first that if people want performance they should use the good API :). > > Maybe the ZnBuffered stream could ask its underlying stream for its #preferredInternalBufferSizeForUpToAll. But that would contaminate all the stream hierarchy... Is it worth it? Dunno... > > > > > ----- > > Cheers, > > Sean > > -- > > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html > > > > > > > > -- > > Guille Polito > Research Engineer > > Centre de Recherche en Informatique, Signal et Automatique de Lille > CRIStAL - UMR 9189 > French National Center for Scientific Research - http://www.cnrs.fr > > Web: http://guillep.github.io > Phone: +33 06 52 70 66 13 > >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Denis Kudriashov

2018-03-20 10:15 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > We can cover this case by extra variable maxBufferSize. > Operations like upToEnd will relocate maximum buffer. #next: related methods can be also improved by it too. We will need extra method: ZnBufferedReadStream>>nextBuffer: requiredSize newBufferSize := requiredSize min: maxBufferSize max: minBufferSize. buffer size = newBufferSize ifFalse: [ buffer := self collectionSpecies new: size]. limit := stream readInto: buffer startingAt: 1 count: buffer size. position := 1 But it will require another variable minBufferSize. By default it would be equal to given size in method #sizeBuffer:. And > special users will specify concrete value. FileHandler will use file size > for this. > It will also improve buffer size for small files where we do not need big > buffer. > > 2018-03-20 9:57 GMT+01:00 Guillermo Polito <guillermopolito(a)gmail.com>: > >> >> >> On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> >> wrote: >> >>> >>> >>> > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> >>> wrote: >>> > >>> > Guillermo Polito wrote >>> >> I've done also some profiling and it does not look like we've lost in >>> >> performance either (reading and decoding a 35MB file): >>> > >>> > Your 1st (new API) example: "0:00:00:01.535" >>> > Your 2nd (old API) example:"0:00:00:01.732" >>> > *** but *** >>> > Your 1st (new API) example using the same consuming selector as >>> example #1 >>> > (#upToEnd): "0:00:00:07.816" or 4.5x slower >>> >>> Yes and no ;-) >>> >>> Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do >>> basically the same as example 1 using #next: >>> >>> The Zn #upToEnd implementation does not use size knowledge (as streams >>> are infinite). >>> >>> As a side note: using #size and #position[:] on variable encoded streams >>> like UTF8 is *very* dangerous and should be avoided. Counting/indexing >>> bytes in the underlying stream is *not* the same as counting/indexing >>> decoded characters. It can be the source of subtle bugs. >>> >> >> Sean, >> >> Think that the Buffered stream can be reused in the context of sockets >> too. And there size has no meaning. >> >> Basically the cost in there is on growing and copying the internal buffer >> used to read. This buffer for the 35M file ends up being ~35M, so it has to >> grow and copy several times... >> Using #next: will preallocate the good size for the internal buffer. >> >> Now, I'd say first that if people want performance they should use the >> good API :). >> >> Maybe the ZnBuffered stream could ask its underlying stream for its >> #preferredInternalBufferSizeForUpToAll. But that would contaminate all >> the stream hierarchy... Is it worth it? Dunno... >> >> >>> >>> >>> > ----- >>> > Cheers, >>> > Sean >>> > -- >>> > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >>> > >>> >>> >>> >> >> >> -- >> >> >> >> Guille Polito >> >> Research Engineer >> >> Centre de Recherche en Informatique, Signal et Automatique de Lille >> >> CRIStAL - UMR 9189 >> >> French National Center for Scientific Research - *http://www.cnrs.fr >> <http://www.cnrs.fr>* >> >> >> *Web:* *http://guillep.github.io* <http://guillep.github.io> >> >> *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> >> > >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Denis Kudriashov

We can cover this case by extra variable maxBufferSize. By default it would be equal to given size in method #sizeBuffer:. And special users will specify concrete value. FileHandler will use file size for this. It will also improve buffer size for small files where we do not need big buffer. 2018-03-20 9:57 GMT+01:00 Guillermo Polito <guillermopolito(a)gmail.com>: > > > On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > >> >> >> > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> >> wrote: >> > >> > Guillermo Polito wrote >> >> I've done also some profiling and it does not look like we've lost in >> >> performance either (reading and decoding a 35MB file): >> > >> > Your 1st (new API) example: "0:00:00:01.535" >> > Your 2nd (old API) example:"0:00:00:01.732" >> > *** but *** >> > Your 1st (new API) example using the same consuming selector as example >> #1 >> > (#upToEnd): "0:00:00:07.816" or 4.5x slower >> >> Yes and no ;-) >> >> Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do >> basically the same as example 1 using #next: >> >> The Zn #upToEnd implementation does not use size knowledge (as streams >> are infinite). >> >> As a side note: using #size and #position[:] on variable encoded streams >> like UTF8 is *very* dangerous and should be avoided. Counting/indexing >> bytes in the underlying stream is *not* the same as counting/indexing >> decoded characters. It can be the source of subtle bugs. >> > > Sean, > > Think that the Buffered stream can be reused in the context of sockets > too. And there size has no meaning. > > Basically the cost in there is on growing and copying the internal buffer > used to read. This buffer for the 35M file ends up being ~35M, so it has to > grow and copy several times... > Using #next: will preallocate the good size for the internal buffer. > > Now, I'd say first that if people want performance they should use the > good API :). > > Maybe the ZnBuffered stream could ask its underlying stream for its # > preferredInternalBufferSizeForUpToAll. But that would contaminate all the > stream hierarchy... Is it worth it? Dunno... > > >> >> >> > ----- >> > Cheers, >> > Sean >> > -- >> > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >> > >> >> >> > > > -- > > > > Guille Polito > > Research Engineer > > Centre de Recherche en Informatique, Signal et Automatique de Lille > > CRIStAL - UMR 9189 > > French National Center for Scientific Research - *http://www.cnrs.fr > <http://www.cnrs.fr>* > > > *Web:* *http://guillep.github.io* <http://guillep.github.io> > > *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Guillermo Polito

On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> > wrote: > > > > Guillermo Polito wrote > >> I've done also some profiling and it does not look like we've lost in > >> performance either (reading and decoding a 35MB file): > > > > Your 1st (new API) example: "0:00:00:01.535" > > Your 2nd (old API) example:"0:00:00:01.732" > > *** but *** > > Your 1st (new API) example using the same consuming selector as example > #1 > > (#upToEnd): "0:00:00:07.816" or 4.5x slower > > Yes and no ;-) > > Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do > basically the same as example 1 using #next: > > The Zn #upToEnd implementation does not use size knowledge (as streams are > infinite). > > As a side note: using #size and #position[:] on variable encoded streams > like UTF8 is *very* dangerous and should be avoided. Counting/indexing > bytes in the underlying stream is *not* the same as counting/indexing > decoded characters. It can be the source of subtle bugs. > Sean, Think that the Buffered stream can be reused in the context of sockets too. And there size has no meaning. Basically the cost in there is on growing and copying the internal buffer used to read. This buffer for the 35M file ends up being ~35M, so it has to grow and copy several times... Using #next: will preallocate the good size for the internal buffer. Now, I'd say first that if people want performance they should use the good API :). Maybe the ZnBuffered stream could ask its underlying stream for its #preferredInternalBufferSizeForUpToAll. But that would contaminate all the stream hierarchy... Is it worth it? Dunno... > > > > ----- > > Cheers, > > Sean > > -- > > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html > > > > > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Sven Van Caekenberghe

> On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > > Guillermo Polito wrote >> I've done also some profiling and it does not look like we've lost in >> performance either (reading and decoding a 35MB file): > > Your 1st (new API) example: "0:00:00:01.535" > Your 2nd (old API) example:"0:00:00:01.732" > *** but *** > Your 1st (new API) example using the same consuming selector as example #1 > (#upToEnd): "0:00:00:07.816" or 4.5x slower Yes and no ;-) Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do basically the same as example 1 using #next: The Zn #upToEnd implementation does not use size knowledge (as streams are infinite). As a side note: using #size and #position[:] on variable encoded streams like UTF8 is *very* dangerous and should be avoided. Counting/indexing bytes in the underlying stream is *not* the same as counting/indexing decoded characters. It can be the source of subtle bugs. > ----- > Cheers, > Sean > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

March 20, 2018