- Pharo-users - lists.pharo.org

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Thierry Goubier

2018-03-20 13:20 GMT+01:00 Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no>: > EstebanLM wrote >>> On 20 Mar 2018, at 12:56, Henrik Sperre Johansen < > >> henrik.s.johansen@ > >> > wrote: >>> >>> This is great! >>> And, +1 on the need for some more convenience constructors... >>> >>> Word of warning (which would be nice to mention in the guide); using >>> writeStreamDo: and manually constructed buffered writers will not #flush >>> the >>> buffered stream automatically on close, so it's crucial this is done >>> manually. >> >> wellâ¦ in fact it should flush it, isnât? >> >>> > > The binary filestream flushes to disk on each write operation iirc (or, at > least when it is being closed), but it has no way to know it is wrapped by a > buffered stream which also needs to be flushed to actually write the > remaining buffer contents to file. Could a binary filestream that has flush requirements (and close requirements as well) be made aware of the presence of the buffered stream above it to ensure a close / flush is done correctly? Could structures where you have multiple buffered streams concurrently opened on a single binary file stream considered as not exactly correct? Thierry > > Cheers, > Henry > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Henrik Sperre Johansen

EstebanLM wrote >> On 20 Mar 2018, at 12:56, Henrik Sperre Johansen < > henrik.s.johansen@ > > wrote: >> >> This is great! >> And, +1 on the need for some more convenience constructors... >> >> Word of warning (which would be nice to mention in the guide); using >> writeStreamDo: and manually constructed buffered writers will not #flush >> the >> buffered stream automatically on close, so it's crucial this is done >> manually. > > wellâ¦ in fact it should flush it, isnât? > >> The binary filestream flushes to disk on each write operation iirc (or, at least when it is being closed), but it has no way to know it is wrapped by a buffered stream which also needs to be flushed to actually write the remaining buffer contents to file. Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Esteban Lorenzano

> On 20 Mar 2018, at 12:56, Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > This is great! > And, +1 on the need for some more convenience constructors... > > Word of warning (which would be nice to mention in the guide); using > writeStreamDo: and manually constructed buffered writers will not #flush the > buffered stream automatically on close, so it's crucial this is done > manually. wellâ¦ in fact it should flush it, isnât? > > charGen := Random seed: 16543986234. > output := String new: 1024*1024*50. > 1 to: output size do: [ :ix | output at: ix put: ((charGen nextInt: 256) - > 1) asCharacter ]. > > [(File named: 'utf16test.txt') writeStreamDo: [:ws | |encStream| > encStream := ZnCharacterWriteStream > on: (ZnBufferedWriteStream on: ws) > encoding: 'utf16'. > encStream nextPutAll: output > ]] timeToRun. > > FileSize: 102 336 KB. > > [(File named: 'utf16test2.txt') writeStreamDo: [:ws | |encStream| > encStream := ZnCharacterWriteStream > on: (ZnBufferedWriteStream on: ws) > encoding: 'utf16'. > encStream nextPutAll: output;flush > ]] timeToRun. > > FileSize: 102 400 KB. > > It's about 4 times slower than the corresponding > > [FileStream forceNewFileNamed: 'utf16test2.txt' do: [:ws | > ws converter: UTF16TextConverter new. > ws nextPutAll: output > ]] timeToRun > > but it's 100MB of UTF16, and 4 secs vs 1, so probably not relevant to real > use cases . is always relevant. I wonder why would be 4x to the 2nd one. Esteban > > Another addition to the guide valuable to those of use still stuck in the > stone age, would be code equivalents to FileStream methods: > forceNewFileNamed:do: > newFileNamed:do: > oldFileNamed:do: > > Cheers, > Henry > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Henrik Sperre Johansen

This is great! And, +1 on the need for some more convenience constructors... Word of warning (which would be nice to mention in the guide); using writeStreamDo: and manually constructed buffered writers will not #flush the buffered stream automatically on close, so it's crucial this is done manually. charGen := Random seed: 16543986234. output := String new: 1024*1024*50. 1 to: output size do: [ :ix | output at: ix put: ((charGen nextInt: 256) - 1) asCharacter ]. [(File named: 'utf16test.txt') writeStreamDo: [:ws | |encStream| encStream := ZnCharacterWriteStream on: (ZnBufferedWriteStream on: ws) encoding: 'utf16'. encStream nextPutAll: output ]] timeToRun. FileSize: 102 336 KB. [(File named: 'utf16test2.txt') writeStreamDo: [:ws | |encStream| encStream := ZnCharacterWriteStream on: (ZnBufferedWriteStream on: ws) encoding: 'utf16'. encStream nextPutAll: output;flush ]] timeToRun. FileSize: 102 400 KB. It's about 4 times slower than the corresponding [FileStream forceNewFileNamed: 'utf16test2.txt' do: [:ws | ws converter: UTF16TextConverter new. ws nextPutAll: output ]] timeToRun but it's 100MB of UTF16, and 4 secs vs 1, so probably not relevant to real use cases . Another addition to the guide valuable to those of use still stuck in the stone age, would be code equivalents to FileStream methods: forceNewFileNamed:do: newFileNamed:do: oldFileNamed:do: Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Sven Van Caekenberghe

I would very much prefer to take things slowly. First we have to make the whole migration work, we are not yet there, there are lots of important things to fix. If there are serious performance regressions, then yes we have to fix them. But improving things or adding features should be done when everything is stable again. > On 20 Mar 2018, at 10:31, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > 2018-03-20 10:15 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > We can cover this case by extra variable maxBufferSize. > > Operations like upToEnd will relocate maximum buffer. > #next: related methods can be also improved by it too. We will need extra method: > > ZnBufferedReadStream>>nextBuffer: requiredSize > newBufferSize := requiredSize min: maxBufferSize max: minBufferSize. > buffer size = newBufferSize ifFalse: [ > buffer := self collectionSpecies new: size]. > limit := stream readInto: buffer startingAt: 1 count: buffer size. > position := 1 > > But it will require another variable minBufferSize. > > By default it would be equal to given size in method #sizeBuffer:. And special users will specify concrete value. FileHandler will use file size for this. > It will also improve buffer size for small files where we do not need big buffer. > > 2018-03-20 9:57 GMT+01:00 Guillermo Polito <guillermopolito(a)gmail.com>: > > > On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > > > > Guillermo Polito wrote > >> I've done also some profiling and it does not look like we've lost in > >> performance either (reading and decoding a 35MB file): > > > > Your 1st (new API) example: "0:00:00:01.535" > > Your 2nd (old API) example:"0:00:00:01.732" > > *** but *** > > Your 1st (new API) example using the same consuming selector as example #1 > > (#upToEnd): "0:00:00:07.816" or 4.5x slower > > Yes and no ;-) > > Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do basically the same as example 1 using #next: > > The Zn #upToEnd implementation does not use size knowledge (as streams are infinite). > > As a side note: using #size and #position[:] on variable encoded streams like UTF8 is *very* dangerous and should be avoided. Counting/indexing bytes in the underlying stream is *not* the same as counting/indexing decoded characters. It can be the source of subtle bugs. > > Sean, > > Think that the Buffered stream can be reused in the context of sockets too. And there size has no meaning. > > Basically the cost in there is on growing and copying the internal buffer used to read. This buffer for the 35M file ends up being ~35M, so it has to grow and copy several times... > Using #next: will preallocate the good size for the internal buffer. > > Now, I'd say first that if people want performance they should use the good API :). > > Maybe the ZnBuffered stream could ask its underlying stream for its #preferredInternalBufferSizeForUpToAll. But that would contaminate all the stream hierarchy... Is it worth it? Dunno... > > > > > ----- > > Cheers, > > Sean > > -- > > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html > > > > > > > > -- > > Guille Polito > Research Engineer > > Centre de Recherche en Informatique, Signal et Automatique de Lille > CRIStAL - UMR 9189 > French National Center for Scientific Research - http://www.cnrs.fr > > Web: http://guillep.github.io > Phone: +33 06 52 70 66 13 > >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Denis Kudriashov

2018-03-20 10:15 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > We can cover this case by extra variable maxBufferSize. > Operations like upToEnd will relocate maximum buffer. #next: related methods can be also improved by it too. We will need extra method: ZnBufferedReadStream>>nextBuffer: requiredSize newBufferSize := requiredSize min: maxBufferSize max: minBufferSize. buffer size = newBufferSize ifFalse: [ buffer := self collectionSpecies new: size]. limit := stream readInto: buffer startingAt: 1 count: buffer size. position := 1 But it will require another variable minBufferSize. By default it would be equal to given size in method #sizeBuffer:. And > special users will specify concrete value. FileHandler will use file size > for this. > It will also improve buffer size for small files where we do not need big > buffer. > > 2018-03-20 9:57 GMT+01:00 Guillermo Polito <guillermopolito(a)gmail.com>: > >> >> >> On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> >> wrote: >> >>> >>> >>> > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> >>> wrote: >>> > >>> > Guillermo Polito wrote >>> >> I've done also some profiling and it does not look like we've lost in >>> >> performance either (reading and decoding a 35MB file): >>> > >>> > Your 1st (new API) example: "0:00:00:01.535" >>> > Your 2nd (old API) example:"0:00:00:01.732" >>> > *** but *** >>> > Your 1st (new API) example using the same consuming selector as >>> example #1 >>> > (#upToEnd): "0:00:00:07.816" or 4.5x slower >>> >>> Yes and no ;-) >>> >>> Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do >>> basically the same as example 1 using #next: >>> >>> The Zn #upToEnd implementation does not use size knowledge (as streams >>> are infinite). >>> >>> As a side note: using #size and #position[:] on variable encoded streams >>> like UTF8 is *very* dangerous and should be avoided. Counting/indexing >>> bytes in the underlying stream is *not* the same as counting/indexing >>> decoded characters. It can be the source of subtle bugs. >>> >> >> Sean, >> >> Think that the Buffered stream can be reused in the context of sockets >> too. And there size has no meaning. >> >> Basically the cost in there is on growing and copying the internal buffer >> used to read. This buffer for the 35M file ends up being ~35M, so it has to >> grow and copy several times... >> Using #next: will preallocate the good size for the internal buffer. >> >> Now, I'd say first that if people want performance they should use the >> good API :). >> >> Maybe the ZnBuffered stream could ask its underlying stream for its >> #preferredInternalBufferSizeForUpToAll. But that would contaminate all >> the stream hierarchy... Is it worth it? Dunno... >> >> >>> >>> >>> > ----- >>> > Cheers, >>> > Sean >>> > -- >>> > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >>> > >>> >>> >>> >> >> >> -- >> >> >> >> Guille Polito >> >> Research Engineer >> >> Centre de Recherche en Informatique, Signal et Automatique de Lille >> >> CRIStAL - UMR 9189 >> >> French National Center for Scientific Research - *http://www.cnrs.fr >> <http://www.cnrs.fr>* >> >> >> *Web:* *http://guillep.github.io* <http://guillep.github.io> >> >> *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> >> > >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Denis Kudriashov

We can cover this case by extra variable maxBufferSize. By default it would be equal to given size in method #sizeBuffer:. And special users will specify concrete value. FileHandler will use file size for this. It will also improve buffer size for small files where we do not need big buffer. 2018-03-20 9:57 GMT+01:00 Guillermo Polito <guillermopolito(a)gmail.com>: > > > On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > >> >> >> > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> >> wrote: >> > >> > Guillermo Polito wrote >> >> I've done also some profiling and it does not look like we've lost in >> >> performance either (reading and decoding a 35MB file): >> > >> > Your 1st (new API) example: "0:00:00:01.535" >> > Your 2nd (old API) example:"0:00:00:01.732" >> > *** but *** >> > Your 1st (new API) example using the same consuming selector as example >> #1 >> > (#upToEnd): "0:00:00:07.816" or 4.5x slower >> >> Yes and no ;-) >> >> Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do >> basically the same as example 1 using #next: >> >> The Zn #upToEnd implementation does not use size knowledge (as streams >> are infinite). >> >> As a side note: using #size and #position[:] on variable encoded streams >> like UTF8 is *very* dangerous and should be avoided. Counting/indexing >> bytes in the underlying stream is *not* the same as counting/indexing >> decoded characters. It can be the source of subtle bugs. >> > > Sean, > > Think that the Buffered stream can be reused in the context of sockets > too. And there size has no meaning. > > Basically the cost in there is on growing and copying the internal buffer > used to read. This buffer for the 35M file ends up being ~35M, so it has to > grow and copy several times... > Using #next: will preallocate the good size for the internal buffer. > > Now, I'd say first that if people want performance they should use the > good API :). > > Maybe the ZnBuffered stream could ask its underlying stream for its # > preferredInternalBufferSizeForUpToAll. But that would contaminate all the > stream hierarchy... Is it worth it? Dunno... > > >> >> >> > ----- >> > Cheers, >> > Sean >> > -- >> > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >> > >> >> >> > > > -- > > > > Guille Polito > > Research Engineer > > Centre de Recherche en Informatique, Signal et Automatique de Lille > > CRIStAL - UMR 9189 > > French National Center for Scientific Research - *http://www.cnrs.fr > <http://www.cnrs.fr>* > > > *Web:* *http://guillep.github.io* <http://guillep.github.io> > > *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> >

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Guillermo Polito

On Tue, Mar 20, 2018 at 8:57 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> > wrote: > > > > Guillermo Polito wrote > >> I've done also some profiling and it does not look like we've lost in > >> performance either (reading and decoding a 35MB file): > > > > Your 1st (new API) example: "0:00:00:01.535" > > Your 2nd (old API) example:"0:00:00:01.732" > > *** but *** > > Your 1st (new API) example using the same consuming selector as example > #1 > > (#upToEnd): "0:00:00:07.816" or 4.5x slower > > Yes and no ;-) > > Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do > basically the same as example 1 using #next: > > The Zn #upToEnd implementation does not use size knowledge (as streams are > infinite). > > As a side note: using #size and #position[:] on variable encoded streams > like UTF8 is *very* dangerous and should be avoided. Counting/indexing > bytes in the underlying stream is *not* the same as counting/indexing > decoded characters. It can be the source of subtle bugs. > Sean, Think that the Buffered stream can be reused in the context of sockets too. And there size has no meaning. Basically the cost in there is on growing and copying the internal buffer used to read. This buffer for the 35M file ends up being ~35M, so it has to grow and copy several times... Using #next: will preallocate the good size for the internal buffer. Now, I'd say first that if people want performance they should use the good API :). Maybe the ZnBuffered stream could ask its underlying stream for its #preferredInternalBufferSizeForUpToAll. But that would contaminate all the stream hierarchy... Is it worth it? Dunno... > > > > ----- > > Cheers, > > Sean > > -- > > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html > > > > > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

March 20, 2018

Re: [Pharo-users] New Files in Pharo - Migration Guide, How To's and examples

by Sven Van Caekenberghe

> On 20 Mar 2018, at 00:19, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > > Guillermo Polito wrote >> I've done also some profiling and it does not look like we've lost in >> performance either (reading and decoding a 35MB file): > > Your 1st (new API) example: "0:00:00:01.535" > Your 2nd (old API) example:"0:00:00:01.732" > *** but *** > Your 1st (new API) example using the same consuming selector as example #1 > (#upToEnd): "0:00:00:07.816" or 4.5x slower Yes and no ;-) Look at MultiByteFileStream>>#upToEnd where they cheat a bit and do basically the same as example 1 using #next: The Zn #upToEnd implementation does not use size knowledge (as streams are infinite). As a side note: using #size and #position[:] on variable encoded streams like UTF8 is *very* dangerous and should be avoided. Counting/indexing bytes in the underlying stream is *not* the same as counting/indexing decoded characters. It can be the source of subtle bugs. > ----- > Cheers, > Sean > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

March 20, 2018

Re: [Pharo-users] Zinc URL and ASCII Encoding

by Gareth Cox

Thanks Sven I will have to first deconstruct the url and then reconstruct it as you mentioned. We discussed this in the discord channel and came to the same conclusion. I really like your final solution and will probably go with that. Thanks Gareth IT Manager/Developer email: gareth(a)inspired.org cell: +27 (0)78 374 9035 Inspired Org (PTY) Ltd On 19 Mar 2018 16:55, Sven Van Caekenberghe wrote: > Hi Gareth, > >> On 19 Mar 2018, at 15:23, Gareth Cox <gareth(a)inspired.org >> <mailto:gareth@inspired.org>> wrote: >> >> Hi Sven >> >> Is Zinc being overly fussy with encoding or is ASCII only URL strings >> a standard. >> the following fails in a workspace/playground. >> >> 'http://www.yr.no/place/Mexico/Nuevo_LeÃ³n/Monterrey/forecast.xml >> <http://www.yr.no/place/Mexico/Nuevo_Le%C3%B3n/Monterrey/forecast.xml>' >> asZnUrl >> >> The url comes from the yr.no <http://yr.no> database (via api). It is >> vaild if you paste into a browser. >> Let me know what you think >> >> Thanks >> Gareth > > (I am replying with CC to the ML for future reference). > > When ZnUrl parses a string representation of a URL it is indeed > strict. As per the spec this must be ASCII. > > Browsers typically allow a wider range of input. > > But of course, these URLs are supported, you just have to construct > them differently. If you use the direct construction API, you can use > full Unicode. > > ZnUrl new > Â scheme: #http; > Â host: 'www.yr.no <http://www.yr.no>'; > Â addPathSegments: #('place' 'Mexico' 'Nuevo_LeÃ³n' 'Monterrey' > 'forecast.xml'); > Â yourself. > > You'll see that the external representation of your URL is actually > the following, which is acceptable to the parser. > > 'http://www.yr.no/place/Mexico/Nuevo_Le%C3%B3n/Monterrey/forecast.xml' > asUrl. > > BTW, I would do your request as follows (using XML Support) : > > ZnClient new > Â systemPolicy; > Â http; > Â host: 'www.yr.no <http://www.yr.no>'; > Â addPath: #('place' 'Mexico' 'Nuevo_LeÃ³n' 'Monterrey' 'forecast.xml'); > Â accept: 'text/xml' asMIMEType; > Â contentReader: [ :entity | XMLDOMParser parse: entity contents ]; > Â get. > > In my Pharo 7 image, this gives me a nice coloured, fully parsed DOM: > > > HTH, > > Sven > >

March 20, 2018