Pharo 7 file streams guideline
Hello, I've prepared a draft of a short document that should help you with the transition to the "new" file streams API in Pharo 7. https://github.com/pavel-krivanek/pharoMaterials/blob/master/Filestreams.MD Pull requests are welcome. Cheers, -- Pavel
On 23 July 2018 at 15:38, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hello,
I've prepared a draft of a short document that should help you with the transition to the "new" file streams API in Pharo 7.
https://github.com/pavel-krivanek/pharoMaterials/blob/master/Filestreams.MD
Pull requests are welcome.
Cheers, -- Pavel
Some discussion prior to any pull request (and while I'm away from Pharo machine) I like all the new code examples until "Write a UTF-8 text to STDOUT" and I wonder "Stdio stdout writeStreamDo: [ :stream | stream nextPutAll: 'a â b' ]" would better fit the pattern of the other new code. (presuming "Stdio stdout" returns a FileReference, oherwise maybe "Stdio stdoutRef" or "Stdio stdout asFileReference") Under "Positionable streams", rather than needing "3 timesRepeat: [ stream next ]" would it be worth having #utf8position: to more intention revealing? cheers -ben
On 23 Jul 2018, at 11:13, Ben Coman <btc@openinworld.com> wrote:
On 23 July 2018 at 15:38, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hello,
I've prepared a draft of a short document that should help you with the transition to the "new" file streams API in Pharo 7.
https://github.com/pavel-krivanek/pharoMaterials/blob/master/Filestreams.MD
Pull requests are welcome.
Cheers, -- Pavel
Some discussion prior to any pull request (and while I'm away from Pharo machine)
I like all the new code examples until "Write a UTF-8 text to STDOUT" and I wonder "Stdio stdout writeStreamDo: [ :stream | stream nextPutAll: 'a â b' ]" would better fit the pattern of the other new code. (presuming "Stdio stdout" returns a FileReference, oherwise maybe "Stdio stdoutRef" or "Stdio stdout asFileReference")
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding. Maybe Stdio stdoutAsText might be an idea, but this is so uncommon that I am not sure this is a good idea.
Under "Positionable streams", rather than needing "3 timesRepeat: [ stream next ]" would it be worth having #utf8position: to more intention revealing?
no more new API, please. Positioning in a variable length encoded stream is plain hard - see my other mail.
cheers -ben
Hi Pavel & Sven, Thanks for writing this, it is a great quick reference. On Mon, 23 Jul 2018 at 12:08, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 11:13, Ben Coman <btc@openinworld.com> wrote:
I like all the new code examples until "Write a UTF-8 text to STDOUT" and I wonder "Stdio stdout writeStreamDo: [ :stream | stream nextPutAll: 'a â b' ]" would better fit the pattern of the other new code. (presuming "Stdio stdout" returns a FileReference, oherwise maybe "Stdio stdoutRef" or "Stdio stdout asFileReference")
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
I've written this code enough times that I'd like to see it included. :-) Maybe Stdout utf8Stdout (following the pattern of ByteArray>>utf8Decoded, String>>utf8Encoded) ? Thanks again, Alistair
Hi. I wonder does not stdout and stdin are always about text input/output? I never saw examples when somebody explicitly write raw bytes into these streams. If I am right then it is better to introduce binaryStdout and binaryStdin messages. And make stdout and stdin use most common encoding by default. How it is done in Java? 2018-07-23 19:19 GMT+01:00 Alistair Grant <akgrant0710@gmail.com>:
Hi Pavel & Sven,
Thanks for writing this, it is a great quick reference.
On Mon, 23 Jul 2018 at 12:08, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 11:13, Ben Coman <btc@openinworld.com> wrote:
I like all the new code examples until "Write a UTF-8 text to STDOUT" and I wonder "Stdio stdout writeStreamDo: [ :stream | stream nextPutAll: 'a â b' ]" would better fit the pattern of the other new code. (presuming "Stdio stdout" returns a FileReference, oherwise maybe "Stdio stdoutRef" or "Stdio stdout asFileReference")
Stdio stdout and friends just return a binary stream, hence they need
wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a
good idea.
I've written this code enough times that I'd like to see it included. :-)
Maybe
Stdout utf8Stdout
(following the pattern of ByteArray>>utf8Decoded, String>>utf8Encoded)
?
Thanks again, Alistair
On 24 July 2018 at 02:38, Denis Kudriashov <dionisiydk@gmail.com> wrote:
2018-07-23 19:19 GMT+01:00 Alistair Grant <akgrant0710@gmail.com>:
Hi Pavel & Sven,
Thanks for writing this, it is a great quick reference.
On Mon, 23 Jul 2018 at 12:08, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 11:13, Ben Coman <btc@openinworld.com> wrote:
I like all the new code examples until "Write a UTF-8 text to STDOUT" and I wonder "Stdio stdout writeStreamDo: [ :stream | stream nextPutAll: 'a â b' ]" would better fit the pattern of the other new code. (presuming "Stdio stdout" returns a FileReference, oherwise maybe "Stdio stdoutRef" or "Stdio stdout asFileReference")
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
I've written this code enough times that I'd like to see it included. :-)
Maybe
Stdout utf8Stdout
(following the pattern of ByteArray>>utf8Decoded, String>>utf8Encoded)
?
Thanks again, Alistair
I wonder does not stdout and stdin are always about text input/output? I never saw examples when somebody explicitly write raw bytes into these streams.
Its done... https://subosito.com/posts/imagemagick/ but I guess its not the usual case.
If I am right then it is better to introduce binaryStdout and binaryStdin messages. And make stdout and stdin use most common encoding by default. How it is done in Java?
It might be nice to further the new convention "use file references as entry points to file streams" so the new API can be reused... StdioRef stdout writeStreamDo: [ :stream | stream nextPutAll: 'a â b' ]. StdioRef stdout writeStreamEncoded: 'cp-1250' do: [ :stream | stream nextPutAll: 'PÅÃliÅ¡ žluÅ¥ouÄký kůŠúpÄl Äábelské ódy.' ]. StdioRef stdout binaryWriteStreamDo: [ :stream | stream nextPutAll: #[1 2 3] ]. cheers -ben
On Mon, Jul 23, 2018 at 07:38:16PM +0100, Denis Kudriashov wrote:
Hi.
I wonder does not stdout and stdin are always about text input/output?
No. Consider the case of reading and writing serialized objects on stdin and stdout, possibly between two images sending serialized objects to one another.
I never saw examples when somebody explicitly write raw bytes into these streams.
http://wiki.squeak.org/squeak/6176 I have not checked recently, but it should still work on Pharo with Fuel serialization.
If I am right then it is better to introduce binaryStdout and binaryStdin messages. And make stdout and stdin use most common encoding by default. How it is done in Java?
You are probably right that this is the best default, since it would be the most common case. Dave
2018-07-23 19:19 GMT+01:00 Alistair Grant <akgrant0710@gmail.com>:
Hi Pavel & Sven,
Thanks for writing this, it is a great quick reference.
On Mon, 23 Jul 2018 at 12:08, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 11:13, Ben Coman <btc@openinworld.com> wrote:
I like all the new code examples until "Write a UTF-8 text to STDOUT" and I wonder "Stdio stdout writeStreamDo: [ :stream | stream nextPutAll: 'a ??? b' ]" would better fit the pattern of the other new code. (presuming "Stdio stdout" returns a FileReference, oherwise maybe "Stdio stdoutRef" or "Stdio stdout asFileReference")
Stdio stdout and friends just return a binary stream, hence they need
wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a
good idea.
I've written this code enough times that I'd like to see it included. :-)
Maybe
Stdout utf8Stdout
(following the pattern of ByteArray>>utf8Decoded, String>>utf8Encoded)
?
Thanks again, Alistair
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that - textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing So, Stdio stdout should return return a character write stream with UTF-8 encoding while Stdio binaryStdout should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
On Tue., 24 Jul. 2018, 10:13 Sven Van Caekenberghe, <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
+1 I didn't suggest this earlier because it isn't backward compatible. But I do think it is the better solution. Cheers, Alistair (on phone)
On Tue, 24 Jul 2018 at 11:39, Alistair Grant <akgrant0710@gmail.com> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote: So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
+1
I didn't suggest this earlier because it isn't backward compatible. But I do think it is the better solution.
+2 I had a look at Stdio recently for Clap. The current implementation with Stdio stdout returning the binary stream is a bit confusing, but at least you can wrap it. The above proposition with an explicit binaryStdout for the lower level uncommon case would be much clearer indeed. Related issue: command line arguments come from VM system attributes as ByteStrings⦠and thus interpreted as iso-8859-1, which is incorrect in most cases nowadays, even though it seems to work as long as you only use ASCII. Decoding them is easy enough, but it requires two copies (asByteString utf8Decoded)
On 25 Jul 2018, at 13:39, Damien Pollet <damien.pollet+pharo@gmail.com> wrote:
On Tue, 24 Jul 2018 at 11:39, Alistair Grant <akgrant0710@gmail.com> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote: So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
+1
I didn't suggest this earlier because it isn't backward compatible. But I do think it is the better solution.
+2
I had a look at Stdio recently for Clap. The current implementation with Stdio stdout returning the binary stream is a bit confusing, but at least you can wrap it. The above proposition with an explicit binaryStdout for the lower level uncommon case would be much clearer indeed.
Related issue: command line arguments come from VM system attributes as ByteStrings⦠and thus interpreted as iso-8859-1, which is incorrect in most cases nowadays, even though it seems to work as long as you only use ASCII. Decoding them is easy enough, but it requires two copies (asByteString utf8Decoded)
Yes this is a really big issue. Anything coming in as command line arg or environment variable (or clipboard) is in a basically unknown OS determined encoding. I would assume/hope the UTF-8 is the sensible default today, but apparently not. And it is hard to find a cross platform solution. We've had serious issues already with this, like $HOME set to a non-ASCII path that then breaks almost everything.
On Wed, 25 Jul 2018 at 13:48, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 25 Jul 2018, at 13:39, Damien Pollet <damien.pollet+pharo@gmail.com> wrote: Related issue: command line arguments come from VM system attributes as ByteStrings⦠and thus interpreted as iso-8859-1, which is incorrect in most cases nowadays, even though it seems to work as long as you only use ASCII. Decoding them is easy enough, but it requires two copies (asByteString utf8Decoded)
Yes this is a really big issue. Anything coming in as command line arg or environment variable (or clipboard) is in a basically unknown OS determined encoding. I would assume/hope the UTF-8 is the sensible default today, but apparently not. And it is hard to find a cross platform solution.
My point here was that it would make more sense for those to be passed into the image as ByteArrays, revealing the fact that their encoding is unknown. Currently the bytes are correct, but since they've been shoved into ByteStrings by the VM, the characters will be wrong unless your system happens to be using Latin 1. I suppose we can either have a setting for decoding (since it's pretty much arbitrary), or heuristics like checking LC_CTYPE or whatever. Pablo mentioned the Locale class, but it doesn't seem to detect anything correct from the environment.
On Wed, Jul 25, 2018 at 02:20:30PM +0200, Damien Pollet wrote:
On Wed, 25 Jul 2018 at 13:48, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 25 Jul 2018, at 13:39, Damien Pollet <damien.pollet+pharo@gmail.com> wrote: Related issue: command line arguments come from VM system attributes as ByteStrings??? and thus interpreted as iso-8859-1, which is incorrect in most cases nowadays, even though it seems to work as long as you only use ASCII. Decoding them is easy enough, but it requires two copies (asByteString utf8Decoded)
Yes this is a really big issue. Anything coming in as command line arg or environment variable (or clipboard) is in a basically unknown OS determined encoding. I would assume/hope the UTF-8 is the sensible default today, but apparently not. And it is hard to find a cross platform solution.
My point here was that it would make more sense for those to be passed into the image as ByteArrays, revealing the fact that their encoding is unknown. Currently the bytes are correct, but since they've been shoved into ByteStrings by the VM, the characters will be wrong unless your system happens to be using Latin 1.
That sounds right to me. Having said that, there should be no need to change the VM interface to do this. A ByteString is by definition an array of 8 bit wide characters, and conversion between ByteString and ByteArray is trivial. Any necessary changes can be done without touching the VM. Dave
I suppose we can either have a setting for decoding (since it's pretty much arbitrary), or heuristics like checking LC_CTYPE or whatever. Pablo mentioned the Locale class, but it doesn't seem to detect anything correct from the environment.
Hi Sven⦠a couple questions: - is there a preferred order of composition between the encoding and buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications. - what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ? On Tue, 24 Jul 2018 at 10:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
On Tue, 31 Jul 2018 at 18:28, Damien Pollet <damien.pollet+pharo@gmail.com> wrote:
Hi Sven⦠a couple questions:
For context, I'm considering options in Clap, for providing accessors to Stdio that: - are convenient in most cases - discourage users from explicitly referencing the Stdio global (so that one can inject other streams instead if needed) For instance, it seems to make sense to provide stdout already wrapped with ZnNewLineWriterStream, but that precludes users from buffering. - is there a preferred order of composition between the encoding and
buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications.
- what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ?
On Tue, 24 Jul 2018 at 10:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
On Tue, Jul 31, 2018 at 6:41 PM Damien Pollet <damien.pollet+pharo@gmail.com> wrote:
On Tue, 31 Jul 2018 at 18:28, Damien Pollet <damien.pollet+pharo@gmail.com> wrote:
Hi Sven⦠a couple questions:
For context, I'm considering options in Clap, for providing accessors to Stdio that: - are convenient in most cases - discourage users from explicitly referencing the Stdio global (so that one can inject other streams instead if needed)
Yes. And to me it's ok if Clap defines his own good usages of streams. Clap's usage of stream is not a typical file access.
For instance, it seems to make sense to provide stdout already wrapped with ZnNewLineWriterStream,
Also, ZnNewLineWriterStream needs only to be used when interacting with the external word. The rationale of keeping it separate is twofold: - Zn* streams are meant to be reusable with other kind of streams that are not files (such as streams in memory) where the newline conventions do not need to be *always* enforced because the image has internally its own convention (crs). - Zn* streams are meant to be reusable with other kind of streams that are not files (such as sockets) where maybe we want *also* to enforce line ending convention.
but that precludes users from buffering.
Yeap. This makes me remember I tried a buffered file stream on /dev/urandom and the buffering introduced some funny effects :)
- is there a preferred order of composition between the encoding and
buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications.
- what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ?
On Tue, 24 Jul 2018 at 10:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
On Wed, Aug 1, 2018 at 11:19 AM Guillermo Polito <guillermopolito@gmail.com> wrote:
On Tue, Jul 31, 2018 at 6:41 PM Damien Pollet < damien.pollet+pharo@gmail.com> wrote:
On Tue, 31 Jul 2018 at 18:28, Damien Pollet < damien.pollet+pharo@gmail.com> wrote:
Hi Sven⦠a couple questions:
For context, I'm considering options in Clap, for providing accessors to Stdio that: - are convenient in most cases - discourage users from explicitly referencing the Stdio global (so that one can inject other streams instead if needed)
Yes. And to me it's ok if Clap defines his own good usages of streams. Clap's usage of stream is not a typical file access.
For instance, it seems to make sense to provide stdout already wrapped with ZnNewLineWriterStream,
Also, ZnNewLineWriterStream needs only to be used when interacting with the external word. The rationale of keeping it separate is twofold: - Zn* streams are meant to be reusable with other kind of streams that are not files (such as streams in memory) where the newline conventions do not need to be *always* enforced because the image has internally its own convention (crs). - Zn* streams are meant to be reusable with other kind of streams that are not files (such as sockets) where maybe we want *also* to enforce line ending convention.
And I'll just add that this also makes it simple to "skip" line end convention transformations.
but that precludes users from buffering.
Yeap. This makes me remember I tried a buffered file stream on /dev/urandom and the buffering introduced some funny effects :)
- is there a preferred order of composition between the encoding and
buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications.
- what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ?
On Tue, 24 Jul 2018 at 10:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
On Tue, Jul 31, 2018 at 6:29 PM Damien Pollet <damien.pollet+pharo@gmail.com> wrote:
Hi Sven⦠a couple questions:
- is there a preferred order of composition between the encoding and buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications.
Well, I'd say that we could check if performance-wise there is a difference... I don't think there will be much of a difference, but, who knows ^^.
- what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ?
Check what we did in FileReference & co. Opening a File reference returns by default a utf8 buffered stream (in that order). And we have convenience methods to specify other encodings and to get directly the binary stream (which will be buffered). The idea is that FileSystem (among others like managing memory file systems) provides a high level API with convenience methods to avoid putting the burden of the composition in the user. The File package stays small and flexible providing direct access to the physical file system. Following this same idea, to me Clap should define several convenient ways to access standard input/output. Like that, other Stdio users can define their own too. Also, maybe Clap can provide the same API as FileSystem (#writeStreamEncoded:do:, #readStreamEncoded:do: & co) just for coherence?
On Tue, 24 Jul 2018 at 10:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Guillermo Polito wrote
On Tue, Jul 31, 2018 at 6:29 PM Damien Pollet <
damien.pollet+pharo@
> wrote:
Hi Sven⦠a couple questions:
- is there a preferred order of composition between the encoding and buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications.
Well, I'd say that we could check if performance-wise there is a difference... I don't think there will be much of a difference, but, who knows ^^.
It can be a world of difference, depending on what operations are expensive on the terminal stream. For example, with a buffer size N, file <-> buffer <-> utf8 encoding does 1 N-byte write/read whenever buffer fills/need filling, file <-> utf8 encoding <-> buffer does N 1-4 byte writes/reads whenever buffer fills/need filling. TLDR; Always put the buffering between where small reads/writes occur (encoding, code doing looped #nextPut:'s), and where reads/writes are expensive (files, sockets, etc). Guillermo Polito wrote
- what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ?
Check what we did in FileReference & co. Opening a File reference returns by default a utf8 buffered stream (in that order). And we have convenience methods to specify other encodings and to get directly the binary stream (which will be buffered).
The idea is that FileSystem (among others like managing memory file systems) provides a high level API with convenience methods to avoid putting the burden of the composition in the user. The File package stays small and flexible providing direct access to the physical file system.
Following this same idea, to me Clap should define several convenient ways to access standard input/output. Like that, other Stdio users can define their own too.
Also, maybe Clap can provide the same API as FileSystem (#writeStreamEncoded:do:, #readStreamEncoded:do: & co) just for coherence?
The problem with using buffered write stream composition in a non-scoped manner, is you have to remember to explicitly #close the streams in order for buffers to flush correctly. That is, unless one could have a #finalize on buffered streams; and ensure it runs before finalizer on the wrapped stream (which would cause the terminal to close, and subsequent #flush to fail), I'm not sure if that's possible/how it already works... Cheers, Henry -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
On 31 Jul 2018, at 18:28, Damien Pollet <damien.pollet+pharo@gmail.com> wrote:
Hi Sven⦠a couple questions:
Very interesting questions, Damien.
- is there a preferred order of composition between the encoding and buffering streams ? If yes, it the same for read and write stream, or reversed ? E.g. if Stdio binaryStdin was implemented, Stdio stdin should be decoded, but buffering it as well would be a problem for interactive applications.
Buffering typically makes a huge (an order of magnitude) difference since in most cases, bathing up a number of more expensive operations is a good idea. How big a difference depends on the access pattern (many small operations are more costly and big IO operation will not see much difference - actually ZnBuffered[Read|Write]Stream bypasses the buffer if more than have the buffer size is requested). Buffering makes sense both at the lower binary level as well as at the character level. Benchmarks will tell. I never thought about having a different buffering approach to reading and writing, but indeed that could make sense too. There is also a cost to buffering: an additional indirection, management and memory usage.
- what's your opinion on convenience composition messages, e.g. aBinaryStream buffered decoded: 'utf-8' ?
I like these kinds of messages, but I keep saying that I prefer smaller stream APIs, not larger ones, hence I am hesitant to add more API (since every new stream has to understand these new cool messages).
On Tue, 24 Jul 2018 at 10:13, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 12:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Stdio stdout and friends just return a binary stream, hence they need wrapping for encoding.
Maybe
Stdio stdoutAsText
might be an idea, but this is so uncommon that I am not sure this is a good idea.
Given all remarks and comments (thanks BTW), I now think that
- textual stdio streams are the more common case - binary stdio streams are the primitive ones that are seldom used - another encoding than UTF-8 seems uncommon - these are streams that exist and need no real opening/closing
So,
Stdio stdout
should return return a character write stream with UTF-8 encoding while
Stdio binaryStdout
should be the lower level binary one. This would be more in line with the other streams. A non-UTF-8 encoding can be used as per Pavel's example.
On 23 Jul 2018, at 09:38, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hello,
I've prepared a draft of a short document that should help you with the transition to the "new" file streams API in Pharo 7.
https://github.com/pavel-krivanek/pharoMaterials/blob/master/Filestreams.MD
Pull requests are welcome.
Cheers, -- Pavel
Great contribution, thanks ! I see no factual errors, everything is clear, cookbook style. The last two sections might be more confusing than helpful. I would go more for 'all streams are buffered, don't worry' and 'beware positioning is byte based'. Regarding positioning, some additional points that might no be so well known. - Positioning in a variable length encoded stream is plain hard (and basically comes down to linear searching) - All ZnCharacterEncoders understand #backOnStream: that does the right thing - There is also ZnPositionableReadStream that can help (read the class comment) Sven
I am a little confused by Filestreams.MD #position and #position: report and set the number of past items. So when you open a stream, #position is 0. So why does the first "Positionable streams" example use ... position: 4 ... while the second one skips 3 characters? ... position: 3 ... would be just as broken. Oh, I think a clarification is needed when talking about UTF-8. To the best of my knowledge you don't need a Byte-Order-Mark at the beginning of a UTF-8 stream because there is no byte order issue to result, but apparently many Windows programs like to add one. Does/will Pharo add one when writing a UTF-8 file? Does/will it skip one when reading a UTF-8 file? I find the new approach produces unattractive code. I can do all of the examples simply in a system that (a) implements the ANSI Smalltalk FileStream class methods (b) supports '/dev/stdin' and '/dev/stdout' as file names (c) interprets the external type #'text' either as #'utf8' or as "whatever $LC_CTYPE says" and also supports #'utf8' and #'cp1250' as external types. Oh, and defines Object>>bindOwn: aBlock ^(self respondsTo: #close) ifTrue: [aBlock ensure: [self close]] ifFalse: [aBlock value] What document should I read to get a mental model of the new system and understand its rationale? On 23 July 2018 at 19:38, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hello,
I've prepared a draft of a short document that should help you with the transition to the "new" file streams API in Pharo 7.
https://github.com/pavel-krivanek/pharoMaterials/blob/ master/Filestreams.MD
Pull requests are welcome.
Cheers, -- Pavel
On 23 Jul 2018, at 18:52, Richard O'Keefe <raoknz@gmail.com> wrote:
Oh, I think a clarification is needed when talking about UTF-8.
Why ?
To the best of my knowledge you don't need a Byte-Order-Mark at the beginning of a UTF-8 stream because there is no byte order issue to result,
Nothing was said about BOMs.
but apparently many Windows programs like to add one.
Apparently yes.
Does/will Pharo add one when writing a UTF-8 file?
No
Does/will it skip one when reading a UTF-8 file?
Yes
I find the new approach produces unattractive code. I can do all of the examples simply in a system that (a) implements the ANSI Smalltalk FileStream class methods
Given a FileReference object of some kind you ask for the streams you need. Seems pretty OO if you ask me. Do you prefer a global class side factory facade ? Both approaches are not exclusive per se, we just want a clear break/difference (because the resulting streams are not 100% the same).
(b) supports '/dev/stdin' and '/dev/stdout' as file names
Maybe, with special casing. I like 'Stdio stdout' better as it is more cross platform.
(c) interprets the external type #'text' either as #'utf8' LC_CTYPE
Maybe, but that is not very Smalltalk like is it ? You want users to set environment variables ? Anyway this is less important point as UTF-8 is (should be) the general default.
What document should I read to get a mental model of the new system and understand its rationale?
ML discussions over the year, I guess. We are generally against big hairy complex classes and prefer simpler ones.
Yes there was no mention of BOMs, but there *was* mention of #position, and the presence or absence of byte order marks makes a difference. As for mailing list discussions over a year, that is not the kind of single coherent source I was hoping for. As someone *using* the system classes, I don't give a damn how big they are. What I care about is the complexity of *my* code, and it looks as though the new interface will make my code bigger, more error-prone, and less portable. As for the "big hairy classes" sneer, the file stream classes put together in my system are about the same size as the Dictionary class proper. I don't "want to make users set environment variables". What I was suggesting was that if a user does go to the trouble of setting relevant environment variables, the system should have the decency to pay attention to them. handling /dev/std... is pretty trivial in UNIX; in Windows nothing about stdin is easy (XP, Vista, 7, 8, and 10 differ amongst themselves in several annoying ways). I can make no sense of the comment that interpreting #'text' as #utf8 (always, as a design choice) or as whatever the user chose for $LC_CTYPE (always, as a design choice) is "not very Smalltalk like". Both design choices are fully consistent with the standard -- to the limited extent that UTF8 processing *can* be consistent with the standard. On 24 July 2018 at 05:40, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 18:52, Richard O'Keefe <raoknz@gmail.com> wrote:
Oh, I think a clarification is needed when talking about UTF-8.
Why ?
To the best of my knowledge you don't need a Byte-Order-Mark at the beginning of a UTF-8 stream because there is no byte order issue to result,
Nothing was said about BOMs.
but apparently many Windows programs like to add one.
Apparently yes.
Does/will Pharo add one when writing a UTF-8 file?
No
Does/will it skip one when reading a UTF-8 file?
Yes
I find the new approach produces unattractive code. I can do all of the examples simply in a system that (a) implements the ANSI Smalltalk FileStream class methods
Given a FileReference object of some kind you ask for the streams you need. Seems pretty OO if you ask me.
Do you prefer a global class side factory facade ?
Both approaches are not exclusive per se, we just want a clear break/difference (because the resulting streams are not 100% the same).
(b) supports '/dev/stdin' and '/dev/stdout' as file names
Maybe, with special casing. I like 'Stdio stdout' better as it is more cross platform.
(c) interprets the external type #'text' either as #'utf8' LC_CTYPE
Maybe, but that is not very Smalltalk like is it ? You want users to set environment variables ? Anyway this is less important point as UTF-8 is (should be) the general default.
What document should I read to get a mental model of the new system and understand its rationale?
ML discussions over the year, I guess.
We are generally against big hairy complex classes and prefer simpler ones.
Richard, I am only engaging in a discussion with you in order to explain what we did and why. The changes that we did were years in the making and are now being pushed into the system. The discussions happened long ago, we are not going to revert them. Of course you are entitled to have your own opinion and disagree. I do not know what 'your system' is that you are referring to. Pharo has a particular philosophy that includes a complex relationship with the concepts of Smalltalk and especially ANSI Smalltalk. In one sentence, we want to have to liberty to make (breaking) changes and don't want to be stuck in the past. At the same time we want our user base to be able to follow by adapting their code. Regards, Sven
On 26 Jul 2018, at 17:06, Richard O'Keefe <raoknz@gmail.com> wrote:
Yes there was no mention of BOMs, but there *was* mention of #position, and the presence or absence of byte order marks makes a difference.
As for mailing list discussions over a year, that is not the kind of single coherent source I was hoping for.
As someone *using* the system classes, I don't give a damn how big they are. What I care about is the complexity of *my* code, and it looks as though the new interface will make my code bigger, more error-prone, and less portable. As for the "big hairy classes" sneer, the file stream classes put together in my system are about the same size as the Dictionary class proper.
I don't "want to make users set environment variables". What I was suggesting was that if a user does go to the trouble of setting relevant environment variables, the system should have the decency to pay attention to them.
handling /dev/std... is pretty trivial in UNIX; in Windows nothing about stdin is easy (XP, Vista, 7, 8, and 10 differ amongst themselves in several annoying ways).
I can make no sense of the comment that interpreting #'text' as #utf8 (always, as a design choice) or as whatever the user chose for $LC_CTYPE (always, as a design choice) is "not very Smalltalk like". Both design choices are fully consistent with the standard -- to the limited extent that UTF8 processing *can* be consistent with the standard.
On 24 July 2018 at 05:40, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 18:52, Richard O'Keefe <raoknz@gmail.com> wrote:
Oh, I think a clarification is needed when talking about UTF-8.
Why ?
To the best of my knowledge you don't need a Byte-Order-Mark at the beginning of a UTF-8 stream because there is no byte order issue to result,
Nothing was said about BOMs.
but apparently many Windows programs like to add one.
Apparently yes.
Does/will Pharo add one when writing a UTF-8 file?
No
Does/will it skip one when reading a UTF-8 file?
Yes
I find the new approach produces unattractive code. I can do all of the examples simply in a system that (a) implements the ANSI Smalltalk FileStream class methods
Given a FileReference object of some kind you ask for the streams you need. Seems pretty OO if you ask me.
Do you prefer a global class side factory facade ?
Both approaches are not exclusive per se, we just want a clear break/difference (because the resulting streams are not 100% the same).
(b) supports '/dev/stdin' and '/dev/stdout' as file names
Maybe, with special casing. I like 'Stdio stdout' better as it is more cross platform.
(c) interprets the external type #'text' either as #'utf8' LC_CTYPE
Maybe, but that is not very Smalltalk like is it ? You want users to set environment variables ? Anyway this is less important point as UTF-8 is (should be) the general default.
What document should I read to get a mental model of the new system and understand its rationale?
ML discussions over the year, I guess.
We are generally against big hairy complex classes and prefer simpler ones.
Am 26.07.2018 um 17:59 schrieb Sven Van Caekenberghe <sven@stfx.eu>:
Richard,
I am only engaging in a discussion with you in order to explain what we did and why. The changes that we did were years in the making and are now being pushed into the system. The discussions happened long ago, we are not going to revert them.
Of course you are entitled to have your own opinion and disagree.
I do not know what 'your system' is that you are referring to.
Pharo has a particular philosophy that includes a complex relationship with the concepts of Smalltalk and especially ANSI Smalltalk. In one sentence, we want to have to liberty to make (breaking) changes and don't want to be stuck in the past. At the same time we want our user base to be able to follow by adapting their code.
+1 Norbert
Regards,
Sven
On 26 Jul 2018, at 17:06, Richard O'Keefe <raoknz@gmail.com> wrote:
Yes there was no mention of BOMs, but there *was* mention of #position, and the presence or absence of byte order marks makes a difference.
As for mailing list discussions over a year, that is not the kind of single coherent source I was hoping for.
As someone *using* the system classes, I don't give a damn how big they are. What I care about is the complexity of *my* code, and it looks as though the new interface will make my code bigger, more error-prone, and less portable. As for the "big hairy classes" sneer, the file stream classes put together in my system are about the same size as the Dictionary class proper.
I don't "want to make users set environment variables". What I was suggesting was that if a user does go to the trouble of setting relevant environment variables, the system should have the decency to pay attention to them.
handling /dev/std... is pretty trivial in UNIX; in Windows nothing about stdin is easy (XP, Vista, 7, 8, and 10 differ amongst themselves in several annoying ways).
I can make no sense of the comment that interpreting #'text' as #utf8 (always, as a design choice) or as whatever the user chose for $LC_CTYPE (always, as a design choice) is "not very Smalltalk like". Both design choices are fully consistent with the standard -- to the limited extent that UTF8 processing *can* be consistent with the standard.
On 24 July 2018 at 05:40, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 23 Jul 2018, at 18:52, Richard O'Keefe <raoknz@gmail.com> wrote:
Oh, I think a clarification is needed when talking about UTF-8.
Why ?
To the best of my knowledge you don't need a Byte-Order-Mark at the beginning of a UTF-8 stream because there is no byte order issue to result,
Nothing was said about BOMs.
but apparently many Windows programs like to add one.
Apparently yes.
Does/will Pharo add one when writing a UTF-8 file?
No
Does/will it skip one when reading a UTF-8 file?
Yes
I find the new approach produces unattractive code. I can do all of the examples simply in a system that (a) implements the ANSI Smalltalk FileStream class methods
Given a FileReference object of some kind you ask for the streams you need. Seems pretty OO if you ask me.
Do you prefer a global class side factory facade ?
Both approaches are not exclusive per se, we just want a clear break/difference (because the resulting streams are not 100% the same).
(b) supports '/dev/stdin' and '/dev/stdout' as file names
Maybe, with special casing. I like 'Stdio stdout' better as it is more cross platform.
(c) interprets the external type #'text' either as #'utf8' LC_CTYPE
Maybe, but that is not very Smalltalk like is it ? You want users to set environment variables ? Anyway this is less important point as UTF-8 is (should be) the general default.
What document should I read to get a mental model of the new system and understand its rationale?
ML discussions over the year, I guess.
We are generally against big hairy complex classes and prefer simpler ones.
participants (11)
-
Alistair Grant -
Ben Coman -
Damien Pollet -
David T. Lewis -
Denis Kudriashov -
Guillermo Polito -
Henrik Sperre Johansen -
Norbert Hartl -
Pavel Krivanek -
Richard O'Keefe -
Sven Van Caekenberghe