File/Stream changes: one Integer decoder/encoder to rule them all
Hi, After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference. As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before. Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass). When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality. (Incidentally this would be an ideal use of a Trait, can we use them again/still ?) The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests. For example, int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true or non-aliases ones like binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false I think I nailed it, but I would love a second pair of eyes to check what I did. The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist. Sven
I should that I love twice the title of your email. - yes if we can have one good instead of multiple not so good. This is wonderful. Now I'm not expert in that and brain dead after car, flight, rer, metro, train to get back home On Fri, Apr 20, 2018 at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
Hi, I am so looking forward to playing with this (as a user). For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo. Thanks a lot! Cheers, Doru
On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
-- www.tudorgirba.com www.feenk.com "From an abstract enough point of view, any two things are similar."
On 22 Apr 2018, at 10:22, Tudor Girba <tudor@tudorgirba.com> wrote:
Hi,
I am so looking forward to playing with this (as a user).
For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.
Well it should not be, not even in older Pharo versions, as long as you use everything correctly. But the latest changes are a bold step forward.
Thanks a lot!
Cheers, Doru
On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
-- www.tudorgirba.com www.feenk.com
"From an abstract enough point of view, any two things are similar."
Thanks a lot Sven. This is great to see all this positive energy to improve this part.
From a core library stand point, streams were the last pieces that needed to be revisited. This is great to see this happening.
Stef On Sun, Apr 22, 2018 at 11:04 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 22 Apr 2018, at 10:22, Tudor Girba <tudor@tudorgirba.com> wrote:
Hi,
I am so looking forward to playing with this (as a user).
For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.
Well it should not be, not even in older Pharo versions, as long as you use everything correctly.
But the latest changes are a bold step forward.
Thanks a lot!
Cheers, Doru
On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
-- www.tudorgirba.com www.feenk.com
"From an abstract enough point of view, any two things are similar."
Thatâs exactly my feeling as well. Doru
On Apr 22, 2018, at 6:41 PM, Stephane Ducasse <stepharo.self@gmail.com> wrote:
Thanks a lot Sven. This is great to see all this positive energy to improve this part. From a core library stand point, streams were the last pieces that needed to be revisited. This is great to see this happening.
Stef
On Sun, Apr 22, 2018 at 11:04 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 22 Apr 2018, at 10:22, Tudor Girba <tudor@tudorgirba.com> wrote:
Hi,
I am so looking forward to playing with this (as a user).
For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.
Well it should not be, not even in older Pharo versions, as long as you use everything correctly.
But the latest changes are a bold step forward.
Thanks a lot!
Cheers, Doru
On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
-- www.tudorgirba.com www.feenk.com
"From an abstract enough point of view, any two things are similar."
-- www.tudorgirba.com www.feenk.com "There are no old things, there are only old ways of looking at them."
On Fri, Apr 20, 2018 at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
Sorry I arrive late. I'm thinking that this is only valid for binary streams, putting it in buffered stream overloads the API... We can leave with it but... Two thoughts that come to my mind about this: - first, when we are decoding like that from a stream, we are probably parsing a binary format, so we want in general to have utility methods to read different sizes. Maybe we could include a Reader class with methods such as * readSignedInt32 * readUnsignedInt32 * readSignedInt64 * readUnsignedInt64 * readFloat64 ... and so on? - Second, that Reader may be (or not) a new decorator alternative. ZnBinaryDecoder? Wrapping a binary (buffered or not) stream. And this would not overload the API ^^. What do you think?
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
2018-05-14 13:22 GMT+02:00 Guillermo Polito <guillermopolito@gmail.com>:
On Fri, Apr 20, 2018 at 5:24 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
For example,
int32 ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
nextWord ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
or non-aliases ones like
binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
I think I nailed it, but I would love a second pair of eyes to check what I did.
Sorry I arrive late.
I'm thinking that this is only valid for binary streams, putting it in buffered stream overloads the API... We can leave with it but... Two thoughts that come to my mind about this:
- first, when we are decoding like that from a stream, we are probably parsing a binary format, so we want in general to have utility methods to read different sizes. Maybe we could include a Reader class with methods such as * readSignedInt32 * readUnsignedInt32 * readSignedInt64 * readUnsignedInt64 * readFloat64
... and so on?
- Second, that Reader may be (or not) a new decorator alternative. ZnBinaryDecoder? Wrapping a binary (buffered or not) stream. And this would not overload the API ^^.
What do you think?
Hi Guillermo, wouldn't a specialized wrapper work for adding this protocol where required? I have to add such BinaryStream wrapper in VW and Squeak http://www.squeaksource.com/STEM.html http://www.cincomsmalltalk.com/publicRepository/SYSEXT-BinaryStream.html For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak... IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
Sven
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13
For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak... IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
I want to move the methods from FFI to the base since years⦠I was told that this is impossible to ever happen because it is not important to do changes like that. Marcus
2018-05-14 14:44 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
For ByteArray, I didn't check recently in Pharo, but I know that it's
quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
I want to move the methods from FFI to the base since years⦠I was told that this is impossible to ever happen because it is not important to do changes like that.
Marcus
Hi Marcus, we all know that those un-important and not-so-much-rewarding changes makes a big difference at the end. like commenting classes, classifying methods, deprecating duplicates, reducing package inter-dependencies, etc... I thought that the defenders of statu quo had two reasons in mind: - the will to keep a FFI-less (locked-in) image - the fear to loose backward compatibility But's it the past, now you are free :)
On 14 May 2018, at 15:09, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
2018-05-14 14:44 GMT+02:00 Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>>:
For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak... IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
I want to move the methods from FFI to the base since years⦠I was told that this is impossible to ever happen because it is not important to do changes like that.
Marcus
Hi Marcus, we all know that those un-important and not-so-much-rewarding changes makes a big difference at the end. like commenting classes, classifying methods, deprecating duplicates, reducing package inter-dependencies, etc... I thought that the defenders of statu quo had two reasons in mind: - the will to keep a FFI-less (locked-in) image - the fear to loose backward compatibility
The idea that âonly big important changesâ are worth doing⦠you know, itâs âinventing the futureâ business, not âcleaning upâ! (Of course revolutionary changes while staying 100% backward compatible⦠because think about the Children!) It would be funny if it would not be sadâ¦
But's it the past, now you are free :)
Not really, because we share that package with Squeak. If I would be free to change it, it would have happened a loong time ago⦠Marcus
2018-05-14 15:15 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
On 14 May 2018, at 15:09, Nicolas Cellier <nicolas.cellier.aka.nice@ gmail.com> wrote:
2018-05-14 14:44 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
For ByteArray, I didn't check recently in Pharo, but I know that it's
quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
I want to move the methods from FFI to the base since years⦠I was told that this is impossible to ever happen because it is not important to do changes like that.
Marcus
Hi Marcus, we all know that those un-important and not-so-much-rewarding changes makes a big difference at the end. like commenting classes, classifying methods, deprecating duplicates, reducing package inter-dependencies, etc... I thought that the defenders of statu quo had two reasons in mind: - the will to keep a FFI-less (locked-in) image - the fear to loose backward compatibility
The idea that âonly big important changesâ are worth doing⦠you know, itâs âinventing the futureâ business, not âcleaning upâ!
(Of course revolutionary changes while staying 100% backward compatible⦠because think about the Children!)
It would be funny if it would not be sadâ¦
But's it the past, now you are free :)
Not really, because we share that package with Squeak. If I would be free to change it, it would have happened a loong time agoâ¦
Marcus
OK, coordinated changes must be performed then, but it's a bit more difficult... It can be a two step process: 1) separate the methods to be re-integrated in core into a separate package 2) move this methods in core, or use the separate package for old images (backward compatibility) It has to be performed both in FFI and Alien.
participants (6)
-
Guillermo Polito -
Marcus Denker -
Nicolas Cellier -
Stephane Ducasse -
Sven Van Caekenberghe -
Tudor Girba