Executive Summary of the recent FileStream Changes
Executive Summary of the recent FileStream Changes In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on. The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed. The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream. The replacements are can be found in packages Files and Zinc-Character-Encoding-Core. Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings. For example, '/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream. gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream. While, '/Users/sven/Desktop/foo.txt' asFileReference readStream. gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream. To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream. There are a couple of more specialised streams to cover special cases (like read and writing at the same time). SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only. Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS. Of course, we are far from done and need more testing, debugging and help from as many people as possible. Sven -- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
Very good initiative! It's worth a few hick-ups. I've tried to reduce RWBinaryOrTextStream usage in Squeak maybe 10 years ago, but you know it very well, the last places which are resisting are the more intricated and convoluted. I call it the SwiisKnifeStream and allways wandered why we would have to carry so many subclasses of Stream, since we have one capable of almost everything! Managing the states in such a hierarchy was an art, if ever torture is an art... and we are loosing a great can of worms for fishing the bugs. So, R.I.P. and no regret! 2018-03-15 21:01 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:
Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
+ 10000 Ah what a relief. Thanks a lot guille for all this good energy. BTW I also like the pattern. We should apply the same to OPAL. Do not mix API and add a wrapper for the backward compatible so that the new API is untouched. I really like that way. Now we should replace cr by newline :) Stef On Thu, Mar 15, 2018 at 11:36 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Very good initiative! It's worth a few hick-ups.
I've tried to reduce RWBinaryOrTextStream usage in Squeak maybe 10 years ago, but you know it very well, the last places which are resisting are the more intricated and convoluted. I call it the SwiisKnifeStream and allways wandered why we would have to carry so many subclasses of Stream, since we have one capable of almost everything! Managing the states in such a hierarchy was an art, if ever torture is an art... and we are loosing a great can of worms for fishing the bugs. So, R.I.P. and no regret!
2018-03-15 21:01 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:
Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
On 15 Mar 2018, at 23:36, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Very good initiative! It's worth a few hick-ups.
Thanks, the really hard work is in the all the using code, writing against simpler APIs. It will be a while before all the dust settles and all performance issues are covered, but we are already quite far.
I've tried to reduce RWBinaryOrTextStream usage in Squeak maybe 10 years ago, but you know it very well, the last places which are resisting are the more intricated and convoluted. I call it the SwiisKnifeStream and allways wandered why we would have to carry so many subclasses of Stream, since we have one capable of almost everything! Managing the states in such a hierarchy was an art, if ever torture is an art... and we are loosing a great can of worms for fishing the bugs. So, R.I.P. and no regret!
2018-03-15 21:01 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>: Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
Sent: Thursday, March 15, 2018 at 4:01 PM From: "Sven Van Caekenberghe" <sven@stfx.eu> To: "Pharo Development List" <pharo-dev@lists.pharo.org> Subject: [Pharo-dev] Executive Summary of the recent FileStream Changes
Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
StandardFileStream, at least, should remain for backwards compatibility and cross-platform compatibility with Squeak. It's a no-frills, non-decoding, non-LE normalizing stream that is heavily depended on.
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
What do you think about this algorithm for encoding detection: http://www.yaml.org/spec/1.2/spec.html#id2771184 I have an implementation (with tests), if you're interested. (I was waiting to propose it until the FileSystem API switched over to using Zn streams and encoders. The TextConverter API doesn't support UTF-32.)
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
Hi Monty We will keep the old stuff around in a kind of deprecated packages. Now in the long term we do not intend to maintain them forever because we will collapse under the load. You see it was either that clean up or modules.... so at the end ***Pharo*** should move on because this is our future. In this vein, we are planning to migrate a selection of the old deprecated methods (of Pharo 30, 40, 50, 60) into a MigrationPackage with deprecatedWithTransform rule to help the migration of the old code to new one. Now of course nobody really stood up to give a hand. Normal this is easier to give us order :) Stef On Fri, Mar 16, 2018 at 7:05 AM, monty <monty2@programmer.net> wrote:
Sent: Thursday, March 15, 2018 at 4:01 PM From: "Sven Van Caekenberghe" <sven@stfx.eu> To: "Pharo Development List" <pharo-dev@lists.pharo.org> Subject: [Pharo-dev] Executive Summary of the recent FileStream Changes
Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
StandardFileStream, at least, should remain for backwards compatibility and cross-platform compatibility with Squeak. It's a no-frills, non-decoding, non-LE normalizing stream that is heavily depended on.
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
What do you think about this algorithm for encoding detection: http://www.yaml.org/spec/1.2/spec.html#id2771184
I have an implementation (with tests), if you're interested. (I was waiting to propose it until the FileSystem API switched over to using Zn streams and encoders. The TextConverter API doesn't support UTF-32.)
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
Stephane Ducasse-3 wrote
we are planning to migrate a selection of the old deprecated methods (of Pharo 30, 40, 50, 60) into a MigrationPackage with deprecatedWithTransform rule to help the migration of the old code to new one.
Cool!!! ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
Sean you can really help on this one. Stef On Fri, Mar 16, 2018 at 5:47 PM, Sean P. DeNigris <sean@clipperadams.com> wrote:
Stephane Ducasse-3 wrote
we are planning to migrate a selection of the old deprecated methods (of Pharo 30, 40, 50, 60) into a MigrationPackage with deprecatedWithTransform rule to help the migration of the old code to new one.
Cool!!!
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
On 16 Mar 2018, at 07:05, monty <monty2@programmer.net> wrote:
Sent: Thursday, March 15, 2018 at 4:01 PM From: "Sven Van Caekenberghe" <sven@stfx.eu> To: "Pharo Development List" <pharo-dev@lists.pharo.org> Subject: [Pharo-dev] Executive Summary of the recent FileStream Changes
Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
StandardFileStream, at least, should remain for backwards compatibility and cross-platform compatibility with Squeak. It's a no-frills, non-decoding, non-LE normalizing stream that is heavily depended on.
Hmm, maybe. The standard (no pun intended) interface to the file system in Pharo has been FileSystem (FileReference) for quite a while. Many packages dealing with either different Pharo versions or different Smalltalk implementations have constructed their own portability facade (heck, I even did it in ZnFileSystemUtils myself). Note however that some aspects (API, behaviour) about the streams themselves changed as well (no longer being bivalent, separating reading/writing, smaller/simpler API, sometimes no positioning).
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
What do you think about this algorithm for encoding detection: http://www.yaml.org/spec/1.2/spec.html#id2771184
I have an implementation (with tests), if you're interested. (I was waiting to propose it until the FileSystem API switched over to using Zn streams and encoders. The TextConverter API doesn't support UTF-32.)
I did a primitive one in ZnCharacterEncoding class>>#detectEncoding: but I am not happy with it. I will read your reference and I am certainly interested in seeing your code !
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
The code is in XMLParser: see XMLEncodingDetector. I can port it, if you think the algorithm is appropriate. The YAML algorithm is actually a less restrictive version of this XML one: https://www.w3.org/TR/REC-xml/#sec-guessing The XML one is "Non-Normative" (ie optional), so I chose to implement the more general YAML algorithm instead.
Sent: Friday, March 16, 2018 at 6:44 AM From: "Sven Van Caekenberghe" <sven@stfx.eu> To: "Pharo Development List" <pharo-dev@lists.pharo.org> Subject: Re: [Pharo-dev] Executive Summary of the recent FileStream Changes
On 16 Mar 2018, at 07:05, monty <monty2@programmer.net> wrote:
Sent: Thursday, March 15, 2018 at 4:01 PM From: "Sven Van Caekenberghe" <sven@stfx.eu> To: "Pharo Development List" <pharo-dev@lists.pharo.org> Subject: [Pharo-dev] Executive Summary of the recent FileStream Changes
Executive Summary of the recent FileStream Changes
In Pharo 7 Guille Polito recently committed a heroic set of changes that we were planning to do for a long time but were afraid to take on.
The idea is to replace a couple of fat, overly complex, multi-functional, do-all classes with a set of simpler single purpose classes that can be combined as needed.
The classes that we want to get rid of can be found in the package DeprecatedFileSystem, in particular FileStream, StandardFileStream, MultiByteFileStream, MultiByteBinaryOrTextStream and RWBinaryOrTextStream.
StandardFileStream, at least, should remain for backwards compatibility and cross-platform compatibility with Squeak. It's a no-frills, non-decoding, non-LE normalizing stream that is heavily depended on.
Hmm, maybe.
The standard (no pun intended) interface to the file system in Pharo has been FileSystem (FileReference) for quite a while. Many packages dealing with either different Pharo versions or different Smalltalk implementations have constructed their own portability facade (heck, I even did it in ZnFileSystemUtils myself).
Note however that some aspects (API, behaviour) about the streams themselves changed as well (no longer being bivalent, separating reading/writing, smaller/simpler API, sometimes no positioning).
The replacements are can be found in packages Files and Zinc-Character-Encoding-Core.
Encoding and decoding characters to and from bytes is done using classes that you wrap around a more primitive binary stream. The same goes for buffering or translating line endings.
For example,
'/Users/sven/Desktop/foo.txt' asFileReference binaryReadStream.
gives you a ZnBufferedWriteStream wrapping a BinaryWriteStream.
While,
'/Users/sven/Desktop/foo.txt' asFileReference readStream.
What do you think about this algorithm for encoding detection: http://www.yaml.org/spec/1.2/spec.html#id2771184
I have an implementation (with tests), if you're interested. (I was waiting to propose it until the FileSystem API switched over to using Zn streams and encoders. The TextConverter API doesn't support UTF-32.)
I did a primitive one in ZnCharacterEncoding class>>#detectEncoding: but I am not happy with it. I will read your reference and I am certainly interested in seeing your code !
gives a ZnCharacterReadStream wrapping a ZnBufferedWriteStream wrapping a BinaryWriteStream.
To translate line endings, we would wrap a ZnCharacterWriteStream using a ZnCrPortableWriteStream.
There are a couple of more specialised streams to cover special cases (like read and writing at the same time).
SocketStream remains another fat, overly complex, multi-functional, do-all class, for which usable replacements exist in the form of ZdcSocketStream and ZdcSecureSocketStream, which are simpler, cleaner and binary only.
Of course, switching is more than replacing one class with a 100% compatible alternative, that would give us the same complex result. The challenge is to use a simpler API as well, to rethink how the streams are used. You know, KISS.
Of course, we are far from done and need more testing, debugging and help from as many people as possible.
Sven
-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
participants (5)
-
monty -
Nicolas Cellier -
Sean P. DeNigris -
Stephane Ducasse -
Sven Van Caekenberghe