Thanks Guille for this great post! I will turn it into a doc :) On Mon, Mar 19, 2018 at 6:10 PM, Guillermo Polito <guillermopolito@gmail.com
wrote:
Well, I'd say it we did it in the name of modularity. And yes, I believe that having separate responsibilities help in designing, testing and ensuring more easily the correctness of each of the parts in isolation.
I've done also some profiling and it does not look like we've lost in performance either (reading and decoding a 35MB file):
[file := Smalltalk sourcesFile fullName. (File named: file) readStreamDo: [ :binaryFile | (ZnCharacterReadStream on: (ZnBufferedReadStream on: binaryFile) encoding: 'utf8') next: binaryFile size. ]] timeToRun. "0:00:00:01.976"
[file := Smalltalk sourcesFile fullName. (MultiByteFileStream fileNamed: file) converter: (TextConverter newForEncoding: 'utf8'); upToEnd ] timeToRun. "0:00:00:02.147"
On Mon, Mar 19, 2018 at 5:51 PM, Richard O'Keefe <raoknz@gmail.com> wrote:
The idea of stacking things like character coding and buffering &c by using layers of wrapping is very much in the spirit of Java and is neither good for performance nor helpful for correctness.
On 20 March 2018 at 05:19, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all,
I've put some minutes summarizing the new APIs provided by the combination of the new File implementation and the Zn encoders. They all basically follow the decorator pattern to stack different responsibilities such as buffering, encoding, line ending convertions.
Please, do not hesitate to give your feedback.
Guille
1. Basic Files
By default files are binary. Not buffered.
(File named: 'name') readStream. (File named: 'name') readStreamDo: [ :stream | ... ]. (File named: 'name') writeStream. (File named: 'name') writeStreamDo: [ :stream | ... ].
2. Encoding
To add encoding, wrap a stream with a corresponding ZnCharacterRead/WriteStream.
"Reading" utf8Encoded := ZnCharacterReadStream on: aBinaryStream encoding: 'utf8'. utf16Encoded := ZnCharacterReadStream on: aBinaryStream encoding: 'utf16'.
"Writing" utf8Encoded := ZnCharacterWriteStream on: aBinaryStream encoding: 'utf8'. utf16Encoded := ZnCharacterWriteStream on: aBinaryStream encoding: 'utf16'.
3. Buffering
To add buffering, wrap a stream with a corresponding ZnBufferedRead/WriteStream.
bufferedReadStream := ZnBufferedReadStream on: aStream. bufferedWriteStream := ZnBufferedWriteStream on: aStream.
It is in general better to buffer the reading on the binary file and apply the encoding on the buffer in memory than the other way around. See
[file := Smalltalk sourcesFile fullName. (File named: file) readStreamDo: [ :binaryFile | (ZnCharacterReadStream on: (ZnBufferedReadStream on: binaryFile) encoding: 'utf8') upToEnd ]] timeToRun. "0:00:00:09.288"
[file := Smalltalk sourcesFile fullName. (File named: file) readStreamDo: [ :binaryFile | (ZnBufferedReadStream on: (ZnCharacterReadStream on: binaryFile encoding: 'utf8')) upToEnd ]] timeToRun. "0:00:00:14.189"
4. File System
By default, file system files are buffered and utf8 encoded to keep backwards compatibility.
'name' asFileReference readStreamDo: [ :bufferedUtf8Stream | ... ]. 'name' asFileReference writeStreamDo: [ :bufferedUtf8Stream | ... ].
FileStream also provides access to plain binary files using the #binaryRead/WriteStream messages. Binary streams are buffered by default also.
'name' asFileReference binaryReadStreamDo: [ :bufferedBinaryStream | ... ]. 'name' asFileReference binaryWriteStreamDo: [ :bufferedBinaryStream | ... ].
If you want a file with another encoding (to come in the PR https://github.com/pharo-project/pharo/pull/1134), you can specify it while obtaining the stream:
'name' asFileReference readStreamEncoded: 'utf16' do: [ :bufferedUtf16Stream | ... ].
'name' asFileReference writeStreamEncoded: 'utf8' do: [ :bufferedUtf16Stream | ... ].
5. Line Ending Conventions
If you want to write files following a specific line ending convention, use the ZnNewLineWriterStream. This stream decorator will transform any line ending (cr, lf, crlf) into a defined line ending. By default it chooses the platform line ending convention.
lineWriter := ZnNewLineWriterStream on: aStream.
If you want to choose another line ending convention you can do:
lineWriter forCr. lineWriter forLf. lineWriter forCrLf. lineWriter forPlatformLineEnding.
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>