Hi,

<short-version>

When doing complex IO, like writing or parsing protocols, to non-memory streams, like file streams (and probably socket streams), buffering can make an important difference. ZnBufferedWriteStream and ZnBufferedReadStream are an easy solution to experience the difference.

</short-version>

And now the long version: first some measurements (code based on latest versions of Zinc, NeoCSV, NeoJSON and STON):

NeoJSON

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ].
[ '/tmp/numbers.json' asFileReference writeStreamDo: [ :stream |
� � � � (NeoJSONWriter on: stream) nextPut: data ] ] timeToRun. 22152

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ].
[ '/tmp/numbers.json' asFileReference writeStreamDo: [ :fstream |
� � � � ZnBufferedWriteStream on: fstream do: [ :stream |
� � � � � � � � (NeoJSONWriter on: stream) nextPut: data ] ] ] timeToRun. 5147

[ '/tmp/numbers.json' asFileReference readStreamDo: [ :stream |
� � � � (NeoJSONReader on: stream) next ] ] timeToRun. 3502

[ '/tmp/numbers.json' asFileReference readStreamDo: [ :fstream |
� � � � ZnBufferedReadStream on: fstream do: [ :stream |
� � � � � � � � (NeoJSONReader on: stream) next ] ] ] timeToRun. 1214

STON

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ].
[ '/tmp/numbers.ston' asFileReference writeStreamDo: [ :stream |
� � � � STON writer on: stream; nextPut: data ] ] timeToRun. 22275

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ].
[ '/tmp/numbers.ston' asFileReference writeStreamDo: [ :fstream |
� � � � ZnBufferedWriteStream on: fstream do: [ :stream |
� � � � � � � � STON writer on: stream; nextPut: data ] ] ] timeToRun. 5501

[ '/tmp/numbers.ston' asFileReference readStreamDo: [ :stream |
� � � � STON reader on: stream; next ] ] timeToRun. 3632

[ '/tmp/numbers.ston' asFileReference readStreamDo: [ :fstream |
� � � � ZnBufferedReadStream on: fstream do: [ :stream |
� � � � � � � � STON reader on: stream; next ] ] ] timeToRun. 1367

NeoCSV

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ].
[ '/tmp/numbers.csv' asFileReference writeStreamDo: [ :stream |
� � � � (NeoCSVWriter on: stream) nextPutAll: data ] ] timeToRun. 20916

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ].
[ '/tmp/numbers.csv' asFileReference writeStreamDo: [ :fstream |
� � � � ZnBufferedWriteStream on: fstream do: [ :stream |
� � � � � � � � (NeoCSVWriter on: stream) nextPutAll: data ] ] ] timeToRun. 5559

[ '/tmp/numbers.csv' asFileReference readStreamDo: [ :stream |
� � � � (NeoCSVReader on: stream)
� � � � � � � � addIntegerField; addIntegerField; addFloatField; addFloatField; addField;
� � � � � � � � upToEnd ] ] timeToRun. 1290

[ '/tmp/numbers.csv' asFileReference readStreamDo: [ :fstream |
� � � � ZnBufferedReadStream on: fstream do: [ :stream |
� � � � � � � � (NeoCSVReader on: stream)
� � � � � � � � � � � � addIntegerField; addIntegerField; addFloatField; addFloatField; addField;
� � � � � � � � � � � � upToEnd ] ] ] timeToRun. �970


The reason for these non-trivial speedups is that buffering is missing and/or buffer management is suboptimal in the standard file stream classes. Simply wrapping the stream with a 64Kb buffer yields results like this. We already knew this for writing (and Fuel adapted it as well), but it seems to make a similar, although smaller difference when reading.

<side-note>

Attentive, curious readers might ask why in this particular benchmark the reading is so much slower than the writing - normally it should be the inverse or at least they should be closer to each other. Lo and behold: another performance issue: float printing (as opposed to parsing) is extremely slow (probably partially due to its reliance on LargeInteger arithmetic) !

Here is a similar benchmark with only Integers:

| data |
data := (1 to: 25000) collect: [ :each |
� � � � { each. each negated. each + 100000. each - 100000. each asWords } ].
[ '/tmp/numbers.csv' asFileReference writeStreamDo: [ :fstream |
� � � � ZnBufferedWriteStream on: fstream do: [ :stream |
� � � � � � � � (NeoCSVWriter on: stream) nextPutAll: data ] ] ] timeToRun. 311

[ '/tmp/numbers.csv' asFileReference readStreamDo: [ :fstream |
� � � � ZnBufferedReadStream on: fstream do: [ :stream |
� � � � � � � � (NeoCSVReader on: stream)
� � � � � � � � � � � � addIntegerField; addIntegerField; addIntegerField; addIntegerField; addField;
� � � � � � � � � � � � upToEnd ] ] ] timeToRun. 396

</side-note>

If ZnBufferedWriteStream or ZnBufferedReadStream were useful in speading up your code, please let us know.

Sven

PS: There is one ZnBufferedReadStream limitation: by design, #position, #position: or #skip: -1 and everything on top of that are NOT supported - IMHO real streams should not (and cannot) provide these operations. A sad consequence is that NumberParser and its subclasses (and thus Number class>>#readFrom:) do not work. NeoJSON and STON do their own number parsing, NeoCSV relies on a new class, NeoNumberParser, that is independently usable, though restricted to normal numbers as opposed to the full Smalltalk syntax.

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill