On Sun, Dec 2, 2012 at 11:09 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
<short-version>
When doing complex IO, like writing or parsing protocols, to non-memory streams, like file streams (and probably socket streams), buffering can make an important difference. ZnBufferedWriteStream and ZnBufferedReadStream are an easy solution to experience the difference.
Yes, I noticed that in Fuel and that's why we use the buffered write stream always. It was a huge difference.
</short-version>
And now the long version: first some measurements (code based on latest versions of Zinc, NeoCSV, NeoJSON and STON):
NeoJSON
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ]. [ '/tmp/numbers.json' asFileReference writeStreamDo: [ :stream | (NeoJSONWriter on: stream) nextPut: data ] ] timeToRun. 22152
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ]. [ '/tmp/numbers.json' asFileReference writeStreamDo: [ :fstream | ZnBufferedWriteStream on: fstream do: [ :stream | (NeoJSONWriter on: stream) nextPut: data ] ] ] timeToRun. 5147
[ '/tmp/numbers.json' asFileReference readStreamDo: [ :stream | (NeoJSONReader on: stream) next ] ] timeToRun. 3502
[ '/tmp/numbers.json' asFileReference readStreamDo: [ :fstream | ZnBufferedReadStream on: fstream do: [ :stream | (NeoJSONReader on: stream) next ] ] ] timeToRun. 1214
STON
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ]. [ '/tmp/numbers.ston' asFileReference writeStreamDo: [ :stream | STON writer on: stream; nextPut: data ] ] timeToRun. 22275
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ]. [ '/tmp/numbers.ston' asFileReference writeStreamDo: [ :fstream | ZnBufferedWriteStream on: fstream do: [ :stream | STON writer on: stream; nextPut: data ] ] ] timeToRun. 5501
[ '/tmp/numbers.ston' asFileReference readStreamDo: [ :stream | STON reader on: stream; next ] ] timeToRun. 3632
[ '/tmp/numbers.ston' asFileReference readStreamDo: [ :fstream | ZnBufferedReadStream on: fstream do: [ :stream | STON reader on: stream; next ] ] ] timeToRun. 1367
NeoCSV
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ]. [ '/tmp/numbers.csv' asFileReference writeStreamDo: [ :stream | (NeoCSVWriter on: stream) nextPutAll: data ] ] timeToRun. 20916
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each reciprocal asFloat. each sqrt. each asWords } ]. [ '/tmp/numbers.csv' asFileReference writeStreamDo: [ :fstream | ZnBufferedWriteStream on: fstream do: [ :stream | (NeoCSVWriter on: stream) nextPutAll: data ] ] ] timeToRun. 5559
[ '/tmp/numbers.csv' asFileReference readStreamDo: [ :stream | (NeoCSVReader on: stream) addIntegerField; addIntegerField; addFloatField; addFloatField; addField; upToEnd ] ] timeToRun. 1290
[ '/tmp/numbers.csv' asFileReference readStreamDo: [ :fstream | ZnBufferedReadStream on: fstream do: [ :stream | (NeoCSVReader on: stream) addIntegerField; addIntegerField; addFloatField; addFloatField; addField; upToEnd ] ] ] timeToRun. 970
The reason for these non-trivial speedups is that buffering is missing and/or buffer management is suboptimal in the standard file stream classes. Simply wrapping the stream with a 64Kb buffer yields results like this. We already knew this for writing (and Fuel adapted it as well), but it seems to make a similar, although smaller difference when reading.
This is really really interesting, because it means a speed up of 2x when reading :) Where can I get the last version of ZnBufferedReadStream?
<side-note>
Attentive, curious readers might ask why in this particular benchmark the reading is so much slower than the writing - normally it should be the inverse or at least they should be closer to each other. Lo and behold: another performance issue: float printing (as opposed to parsing) is extremely slow (probably partially due to its reliance on LargeInteger arithmetic) !
I remember Nicolas Cieller doing something about Floats and printing...maybe he improved something?
Here is a similar benchmark with only Integers:
| data | data := (1 to: 25000) collect: [ :each | { each. each negated. each + 100000. each - 100000. each asWords } ]. [ '/tmp/numbers.csv' asFileReference writeStreamDo: [ :fstream | ZnBufferedWriteStream on: fstream do: [ :stream | (NeoCSVWriter on: stream) nextPutAll: data ] ] ] timeToRun. 311
[ '/tmp/numbers.csv' asFileReference readStreamDo: [ :fstream | ZnBufferedReadStream on: fstream do: [ :stream | (NeoCSVReader on: stream) addIntegerField; addIntegerField; addIntegerField; addIntegerField; addField; upToEnd ] ] ] timeToRun. 396
</side-note>
If ZnBufferedWriteStream or ZnBufferedReadStream were useful in speading up your code, please let us know.
Sven
PS: There is one ZnBufferedReadStream limitation: by design, #position, #position: or #skip: -1 and everything on top of that are NOT supported - IMHO real streams should not (and cannot) provide these operations. A sad consequence is that NumberParser and its subclasses (and thus Number class>>#readFrom:) do not work. NeoJSON and STON do their own number parsing, NeoCSV relies on a new class, NeoNumberParser, that is independently usable, though restricted to normal numbers as opposed to the full Smalltalk syntax.
I want to give it a try for Fuel.
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- Mariano http://marianopeck.wordpress.com