[Pharo-project] Basic tricks for improving a serializer?
Hi guys. Together with Martin we are developing a fast object graph serializer called Fuel. The idea is to use a pickle format similar to VW Parcels. Right now we are able to correctly serialize (almost) all kind of objects (from different classes and so on). The algorithm is working quite well and we are already quite fast. We also know how to encode more or less efficient some of the "primitive" classes like Floats, SmallIntegers, ByteString, ByteArray, etc, etc. What we really don't know anything about streams. For example, until now, we were using just a MultiByteFileStream and doing all the time things like #next: #nextPutAll: #nextWordPut: etc... Now Igor told us for example, to use a buffer like this: | bufferStream | bufferStream := ByteArray new writeStream. (FLSerializer on: bufferStream) serialize: anArrayToSerialize. aFileStream nextPutAll: bufferStream contents. and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ? For reading (deserialization) we are not doing nothing special. We just open the file stream and we read with #next #nextString ... etc. Another thing is which kind of file stream should we use. MultiByteFileStream ? StandardFileStream ? So...before spending time in some optimizations, are there "known" things that we should check? Thanks in advance, -- Mariano http://marianopeck.wordpress.com
Mariano, On 17 May 2011, at 15:21, Mariano Martinez Peck wrote:
Now Igor told us for example, to use a buffer like this:
| bufferStream | bufferStream := ByteArray new writeStream.
(FLSerializer on: bufferStream) serialize: anArrayToSerialize.
aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
The disadvantage of the above is that you buffer everything before you write. Furthermore you are growing the buffer all the time (recopying contents) which is not too efficient. Some time ago we had a discussion about this and then I implemented ZnBufferedWriteStream (it should be in Core 1.3), have a look. It was just a quick hack, but it worked in speeding up writing. Sven
On Tue, May 17, 2011 at 3:34 PM, Sven Van Caekenberghe <sven@beta9.be>wrote:
Mariano,
On 17 May 2011, at 15:21, Mariano Martinez Peck wrote:
Now Igor told us for example, to use a buffer like this:
| bufferStream | bufferStream := ByteArray new writeStream.
(FLSerializer on: bufferStream) serialize: anArrayToSerialize.
aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
The disadvantage of the above is that you buffer everything before you write. Furthermore you are growing the buffer all the time (recopying contents) which is not too efficient.
Some time ago we had a discussion about this and then I implemented ZnBufferedWriteStream (it should be in Core 1.3), have a look. It was just a quick hack, but it worked in speeding up writing.
Thanks Sven, what a great point you gave me. I tried it a little and I found a couple of methods missing which I used to use for a normal WriteStream. I will play a little more when I have time and I will ask you again!! -- Mariano http://marianopeck.wordpress.com
Sven, I want to make it work :) so....the missing methods I told you that I need are: #nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut: #contents Implement #contents I guess it is something like: ZnBufferedWriteStream >> contents ^ stream contents Those missing methods I need are implemented PositionableStream. I took the implementation from there and put it in ZnBufferedWriteStream. I just added to them a first line " self flushBufferIfFull." The way I am creating it is like this: stream := ZnBufferedWriteStream on: ((FileDirectory default forceNewFileNamed: 'mariano') binary). So...is this correct Sven? My tests are green :) Still need to try benchmarks. Thanks! Mariano On Tue, May 17, 2011 at 8:18 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Tue, May 17, 2011 at 3:34 PM, Sven Van Caekenberghe <sven@beta9.be>wrote:
Mariano,
On 17 May 2011, at 15:21, Mariano Martinez Peck wrote:
Now Igor told us for example, to use a buffer like this:
| bufferStream | bufferStream := ByteArray new writeStream.
(FLSerializer on: bufferStream) serialize: anArrayToSerialize.
aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
The disadvantage of the above is that you buffer everything before you write. Furthermore you are growing the buffer all the time (recopying contents) which is not too efficient.
Some time ago we had a discussion about this and then I implemented ZnBufferedWriteStream (it should be in Core 1.3), have a look. It was just a quick hack, but it worked in speeding up writing.
Thanks Sven, what a great point you gave me. I tried it a little and I found a couple of methods missing which I used to use for a normal WriteStream. I will play a little more when I have time and I will ask you again!!
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 17 May 2011, at 21:57, Mariano Martinez Peck wrote:
Sven, I want to make it work :)
so....the missing methods I told you that I need are:
#nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut:
I guess these are pretty easy. But I think they clutter the interface of ZnBufferedWriteStream, so maybe you should make a subclass.
#contents
Implement #contents I guess it is something like:
ZnBufferedWriteStream >> contents ^ stream contents
Why to you need #contents ? I would say that it goes a bit against the concept of a stream as a sink of data. I haven't looked, but I would guess that saying #contents to a FileStream is not efficient.
Those missing methods I need are implemented PositionableStream. I took the implementation from there and put it in ZnBufferedWriteStream. I just added to them a first line " self flushBufferIfFull."
That is probably OK, except when your string becomes larger than the buffer. Have a look at #nextPutAll:
The way I am creating it is like this:
stream := ZnBufferedWriteStream on: ((FileDirectory default forceNewFileNamed: 'mariano') binary).
So...is this correct Sven? My tests are green :)
That seems OK
Still need to try benchmarks.
You might also consider playing with making the buffer larger. Sven
On Tue, May 17, 2011 at 10:31 PM, Sven Van Caekenberghe <sven@beta9.be>wrote:
On 17 May 2011, at 21:57, Mariano Martinez Peck wrote:
Sven, I want to make it work :)
so....the missing methods I told you that I need are:
#nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut:
I guess these are pretty easy. But I think they clutter the interface of ZnBufferedWriteStream, so maybe you should make a subclass.
Yeah, don't worry. I can even duplicate the class hehehe
#contents
Implement #contents I guess it is something like:
ZnBufferedWriteStream >> contents ^ stream contents
Why to you need #contents ?
becasue I am an idiot. No, I don't need it. You are correct. Thanks for asking.
I would say that it goes a bit against the concept of a stream as a sink of data. I haven't looked, but I would guess that saying #contents to a FileStream is not efficient.
Those missing methods I need are implemented PositionableStream. I took the implementation from there and put it in ZnBufferedWriteStream. I just added to them a first line " self flushBufferIfFull."
That is probably OK, except when your string becomes larger than the buffer. Have a look at #nextPutAll:
I am not sure if I understood. The following are correct for sure then: #nextNumber:put: #nextInt32Put: #nextWordPut: And #nextStringPut: is like this: nextStringPut: s "Append the string, s, to the receiver. Only used by DataStream. Max size of 64*256*256*256." | length | self flushBufferIfFull. (length := s size) < 192 ifTrue: [self nextPut: length] ifFalse: [self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1)]. self nextPutAll: s asByteArray. ^s which will finally send #nextPutAll: that does correct check.
The way I am creating it is like this:
stream := ZnBufferedWriteStream on: ((FileDirectory default forceNewFileNamed: 'mariano') binary).
So...is this correct Sven? My tests are green :)
That seems OK
Ok.
Still need to try benchmarks.
You might also consider playing with making the buffer larger.
yeah, I was trying that. Just by curious, why you put a 2^X number there? Ahh and for reading we don't need something special because it is already buffered? (I have been told that) Thanks a lot Sven, -- Mariano http://marianopeck.wordpress.com
On 17 May 2011 22:58, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, May 17, 2011 at 10:31 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 17 May 2011, at 21:57, Mariano Martinez Peck wrote:
Sven, I want to make it work :)
so....the missing methods I told you that I need are:
#nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut:
I guess these are pretty easy. But I think they clutter the interface of ZnBufferedWriteStream, so maybe you should make a subclass.
Yeah, don't worry. I can even duplicate the class hehehe
#contents
Implement #contents I guess it is something like:
ZnBufferedWriteStream >> contents ^ stream contents
Why to you need #contents ?
becasue I am an idiot. No, I don't need it. You are correct. Thanks for asking.
I would say that it goes a bit against the concept of a stream as a sink of data. I haven't looked, but I would guess that saying #contents to a FileStream is not efficient.
Those missing methods I need are implemented PositionableStream. I took the implementation from there and  put it in ZnBufferedWriteStream. I just added to them a first line "   self flushBufferIfFull."
That is probably OK, except when your string becomes larger than the buffer. Have a look at #nextPutAll:
I am not sure if I understood. The following are correct for sure then: Â #nextNumber:put: Â #nextInt32Put: Â #nextWordPut:
And #nextStringPut:Â Â is like this:
nextStringPut: s    "Append the string, s, to the receiver. Only used by DataStream. Max size of 64*256*256*256."
   | length |    self flushBufferIfFull.    (length := s size) < 192       ifTrue: [self nextPut: length]       ifFalse:          [self nextPut: (length digitAt: 4)+192.          self nextPut: (length digitAt: 3).          self nextPut: (length digitAt: 2).          self nextPut: (length digitAt: 1)].    self nextPutAll: s asByteArray.    ^s
Sorry, but i can't resist commenting on that. Why, if you demand from stream to implement #nextInt32Put: a the same time, you writing code like this self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1) ? Then just extend your serializer with a notion of 'length' field, which you can use for anything where you need to encode length/size value, but not just for Strings. So, then the above method could be as short as: nextStringPut: s self putLength: s size. self nexPutAll: s asByteArray. and here you have a potential caveat because your string could be WideString .. muhahaha. So, i suggest you to reconsider the way how you serializing strings. Instead what you could do is to extend ByteString and WideString (and perhaps similarily do for ByteSymbol and WideSymbol), the methods which is responsible to turning a receiver in a sequence of bytes, and then simply put it into output stream, whatever it might be. Then you don't need #nextStringPut: because its a) not polymorphic, because apparently serializing ByteString should be different from serializing WideString b) instead you implementing this in Byte/WideString>>serializeToFuelStream: aStream and you done.
which will finally send #nextPutAll:Â that does correct check.
The way I am creating it is like this:
stream := ZnBufferedWriteStream on: ((FileDirectory default forceNewFileNamed: 'mariano') binary).
So...is this correct Sven? Â My tests are green :)
That seems OK
Ok.
Still need to try benchmarks.
You might also consider playing with making the buffer larger.
yeah, I was trying that. Just by curious, why you put a 2^X number there?
Ahh and for reading we don't need something special because it is already buffered? (I have been told that)
Thanks a lot Sven,
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
On Tue, May 17, 2011 at 7:16 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 17 May 2011 22:58, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, May 17, 2011 at 10:31 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 17 May 2011, at 21:57, Mariano Martinez Peck wrote:
Sven, I want to make it work :)
so....the missing methods I told you that I need are:
#nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut:
I guess these are pretty easy. But I think they clutter the interface of ZnBufferedWriteStream, so maybe you should make a subclass.
Yeah, don't worry. I can even duplicate the class hehehe
#contents
Implement #contents I guess it is something like:
ZnBufferedWriteStream >> contents ^ stream contents
Why to you need #contents ?
becasue I am an idiot. No, I don't need it. You are correct. Thanks for asking.
I would say that it goes a bit against the concept of a stream as a sink of data. I haven't looked, but I would guess that saying #contents to a
FileStream
is not efficient.
Those missing methods I need are implemented PositionableStream. I took the implementation from there and put it in ZnBufferedWriteStream. I just added to them a first line " self flushBufferIfFull."
That is probably OK, except when your string becomes larger than the buffer. Have a look at #nextPutAll:
I am not sure if I understood. The following are correct for sure then: #nextNumber:put: #nextInt32Put: #nextWordPut:
And #nextStringPut: is like this:
nextStringPut: s "Append the string, s, to the receiver. Only used by DataStream. Max size of 64*256*256*256."
| length | self flushBufferIfFull. (length := s size) < 192 ifTrue: [self nextPut: length] ifFalse: [self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1)]. self nextPutAll: s asByteArray. ^s
Sorry, but i can't resist commenting on that. Why, if you demand from stream to implement #nextInt32Put: a the same time, you writing code like this
self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1)
? Then just extend your serializer with a notion of 'length' field, which you can use for anything where you need to encode length/size value, but not just for Strings. So, then the above method could be as short as:
nextStringPut: s self putLength: s size. self nexPutAll: s asByteArray.
and here you have a potential caveat because your string could be WideString .. muhahaha.
So, i suggest you to reconsider the way how you serializing strings. Instead what you could do is to extend ByteString and WideString (and perhaps similarily do for ByteSymbol and WideSymbol), the methods which is responsible to turning a receiver in a sequence of bytes, and then simply put it into output stream, whatever it might be.
Then you don't need #nextStringPut: because its a) not polymorphic, because apparently serializing ByteString should be different from serializing WideString b) instead you implementing this in Byte/WideString>>serializeToFuelStream: aStream and you done.
Yes, I am not sure if #nextStringPut: is not polymorphic, but I think that with #serializeToFuelStream: we can avoid converting the WideString to ByteArray just to write the ByteArray to the stream, and instead just write the WideString to the stream. Right?
which will finally send #nextPutAll: that does correct check.
The way I am creating it is like this:
stream := ZnBufferedWriteStream on: ((FileDirectory default forceNewFileNamed: 'mariano') binary).
So...is this correct Sven? My tests are green :)
That seems OK
Ok.
Still need to try benchmarks.
You might also consider playing with making the buffer larger.
yeah, I was trying that. Just by curious, why you put a 2^X number there?
Ahh and for reading we don't need something special because it is already buffered? (I have been told that)
Thanks a lot Sven,
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
On 18.05.2011 09:21, Martin Dias wrote:
On Tue, May 17, 2011 at 7:16 PM, Igor Stasenko <siguctua@gmail.com <mailto:siguctua@gmail.com>> wrote:
On 17 May 2011 22:58, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote: > > > On Tue, May 17, 2011 at 10:31 PM, Sven Van Caekenberghe <sven@beta9.be <mailto:sven@beta9.be>> > wrote: >> >> On 17 May 2011, at 21:57, Mariano Martinez Peck wrote: >> >> > Sven, I want to make it work :) >> > >> > so....the missing methods I told you that I need are: >> > >> > #nextStringPut: >> > #nextNumber:put: >> > #nextInt32Put: >> > #nextWordPut: >> >> I guess these are pretty easy. But I think they clutter the interface of >> ZnBufferedWriteStream, so maybe you should make a subclass. >> > > Yeah, don't worry. I can even duplicate the class hehehe > >> >> > #contents >> > >> > Implement #contents I guess it is something like: >> > >> > ZnBufferedWriteStream >> contents >> > ^ stream contents >> >> Why to you need #contents ? > > becasue I am an idiot. No, I don't need it. You are correct. Thanks for > asking. > >> >> I would say that it goes a bit against the concept of a stream as a sink >> of data. >> I haven't looked, but I would guess that saying #contents to a FileStream >> is not efficient. >> >> > Those missing methods I need are implemented PositionableStream. I took >> > the implementation from there and put it in ZnBufferedWriteStream. >> > I just added to them a first line " self flushBufferIfFull." >> >> That is probably OK, except when your string becomes larger than the >> buffer. Have a look at #nextPutAll: >> > > I am not sure if I understood. The following are correct for sure then: > #nextNumber:put: > #nextInt32Put: > #nextWordPut: > > > And #nextStringPut: is like this: > > nextStringPut: s > "Append the string, s, to the receiver. Only used by DataStream. Max > size of 64*256*256*256." > > | length | > self flushBufferIfFull. > (length := s size) < 192 > ifTrue: [self nextPut: length] > ifFalse: > [self nextPut: (length digitAt: 4)+192. > self nextPut: (length digitAt: 3). > self nextPut: (length digitAt: 2). > self nextPut: (length digitAt: 1)]. > self nextPutAll: s asByteArray. > ^s >
Sorry, but i can't resist commenting on that. Why, if you demand from stream to implement #nextInt32Put: a the same time, you writing code like this
self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1)
? Then just extend your serializer with a notion of 'length' field, which you can use for anything where you need to encode length/size value, but not just for Strings. So, then the above method could be as short as:
nextStringPut: s self putLength: s size. self nexPutAll: s asByteArray.
and here you have a potential caveat because your string could be WideString .. muhahaha.
So, i suggest you to reconsider the way how you serializing strings. Instead what you could do is to extend ByteString and WideString (and perhaps similarily do for ByteSymbol and WideSymbol), the methods which is responsible to turning a receiver in a sequence of bytes, and then simply put it into output stream, whatever it might be.
Then you don't need #nextStringPut: because its a) not polymorphic, because apparently serializing ByteString should be different from serializing WideString b) instead you implementing this in Byte/WideString>>serializeToFuelStream: aStream and you done.
Yes, I am not sure if #nextStringPut: is not polymorphic, but I think that with #serializeToFuelStream: we can avoid converting the WideString to ByteArray just to write the ByteArray to the stream, and instead just write the WideString to the stream. Right?
You don't ever need the asByteArray call to store ByteStrings if you use nextPutAll: on a default stream in binary mode, the primitive handles that just fine for variableByte subclasses. If you are doing the same as DataStream, only ByteStrings are written using nextStringPut: , WideStrings are stored with the default writeWordLike: facilities. Is there a document describing the binary format of Fuel somewhere? Cheers, Henry
On Wed, May 18, 2011 at 1:03 PM, Henrik Sperre Johansen < henrik.s.johansen@veloxit.no> wrote:
On 18.05.2011 09:21, Martin Dias wrote:
On Tue, May 17, 2011 at 7:16 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 17 May 2011 22:58, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, May 17, 2011 at 10:31 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 17 May 2011, at 21:57, Mariano Martinez Peck wrote:
Sven, I want to make it work :)
so....the missing methods I told you that I need are:
#nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut:
I guess these are pretty easy. But I think they clutter the interface
of
ZnBufferedWriteStream, so maybe you should make a subclass.
Yeah, don't worry. I can even duplicate the class hehehe
#contents
Implement #contents I guess it is something like:
ZnBufferedWriteStream >> contents ^ stream contents
Why to you need #contents ?
becasue I am an idiot. No, I don't need it. You are correct. Thanks for asking.
I would say that it goes a bit against the concept of a stream as a
sink
of data. I haven't looked, but I would guess that saying #contents to a FileStream is not efficient.
Those missing methods I need are implemented PositionableStream. I took the implementation from there and put it in ZnBufferedWriteStream. I just added to them a first line " self flushBufferIfFull."
That is probably OK, except when your string becomes larger than the buffer. Have a look at #nextPutAll:
I am not sure if I understood. The following are correct for sure then: #nextNumber:put: #nextInt32Put: #nextWordPut:
And #nextStringPut: is like this:
nextStringPut: s "Append the string, s, to the receiver. Only used by DataStream. Max size of 64*256*256*256."
| length | self flushBufferIfFull. (length := s size) < 192 ifTrue: [self nextPut: length] ifFalse: [self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1)]. self nextPutAll: s asByteArray. ^s
Sorry, but i can't resist commenting on that. Why, if you demand from stream to implement #nextInt32Put: a the same time, you writing code like this
self nextPut: (length digitAt: 4)+192. self nextPut: (length digitAt: 3). self nextPut: (length digitAt: 2). self nextPut: (length digitAt: 1)
? Then just extend your serializer with a notion of 'length' field, which you can use for anything where you need to encode length/size value, but not just for Strings. So, then the above method could be as short as:
nextStringPut: s self putLength: s size. self nexPutAll: s asByteArray.
and here you have a potential caveat because your string could be WideString .. muhahaha.
So, i suggest you to reconsider the way how you serializing strings. Instead what you could do is to extend ByteString and WideString (and perhaps similarily do for ByteSymbol and WideSymbol), the methods which is responsible to turning a receiver in a sequence of bytes, and then simply put it into output stream, whatever it might be.
Then you don't need #nextStringPut: because its a) not polymorphic, because apparently serializing ByteString should be different from serializing WideString b) instead you implementing this in Byte/WideString>>serializeToFuelStream: aStream and you done.
Yes, I am not sure if #nextStringPut: is not polymorphic, but I think that with #serializeToFuelStream: we can avoid converting the WideString to ByteArray just to write the ByteArray to the stream, and instead just write the WideString to the stream. Right?
You don't ever need the asByteArray call to store ByteStrings if you use nextPutAll: on a default stream in binary mode, the primitive handles that just fine for variableByte subclasses.
If you are doing the same as DataStream, only ByteStrings are written using nextStringPut: , WideStrings are stored with the default writeWordLike: facilities.
Is there a document describing the binary format of Fuel somewhere?
Hi Henry. When I sent this email, I was thinking in you :) Depends what you mean by binary format ;) How the whole subgraph is encoded (the order, the references, indexes, etc) is quite hard to follow. This is because of the pickle format. But I guess that if you check this slides: http://rmod.lille.inria.fr/web/pier/software/Fuel you will get it. That being said, to see how Fuel serialize or deserialize a particular class is really easy. Every class has a Fuel extension method called #fuelSerializer that answers the class to serialize/des it. So the easiest way is to check for #fuelSerializer implementors and then map to its serializer class. The relation is not 1 to 1. To load it: Gofer it squeaksource: 'Fuel'; package: 'ConfigurationOfFuel'; load. ((Smalltalk at: #ConfigurationOfFuel) project version: '1.2-baseline') load. If you play with something and you want to run benchmarks, just open a Transcript and evaluate: FLBenchmarks newFullFuelOnly run Thanks Henry for any review you can do. -- Mariano http://marianopeck.wordpress.com
On 18 May 2011 09:21, Martin Dias <tinchodias@gmail.com> wrote:
On Tue, May 17, 2011 at 7:16 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 17 May 2011 22:58, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, May 17, 2011 at 10:31 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 17 May 2011, at 21:57, Mariano Martinez Peck wrote:
Sven, I want to make it work :)
so....the missing methods I told you that I need are:
#nextStringPut: #nextNumber:put: #nextInt32Put: #nextWordPut:
I guess these are pretty easy. But I think they clutter the interface of ZnBufferedWriteStream, so maybe you should make a subclass.
Yeah, don't worry. I can even duplicate the class hehehe
#contents
Implement #contents I guess it is something like:
ZnBufferedWriteStream >> contents ^ stream contents
Why to you need #contents ?
becasue I am an idiot. No, I don't need it. You are correct. Thanks for asking.
I would say that it goes a bit against the concept of a stream as a sink of data. I haven't looked, but I would guess that saying #contents to a FileStream is not efficient.
Those missing methods I need are implemented PositionableStream. I took the implementation from there and  put it in ZnBufferedWriteStream. I just added to them a first line "   self flushBufferIfFull."
That is probably OK, except when your string becomes larger than the buffer. Have a look at #nextPutAll:
I am not sure if I understood. The following are correct for sure then: Â #nextNumber:put: Â #nextInt32Put: Â #nextWordPut:
And #nextStringPut:Â Â is like this:
nextStringPut: s    "Append the string, s, to the receiver. Only used by DataStream. Max size of 64*256*256*256."
   | length |    self flushBufferIfFull.    (length := s size) < 192       ifTrue: [self nextPut: length]       ifFalse:          [self nextPut: (length digitAt: 4)+192.          self nextPut: (length digitAt: 3).          self nextPut: (length digitAt: 2).          self nextPut: (length digitAt: 1)].    self nextPutAll: s asByteArray.    ^s
Sorry, but i can't resist commenting on that. Why, if you demand from stream to implement #nextInt32Put: a the same time, you writing code like this
self nextPut: (length digitAt: 4)+192. Â Â Â Â Â Â self nextPut: (length digitAt: 3). Â Â Â Â Â Â self nextPut: (length digitAt: 2). Â Â Â Â Â Â self nextPut: (length digitAt: 1)
? Then just extend your serializer with a notion of 'length' field, which you can use for anything where you need to encode length/size value, but not just for Strings. So, then the above method could be as short as:
nextStringPut: s  self putLength: s size.  self nexPutAll: s asByteArray.
and here you have a potential caveat because your string could be WideString .. muhahaha.
So, i suggest you to reconsider the way how you serializing strings. Instead what you could do is to extend ByteString and WideString (and perhaps similarily do for ByteSymbol and WideSymbol), the methods which is responsible to turning a receiver in a sequence of bytes, and then simply put it into output stream, whatever it might be.
Then you don't need #nextStringPut: because its a) not polymorphic, because apparently serializing ByteString should be different from serializing WideString b) instead you implementing this in Byte/WideString>>serializeToFuelStream: aStream and you done.
Yes, I am not sure if #nextStringPut: is not polymorphic, but I think that with #serializeToFuelStream: we can avoid converting the WideString to ByteArray just to write the ByteArray to the stream, and instead just write the WideString to the stream. Right?
I'm not sure how you doing that, but i think you could just implement Object>>serializeToFuelStream: where you can handle everything, without bothering implementing separate specific methods for ByteString or WideString. You can just implement a generic serializer which serializing object according to its format (variable bytes, variable pointers or variable words etc). -- Best regards, Igor Stasenko AKA sig.
On Tue, May 17, 2011 at 3:34 PM, Sven Van Caekenberghe <sven@beta9.be>wrote:
Mariano,
On 17 May 2011, at 15:21, Mariano Martinez Peck wrote:
Now Igor told us for example, to use a buffer like this:
| bufferStream | bufferStream := ByteArray new writeStream.
(FLSerializer on: bufferStream) serialize: anArrayToSerialize.
aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
The disadvantage of the above is that you buffer everything before you write. Furthermore you are growing the buffer all the time (recopying contents) which is not too efficient.
Some time ago we had a discussion about this and then I implemented ZnBufferedWriteStream (it should be in Core 1.3), have a look. It was just a quick hack, but it worked in speeding up writing.
Sven, I improved quite a lot using your buffered write stream. What I don't understans is why MultiByteFileStream doesn't perform as good as them. If I see StandardFileStream has an internal buffer called 'buffer1'. So why is that much difference between using MultiByteFileStream directly and a WriteStream (ByteArray new writeSteam ) or ZnBufferedWriteStream ? Thanks -- Mariano http://marianopeck.wordpress.com
Mariano, On 18 May 2011, at 14:29, Mariano Martinez Peck wrote:
Sven, I improved quite a lot using your buffered write stream. What I don't understans is why MultiByteFileStream doesn't perform as good as them. If I see StandardFileStream has an internal buffer called 'buffer1'.
So why is that much difference between using MultiByteFileStream directly and a WriteStream (ByteArray new writeSteam ) or ZnBufferedWriteStream ?
I just looked briefly at the hierarchy, it seems that buffer1 is an output buffer of size 1 ! Have a look at StandardFileStream>>#nextPut: char "Write the given character to this file." rwmode ifFalse: [^ self error: 'Cannot write a read-only file']. collection ifNotNil: [ position < readLimit ifTrue: [ self flushReadBuffer ] ]. buffer1 at: 1 put: char. self primWrite: fileID from: buffer1 startingAt: 1 count: 1. ^ char There is simply no buffering going on. buffer1 is a mystery to me ;-) Sven BTW: the above method flushes the READ buffer while WRITING, there is probably an explanation for this, but I can't think of one.
On 18.05.2011 14:53, Sven Van Caekenberghe wrote:
Mariano,
On 18 May 2011, at 14:29, Mariano Martinez Peck wrote:
Sven, I improved quite a lot using your buffered write stream. What I don't understans is why MultiByteFileStream doesn't perform as good as them. If I see StandardFileStream has an internal buffer called 'buffer1'.
So why is that much difference between using MultiByteFileStream directly and a WriteStream (ByteArray new writeSteam ) or ZnBufferedWriteStream ? Because actually writing to file is expensive. MBFS does it on every nextPut/nextPutAll: send, ZnBuffered only when asked to/buffer fills.
I just looked briefly at the hierarchy, it seems that buffer1 is an output buffer of size 1 !
Have a look at
StandardFileStream>>#nextPut: char "Write the given character to this file."
rwmode ifFalse: [^ self error: 'Cannot write a read-only file']. collection ifNotNil: [ position< readLimit ifTrue: [ self flushReadBuffer ] ]. buffer1 at: 1 put: char. self primWrite: fileID from: buffer1 startingAt: 1 count: 1. ^ char
There is simply no buffering going on. buffer1 is a mystery to me ;-)
Sven The primitive requires an object of variableByte type. If you didn't have a buffer1, you'd need to create a new 1-element ByteArray/String on every call to nextPut: ;)
As for the primitive call always flushing to file, that is a decision which have arguments both in favor and against it. I wasn't there when it was written, so have no idea why they weighed one over the other. If I were to guess, I'd say PLS ("But I wrote it to file! Why isnt it there yet when I open it??") and deterministic behaviour won over performance. Changing it now would probably mean quite a few changes to f.ex. source management if you want the same level of recoverability you are ensured now. That does not mean a buffering of write stream should not be available, of course. There are so many ways to do that though it quickly turns into painting a bikeshed. Cheers, Henry
Mariano, It is early in the morning right here and I might have read too hastily to your post, but from the diagonal reading I gave it seems you're going quickly to reinvent the wheel! Those issues you rise are typical of database (engine) programming. Also, IIRC Smalltalk has another approach to persistence of objects called BOSS, have you given a look at it and check if a mix of Fuel + BOSS would accomplish your intent? -- Cesar Rabak Em 17/05/2011 10:21, Mariano Martinez Peck < marianopeck@gmail.com > escreveu: Hi guys. Together with Martin we are developing a fast object graph serializer called Fuel. The idea is to use a pickle format similar to VW Parcels. Right now we are able to correctly serialize (almost) all kind of objects (from different classes and so on). The algorithm is working quite well and we are already quite fast. We also know how to encode more or less efficient some of the "primitive" classes like Floats, SmallIntegers, ByteString, ByteArray, etc, etc. What we really don't know anything about streams. For example, until now, we were using just a MultiByteFileStream and doing all the time things like #next: #nextPutAll: #nextWordPut: etc... Now Igor told us for example, to use a buffer like this: | bufferStream | bufferStream := ByteArray new writeStream. (FLSerializer on: bufferStream) serialize: anArrayToSerialize. aFileStream nextPutAll: bufferStream contents. and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ? For reading (deserialization) we are not doing nothing special. We just open the file stream and we read with #next #nextString ... etc. Another thing is which kind of file stream should we use. MultiByteFileStream ? StandardFileStream ? So...before spending time in some optimizations, are there "known" things that we should check? Thanks in advance, -- Mariano http://marianopeck.wordpress.com
On 17 May 2011 15:51, <csrabak@bol.com.br> wrote:
Mariano,
It is early in the morning right here and I might have read too hastily to your post, but from the diagonal reading I gave it seems you're going quickly to reinvent the wheel!
Those issues you rise are typical of database (engine) programming. Â Also, IIRC Smalltalk has another approach to persistence of objects called BOSS, have you given a look at it and check if a mix of Fuel + BOSS would accomplish your intent?
I think its implemented from scratch for one big reason: devil is always in details. Object format, streams, protocol and dozen of various things may differ from one smalltalk dialect to another one. Also, i assume most of us are never heard about BOSS. Just googled it and found couple citations which related to VW world, but not to Squeak/Pharo. Is it runs on Pharo? What its license? But by saying that, i'm a of course in favor of having more and more cross-dialect solutions.
-- Cesar Rabak
Em 17/05/2011 10:21, Mariano Martinez Peck < marianopeck@gmail.com > escreveu: Hi guys. Together with Martin we are developing a fast object graph serializer called Fuel. The idea is to use a pickle format similar to VW Parcels. Right now we are able to correctly serialize (almost) all kind of objects (from different classes and so on). The algorithm is working quite well and we are already quite fast. We also know how to encode more or less efficient some of the "primitive" classes like Floats, SmallIntegers, ByteString, ByteArray, etc, etc.
What we really don't know anything about streams. For example, until now, we were using just a MultiByteFileStream and doing all the time things like #next:Â #nextPutAll:Â #nextWordPut: etc...
Now Igor told us for example, to use a buffer like this:
 | bufferStream |  bufferStream := ByteArray new writeStream.
 (FLSerializer on: bufferStream)  serialize: anArrayToSerialize.
 aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
For reading (deserialization) we are not doing nothing special. We just open the file stream and we read with #next #nextString ... etc.
Another thing is which kind of file stream should we use. MultiByteFileStream ? StandardFileStream ?
So...before spending time in some optimizations, are there "known" things that we should check?
Thanks in advance,
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
On 17 May 2011 15:51, <csrabak@bol.com.br> wrote:
Mariano,
It is early in the morning right here and I might have read too hastily to your post, but from the diagonal reading I gave it seems you're going quickly to reinvent the wheel!
Those issues you rise are typical of database (engine) programming. Â Also, IIRC Smalltalk has another approach to persistence of objects called BOSS, have you given a look at it and check if a mix of Fuel + BOSS would accomplish your intent?
I think its implemented from scratch for one big reason: devil is always in details. Object format, streams, protocol and dozen of various things may differ from one smalltalk dialect to another one. Also, i assume most of us are never heard about BOSS. Just googled it and found couple citations which related to VW world, but not to Squeak/Pharo. Is it runs on Pharo? What its license? But by saying that, i'm a of course in favor of having more and more cross-dialect solutions.
-- Cesar Rabak
Em 17/05/2011 10:21, Mariano Martinez Peck < marianopeck@gmail.com > escreveu: Hi guys. Together with Martin we are developing a fast object graph serializer called Fuel. The idea is to use a pickle format similar to VW Parcels. Right now we are able to correctly serialize (almost) all kind of objects (from different classes and so on). The algorithm is working quite well and we are already quite fast. We also know how to encode more or less efficient some of the "primitive" classes like Floats, SmallIntegers, ByteString, ByteArray, etc, etc.
What we really don't know anything about streams. For example, until now, we were using just a MultiByteFileStream and doing all the time things like #next:Â #nextPutAll:Â #nextWordPut: etc...
Now Igor told us for example, to use a buffer like this:
 | bufferStream |  bufferStream := ByteArray new writeStream.
 (FLSerializer on: bufferStream)  serialize: anArrayToSerialize.
 aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
For reading (deserialization) we are not doing nothing special. We just open the file stream and we read with #next #nextString ... etc.
Another thing is which kind of file stream should we use. MultiByteFileStream ? StandardFileStream ?
So...before spending time in some optimizations, are there "known" things that we should check?
Thanks in advance,
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
Mariano,
It is early in the morning right here and I might have read too hastily to your post, but from the diagonal reading I gave it seems you're going quickly to reinvent the wheel!
Those issues you rise are typical of database (engine) programming. Also, IIRC Smalltalk has another approach to persistence of objects called BOSS, have you given a look at it and check if a mix of Fuel + BOSS would accomplish your intent?
I think its implemented from scratch for one big reason: devil is always in details. Object format, streams, protocol and dozen of various things may differ from one smalltalk dialect to another one.
Also, i assume most of us are never heard about BOSS. Just googled it and found couple citations which related to VW world, but not to Squeak/Pharo. Is it runs on Pharo? What its license?
boss is a library in VW.
But by saying that, i'm a of course in favor of having more and more cross-dialect solutions.
for me larger autonomy. We do not invent by being forced to be compatible. If we are successful people will copy. Stef
On Tue, May 17, 2011 at 3:51 PM, <csrabak@bol.com.br> wrote:
Mariano,
It is early in the morning right here and I might have read too hastily to your post, but from the diagonal reading I gave it seems you're going quickly to reinvent the wheel!
Hi. Good evenening. In fact, we are not reinventing the whele. As I said, Fuel takes the concepts of Parcels.
Those issues you rise are typical of database (engine) programming.
Database programming are just ONE usage of serializacion. But there are much more: - persistance. - session management in a webframework. - remote messages and whatever remote. - code versioning In summary, whenever you need to take a subgraph and get a chunk of bytes from it and vice-versa. Also, IIRC Smalltalk has another approach to persistence of objects called
BOSS, have you given a look at it and check if a mix of Fuel + BOSS would accomplish your intent?
Yes. It is in VW. I promise we have analyzed all serializers available: Magma serializer, MC2 serializer, BOSS, ReferenceStream, ImageSegment, etc. Some of them, like BOSS focus on portability. I don't care at all in portability. Some other do not even support class reshape or they are too slow. If it works in other dialects, perfect. But I won't loose performance just because of that. The main focus of Fuel is to be FAST. And even more, be fast at deserializacion time (like Parcels). The idea is that at serializacion time you can do some calculus and groups objects some way so that the deserializacion is really fast.
-- Cesar Rabak
Em 17/05/2011 10:21, Mariano Martinez Peck < marianopeck@gmail.com > escreveu: Hi guys. Together with Martin we are developing a fast object graph serializer called Fuel. The idea is to use a pickle format similar to VW Parcels. Right now we are able to correctly serialize (almost) all kind of objects (from different classes and so on). The algorithm is working quite well and we are already quite fast. We also know how to encode more or less efficient some of the "primitive" classes like Floats, SmallIntegers, ByteString, ByteArray, etc, etc.
What we really don't know anything about streams. For example, until now, we were using just a MultiByteFileStream and doing all the time things like #next: #nextPutAll: #nextWordPut: etc...
Now Igor told us for example, to use a buffer like this:
| bufferStream | bufferStream := ByteArray new writeStream.
(FLSerializer on: bufferStream) serialize: anArrayToSerialize.
aFileStream nextPutAll: bufferStream contents.
and that it at least 2 times faster than we were doing.... I guess it is because it goes to the disk only once. But MultiByteFileStream uses a buffer, doesn't it ?
For reading (deserialization) we are not doing nothing special. We just open the file stream and we read with #next #nextString ... etc.
Another thing is which kind of file stream should we use. MultiByteFileStream ? StandardFileStream ?
So...before spending time in some optimizations, are there "known" things that we should check?
Thanks in advance,
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
participants (7)
-
csrabak＠bol.com.br -
Henrik Sperre Johansen -
Igor Stasenko -
Mariano Martinez Peck -
Martin Dias -
Stéphane Ducasse -
Sven Van Caekenberghe