[Pharo-project] Any serializer can customize serialization strategy?
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process? With external I mean it could be in a separate binary (I could arrange the binding though). Thanks Hernán
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do: String >> fuelSubstitution ^ ExternalCompressor compress: self String >> fuelAccept: aVisitor ^ (ExternalCompressor shouldCompressString: self) ifTrue: [aVisitor visitSubstitution: self] ifFalse: [super fuelAccept: aVisitor] But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString: asnwers true to the compressed string, then you will end up in an infinitive loop. See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are going to compress it is better to do it during serialization. With external I mean it could
be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
I forgot to say something..... notice that using this hook is slower than a plain serialization. So what I mean is that if the strings are not that big, then it may happens that the usage of the hook + the compression is slower than a plain serialization + the regular string. I let you know because you may not expect that. I should benchmark the overhead of the hook.... Cheers On Mon, Oct 10, 2011 at 9:54 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do:
String >> fuelSubstitution ^ ExternalCompressor compress: self
String >> fuelAccept: aVisitor ^ (ExternalCompressor shouldCompressString: self) ifTrue: [aVisitor visitSubstitution: self] ifFalse: [super fuelAccept: aVisitor]
But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString: asnwers true to the compressed string, then you will end up in an infinitive loop.
See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are going to compress it is better to do it during serialization.
With external I mean it could
be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Thanks Mariano, That's pretty much what I want, but... why to expose the visitor interface? ;) Just to give a little feedback of what I have in my mind: mySerializer setEngine: ExternalCompressor for: ( MyClass -> #myStringIVar ) or mySerializer setEngine: ExternalCompressor for: incomingStream Now with machines generating daily 6 TBytes of raw data, one of the keys will be data compression. 2011/10/10 Mariano Martinez Peck <marianopeck@gmail.com>:
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do:
String >> fuelSubstitution ^ ExternalCompressor compress: self
String >> fuelAccept: aVisitor ^ (ExternalCompressor shouldCompressString: self) ifTrue: [aVisitor visitSubstitution: self] ifFalse: [super fuelAccept: aVisitor]
But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString:Â asnwers true to the compressed string, then you will end up in an infinitive loop.
See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are going to compress it is better to do it during serialization.
With external I mean it could be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
On Tue, Oct 11, 2011 at 12:56 AM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
Thanks Mariano,
That's pretty much what I want, but... why to expose the visitor interface? ;)
Good question. Honesty, it was the only way we have found to do it with Fuel. I think (but I am not sure) that one of the drawbacks of the pickle format is that it analysis phase (as we call it in Fuel) make some hooks more complicated to implement than a classic serializer.
Just to give a little feedback of what I have in my mind:
mySerializer setEngine: ExternalCompressor for: ( MyClass -> #myStringIVar )
Well, you can also create an specific Fuel cluster for MyClass that does exactly that. Basically you have to create FLMyClassCluster, implement methods like #serializeWith: and #materializeFrom: and there you serialize/materialize the specific #myStringIVar with a special way and the rest as normal. Then you subclass the FLDefaultMapper, say YourMapper and you implement #visitXXX Then you have to implement MyClass >> visitXXX. Of course, you should be able to generalize such cluster and be able to reuse it for different classes or instVars. I am not saying this is the best way. I am just telling you one possible solution you can do by extending Fuel for your needs. Of course, you may need to understand how Fuel works first. But of course, we can help if you want.
or
mySerializer setEngine: ExternalCompressor for: incomingStream
Now with machines generating daily 6 TBytes of raw data, one of the keys will be data compression.
it looks like ;)
2011/10/10 Mariano Martinez Peck <marianopeck@gmail.com>:
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do:
String >> fuelSubstitution ^ ExternalCompressor compress: self
String >> fuelAccept: aVisitor ^ (ExternalCompressor shouldCompressString: self) ifTrue: [aVisitor visitSubstitution: self] ifFalse: [super fuelAccept: aVisitor]
But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString: asnwers true to the compressed string, then you will end up in an infinitive loop.
See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are
going
to compress it is better to do it during serialization.
With external I mean it could be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Hi Hernan, Could your "specific Strings" be instances of a variant of RemoteString that compresses and decompresses? Then they will occupy less memory in your image and it would be easier to serializer and deserialize the proxy object. Thanks for the feedback, I also don't like the visitor interface we have for customizations. In fact, I think it is not a visitor but a double-dispatch. And I would prefer to merge the two hooks Mariano proposed to something like: String >> fuelMapOn: aDefaultMapper (ExternalCompressor shouldCompressString: self) ifTrue: [aDefaultMapper mapSubstitution: self by: (ExternalCompressor compress: self)] ifFalse: [super fuelMapOn: aDefaultMapper] Do you like more this way? Martin On Tue, Oct 11, 2011 at 4:10 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Tue, Oct 11, 2011 at 12:56 AM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
Thanks Mariano,
That's pretty much what I want, but... why to expose the visitor interface? ;)
Good question. Honesty, it was the only way we have found to do it with Fuel. I think (but I am not sure) that one of the drawbacks of the pickle format is that it analysis phase (as we call it in Fuel) make some hooks more complicated to implement than a classic serializer.
Just to give a little feedback of what I have in my mind:
mySerializer setEngine: ExternalCompressor for: ( MyClass -> #myStringIVar )
Well, you can also create an specific Fuel cluster for MyClass that does exactly that. Basically you have to create FLMyClassCluster, implement methods like #serializeWith: and #materializeFrom: and there you serialize/materialize the specific #myStringIVar with a special way and the rest as normal. Then you subclass the FLDefaultMapper, say YourMapper and you implement #visitXXX Then you have to implement MyClass >> visitXXX. Of course, you should be able to generalize such cluster and be able to reuse it for different classes or instVars.
I am not saying this is the best way. I am just telling you one possible solution you can do by extending Fuel for your needs. Of course, you may need to understand how Fuel works first. But of course, we can help if you want.
or
mySerializer setEngine: ExternalCompressor for: incomingStream
Now with machines generating daily 6 TBytes of raw data, one of the keys will be data compression.
it looks like ;)
2011/10/10 Mariano Martinez Peck <marianopeck@gmail.com>:
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do:
String >> fuelSubstitution ^ ExternalCompressor compress: self
String >> fuelAccept: aVisitor ^ (ExternalCompressor shouldCompressString: self) ifTrue: [aVisitor visitSubstitution: self] ifFalse: [super fuelAccept: aVisitor]
But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString: asnwers true to the compressed string, then you will end up in an infinitive loop.
See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are
going
to compress it is better to do it during serialization.
With external I mean it could be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Hi Martin 2011/10/13 Martin Dias <tinchodias@gmail.com>:
Hi Hernan, Could your "specific Strings" be instances of a variant of RemoteString that compresses and decompresses? Then they will occupy less memory in your image and it would be easier to serializer and deserialize the proxy object.
I'd like to keep my objects cross-dialect, so if that's not an issue it would be ok to use a RemoteString. I don't see why it would be easier to serialize/deserialize?
Thanks for the feedback, I also don't like the visitor interface we have for customizations. In fact, I think it is not a visitor but a double-dispatch. And I would prefer to merge the two hooks Mariano proposed to something like: String >> fuelMapOn: aDefaultMapper (ExternalCompressor shouldCompressString: self) ifTrue: [aDefaultMapper mapSubstitution: self by: (ExternalCompressor compress: self)] ifFalse: [super fuelMapOn: aDefaultMapper]
Do you like more this way?
Absolutely :)
Martin
On Tue, Oct 11, 2011 at 4:10 PM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Oct 11, 2011 at 12:56 AM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Thanks Mariano,
That's pretty much what I want, but... why to expose the visitor interface? ;)
Good question. Honesty, it was the only way we have found to do it with Fuel. I think (but I am not sure) that one of the drawbacks of the pickle format is that it analysis phase (as we call it in Fuel) make some hooks more complicated to implement than a classic serializer.
Just to give a little feedback of what I have in my mind:
mySerializer setEngine: ExternalCompressor for: ( MyClass -> #myStringIVar )
Well, you can also create an specific Fuel cluster for MyClass that does exactly that. Basically you have to create FLMyClassCluster, implement methods like #serializeWith: and #materializeFrom:Â and there you serialize/materialize the specific #myStringIVar with a special way and the rest as normal. Then you subclass the FLDefaultMapper, say YourMapper and you implement #visitXXX Then you have to implement MyClass >> visitXXX. Of course, you should be able to generalize such cluster and be able to reuse it for different classes or instVars.
I am not saying this is the best way. I am just telling you one possible solution you can do by extending Fuel for your needs. Of course, you may need to understand how Fuel works first. But of course, we can help if you want.
or
mySerializer setEngine: ExternalCompressor for: incomingStream
Now with machines generating daily 6 TBytes of raw data, one of the keys will be data compression.
it looks like ;)
2011/10/10 Mariano Martinez Peck <marianopeck@gmail.com>:
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do:
String >> fuelSubstitution    ^ ExternalCompressor compress: self
String >> fuelAccept: aVisitor   ^ (ExternalCompressor shouldCompressString: self)   ifTrue: [aVisitor visitSubstitution: self]   ifFalse: [super fuelAccept: aVisitor]
But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString:Â asnwers true to the compressed string, then you will end up in an infinitive loop.
See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are going to compress it is better to do it during serialization.
With external I mean it could be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On Fri, Oct 14, 2011 at 2:40 AM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
Hi Martin
2011/10/13 Martin Dias <tinchodias@gmail.com>:
Hi Hernan, Could your "specific Strings" be instances of a variant of RemoteString that compresses and decompresses? Then they will occupy less memory in your image and it would be easier to serializer and deserialize the proxy object.
I'd like to keep my objects cross-dialect, so if that's not an issue it would be ok to use a RemoteString. I don't see why it would be easier to serialize/deserialize?
I don't know it has sense, also it depends on your case... but if MyRemoteString just has a instVar 'filename' or 'fileIndex' to fetch the real string from disk when necessary... it should be easy to serialize instances of such MyRemoteString.
Thanks for the feedback, I also don't like the visitor interface we have for customizations. In fact, I think it is not a visitor but a double-dispatch. And I would prefer to merge the two hooks Mariano proposed to something like: String >> fuelMapOn: aDefaultMapper (ExternalCompressor shouldCompressString: self) ifTrue: [aDefaultMapper mapSubstitution: self by: (ExternalCompressor compress: self)] ifFalse: [super fuelMapOn: aDefaultMapper]
Do you like more this way?
Absolutely :)
thanks... probably for Fuel 1.8!
Martin
On Tue, Oct 11, 2011 at 4:10 PM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Oct 11, 2011 at 12:56 AM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Thanks Mariano,
That's pretty much what I want, but... why to expose the visitor interface? ;)
Good question. Honesty, it was the only way we have found to do it with Fuel. I think (but I am not sure) that one of the drawbacks of the
pickle
format is that it analysis phase (as we call it in Fuel) make some hooks more complicated to implement than a classic serializer.
Just to give a little feedback of what I have in my mind:
mySerializer setEngine: ExternalCompressor for: ( MyClass -> #myStringIVar )
Well, you can also create an specific Fuel cluster for MyClass that does exactly that. Basically you have to create FLMyClassCluster, implement methods like #serializeWith: and #materializeFrom: and there you serialize/materialize the specific #myStringIVar with a special way and the rest as normal. Then you subclass the FLDefaultMapper, say YourMapper and you implement #visitXXX Then you have to implement MyClass >> visitXXX. Of course, you should be able to generalize such cluster and be able to reuse it for different classes or instVars.
I am not saying this is the best way. I am just telling you one possible solution you can do by extending Fuel for your needs. Of course, you may need to understand how Fuel works first. But of course, we can help if you want.
or
mySerializer setEngine: ExternalCompressor for: incomingStream
Now with machines generating daily 6 TBytes of raw data, one of the keys will be data compression.
it looks like ;)
2011/10/10 Mariano Martinez Peck <marianopeck@gmail.com>:
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
In Fuel you can do:
String >> fuelSubstitution ^ ExternalCompressor compress: self
String >> fuelAccept: aVisitor ^ (ExternalCompressor shouldCompressString: self) ifTrue: [aVisitor visitSubstitution: self] ifFalse: [super fuelAccept: aVisitor]
But the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString: asnwers true to the compressed string, then you will end up in an infinitive loop.
See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are going to compress it is better to do it during serialization.
With external I mean it could be in a separate binary (I could arrange the binding though).
I am not sure if I understood.
Thanks
Hernán
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Hi, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Is there any serializer (Fuel/StOMP/Ma object serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process?
Mariano Martinez Peck <marianopeck@gmail.com> wrote:
I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods
Yes. StOMP does support such kind of custom serialization. Pattern 1: Using Memento. StOMP has two basic hook methods (stompWriteValue/stompReadValue). With a memento pattern, you can customize serialization/deserialization behavior easily. SpecificString>> stompWriteValue ^ StringCompressedMemento on: self StringCompressedMemento>> stompReadValue ^ SpecificString on: self uncompressToString SpecificString is a mere wrapper class (holding original string). And StringCompressedMemento should implement compress/uncompress logic. Pattern 2: Subclassing Writer/Reader Pattern 1 is usable, but it is basically suitable for pointer objects. For String-like objects, defining a new String wrapper and Memento is overwhelming. You can also subclass StompWriter/Reader for adding custom serialization/deserialization behaviors. MyStompWriter>>writeString: aString (self isSpecificString: aString) ifTrue: [^super writeString: (self compressString: aString)]. ^ super writeString: string MyStompReader>>readByteString | rawString | rawString := self basicReadObject asString. ^ (self isCompressedString: rawString) ifTrue: [self uncompressString: rawString] ifFalse: [rawString] You have to add some marking headers to the original string for determining it should be compressed/uncompressed. (Yes. this is a little awkward, but works well). I've attached working sample codes. So please see for details. Best regards, -- [:masashi | ^umezawa]
participants (4)
-
Hernán Morales Durand -
Mariano Martinez Peck -
Martin Dias -
Masashi UMEZAWA