[Pharo-project] Fuel - a fast object deployment tool
Hi all Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects. There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community. = Pickle format = The pickle format and the serialization algorithm main idea, is explained in this slides: http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example = Current features = - Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests = Next steps = - Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior) = Download = In a Pharo 1.1 or 1.1.1 evaluate: Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load. = Benchmarks = You can run benchmarks executing this line (results in Transcript): FLBenchmarks newBasic run. Thank you! Martin Dias
Thanks martin. Stef On Dec 8, 2010, at 5:50 PM, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
It look cool! Is this a better ImageSegment? Alexandre On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On Dec 8, 2010, at 8:42 PM, Alexandre Bergel wrote:
It look cool! Is this a better ImageSegment?
there is no VM code implied and you do not get all the problems of imageSegment. You should ask is it better than SmartRefStreams. Yes Now the code has not been optimized
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
It look cool! Is this a better ImageSegment?
there is no VM code implied and you do not get all the problems of imageSegment. You should ask is it better than SmartRefStreams. Yes Now the code has not been optimized
Ah ok! How to deal with stubs is the difference with ImageSegment then. It looks exciting! Alexandre
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
alex this is good candidate for your profiler :) Stef On Dec 8, 2010, at 9:05 PM, Alexandre Bergel wrote:
It look cool! Is this a better ImageSegment?
there is no VM code implied and you do not get all the problems of imageSegment. You should ask is it better than SmartRefStreams. Yes Now the code has not been optimized
Ah ok! How to deal with stubs is the difference with ImageSegment then.
It looks exciting!
Alexandre
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On Wed, Dec 8, 2010 at 9:05 PM, Alexandre Bergel <alexandre@bergel.eu>wrote:
It look cool! Is this a better ImageSegment?
there is no VM code implied and you do not get all the problems of imageSegment. You should ask is it better than SmartRefStreams. Yes Now the code has not been optimized
Ah ok! How to deal with stubs is the difference with ImageSegment then.
The difference is that in Fuel there is no managment for "shared objects". Fuel is not for swapping (there are no stubs/proxies and becomes), and the idea is that you write on a file, ALL (not only those objects that are ONLY accessible by the roots, like in ImageSegment) the objects reachable from user defined roots. Fuel is similar to ReferenceStream and subclasses, a XML serializer, or any serializers, etc...but MOSTLY, to Parcels. It is very similar to VisualWorks parcel. Another important thing is that it is binary. The idea is that maybe in a future we have a Monticello that doesn't need a mcz (code) + Compiler, but instead, just load binary code (fuel files). This would be muuuch faster that current Monticello, and even more, for minimal images, we wouldn't need a Compiler. Finally, Fuel is designed (like Parcels) to be very fast at loading time :) Cheers Mariano
It looks exciting!
Alexandre
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On Wed, 8 Dec 2010, Mariano Martinez Peck wrote:
On Wed, Dec 8, 2010 at 9:05 PM, Alexandre Bergel <alexandre@bergel.eu>wrote:
It look cool! Is this a better ImageSegment?
there is no VM code implied and you do not get all the problems of imageSegment. You should ask is it better than SmartRefStreams. Yes Now the code has not been optimized
Ah ok! How to deal with stubs is the difference with ImageSegment then.
The difference is that in Fuel there is no managment for "shared objects". Fuel is not for swapping (there are no stubs/proxies and becomes), and the idea is that you write on a file, ALL (not only those objects that are ONLY accessible by the roots, like in ImageSegment) the objects reachable from user defined roots.
Fuel is similar to ReferenceStream and subclasses, a XML serializer, or any serializers, etc...but MOSTLY, to Parcels. It is very similar to VisualWorks parcel.
Another important thing is that it is binary. The idea is that maybe in a future we have a Monticello that doesn't need a mcz (code) + Compiler, but instead, just load binary code (fuel files). This would be muuuch faster that current Monticello, and even more, for minimal images, we wouldn't need a Compiler. Finally, Fuel is designed (like Parcels) to be very fast at loading time :)
Well, I doubt if speed is really important, since you're loading everything at most once. And the current tools are really fast IMHO. The following numbers are from Squeak: [ Compiler recompileAll ] timeToRun. 29083. CompiledMethod allInstances size. 60701. [ CompiledMethod allInstancesDo: [ :each | each getSource ] ] timeToRun. 1133. So the compiler is compiling 2087 methods per second on average. You can load 53575 methods per second from a file on average. If it's zipped, than it may be a bit slower, say a factor of 2-3x slowdown. So you can still load and compile more than 1800 methods per second. I guess thats fast enough. Even if Fuel can be 10x faster, it doesn't really make a difference IMHO. Levente
Cheers
Mariano
It looks exciting!
Alexandre
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On Dec 8, 2010, at 22:33 , Levente Uzonyi wrote:
On Wed, 8 Dec 2010, Mariano Martinez Peck wrote:
Well, I doubt if speed is really important, since you're loading everything at most once. And the current tools are really fast IMHO. The following numbers are from Squeak: [ Compiler recompileAll ] timeToRun. 29083. CompiledMethod allInstances size. 60701. [ CompiledMethod allInstancesDo: [ :each | each getSource ] ] timeToRun. 1133. So the compiler is compiling 2087 methods per second on average. You can load 53575 methods per second from a file on average. If it's zipped, than it may be a bit slower, say a factor of 2-3x slowdown. So you can still load and compile more than 1800 methods per second. I guess thats fast enough.
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
It would be interesting to thoroughly profile MC to figure where it spends all its time (with large projects it gets very very slow, like several minutes to just show the merge diffs between two branches). Adrian
On Wed, Dec 8, 2010 at 10:38 PM, Adrian Lienhard <adi@netstyle.ch> wrote:
On Dec 8, 2010, at 22:33 , Levente Uzonyi wrote:
So the compiler is compiling 2087 methods per second on average. You can load 53575 methods per second from a file on average. If it's zipped, than it may be a bit slower, say a factor of 2-3x slowdown. So you can still load and compile more than 1800 methods per second. I guess thats fast enough.
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
On Wed, 8 Dec 2010, Mariano Martinez Peck wrote:
Well, I doubt if speed is really important, since you're loading everything at most once. And the current tools are really fast IMHO. The following numbers are from Squeak: [ Compiler recompileAll ] timeToRun. 29083. CompiledMethod allInstances size. 60701. [ CompiledMethod allInstancesDo: [ :each | each getSource ] ] timeToRun.
It would be interesting to thoroughly profile MC to figure where it spends all its time (with large projects it gets very very slow, like several minutes to just show the merge diffs between two branches).
Exactly. That's why I was suggesting that maybe with a fast binary serializer this can be much faster.
On Wed, 8 Dec 2010, Adrian Lienhard wrote:
On Dec 8, 2010, at 22:33 , Levente Uzonyi wrote:
On Wed, 8 Dec 2010, Mariano Martinez Peck wrote:
Well, I doubt if speed is really important, since you're loading everything at most once. And the current tools are really fast IMHO. The following numbers are from Squeak: [ Compiler recompileAll ] timeToRun. 29083. CompiledMethod allInstances size. 60701. [ CompiledMethod allInstancesDo: [ :each | each getSource ] ] timeToRun. 1133. So the compiler is compiling 2087 methods per second on average. You can load 53575 methods per second from a file on average. If it's zipped, than it may be a bit slower, say a factor of 2-3x slowdown. So you can still load and compile more than 1800 methods per second. I guess thats fast enough.
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
It would be interesting to thoroughly profile MC to figure where it spends all its time (with large projects it gets very very slow, like several minutes to just show the merge diffs between two branches).
I guess those days are over when MC spends minutes doing this, it's at most a few seconds for large packages. The 1.5MB Morphic package of Squeak can be compared to another really old version (changes) in 3 seconds. According to MessageTally 50% of the time is spend in getting the timeStamp for the methods. Levente
Adrian
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
It would be interesting to thoroughly profile MC to figure where it spends all its time (with large projects it gets very very slow, like several minutes to just show the merge diffs between two branches).
I guess those days are over when MC spends minutes doing this, it's at most a few seconds for large packages. The 1.5MB Morphic package of Squeak can be compared to another really old version (changes) in 3 seconds. According to MessageTally 50% of the time is spend in getting the timeStamp for the methods.
Sure, that's for one package. But we don't just put everything in one package; we have like 20 packages for a large project. 20 x 3s is already a minute. Large ancestry and HTTP repository also has an influence. I've done several small optimizations (like adding caching) to speed things up. But still, MC speed in our case is rather measured in minutes than seconds :( Adrian
On Fri, 10 Dec 2010, Adrian Lienhard wrote:
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
It would be interesting to thoroughly profile MC to figure where it spends all its time (with large projects it gets very very slow, like several minutes to just show the merge diffs between two branches).
I guess those days are over when MC spends minutes doing this, it's at most a few seconds for large packages. The 1.5MB Morphic package of Squeak can be compared to another really old version (changes) in 3 seconds. According to MessageTally 50% of the time is spend in getting the timeStamp for the methods.
Sure, that's for one package. But we don't just put everything in one package; we have like 20 packages for a large project. 20 x 3s is already a minute. Large ancestry and HTTP repository also has an influence. I've done several small optimizations (like adding caching) to speed things up. But still, MC speed in our case is rather measured in minutes than seconds :(
Note that this is not package loading, but viewing the changes. And the 3 seconds is for a 1.5MB package. I doubt you have 20 x 1.5MB packages. Btw Keith's MC1.5/1.6 is really impressive. With Squeak 3.10.2 (no closures, no cog, no buffered files, etc) it takes 6.8 seconds to load RoelTyper + OCompletion. In a current Squeak image that takes 4.3 seconds with Cog. Levente
Adrian
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel. Stef
The difference is that in Fuel there is no managment for "shared objects". Fuel is not for swapping (there are no stubs/proxies and becomes), and the idea is that you write on a file, ALL (not only those objects that are ONLY accessible by the roots, like in ImageSegment) the objects reachable from user defined roots.
Fuel is similar to ReferenceStream and subclasses, a XML serializer, or any serializers, etc...but MOSTLY, to Parcels. It is very similar to VisualWorks parcel.
Another important thing is that it is binary. The idea is that maybe in a future we have a Monticello that doesn't need a mcz (code) + Compiler, but instead, just load binary code (fuel files). This would be muuuch faster that current Monticello, and even more, for minimal images, we wouldn't need a Compiler. Finally, Fuel is designed (like Parcels) to be very fast at loading time :)
Well, I doubt if speed is really important, since you're loading everything at most once. And the current tools are really fast IMHO. The following numbers are from Squeak: [ Compiler recompileAll ] timeToRun. 29083. CompiledMethod allInstances size. 60701. [ CompiledMethod allInstancesDo: [ :each | each getSource ] ] timeToRun. 1133. So the compiler is compiling 2087 methods per second on average. You can load 53575 methods per second from a file on average. If it's zipped, than it may be a bit slower, say a factor of 2-3x slowdown. So you can still load and compile more than 1800 methods per second. I guess thats fast enough.
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
Levente
Cheers
Mariano
It looks exciting!
Alexandre
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package? Levente
Stef
The difference is that in Fuel there is no managment for "shared objects". Fuel is not for swapping (there are no stubs/proxies and becomes), and the idea is that you write on a file, ALL (not only those objects that are ONLY accessible by the roots, like in ImageSegment) the objects reachable from user defined roots.
Fuel is similar to ReferenceStream and subclasses, a XML serializer, or any serializers, etc...but MOSTLY, to Parcels. It is very similar to VisualWorks parcel.
Another important thing is that it is binary. The idea is that maybe in a future we have a Monticello that doesn't need a mcz (code) + Compiler, but instead, just load binary code (fuel files). This would be muuuch faster that current Monticello, and even more, for minimal images, we wouldn't need a Compiler. Finally, Fuel is designed (like Parcels) to be very fast at loading time :)
Well, I doubt if speed is really important, since you're loading everything at most once. And the current tools are really fast IMHO. The following numbers are from Squeak: [ Compiler recompileAll ] timeToRun. 29083. CompiledMethod allInstances size. 60701. [ CompiledMethod allInstancesDo: [ :each | each getSource ] ] timeToRun. 1133. So the compiler is compiling 2087 methods per second on average. You can load 53575 methods per second from a file on average. If it's zipped, than it may be a bit slower, say a factor of 2-3x slowdown. So you can still load and compile more than 1800 methods per second. I guess thats fast enough.
Even if Fuel can be 10x faster, it doesn't really make a difference IMHO.
Levente
Cheers
Mariano
It looks exciting!
Alexandre
Alexandre
On 8 Dec 2010, at 13:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time. Stef
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time.
Do you think MC is "slow" because the Compiler is "slow"? Levente
Stef
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time.
Do you think MC is "slow" because the Compiler is "slow"?
not totally but I do not understand why this is obligatory to compile everything all the time. Then Opal will be slower than the old comcrapiler.
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time.
Do you think MC is "slow" because the Compiler is "slow"?
not totally but I do not understand why this is obligatory to compile everything all the time. Then Opal will be slower than the old comcrapiler.
I did a quick test where it took 4294ms to load RoelTyper + OCompletion from disk (no socket creation, no network latency, etc). These packages contain 790 methods. 5.1% of the total time was spent for compilation, that's 219ms. The rest is used by other stuff like: - loading the files from disk - writing the source code to disk (sources/changes files) - evaluating class side #initialize methods - processing system change notifications Another 51ms was spent in creating 63 classes and 107ms to install the CompiledMethods to the classes. So that's 219+51+107 = 377ms for creating all classes and their methods. The rest is administration which can not be avoided by binary loading. Even if binary loading is 10x faster than compiling the code (which I doubt), then you save 339ms. So it would take only 3917ms to load these packages. That would save you 7.9% of the total time. Levente
thanks for the analysis so we will have to speed it up too.
It does! It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time.
Do you think MC is "slow" because the Compiler is "slow"?
not totally but I do not understand why this is obligatory to compile everything all the time. Then Opal will be slower than the old comcrapiler.
I did a quick test where it took 4294ms to load RoelTyper + OCompletion from disk (no socket creation, no network latency, etc). These packages contain 790 methods. 5.1% of the total time was spent for compilation, that's 219ms. The rest is used by other stuff like: - loading the files from disk - writing the source code to disk (sources/changes files) - evaluating class side #initialize methods - processing system change notifications
Another 51ms was spent in creating 63 classes and 107ms to install the CompiledMethods to the classes. So that's 219+51+107 = 377ms for creating all classes and their methods. The rest is administration which can not be avoided by binary loading.
Even if binary loading is 10x faster than compiling the code (which I doubt), then you save 339ms. So it would take only 3917ms to load these packages. That would save you 7.9% of the total time.
Levente
Hi Levente, 2010/12/9 Levente Uzonyi <leves@elte.hu>
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
It does!
It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time.
Do you think MC is "slow" because the Compiler is "slow"?
not totally but I do not understand why this is obligatory to compile everything all the time. Then Opal will be slower than the old comcrapiler.
I did a quick test where it took 4294ms to load RoelTyper + OCompletion from disk (no socket creation, no network latency, etc). These packages contain 790 methods. 5.1% of the total time was spent for compilation, that's 219ms. The rest is used by other stuff like: - loading the files from disk - writing the source code to disk (sources/changes files) - evaluating class side #initialize methods - processing system change notifications
Another 51ms was spent in creating 63 classes and 107ms to install the CompiledMethods to the classes. So that's 219+51+107 = 377ms for creating all classes and their methods. The rest is administration which can not be avoided by binary loading.
Even if binary loading is 10x faster than compiling the code (which I doubt), then you save 339ms. So it would take only 3917ms to load these packages. That would save you 7.9% of the total time.
One of the important features of the VW parcels work is that one does not write the source code to the changes file. Instead teh system has a SourceFileManager that works like a dictionary mapping file indices to source files, so instead of the two element SourceFileArray one has an arbitrarily large collection of files. When a parcel file is loaded its source file is added to the SFM which returns an index and then all the file pointers in the methods in the parcel are swizzled to refer to their sources' position in their files' index. We changed the format of file pointers so something resembling four floating-point formats so we can have lots of small files (more space for file indices) and a few large files (more space for file offsets) before one overflows into large integer file pointers. Now Igor's trailer work makes this approach feasible but one wouldn't need the funky floating-point format stuff because one could easily allocate, say, 5 bytes to the file pointer, two for the file index for a maximum of 64k source files, and 3 for the file offset for a maximum of 16m of source per parcel source file (I think splitting the 40 bits as 14:26 might be better but that's details). One needs to predetermine the size of the file pointer so that the trailer can be modified in place since changing the length of a trailer means allocating a new comp[iled method and that will be slow. If this approach is taken how much does that change your analysis? Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files. There are downsides. Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps. best Eliot
Levente
On Fri, 10 Dec 2010, Eliot Miranda wrote:
Hi Levente,
2010/12/9 Levente Uzonyi <leves@elte.hu>
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
It does!
It seems that you did not work in VW2.5 and 3.0 and when parcels arrived loading was realllllly a big difference I do not see why this would not the same with Fuel.
No I didn't, but the version number of VW is around 7.x now, so I guess the CPUs and VMs are now several times faster. Does it really matter if it takes 200ms or 20ms to load a package?
When I load a package with MC so far I still notice it and I would not even think about it. It should load in a unnoticeable amount of time.
Do you think MC is "slow" because the Compiler is "slow"?
not totally but I do not understand why this is obligatory to compile everything all the time. Then Opal will be slower than the old comcrapiler.
I did a quick test where it took 4294ms to load RoelTyper + OCompletion from disk (no socket creation, no network latency, etc). These packages contain 790 methods. 5.1% of the total time was spent for compilation, that's 219ms. The rest is used by other stuff like: - loading the files from disk - writing the source code to disk (sources/changes files) - evaluating class side #initialize methods - processing system change notifications
Another 51ms was spent in creating 63 classes and 107ms to install the CompiledMethods to the classes. So that's 219+51+107 = 377ms for creating all classes and their methods. The rest is administration which can not be avoided by binary loading.
Even if binary loading is 10x faster than compiling the code (which I doubt), then you save 339ms. So it would take only 3917ms to load these packages. That would save you 7.9% of the total time.
One of the important features of the VW parcels work is that one does not write the source code to the changes file. Instead teh system has a SourceFileManager that works like a dictionary mapping file indices to source files, so instead of the two element SourceFileArray one has an arbitrarily large collection of files. When a parcel file is loaded its source file is added to the SFM which returns an index and then all the file pointers in the methods in the parcel are swizzled to refer to their sources' position in their files' index. We changed the format of file pointers so something resembling four floating-point formats so we can have lots of small files (more space for file indices) and a few large files (more space for file offsets) before one overflows into large integer file pointers.
Now Igor's trailer work makes this approach feasible but one wouldn't need the funky floating-point format stuff because one could easily allocate, say, 5 bytes to the file pointer, two for the file index for a maximum of 64k source files, and 3 for the file offset for a maximum of 16m of source per parcel source file (I think splitting the 40 bits as 14:26 might be better but that's details). One needs to predetermine the size of the file pointer so that the trailer can be modified in place since changing the length of a trailer means allocating a new comp[iled method and that will be slow.
If this approach is taken how much does that change your analysis?
Writing the sources takes 1.6% of total time according to MessageTally. Significant amount of time is spent in finalization (12.8%) and GCs (15%). But note that MessageTally is not reliable nowadays.
Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files.
There are downsides. Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps.
Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels. Levente
best Eliot
Levente
If this approach is taken how much does that change your analysis?
But note that MessageTally is not reliable nowadays.
Why ? since when ?
Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files.
There are downsides. Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps.
Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.
Levente
best Eliot
Levente
On Fri, 10 Dec 2010, Mariano Martinez Peck wrote:
If this approach is taken how much does that change your analysis?
But note that MessageTally is not reliable nowadays.
Why ? since when ?
It's like this since a long time, but it seems to be worse with Cog. See http://bugs.squeak.org/view.php?id=7515 for details. Levente
Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files.
There are downsides. Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps.
Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.
Levente
best Eliot
Levente
2010/12/10 Levente Uzonyi <leves@elte.hu> > On Fri, 10 Dec 2010, Eliot Miranda wrote: > > Hi Levente, >> >> 2010/12/9 Levente Uzonyi <leves@elte.hu> >> >> On Thu, 9 Dec 2010, Stéphane Ducasse wrote: >>> >>> It does! >>> >>>> It seems that you did not work in VW2.5 and 3.0 and when parcels >>>>>>>> arrived loading was realllllly a big difference >>>>>>>> I do not see why this would not the same with Fuel. >>>>>>>> >>>>>>>> >>>>>>> No I didn't, but the version number of VW is around 7.x now, so I >>>>>>> guess >>>>>>> the CPUs and VMs are now several times faster. Does it really matter >>>>>>> if it >>>>>>> takes 200ms or 20ms to load a package? >>>>>>> >>>>>>> >>>>>> When I load a package with MC so far I still notice it and I would not >>>>>> even think about it. >>>>>> It should load in a unnoticeable amount of time. >>>>>> >>>>>> >>>>> Do you think MC is "slow" because the Compiler is "slow"? >>>>> >>>>> >>>> not totally but I do not understand why this is obligatory to compile >>>> everything all the time. >>>> Then Opal will be slower than the old comcrapiler. >>>> >>>> >>>> I did a quick test where it took 4294ms to load RoelTyper + OCompletion >>> from disk (no socket creation, no network latency, etc). These packages >>> contain 790 methods. 5.1% of the total time was spent for compilation, >>> that's 219ms. The rest is used by other stuff like: >>> - loading the files from disk >>> - writing the source code to disk (sources/changes files) >>> - evaluating class side #initialize methods >>> - processing system change notifications >>> >>> Another 51ms was spent in creating 63 classes and 107ms to install the >>> CompiledMethods to the classes. So that's 219+51+107 = 377ms for creating >>> all classes and their methods. The rest is administration which can not >>> be >>> avoided by binary loading. >>> >>> Even if binary loading is 10x faster than compiling the code (which I >>> doubt), then you save 339ms. So it would take only 3917ms to load these >>> packages. That would save you 7.9% of the total time. >>> >>> >> One of the important features of the VW parcels work is that one does not >> write the source code to the changes file. Instead teh system has a >> SourceFileManager that works like a dictionary mapping file indices to >> source files, so instead of the two element SourceFileArray one has an >> arbitrarily large collection of files. When a parcel file is loaded its >> source file is added to the SFM which returns an index and then all the >> file >> pointers in the methods in the parcel are swizzled to refer to their >> sources' position in their files' index. We changed the format of file >> pointers so something resembling four floating-point formats so we can >> have >> lots of small files (more space for file indices) and a few large files >> (more space for file offsets) before one overflows into large integer file >> pointers. >> >> Now Igor's trailer work makes this approach feasible but one wouldn't need >> the funky floating-point format stuff because one could easily allocate, >> say, 5 bytes to the file pointer, two for the file index for a maximum of >> 64k source files, and 3 for the file offset for a maximum of 16m of source >> per parcel source file (I think splitting the 40 bits as 14:26 might be >> better but that's details). One needs to predetermine the size of the >> file >> pointer so that the trailer can be modified in place since changing the >> length of a trailer means allocating a new comp[iled method and that will >> be >> slow. >> >> If this approach is taken how much does that change your analysis? >> > > Writing the sources takes 1.6% of total time according to MessageTally. > Significant amount of time is spent in finalization (12.8%) and GCs (15%). > But note that MessageTally is not reliable nowadays. > > > >> Note that not writing source to the changes file has ancilliary benefits; >> change recovery is now not polluted with package loads and the changes >> file >> does not grow as packages are added, only as one's changes are made. >> Unloading a package doesn't leave garbage in the changes files. >> >> There are downsides. Deploying a development image means deploying all >> the >> associated parcel source files as well, and for this a >> platform-independent >> Filename abstraction really helps. >> > > Thanks for the explanation. I wonder how the previous versions of a method > can be found using parcels. > I hacked a dreadful implementation of overrides in vw3.0 and I don't think things are much better now. But in http://www.mail-archive.com/pharo-project@lists.gforge.inria.fr/msg17714.htmlI sketched how I think it should be done: Maintain a global package load order (a stack of loaded packages, removing interior elements on unload). Maintain a dictionary from method reference to set of package/method pairs for each method that is overridden. When a package is removed search overrides and compute the overridden methods to be reinstalled, computing the uppermost method depending on the new package order. So to answer your question, one finds the previous versions directly in the overrides dictionary, and sorts the results according to the current package load order. best Eliot > > > Levente > > >> best >> Eliot >> >> >> >>> Levente >>> >>
On Fri, 10 Dec 2010 21:11:47 +0100, Eliot Miranda <eliot.miranda@gmail.com> wrote: Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files. There are downsides. Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps.
Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.
I hacked a dreadful implementation of overrides in vw3.0 and I don't think things are much better now. But in http://www.mail-archive.com/pharo-project@lists.gforge.inria.fr/msg17714.htm... I sketched how I think it should be done:
Maintain a global package load order (a stack of loaded packages, removing interior elements on unload).
Maintain a dictionary from method reference to set of package/method pairs for each method that is overridden. When a package is removed search overrides and compute the overridden methods to be reinstalled, computing the uppermost method depending on the new package order.
So to answer your question, one finds the previous versions directly in the overrides dictionary, and sorts the results according to the current package load order. Wasn't Levente asking about "regular" replacing of methods? I thought his question was about that; if the source is not in the changes file but in a parcel source file, then when I save a new version of a method and want to look for the old one, that will not be in the changes file. When I build up a new image from parcels, the load script copies all the sources to the changes file, so I have easy access to the history. Just wondering... Peter
best
Eliot
Levente
best Eliot
Levente
On Wed, Dec 15, 2010 at 5:23 AM, Peter van Rooijen <peter@vanrooijen.com>wrote:
On Fri, 10 Dec 2010 21:11:47 +0100, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files.
There are downsides. Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps.
Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.
I hacked a dreadful implementation of overrides in vw3.0 and I don't think things are much better now. But in http://www.mail-archive.com/pharo-project@lists.gforge.inria.fr/msg17714.htm... sketched how I think it should be done:
Maintain a global package load order (a stack of loaded packages, removing interior elements on unload). Maintain a dictionary from method reference to set of package/method pairs for each method that is overridden. When a package is removed search overrides and compute the overridden methods to be reinstalled, computing the uppermost method depending on the new package order.
So to answer your question, one finds the previous versions directly in the overrides dictionary, and sorts the results according to the current package load order.
Wasn't Levente asking about "regular" replacing of methods?
I thought his question was about that; if the source is not in
the changes file but in a parcel source file, then when I save
a new version of a method and want to look for the old one,
that will not be in the changes file.
No. Anything you write goes on the changes files including new method definitions, doits, reorganizations, method removals class removals etc.
When I build up a new image
from parcels, the load script copies all the sources to the
changes file, so I have easy access to the history.
Why bother? A published parcel is an immutable versioned artifact. It doesn't change over time, so, like a release sources file, one can always rely on its contents. HTH Eliot
Just wondering...
Peter
best Eliot
Levente
best Eliot
Levente
On 15 December 2010 19:30, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Wed, Dec 15, 2010 at 5:23 AM, Peter van Rooijen <peter@vanrooijen.com> wrote:
On Fri, 10 Dec 2010 21:11:47 +0100, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Note that not writing source to the changes file has ancilliary benefits; change recovery is now not polluted with package loads and the changes file does not grow as packages are added, only as one's changes are made. Unloading a package doesn't leave garbage in the changes files.
There are downsides. Â Deploying a development image means deploying all the associated parcel source files as well, and for this a platform-independent Filename abstraction really helps.
Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.
I hacked a dreadful implementation of overrides in vw3.0 and I don't think things are much better now.  But in http://www.mail-archive.com/pharo-project@lists.gforge.inria.fr/msg17714.htm... I sketched how I think it should be done: Maintain a global package load order (a stack of loaded packages, removing interior elements on unload). Maintain a dictionary from method reference to set of package/method pairs for each method that is overridden. When a package is removed search overrides and compute the overridden methods to be reinstalled, computing the uppermost method depending on the new package order. So to answer your question, one finds the previous versions directly in the overrides dictionary, and sorts the results according to the current package load order.
Wasn't Levente asking about "regular" replacing of methods?
I thought his question was about that; if the source is not in
the changes file but in a parcel source file, then when I save
a new version of a method and want to look for the old one,
that will not be in the changes file.
No. Â Anything you write goes on the changes files including new method definitions, doits, reorganizations, method removals class removals etc.
When I build up a new image
from parcels, the load script copies all the sources to the
changes file, so I have easy access to the history.
Why bother? Â A published parcel is an immutable versioned artifact. Â It doesn't change over time, so, like a release sources file, one can always rely on its contents.
+1 Instead of putting all methods and classes from package into .changes it could be just a simple doit, like: MC load: 'blahblah.mcz' It could be somewhat different during merge.
HTH Eliot
-- Best regards, Igor Stasenko AKA sig.
BTW when giving feedback consider that the guy doing that is spending a lot of time and this will be his master and that the code was not optimize and that there is no dedicated primitive in play. So we will see at the end and I was thinking that our little community would be much more positive but we will continue because we believe that there is some value in that. Stef
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
BTW when giving feedback consider that the guy doing that is spending a lot of time and this will be his master and that the code was not optimize and that there is no dedicated primitive in play.
So we will see at the end and I was thinking that our little community would be much more positive but we will continue because we believe that there is some value in that.
Don't get me wrong, I'm not saying that Fuel is not useful. I'm saying that improving code loading performance is not that important. Levente
Stef
2010/12/9 Levente Uzonyi <leves@elte.hu>:
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
BTW when giving feedback consider that the guy doing that is spending a lot of time and this will be his master and that the code was not optimize and that there is no dedicated primitive in play.
So we will see at the end and I was thinking that our little community would be much more positive but we will continue because we believe that there is some value in that.
Don't get me wrong, I'm not saying that Fuel is not useful. I'm saying that improving code loading performance is not that important.
Sort of. But what is most important, i think that you can exchange objects between images. MC really allows you to exchange only with source code, while with Fuel, i think you could put any object/data into binary package, and don't bother with inventing the pervasive ways how to recreate complex (or big) data structures from array literals :) Another interesting aspect of binary format is that you can give binary to people without disclosing source code.. (waving to corporate world ;)
Levente
Stef
-- Best regards, Igor Stasenko AKA sig.
Hi all, Thank you for the discussion, is very interesting for me. Thanks Adrian, I don't have benchmarks with ImageSegment, so I like to see the numbers, how could I reproduce that benchmarks? it would be useful for me. I think that ImageSegment is the best if fits the needs. Maybe if in a future we implement using primitives some things it could be more comparable. As someones said, Fuel is more similar to Parcels, in fact we started working based on this paper: http://scg.unibe.ch/archive/papers/Mira05aParcels.pdf About if "objects are much more times loaded than stored", yes... Fuel is not the universal serialization solution, but I believe it could be useful in some cases. Beside of the speed, I think that is also important the "user experience" part of the purpose ("is worth to spend time while storing in order to have faster loading and user experience"), in the sense of giving more features that makes easier the life of the developer to share their object structures. Maybe Fuel could give more flexibility than ImageSegment to select which objects of the graph store and which not. It could let the user define custom rules. For example if FuelFameExtension is installed, when Fuel stores an object first looks at the Fame metadescription of that object and then only stores the "not derived attributes" of that object. That was necessary to export and import properly Moose models. Martin On Thu, Dec 9, 2010 at 3:58 PM, Igor Stasenko <siguctua@gmail.com> wrote:
2010/12/9 Levente Uzonyi <leves@elte.hu>:
On Thu, 9 Dec 2010, Stéphane Ducasse wrote:
BTW when giving feedback consider that the guy doing that is spending a lot of time and this will be his master and that the code was not optimize and that there is no dedicated primitive in play.
So we will see at the end and I was thinking that our little community would be much more positive but we will continue because we believe that there is some value in that.
Don't get me wrong, I'm not saying that Fuel is not useful. I'm saying that improving code loading performance is not that important.
Sort of. But what is most important, i think that you can exchange objects between images. MC really allows you to exchange only with source code, while with Fuel, i think you could put any object/data into binary package, and don't bother with inventing the pervasive ways how to recreate complex (or big) data structures from array literals :)
Another interesting aspect of binary format is that you can give binary to people without disclosing source code.. (waving to corporate world ;)
Levente
Stef
-- Best regards, Igor Stasenko AKA sig.
On Dec 9, 2010, at 16:25 , Martin Dias wrote:
Hi all,
Thank you for the discussion, is very interesting for me.
Thanks Adrian, I don't have benchmarks with ImageSegment, so I like to see the numbers, how could I reproduce that benchmarks? it would be useful for me.
This is the essential part of our code (in a subclass of ImageSegment) that writes out our model starting from the root anObject basicSave: anObject | stream temp symbolHolder | symbolHolder := Symbol allSymbols. self copyFromRoots: (Array with: anObject) sizeHint: self fileSize // 2 areUnique: true. state = #activeCopy ifFalse: [ ^ self logger error: 'wrong serializer state' ]. temp := endMarker. endMarker := nil. stream := FileStream forceNewFileNamed: fileName. [ stream fileOutClass: nil andObject: self ] ensure: [ stream close ]. endMarker := temp. Loading is done as follows: | stream streamContents mode object | stream := aDirectory oldFileOrNoneNamed: filename. streamContents := stream fileInObjectAndCode. ^ streamContents install I just timed how long it takes to run these method. HTH, Adrian
Ok even if I disagree. We could imagine a system where we do not need the compiler to be present to load code.
BTW when giving feedback consider that the guy doing that is spending a lot of time and this will be his master and that the code was not optimize and that there is no dedicated primitive in play.
So we will see at the end and I was thinking that our little community would be much more positive but we will continue because we believe that there is some value in that.
Don't get me wrong, I'm not saying that Fuel is not useful. I'm saying that improving code loading performance is not that important.
Levente
Stef
Hi Martin, I took some application for which we use image segments to test Fuel - With Fuel serializing and writing to disk took 330s. File size is 16.1MB - With image segments saving takes 4s and the file size is 2.4MB - When loading, I got a low space warning because the primitive newMethod:header: failed. - Loading with image segments takes 11 seconds I wonder what your rational is for "objects are much more times loaded than stored". In our case its exactly the other way round. We store very often (like every couple of minutes if there are changes), but only load when we restart an image (this may be weeks). HTH, Adrian On Dec 8, 2010, at 17:50 , Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
On Wed, Dec 8, 2010 at 10:33 PM, Adrian Lienhard <adi@netstyle.ch> wrote:
Hi Martin,
I took some application for which we use image segments to test Fuel
- With Fuel serializing and writing to disk took 330s. File size is 16.1MB - With image segments saving takes 4s and the file size is 2.4MB
but how are you using ImageSegment? just the primitive? because in order to compare it to Fuel, you should write all objects, including "outPointers". So you should use #writeForExportOn: or similar... funny it is only 4s in IS, since it has to do a #become, a full GC mark phase, etc...
- When loading, I got a low space warning because the primitive newMethod:header: failed. - Loading with image segments takes 11 seconds
I wonder what your rational is for "objects are much more times loaded than stored".
That's Fuel purpose. And it is useful for version systems, where you may commit once (a specfic version for example), but load hundred of times.
In our case its exactly the other way round. We store very often (like every couple of minutes if there are changes), but only load when we restart an image (this may be weeks).
So I guess Fuel is not the best approach for you :)
HTH, Adrian
On Dec 8, 2010, at 17:50 , Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
On Dec 8, 2010, at 22:49 , Mariano Martinez Peck wrote:
On Wed, Dec 8, 2010 at 10:33 PM, Adrian Lienhard <adi@netstyle.ch> wrote:
Hi Martin,
I took some application for which we use image segments to test Fuel
- With Fuel serializing and writing to disk took 330s. File size is 16.1MB - With image segments saving takes 4s and the file size is 2.4MB
but how are you using ImageSegment? just the primitive? because in order to compare it to Fuel, you should write all objects, including "outPointers". So you should use #writeForExportOn: or similar...
yes, this includes outPointers serialized with reference stream and writing to disk. Adrian BTW, I was just providing the numbers that I gathered when looking at Fuel (to see whether that could be interesting for our use case to replace image segments). This was not to say that Fuel is not on the right way or anything, but I though the numbers would be interesting for Martin because they show a real-world use case with a large number of objects. I know that Fuel is in an early stage of development and it doesn't (yet?) have a primitive/plugin to speed things up.
sure I understood that like that. I think that the use case for fuel is fast loading like VW parcels.
Hi Martin,
I took some application for which we use image segments to test Fuel
- With Fuel serializing and writing to disk took 330s. File size is 16.1MB - With image segments saving takes 4s and the file size is 2.4MB
but how are you using ImageSegment? just the primitive? because in order to compare it to Fuel, you should write all objects, including "outPointers". So you should use #writeForExportOn: or similar...
yes, this includes outPointers serialized with reference stream and writing to disk.
Adrian
BTW, I was just providing the numbers that I gathered when looking at Fuel (to see whether that could be interesting for our use case to replace image segments). This was not to say that Fuel is not on the right way or anything, but I though the numbers would be interesting for Martin because they show a real-world use case with a large number of objects. I know that Fuel is in an early stage of development and it doesn't (yet?) have a primitive/plugin to speed things up.
Hi Martin, Nice project. I noticed that you have a package FuelFameExtension. Is this done for the Fame meta engine? If yes, I would be interested in testing it, especially that in the context of Moose we do load the objects significantly more often than we store them :). Cheers, Doru On 8 Dec 2010, at 17:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- www.tudorgirba.com "Sometimes the best solution is not the best solution."
Hi Tudor, Thank you. Yes, just to test the project in a real case, I created a package that uses Fame metadescription like MSE to export and import moose models. In the main page I talk a bit about the extension, but I didn't mentioned it in the summary mail. It needs to be better tested... and a progress bar like MSE! In Moose 4.1: Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelFameExtension-MartinDias.10'; load. And opening a new MoosePanel it will have the export to FL and import from FL options. I would like to have your feedback! Martin On Thu, Dec 9, 2010 at 10:02 AM, Tudor Girba <tudor.girba@gmail.com> wrote:
Hi Martin,
Nice project.
I noticed that you have a package FuelFameExtension. Is this done for the Fame meta engine? If yes, I would be interested in testing it, especially that in the context of Moose we do load the objects significantly more often than we store them :).
Cheers, Doru
On 8 Dec 2010, at 17:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new    squeaksource: 'Fuel';    version: 'Fuel-MartinDias.74';    version: 'FuelBenchmarks-MartinDias.4';    load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
-- www.tudorgirba.com
"Sometimes the best solution is not the best solution."
Hi Martin,
Nice project.
I noticed that you have a package FuelFameExtension. Is this done for the Fame meta engine? If yes, I would be interested in testing it, especially that in the context of Moose we do load the objects significantly more often than we store them :).
:) We tried to save and load some large moose models to see if it could be used for that :)
Hi Mariano and Martin, some feedback. The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me. I would just compute this map at the start of serialization. It'll take hardly any time. Whereas there's lots of scope for this information, if cached, to become stale over time. Find attached. If you're happy with this then before or after remember to evaluate SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper best Eliot
On Wed, Jun 8, 2011 at 11:59 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Mariano and Martin,
some feedback. The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me. I would just compute this map at the start of serialization. It'll take hardly any time. Whereas there's lots of scope for this information, if cached, to become stale over time.
Find attached. If you're happy with this then before or after remember to evaluate SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper
and the same goes for FLGlobalMapper. Also remember SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLGlobalMapper
best Eliot
Hi Eliot, I am glad to receive your feedback. I definitely agree with that, your changes have been added on last version. The reason for that cached information was mainly to support a Mariano's project requirement: he needs to frequently do lots of small serializations. But in any case, that cache should be managed externally of Fuel. In my computer, this bench: [ FLGlobalMapper newWith: nil ] bench. went from '402,000 per second' to '35.8 per second'. But in the common use case of a serializer it is really imperceptible. Thank you very much. Martin On Wed, Jun 8, 2011 at 5:20 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
On Wed, Jun 8, 2011 at 11:59 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Mariano and Martin,
some feedback. The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me. I would just compute this map at the start of serialization. It'll take hardly any time. Whereas there's lots of scope for this information, if cached, to become stale over time.
Find attached. If you're happy with this then before or after remember to evaluate SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper
and the same goes for FLGlobalMapper. Also remember SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLGlobalMapper
best Eliot
On Thu, Jun 9, 2011 at 9:18 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Eliot,
I am glad to receive your feedback. I definitely agree with that, your changes have been added on last version.
The reason for that cached information was mainly to support a Mariano's project requirement: he needs to frequently do lots of small serializations.
Exactly. I was the guilty. In fact, at the beginning this was exactly as you suggested now. And I ask Martin to change it heheheh.
But in any case, that cache should be managed externally of Fuel.
Yes, I can do that.
In my computer, this bench: [ FLGlobalMapper newWith: nil ] bench.
went from '402,000 per second' to '35.8 per second'. But in the common use case of a serializer it is really imperceptible.
Thank you very much. Martin
On Wed, Jun 8, 2011 at 5:20 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
On Wed, Jun 8, 2011 at 11:59 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Mariano and Martin,
some feedback. The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me. I would just compute this map at the start of serialization. It'll take hardly any time. Whereas there's lots of scope for this information, if cached, to become stale over time.
Find attached. If you're happy with this then before or after remember to evaluate SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper
and the same goes for FLGlobalMapper. Also remember SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLGlobalMapper
best Eliot
-- Mariano http://marianopeck.wordpress.com
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
Having not looked at Fuel at all... Has anyone done any serious work with Paul Baumann's SRP? http://sourceforge.net/projects/srp/ http://web.archive.org/web/20060201011511/http://wiki.cs.uiuc.edu/CampSmallt... I found it to be an extremely innovative serialisation mechanism, then again it has been a few years since I looked at it.
On 9 June 2011 03:23, Sean Malloy <smalloy@gmail.com> wrote:
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
Having not looked at Fuel at all...
Has anyone done any serious work with Paul Baumann's SRP?
And also with MessagePack (http://msgpack.org/), a highly space/time efficient binary serialization protocol. There is an implementation on SqueakSource at http://www.squeaksource.com/MessagePack and a high-level object-serialization protocol on top of that at http://www.squeaksource.com/STOMP. Lukas -- Lukas Renggli www.lukas-renggli.ch
do you know what is the encoding scheme under message pack? Huffman like? It would be a nice student projects to get one version for Smalltalk Stef On Jun 9, 2011, at 6:51 AM, Lukas Renggli wrote:
On 9 June 2011 03:23, Sean Malloy <smalloy@gmail.com> wrote:
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
Having not looked at Fuel at all...
Has anyone done any serious work with Paul Baumann's SRP?
And also with MessagePack (http://msgpack.org/), a highly space/time efficient binary serialization protocol. There is an implementation on SqueakSource at http://www.squeaksource.com/MessagePack and a high-level object-serialization protocol on top of that at http://www.squeaksource.com/STOMP.
Lukas
-- Lukas Renggli www.lukas-renggli.ch
On 9 June 2011 09:40, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
do  you know what is the encoding scheme under message pack?
It is very simple (like JSON and serializes only booleans, integers, floats, strings, arrays and dictionaries) and is described here: - design: http://wiki.msgpack.org/display/MSGPACK/Design+of+Serialization - format: http://wiki.msgpack.org/display/MSGPACK/Format+specification.
Huffman like?
Kind of, but static. Common objects are serialized with very few bits.
It would be a nice student projects to get one version for Smalltalk
I posted the links in my previous mail, there is already an implementation on SqueakSource (http://www.squeaksource.com/MessagePack). Also there is a high-level implementation that can serialize any Smalltalk object (http://www.squeaksource.com/STOMP). It seems to handle class changes too. Lukas -- Lukas Renggli www.lukas-renggli.ch
On Jun 9, 2011, at 7:01 PM, Lukas Renggli wrote:
On 9 June 2011 09:40, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
do you know what is the encoding scheme under message pack?
It is very simple (like JSON and serializes only booleans, integers, floats, strings, arrays and dictionaries) and is described here:
- design: http://wiki.msgpack.org/display/MSGPACK/Design+of+Serialization - format: http://wiki.msgpack.org/display/MSGPACK/Format+specification.
Huffman like?
Kind of, but static. Common objects are serialized with very few bits.
It would be a nice student projects to get one version for Smalltalk
I posted the links in my previous mail, there is already an implementation on SqueakSource (http://www.squeaksource.com/MessagePack). Also there is a high-level implementation that can serialize any Smalltalk object (http://www.squeaksource.com/STOMP). It seems to handle class changes too.
Ok I missed that.
Lukas
-- Lukas Renggli www.lukas-renggli.ch
Hi
Has anyone done any serious work with Paul Baumann's SRP?
And also with MessagePack (http://msgpack.org/), a highly space/time efficient binary serialization protocol. There is an implementation on SqueakSource at http://www.squeaksource.com/MessagePack and a high-level object-serialization protocol on top of that at http://www.squeaksource.com/STOMP.
I have never used SRP or MessagePack, both seems great. An important difference of Fuel is that it is not focused in cross-dialect or cross-language object exchange.
Hi Martin and Mariano, a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects? best Eliot
On Wed, Jun 8, 2011 at 10:35 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Hi, We are close to have this feature, but still is an open issue. The idea is to to support this "statically" via: ClassWithTransients >> fuelTransientInstanceVariables ^ #(myTransientValue1 myTransientValue1) or more dinamically via: FLSerializationSettings newDefault onClass: ClassWithTransients declareTransients: #(myTransientValue1 myTransientValue1). Do you have a better idea? Cheers Martin
best Eliot
On Thu, Jun 9, 2011 at 12:36 AM, Martin Dias <tinchodias@gmail.com> wrote:
On Wed, Jun 8, 2011 at 10:35 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Hi,
We are close to have this feature, but still is an open issue.
The idea is to to support this "statically" via:
ClassWithTransients >> fuelTransientInstanceVariables ^ #(myTransientValue1 myTransientValue1)
or more dinamically via:
FLSerializationSettings newDefault onClass: ClassWithTransients declareTransients: #(myTransientValue1 myTransientValue1).
Do you have a better idea?
No. The above looks fine.
Cheers Martin
best Eliot
On Thu, Jun 9, 2011 at 1:26 PM, Stéphane Ducasse <stephane.ducasse@inria.fr>wrote:
FLSerializationSettings newDefault onClass: ClassWithTransients declareTransients: #(myTransientValue1 myTransientValue1).
Do you have a better idea?
I'm dead tired and not having clear idea but is it better to mention what should be save or what should not be saved?
IMO, what should not, since this is the exception. Most times one wants to save all inst vars. e.
Stef
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side. Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization." ^#() MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1') The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side... thanks -- Mariano http://marianopeck.wordpress.com
On 13.06.2011 19:16, Mariano Martinez Peck wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
You can have variables on both instances and classes. I believe in SIXX you send #sixxIgnorableInstVarNames to the object you want to serialize. IE. on class side you define ignorable class instance variables, while on instance side you define ignorable instance variables. Cheers, Henry
Hi Martin & Mariano, regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework. We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension. In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work. Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much). As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper: FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers' It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest: FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class] It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set). FLGlueMapper>>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil: FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters' FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream super serialize: nil on: aWriteStream So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this: FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class] FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set. For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set. I think both Yaron and I feel the Fuel framework is comprehensible and flexible. We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much. Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g. MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser] This makes it easier to find a specific unique cluster to handle a group of objects specially. Lastly, the class-side cluster ids are a bit of a pain. It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice. You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization. again thanks for a great framework. best, Eliot On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
On Jun 15, 2011, at 8:29 PM, Eliot Miranda wrote:
Hi Martin & Mariano,
regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework.
I'm happy to see that we picked up good projects and guys: my consulting money is well spent I'm happy.
We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension.
In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars;
yes I imagine that you can have different cases for the same class.
there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work.
Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved.
yes like TranscriptStream
Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much).
Yes I have the impression that we need both too.
As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper:
FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers'
It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest:
FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class]
It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set).
FLGlueMapper>>visitObject: anObject
analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil:
FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters'
FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream
super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this:
FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class]
FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set.
For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
I think both Yaron and I feel the Fuel framework is comprehensible and flexible. We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
No thank you for the feedback. We are writing two papers and it will help the master of Martin and probably helping PhD funding if we can say that people really use his work.
Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g.
MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser]
This makes it easier to find a specific unique cluster to handle a group of objects specially.
Lastly, the class-side cluster ids are a bit of a pain. It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice.
You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
We will have to think about that What is important is that Fuel should support change shape and evolution.
again thanks for a great framework.
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote: Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
On Wed, Jun 15, 2011 at 8:29 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin & Mariano,
regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework.
Thanks Eliot. These are nice words. We are used to receive critics like "yet another serializer?". So it is good for us to know you like it and that could port it even to Newspeak. That means, at some point, that Fuel design is good. And this is good because we spend a lot of time in design, in good class names, good method names, class comments, tests, benchmarks, etc.
We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension.
In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved.
Exactly. At least this was implemented this week in Fuel-MartinDias.259. Basically, you can define a class method #fuelIgnoredInstanceVariableNames and all instances of such class will ignore those instVarNames. Example: MyClass class >> fuelIgnoredInstanceVariableNames ^ #(instVar1) But this can be clumsy; there may be references to objects one wants to
prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work.
+1 Do you have an example? For example, you may don't want to serialize classes of #specialObjectsArray?
Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil.
Yes, I really want that. I discussed with Dale because this is what they have in GemStone. There, they have DbTransient and they can do "aClass makeInstancesDbTransient". After that, all instnaces of aClass will be ignored. In addition, they have TransientValue, which is a class that it is always ignored. So you can use an instance of TransientValue to wrap the actual value want to be transient ... kind of value holder What do you think about it ? it sounds really similar to what you suggest :)
The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much).
+999. GemStone has exactly that so I am not surprise that you come to the same conclusion :) So...to sum up, we want: - classes whose instances are always ignored. Say #toBePrunedClasses or #makeInstancesDbTransient - particular references that we want to ignore. Say TransientValue or something like that.
As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper:
FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers'
It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest:
FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class]
It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set).
FLGlueMapper>>visitObject: anObject
analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil:
FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters'
FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream
super serialize: nil on: aWriteStream
I understand. But I have a question. This way, imagine an object whose class is in #prunedObjectClasses. So, you will serialize it as nil. BUT, what happens with the references from that object ? are those references still analyzed ? Because probably you don't want that. But you should take care of #referencesOf: anObject do: aBlock In this case you are lucky because FLNilTrueFalseCluster uses the empty implementation of FLCluster. So...no problem I think. Martin ? Check what we do in: FLObjectCluster >> referencesOf: anObject do: aBlock | ignoredInstanceVariableNames | ignoredInstanceVariableNames := theClass fuelIgnoredInstanceVariableNames. theClass instVarNamesAndOffsetsDo: [:name :index | (ignoredInstanceVariableNames includes: name) ifFalse: [ aBlock value: (anObject instVarAt: index) ]] so of course, we don't follow those instance variables
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this:
FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class]
FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set.
Yes, in fact, when you talk about #toBePrunedClasses you mean something like this: Behavior >> toBePrunedClasses FLGlueMapper addPrunnedClass: self or something like that. I am right? what I mean is that #toBePrunedClasses put the class in the array 'prunedObjectClasses' of FLGlueMapper.
For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
That's a good idea, because those mappers are evaluated for EVERY single object. If 80% of the times it is FLDefaultMapper, then we have to pay the cost of the #accepts: of all the rest of the mappers.
I think both Yaron and I feel the Fuel framework is comprehensible and flexible.
Again. Thanks. This is what people don't see when using other serializers.
We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
Thanks to you for the wonderful feedback.
Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g.
MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser]
This makes it easier to find a specific unique cluster to handle a group of objects specially.
I understand. Do you have an example?
Lastly, the class-side cluster ids are a bit of a pain.
Yes, we know :(
It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag?
Good questions. I will let Martin to answer ;)
Also, some class-side code to assign an unused id would be nice.
+999
You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
Thanks, good idea.
again thanks for a great framework.
Thanks for the feedback. Now, a couple of things I would like to comment: 1) we are preparing a paper right now so we are a concentrated there instead of the code. In addition, we have 5 failing tests that we should fix ;) So...as far as we finish with that we will continue with all these things you are talking about. 2) I will open issues in our bug tracker for everything we discuss in this thread. 3) Do you have something in mind so that we can ease your port? I mean...if we continue the development...do you plan to get new versions in the future? how are you going to do that? Thanks a lot, Mariano
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Hi Eliot, I am very happy to read your mail. On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin & Mariano,
regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework. We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension.
In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work.
Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much).
As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper:
FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers'
It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest:
FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class]
It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set).
FLGlueMapper>>visitObject: anObject
analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil:
FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters'
FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream
super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this:
FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class]
FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set.
For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel. We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
I think both Yaron and I feel the Fuel framework is comprehensible and flexible. We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
:-) thank you!
Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g.
MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser]
This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
Lastly, the class-side cluster ids are a bit of a pain. It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice.
You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient. Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for example if your Glue cluster id is the same of the Moose cluster id. I added an issue in our tracker. If it makes sense, maybe the weak array you suggest can be also used to avoid instantiating lots of FLObjectCluster like we are doing in Object: fuelCluster ^ self class isVariable ifTrue: [ FLVariableObjectCluster for: self class ] ifFalse: [ FLFixedObjectCluster for: self class ] the second time you send fuelCluster to an object, it can reuse the cluster instance.
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere? regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
On Fri, Jun 17, 2011 at 1:26 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Eliot, I am very happy to read your mail.
On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin & Mariano,
regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework. We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension.
In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work.
Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much).
As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper:
FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers'
It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest:
FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class]
It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set).
FLGlueMapper>>visitObject: anObject
analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil:
FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters'
FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream
super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this:
FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class]
FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects
and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set.
For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel.
We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
That was an idea Yaron came up with. That instead of using fuelIgnoredInstanceVariableNames one uses e.g. Object>>objectToSerialize ^self and then if one wants to prune specific inst vars in MyClass one implements MyClass>>objectToSerialize ^self shallowCopy prepareForSerialization MyClass>>prepareForSerialization instVarIDontWantToSerialize := nil. ^self and for objects one doesn't want to serlalize one implements MyNotToBeSerializedClass>>objectToSerialize ^nil So its more general. But I would pass the analyser in as an argument, which would allow things like MyPerhapsNotToBeSerializedClass>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse: [self] ifTrue: [nil] which would of course be the default in Object: Object>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse:: [self]
I think both Yaron and I feel the Fuel framework is comprehensible and flexible. We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
:-) thank you!
Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g.
MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser]
This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
To be honest I'm not sure. But passing in the analyser in things like fuelCluster or objectToSerialize is I think a good idea as it provides a convenient communication path which in turn provides considerable flexibility.
Lastly, the class-side cluster ids are a bit of a pain. It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice.
You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient.
Yes, but how inefficient? What's the size of all the cluster names? FLCluster allSubclasses inject: 0 into: [:t :c| t + c name size + 1] 670 So you'd add less than a kilobyte to the size of each serialization and get complete freedom from ids. Something to think about.
Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for example if your Glue cluster id is the same of the Moose cluster id.
I added an issue in our tracker.
If it makes sense, maybe the weak array you suggest can be also used to avoid instantiating lots of FLObjectCluster like we are doing in Object:
fuelCluster ^ self class isVariable ifTrue: [ FLVariableObjectCluster for: self class ] ifFalse: [ FLFixedObjectCluster for: self class ]
the second time you send fuelCluster to an object, it can reuse the cluster instance.
Right. I think that's important, and is one reason why I think passing in the analyser is important, because it allows certain objects to discover existing clusters in the analyzer and join them if they want to, instead of having to invent and maintain their own cluster uniquing solution .
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere?
No, and its extremely proprietary :) Newspeak however is available and we may end up maintaining a port of Fuel for Newspeak. best regards, Eliot
regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com>wrote:
Hi Martin and Mariano,
a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 1:26 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Eliot, I am very happy to read your mail.
On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin & Mariano,    regarding filtering.  Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework.  We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary.  The second extension is the ability to map specific objects to nil, to prune objects on the way out.  I want to discuss this latter extension. In our data set we have a set of references to objects that are logically not persistent and hence not to be saved.  I'm sure that this will be a common case.  The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil.  One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved.  But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work. Here are two more general schemes.  VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser.  One is a set of classes whose instances are not to be saved.  Any reference to an instance of a class in the toBePrunedClasses set is saved as nil.  The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil.  Why have both?  It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much). As an example here's how we implemented pruning.  Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper: FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers' It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest: FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer.  We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes.  We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class] It prunes by mapping instances of the prunedObjectClasses to a special cluster.  It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set). FLGlueMapper>>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil: FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters' FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this: FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class] FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set. For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel.
We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
That was an idea Yaron came up with.  That instead of using fuelIgnoredInstanceVariableNames one uses e.g. Object>>objectToSerialize    ^self and then if one wants to prune specific inst vars in MyClass one implements MyClass>>objectToSerialize    ^self shallowCopy prepareForSerialization
Hi Eliot, I'm not convinced by the shallowCopy solution, except for the simple structures. If object graph is complex (have share nodes, loops, ...) then you gonna end up in a replication problem equivalent to the one Fuel is trying to solve. Nicolas
MyClass>>prepareForSerialization    instVarIDontWantToSerialize := nil.    ^self and for objects one doesn't want to serlalize one implements MyNotToBeSerializedClass>>objectToSerialize    ^nil So its more general.  But I would pass the analyser in as an argument, which would allow things like MyPerhapsNotToBeSerializedClass>>objectToSerializeIn: anFLAnalyser    ^(anFLAnalyser shouldPrune: self)      ifFalse: [self]      ifTrue: [nil] which would of course be the default in Object: Object>>objectToSerializeIn: anFLAnalyser    ^(anFLAnalyser shouldPrune: self) ifFalse:: [self]
I think both Yaron and I feel the Fuel framework is comprehensible and flexible. Â We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Â Thank you very much.
:-) thank you!
Finally, a couple of things.  First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g. MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser] This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
To be honest I'm not sure. Â But passing in the analyser in things like fuelCluster or objectToSerialize is I think a good idea as it provides a convenient communication path which in turn provides considerable flexibility.
Lastly, the class-side cluster ids are a bit of a pain. Â It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? Â For example, are clusters ordered by id, or is this just an integer tag? Â Also, some class-side code to assign an unused id would be nice. You might think of virtualizing the id scheme. Â For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Â Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient.
Yes, but how inefficient? Â What's the size of all the cluster names? Â Â Â FLCluster allSubclasses inject: 0 into: [:t :c| t + c name size + 1] 670
So you'd add less than a kilobyte to the size of each serialization and get complete freedom from ids. Â Something to think about.
Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for example if your Glue cluster id is the same of the Moose cluster id.
I added an issue in our tracker.
If it makes sense, maybe the weak array you suggest can be also used to avoid instantiating lots of FLObjectCluster like we are doing in Object:
fuelCluster    ^ self class isVariable       ifTrue: [ FLVariableObjectCluster for: self class ]       ifFalse: [ FLFixedObjectCluster for: self class ]
the second time you send fuelCluster to an object, it can reuse the cluster instance.
Right. Â I think that's important, and is one reason why I think passing in the analyser is important, because it allows certain objects to discover existing clusters in the analyzer and join them if they want to, instead of having to invent and maintain their own cluster uniquing solution .
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere?
No, and its extremely proprietary :) Â Newspeak however is available and we may end up maintaining a port of Fuel for Newspeak. best regards, Eliot
regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin and Mariano, Â Â Â a couple of questions. Â What's the right way to exclude certain objects from the serialization? Â Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames    "Indicates which variables have to be ignored during serialization."
   ^#()
MyClass class >> fuelIgnoredInstanceVariableNames  ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
On Fri, Jun 17, 2011 at 2:39 PM, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 1:26 AM, Martin Dias <tinchodias@gmail.com>
wrote:
Hi Eliot, I am very happy to read your mail.
On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com
wrote:
Hi Martin & Mariano, regarding filtering. Yesterday my colleague Yaron and I
successfully
finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework. We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension. In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work. Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much). As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper: FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers' It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest: FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class] It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set). FLGlueMapper>>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil: FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters' FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this: FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class] FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set. For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel.
We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
That was an idea Yaron came up with. That instead of using fuelIgnoredInstanceVariableNames one uses e.g. Object>>objectToSerialize ^self and then if one wants to prune specific inst vars in MyClass one implements MyClass>>objectToSerialize ^self shallowCopy prepareForSerialization
Hi Eliot,
I'm not convinced by the shallowCopy solution, except for the simple structures. If object graph is complex (have share nodes, loops, ...) then you gonna end up in a replication problem equivalent to the one Fuel is trying to solve.
The assumption is that the analyser would create a maximum of one proxy per object in the graph (default, no proxy) and that it would map objects with proxies to their proxies. So if proxies only nilled out inst vars I don't see a problem. What's attractive about this is that it provides a general solution to a couple of problems, a) how to replace a class of objects by some substitute (e.g. nil), b) how to prune state that needn't be saved. It is also conceptually simple; one just creates a proxy instance; no defining metadata, such as inst var names, and hence the code is always up-to-date (e.g. a class redefine won't automatically uncover renamed inst vars in serialization metadata).
Nicolas
MyClass>>prepareForSerialization instVarIDontWantToSerialize := nil. ^self and for objects one doesn't want to serlalize one implements MyNotToBeSerializedClass>>objectToSerialize ^nil So its more general. But I would pass the analyser in as an argument, which would allow things like MyPerhapsNotToBeSerializedClass>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse: [self] ifTrue: [nil] which would of course be the default in Object: Object>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse:: [self]
I think both Yaron and I feel the Fuel framework is comprehensible and flexible. We enjoyed using it and while we took two passes at coming
up
with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
:-) thank you!
Finally, a couple of things. First, it may be more flexible to
implement
fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g. MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser] This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
To be honest I'm not sure. But passing in the analyser in things like fuelCluster or objectToSerialize is I think a good idea as it provides a convenient communication path which in turn provides considerable flexibility.
Lastly, the class-side cluster ids are a bit of a pain. It would be
nice
to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice. You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient.
Yes, but how inefficient? What's the size of all the cluster names? FLCluster allSubclasses inject: 0 into: [:t :c| t + c name size + 1] 670
So you'd add less than a kilobyte to the size of each serialization and get complete freedom from ids. Something to think about.
Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for example if your Glue cluster id is the same of the Moose cluster id.
I added an issue in our tracker.
If it makes sense, maybe the weak array you suggest can be also used to avoid instantiating lots of FLObjectCluster like we are doing in Object:
fuelCluster ^ self class isVariable ifTrue: [ FLVariableObjectCluster for: self class ] ifFalse: [ FLFixedObjectCluster for: self class ]
the second time you send fuelCluster to an object, it can reuse the cluster instance.
Right. I think that's important, and is one reason why I think passing in the analyser is important, because it allows certain objects to discover existing clusters in the analyzer and join them if they want to, instead of having to invent and maintain their own cluster uniquing solution .
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere?
No, and its extremely proprietary :) Newspeak however is available and we may end up maintaining a port of Fuel for Newspeak. best regards, Eliot
regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <
eliot.miranda@gmail.com>
wrote:
Hi Martin and Mariano, a couple of questions. What's the right way to exclude certain objects from the serialization? Is there a way of excluding certain
inst
vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
-- best, Eliot
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 2:39 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 1:26 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Eliot, I am very happy to read your mail.
On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin & Mariano,    regarding filtering.  Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it to save and restore our data sets; thank you, its a cool framework.  We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary.  The second extension is the ability to map specific objects to nil, to prune objects on the way out.  I want to discuss this latter extension. In our data set we have a set of references to objects that are logically not persistent and hence not to be saved.  I'm sure that this will be a common case.  The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil.  One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved.  But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work. Here are two more general schemes.  VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser.  One is a set of classes whose instances are not to be saved.  Any reference to an instance of a class in the toBePrunedClasses set is saved as nil.  The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil.  Why have both?  It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much). As an example here's how we implemented pruning.  Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper: FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers' It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest: FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer.  We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes.  We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class] It prunes by mapping instances of the prunedObjectClasses to a special cluster.  It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set). FLGlueMapper>>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil: FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters' FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this: FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class] FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set. For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel.
We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
That was an idea Yaron came up with.  That instead of using fuelIgnoredInstanceVariableNames one uses e.g. Object>>objectToSerialize    ^self and then if one wants to prune specific inst vars in MyClass one implements MyClass>>objectToSerialize    ^self shallowCopy prepareForSerialization
Hi Eliot,
I'm not convinced by the shallowCopy solution, except for the simple structures. If object graph is complex (have share nodes, loops, ...) then you gonna end up in a replication problem equivalent to the one Fuel is trying to solve.
The assumption is that the analyser would create a maximum of one proxy per object in the graph (default, no proxy) and that it would map objects with proxies to their proxies. Â So if proxies only nilled out inst vars I don't see a problem. Â What's attractive about this is that it provides a general solution to a couple of problems, a) how to replace a class of objects by some substitute (e.g. nil), b) how to prune state that needn't be saved. Â It is also conceptually simple; one just creates a proxy instance; no defining metadata, such as inst var names, and hence the code is always up-to-date (e.g. a class redefine won't automatically uncover renamed inst vars in serialization metadata).
Ah, OK, it occurs after the graph analysis, which I did not catch at first read. Now I understand better. Nicolas
Nicolas
MyClass>>prepareForSerialization    instVarIDontWantToSerialize := nil.    ^self and for objects one doesn't want to serlalize one implements MyNotToBeSerializedClass>>objectToSerialize    ^nil So its more general.  But I would pass the analyser in as an argument, which would allow things like MyPerhapsNotToBeSerializedClass>>objectToSerializeIn: anFLAnalyser    ^(anFLAnalyser shouldPrune: self)      ifFalse: [self]      ifTrue: [nil] which would of course be the default in Object: Object>>objectToSerializeIn: anFLAnalyser    ^(anFLAnalyser shouldPrune: self) ifFalse:: [self]
I think both Yaron and I feel the Fuel framework is comprehensible and flexible. Â We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Â Thank you very much.
:-) thank you!
Finally, a couple of things.  First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying to override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g. MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser] This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
To be honest I'm not sure. Â But passing in the analyser in things like fuelCluster or objectToSerialize is I think a good idea as it provides a convenient communication path which in turn provides considerable flexibility.
Lastly, the class-side cluster ids are a bit of a pain. Â It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? Â For example, are clusters ordered by id, or is this just an integer tag? Â Also, some class-side code to assign an unused id would be nice. You might think of virtualizing the id scheme. Â For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Â Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient.
Yes, but how inefficient? Â What's the size of all the cluster names? Â Â Â FLCluster allSubclasses inject: 0 into: [:t :c| t + c name size + 1] 670
So you'd add less than a kilobyte to the size of each serialization and get complete freedom from ids. Â Something to think about.
Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for example if your Glue cluster id is the same of the Moose cluster id.
I added an issue in our tracker.
If it makes sense, maybe the weak array you suggest can be also used to avoid instantiating lots of FLObjectCluster like we are doing in Object:
fuelCluster    ^ self class isVariable       ifTrue: [ FLVariableObjectCluster for: self class ]       ifFalse: [ FLFixedObjectCluster for: self class ]
the second time you send fuelCluster to an object, it can reuse the cluster instance.
Right. Â I think that's important, and is one reason why I think passing in the analyser is important, because it allows certain objects to discover existing clusters in the analyzer and join them if they want to, instead of having to invent and maintain their own cluster uniquing solution .
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere?
No, and its extremely proprietary :) Â Newspeak however is available and we may end up maintaining a port of Fuel for Newspeak. best regards, Eliot
regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin and Mariano, Â Â Â a couple of questions. Â What's the right way to exclude certain objects from the serialization? Â Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames    "Indicates which variables have to be ignored during serialization."
   ^#()
MyClass class >> fuelIgnoredInstanceVariableNames  ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
-- best, Eliot
I think the substitution should be done during the graph trace. Following with the example, if a proxy replaces an object, the proxy represents a subgraph that is appended and so it should be traced. For that we should keep track of the substitutions. I'm not sure how complex is that but is think it's not so difficult. Seems to be a great idea, we have to try it. I like that avoids writing inst var names as strings. I have no idea if with *slots* implemented then we will be able to return inst vars as first-class objects... but anyway this looks like the a nice solution. So, we have this as a pending issue as well as the id virtualization. Thanks for the ideas and the discussion! Martin On Fri, Jun 17, 2011 at 7:09 PM, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 2:39 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 1:26 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Eliot, I am very happy to read your mail.
On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin & Mariano, regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it
to
save and restore our data sets; thank you, its a cool framework. We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension. In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work. Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much). As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper: FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers' It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest: FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class] It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set). FLGlueMapper>>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil: FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters' FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this: FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class] FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set. For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel.
We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
That was an idea Yaron came up with. That instead of using fuelIgnoredInstanceVariableNames one uses e.g. Object>>objectToSerialize ^self and then if one wants to prune specific inst vars in MyClass one implements MyClass>>objectToSerialize ^self shallowCopy prepareForSerialization
Hi Eliot,
I'm not convinced by the shallowCopy solution, except for the simple structures. If object graph is complex (have share nodes, loops, ...) then you gonna end up in a replication problem equivalent to the one Fuel is trying to solve.
The assumption is that the analyser would create a maximum of one proxy per object in the graph (default, no proxy) and that it would map objects with proxies to their proxies. So if proxies only nilled out inst vars I don't see a problem. What's attractive about this is that it provides a general solution to a couple of problems, a) how to replace a class of objects by some substitute (e.g. nil), b) how to prune state that needn't be saved. It is also conceptually simple; one just creates a proxy instance; no defining metadata, such as inst var names, and hence the code is always up-to-date (e.g. a class redefine won't automatically uncover renamed inst vars in serialization metadata).
Ah, OK, it occurs after the graph analysis, which I did not catch at first read. Now I understand better.
Nicolas
Nicolas
MyClass>>prepareForSerialization instVarIDontWantToSerialize := nil. ^self and for objects one doesn't want to serlalize one implements MyNotToBeSerializedClass>>objectToSerialize ^nil So its more general. But I would pass the analyser in as an argument, which would allow things like MyPerhapsNotToBeSerializedClass>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse: [self] ifTrue: [nil] which would of course be the default in Object: Object>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse:: [self]
I think both Yaron and I feel the Fuel framework is comprehensible
and
flexible. We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
:-) thank you!
Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying
to
override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g. MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser] This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
To be honest I'm not sure. But passing in the analyser in things like fuelCluster or objectToSerialize is I think a good idea as it provides a convenient communication path which in turn provides considerable flexibility.
Lastly, the class-side cluster ids are a bit of a pain. It would be nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to
the
ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice. You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient.
Yes, but how inefficient? What's the size of all the cluster names? FLCluster allSubclasses inject: 0 into: [:t :c| t + c name size + 1] 670
So you'd add less than a kilobyte to the size of each serialization and get complete freedom from ids. Something to think about.
Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for example if your Glue cluster id is the same of the Moose cluster id.
I added an issue in our tracker.
If it makes sense, maybe the weak array you suggest can be also used
to
avoid instantiating lots of FLObjectCluster like we are doing in Object:
fuelCluster ^ self class isVariable ifTrue: [ FLVariableObjectCluster for: self class ] ifFalse: [ FLFixedObjectCluster for: self class ]
the second time you send fuelCluster to an object, it can reuse the cluster instance.
Right. I think that's important, and is one reason why I think passing in the analyser is important, because it allows certain objects to discover existing clusters in the analyzer and join them if they want to, instead of having to invent and maintain their own cluster uniquing solution .
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere?
No, and its extremely proprietary :) Newspeak however is available and we may end up maintaining a port of Fuel for Newspeak. best regards, Eliot
regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin and Mariano, a couple of questions. What's the right way to exclude
certain
objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
-- best, Eliot
I had some problems when I tried to write a package export tool with fuel. I wanted to store the classes without method extensions of other packages. Maybe with the objectToSerializeIn: idea I can write: Class>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldAvoidForeignProtocol: self) ifFalse: [self] ifTrue: [self copyWithoutForeignProtocol] Cheers, Martin On Mon, Jun 20, 2011 at 1:48 AM, Martin Dias <tinchodias@gmail.com> wrote:
I think the substitution should be done during the graph trace. Following with the example, if a proxy replaces an object, the proxy represents a subgraph that is appended and so it should be traced.
For that we should keep track of the substitutions. I'm not sure how complex is that but is think it's not so difficult.
Seems to be a great idea, we have to try it. I like that avoids writing inst var names as strings. I have no idea if with *slots* implemented then we will be able to return inst vars as first-class objects... but anyway this looks like the a nice solution.
So, we have this as a pending issue as well as the id virtualization. Thanks for the ideas and the discussion!
Martin
On Fri, Jun 17, 2011 at 7:09 PM, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 2:39 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
2011/6/17 Eliot Miranda <eliot.miranda@gmail.com>:
On Fri, Jun 17, 2011 at 1:26 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Eliot, I am very happy to read your mail.
On Wed, Jun 15, 2011 at 3:29 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin & Mariano, regarding filtering. Yesterday my colleague Yaron and I successfully finished our port of Fuel to Newspeak and are successfully using it
to
save and restore our data sets; thank you, its a cool framework. We had to implement two extensions, the first of which the ability to save and restore Newspeak classes, which is complex because these are instantiated classes inside instantiated Newspeak modules, not static Smalltalk classes in the Smalltalk dictionary. The second extension is the ability to map specific objects to nil, to prune objects on the way out. I want to discuss this latter extension. In our data set we have a set of references to objects that are logically not persistent and hence not to be saved. I'm sure that this will be a common case. The requirement is for the pickling system to prune certain objects, typically by arranging that when an object graph is pickled, references to the pruned objects are replaced by references to nil. One way of doing this is as described below, by specifiying per-class lists of instance variables whose referents shoudl not be saved. But this can be clumsy; there may be references to objects one wants to prune from e.g. more than one class, in which case one may have to provide multiple lists of the relevant inst vars; there may be references to objects one wants to prune from e.g. collections (e.g. sets and dictionaries) in which case the instance variable list approach just doesn't work. Here are two more general schemes. VFirst, most directly, Fuel could provide two filters, implemented in the default mapper, or the core analyser. One is a set of classes whose instances are not to be saved. Any reference to an instance of a class in the toBePrunedClasses set is saved as nil. The other is a set of instances that are not to be saved, and also any reference to an instance in the toBePruned set is saved as nil. Why have both? It can be convenient and efficient to filter by class (in our case we had many instances of a specific class, all of which should be filtered, and finding them could be time consuming), but filtering by class can be too inflexible, there may indeed be specific instances to exclude (thing for example of part of the object graph that functions as a cache; pruning the specific objects in the cache is the right thing to do; pruning all instances of classes whose instances exist in the cache may prune too much). As an example here's how we implemented pruning. Our system is called Glue, and we start with a mapper for Glue objects, FLGlueMapper: FLMapper subclass: #FLGlueMapper instanceVariableNames: 'prunedObjectClasses newspeakClassesCluster modelClasses' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Mappers' It accepts newspeak objects and filters instances in the prunedObjectsClasses set, and as a side-effect collects certain classes that we need in a manifest: FLGlueMapper>>accepts: anObject "Tells if the received object is handled by this analyzer. We want to hand-off instantiated Newspeak classes to the newspeakClassesCluster, and we want to record other model classes. We want to filter-out instances of any class in prunedObjectClasses." ^anObject isBehavior ifTrue: [(self isInstantiatedNewspeakClass: anObject) ifTrue: [true] ifFalse: [(anObject inheritsFrom: GlueDataObject) ifTrue: [modelClasses add: anObject]. false]] ifFalse: [prunedObjectClasses includes: anObject class] It prunes by mapping instances of the prunedObjectClasses to a special cluster. It can do this in visitObject: since any newspeak objects it is accepting will be visited in its visitClassOrTrait: method (i.e. it's implicit that all arguments to visitObjects: are instances of the prunedObjectsClasses set). FLGlueMapper>>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects FLPrunedObjectsCluster is a specialization of the nil,true,false cluster that maps its objects to nil: FLNilTrueFalseCluster subclass: #FLPrunedObjectsCluster instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Fuel-Core-Clusters' FLPrunedObjectsCluster >>serialize: aPrunedObject on: aWriteStream super serialize: nil on: aWriteStream
So this would generalize by the analyser having an e.g. FLPruningMapper as the first mapper, and this having a prunedObjects and a priunedObjectClasses set and going something like this: FLPruningMapper>>accepts: anObject ^(prunedObjects includes: anObject) or: [prunedObjectClasses includes: anObject class] FLPruningMapper >>visitObject: anObject analyzer mapAndTrace: anObject to: FLPrunedObjectsCluster instance into: analyzer clustersWithBaselevelObjects and then one would provide accessors in FLSerialzer and/or FLAnalyser to add objects and classes to the prunedObjects and prunedObjectClasses set. For efficiency one could arrange that the FLPruningMapper was not added to the sequence of mappers unless and until objects or classes were added to the prunedObjects and prunedObjectClasses set.
Excellent. I love the botanical metaphor of pruning! Of course we can include FLPruningMapper and FLPrunedObjectsCluster in Fuel.
We are also interested in pruning objects but not necessarily replacing them by nil, but for another user defined objects. For example proxies. We can extend the pruning stuff for doing that.
That was an idea Yaron came up with. That instead of using fuelIgnoredInstanceVariableNames one uses e.g. Object>>objectToSerialize ^self and then if one wants to prune specific inst vars in MyClass one implements MyClass>>objectToSerialize ^self shallowCopy prepareForSerialization
Hi Eliot,
I'm not convinced by the shallowCopy solution, except for the simple structures. If object graph is complex (have share nodes, loops, ...) then you gonna end up in a replication problem equivalent to the one Fuel is trying to solve.
The assumption is that the analyser would create a maximum of one proxy per object in the graph (default, no proxy) and that it would map objects with proxies to their proxies. So if proxies only nilled out inst vars I don't see a problem. What's attractive about this is that it provides a general solution to a couple of problems, a) how to replace a class of objects by some substitute (e.g. nil), b) how to prune state that needn't be saved. It is also conceptually simple; one just creates a proxy instance; no defining metadata, such as inst var names, and hence the code is always up-to-date (e.g. a class redefine won't automatically uncover renamed inst vars in serialization metadata).
Ah, OK, it occurs after the graph analysis, which I did not catch at first read. Now I understand better.
Nicolas
Nicolas
MyClass>>prepareForSerialization instVarIDontWantToSerialize := nil. ^self and for objects one doesn't want to serlalize one implements MyNotToBeSerializedClass>>objectToSerialize ^nil So its more general. But I would pass the analyser in as an
argument,
which would allow things like MyPerhapsNotToBeSerializedClass>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse: [self] ifTrue: [nil] which would of course be the default in Object: Object>>objectToSerializeIn: anFLAnalyser ^(anFLAnalyser shouldPrune: self) ifFalse:: [self]
I think both Yaron and I feel the Fuel framework is comprehensible
and
flexible. We enjoyed using it and while we took two passes at coming up with the pruning scheme we liked (our first was based on not serializing specific ins vars and was much more complex than our second, based on pruning instances of specific classes) we got there quickly and will very little frustration along the way. Thank you very much.
:-) thank you!
Finally, a couple of things. First, it may be more flexible to implement fuelCluster as fuelClusterIn: anFLAnalyser so that if one is trying
to
override certain parts of the mapping framework an implementation can access the analyser to find existing clusters, e.g. MyClass>>fuelClusterIn: anFLAnalyser ^self shouldBeInASpecialCluster ifTrue: [anFLAnalyser clusterWithId: MySpecialCluster id] ifFalse: [super fuelClusterIn: anFLAnalyser] This makes it easier to find a specific unique cluster to handle a group of objects specially.
I can't imagine a concrete example but I see that it is more flexible... the cluster obtained via double dispatch can be anything polymorphic with MySpecialCluster... that's the point?
To be honest I'm not sure. But passing in the analyser in things like fuelCluster or objectToSerialize is I think a good idea as it provides a convenient communication path which in turn provides considerable flexibility.
Lastly, the class-side cluster ids are a bit of a pain. It would
be
nice to know a) are these byte values or general integer values, i.e. can there be more than 256 types of cluster?, and b) is there any meaning to the ids? For example, are clusters ordered by id, or is this just an integer tag? Also, some class-side code to assign an unused id would be nice. You might think of virtualizing the id scheme. For example, if FLCluster maintained a weak array of all its subclasses then the id of a cluster could be the index in the array, and the array could be cleaned up occasionally. Then each fuel serialization could start with the list of cluster class names and ids, so that specific values of ids are specific to a particular serialization.
I do agree, these ids are an heritage from the first prototypes of fuel, they should be revised. a) yes, it is encoded in only one byte; b) just an integer tag, the only purpose of the id was for decoding fast: read a byte and then look in a dictionary for the corresponding cluster instance. We could even store the cluster class name but that's inefficient.
Yes, but how inefficient? What's the size of all the cluster names? FLCluster allSubclasses inject: 0 into: [:t :c| t + c name size + 1] 670
So you'd add less than a kilobyte to the size of each serialization and get complete freedom from ids. Something to think about.
Virtualizing the id scheme is a good idea. Much more elegant and extensible. The current mechanism not only limits the number of possible clusters, but also "user defined" extensions can collide, for
example
if your Glue cluster id is the same of the Moose cluster id.
I added an issue in our tracker.
If it makes sense, maybe the weak array you suggest can be also used to avoid instantiating lots of FLObjectCluster like we are doing in Object:
fuelCluster ^ self class isVariable ifTrue: [ FLVariableObjectCluster for: self class ] ifFalse: [ FLFixedObjectCluster for: self class ]
the second time you send fuelCluster to an object, it can reuse the cluster instance.
Right. I think that's important, and is one reason why I think passing in the analyser is important, because it allows certain objects to discover existing clusters in the analyzer and join them if they want to, instead of having to invent and maintain their own cluster uniquing solution .
again thanks for a great framework.
Thanks for your words and the feedback. Is Glue published somewhere?
No, and its extremely proprietary :) Newspeak however is available and we may end up maintaining a port of Fuel for Newspeak. best regards, Eliot
regards Martin
best, Eliot
On Mon, Jun 13, 2011 at 10:16 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Martin and Mariano, a couple of questions. What's the right way to exclude
certain
objects from the serialization? Is there a way of excluding certain inst vars from certain objects?
Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.
Behavior >> fuelIgnoredInstanceVariableNames "Indicates which variables have to be ignored during serialization."
^#()
MyClass class >> fuelIgnoredInstanceVariableNames ^ #('instVar1')
The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ? We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...
thanks
-- Mariano http://marianopeck.wordpress.com
-- best, Eliot
On 08.12.2010 17:50, Martin Dias wrote:
Hi all
Last months I and Tristan have been working on Fuel project, an object binary serialization tool. The idea is that objects are much more times loaded than stored, therefore it is worth to spend time while storing in order to have faster loading and user experience. We present an implementation of a pickle format that is based on clustering similar objects.
There is a summary of the project below, but more complete information is available here: http://rmod.lille.inria.fr/web/pier/software/Fuel
The implementation still needs a lot of work to be really useful, optimizations should be done, but we'll be glad to get feedback of the community.
= Pickle format =
The pickle format and the serialization algorithm main idea, is explained in this slides:
http://www.slideshare.net/tinchodias/fuel-serialization-in-an-example
= Current features =
- Class shape changing (when a variable has been added, or removed, or its index changed) - Serialize most of the basic objects. - Serialize (almost) any CompiledMethod - Detection of global or class variables - Support for cyclic object graphs - Tests
= Next steps =
- Improve version checking. - Optimize performance. - Serialize more kinds of objects: -- Class with its complete description. -- Method contexts -- Active block closures -- Continuation - Some improvements for the user: -- pre and post actions to be executed. -- easily say 'this object is singleton'. - Partial loading of a stored graph. - Fast statistics/brief info extraction of a stored graph. - ConfigurationOfFuel. - Be able to deploy materialization behavior only (independent from the serialization behavior)
= Download =
In a Pharo 1.1 or 1.1.1 evaluate:
Gofer new squeaksource: 'Fuel'; version: 'Fuel-MartinDias.74'; version: 'FuelBenchmarks-MartinDias.4'; load.
= Benchmarks =
You can run benchmarks executing this line (results in Transcript):
FLBenchmarks newBasic run.
Thank you! Martin Dias
One thing I do not see mentioned, and feel could use some attention, is thread safety. (aka other threads altering the graph you are serializing) The classic answer would of course be "always run FUEL at highest priority", but if we ever want to move to true multi-core, that's not enough. What would be neat is protecting all mutation of objects in the graph with a Mutex/Monitor whose critical section covers the analysis and serialization, i.e. blocking all other processes that wants to mutate objects in the graph untill serialization is complete. Aside from the behaviour when a marked object is encountered, the process is the same as for immutability, as discussed here: http://forum.world.st/immutability-and-become-Was-Re-squeak-dev-immutability... You could do this image-side as part of the analyze phase, as Eliot's post suggests, but it's not entirely safe when: A child -> B B parent -> A 1 . Process1 protects A with Mutex 2. Process2 calls method on B which does: a) B parent: C b) (B's tmpRef to A) child: somethingElse *Wait for Mutex* 3. Process 1 makes B, C immutable, serializes B with C as parent. 4. Process2 changes A child. It could of course be argued that is a programming/logic error to update B parent ref before child ref in A, (you'd actually have to do extra work to get that order), still it's a hard one to debug if it does happen. (Note to the thread: Following the same logic, I would also say it should be considered an error to keep a mutable cache as part of an immutable object :) ) In fact, you *could* implement it with immutability, injecting handler contexts with behaviour as described above into existing/new processes for any resulting immutability errors. Cheers, Henry
One thing I do not see mentioned, and feel could use some attention, is thread safety. (aka other threads altering the graph you are serializing)
The classic answer would of course be "always run FUEL at highest priority", but if we ever want to move to true multi-core, that's not enough.
Wow. Were you reading our minds?? hehehe. Yes, reading some of the related work papers while writing our own paper, we read about the "atomicy". How to be atomic while analyzing and serializing? Right now, if the graph changes between the first phase (analysis) and the second phase (serializating) you are screw :) I wonder if Eliot fixed that in Parcels? What would be neat is protecting all mutation of objects in the graph with a
Mutex/Monitor whose critical section covers the analysis and serialization, i.e. blocking all other processes that wants to mutate objects in the graph untill serialization is complete.
yes,
Aside from the behaviour when a marked object is encountered, the process is the same as for immutability, as discussed here: http://forum.world.st/**immutability-and-become-Was-** Re-squeak-dev-immutability-**td1597511.html<http://forum.world.st/immutability-and-become-Was-Re-squeak-dev-immutability...>
You could do this image-side as part of the analyze phase, as Eliot's post suggests,
Ok, but in order to do I need the immutable bit in the ObjectHeader, right?
but it's not entirely safe when: A child -> B B parent -> A
1 . Process1 protects A with Mutex 2. Process2 calls method on B which does: a) B parent: C b) (B's tmpRef to A) child: somethingElse *Wait for Mutex* 3. Process 1 makes B, C immutable, serializes B with C as parent. 4. Process2 changes A child.
It could of course be argued that is a programming/logic error to update B parent ref before child ref in A, (you'd actually have to do extra work to get that order), still it's a hard one to debug if it does happen. (Note to the thread: Following the same logic, I would also say it should be considered an error to keep a mutable cache as part of an immutable object :) )
In fact, you *could* implement it with immutability, injecting handler contexts with behaviour as described above into existing/new processes for any resulting immutability errors.
wow. Seems a really complicated stuff to deal with :( Immutability is not yet present in default VMs -- Mariano http://marianopeck.wordpress.com
On 20. juni 2011, at 15:59, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
One thing I do not see mentioned, and feel could use some attention, is thread safety. (aka other threads altering the graph you are serializing) The classic answer would of course be "always run FUEL at highest priority", but if we ever want to move to true multi-core, that's not enough.
Wow. Were you reading our minds?? hehehe. Yes, reading some of the related work papers while writing our own paper, we read about the "atomicy". How to be atomic while analyzing and serializing? Right now, if the graph changes between the first phase (analysis) and the second phase (serializating) you are screw :)
I wonder if Eliot fixed that in Parcels?
The classical way? ;) What I describe below is one way of achieving atomicity (in the strict sense if using uninterruptible vm functionality)
What would be neat is protecting all mutation of objects in the graph with a Mutex/Monitor whose critical section covers the analysis and serialization, i.e. blocking all other processes that wants to mutate objects in the graph untill serialization is complete.
yes,
Aside from the behaviour when a marked object is encountered, the process is the same as for immutability, as discussed here: http://forum.world.st/immutability-and-become-Was-Re-squeak-dev-immutability...
You could do this image-side as part of the analyze phase, as Eliot's post suggests,
Ok, but in order to do I need the immutable bit in the ObjectHeader, right?
Immutability as discussed there? Yes. Providing serialization thread-saferness? No. You could also use method wrappers (of all methods mutating state), like other potential users of immutability do currently. I'd imagine immutability would be a much lower overhead in the common case of no other threads doing access.
but it's not entirely safe when: A child -> B B parent -> A
1 . Process1 protects A with Mutex 2. Process2 calls method on B which does: a) B parent: C b) (B's tmpRef to A) child: somethingElse *Wait for Mutex* 3. Process 1 makes B, C immutable, serializes B with C as parent. 4. Process2 changes A child.
It could of course be argued that is a programming/logic error to update B parent ref before child ref in A, (you'd actually have to do extra work to get that order), still it's a hard one to debug if it does happen. (Note to the thread: Following the same logic, I would also say it should be considered an error to keep a mutable cache as part of an immutable object :) )
In fact, you *could* implement it with immutability, injecting handler contexts with behaviour as described above into existing/new processes for any resulting immutability errors.
wow. Seems a really complicated stuff to deal with :(
Not really, 2 threads performing 2 sequences of operation in a specific order is really the simplest kind of treading problem one can imagine.
I guess it comes down to the presenter :/
Immutability is not yet present in default VMs True. It has been done though, and iirc came down to newspeak license (resolved?) and noone to review and actually integrate. So for a comparative test of approaches it might not be too much work.
Cheers, Henry
participants (15)
-
Adrian Lienhard -
Alexandre Bergel -
Eliot Miranda -
Henrik Johansen -
Henrik Sperre Johansen
-
Igor Stasenko -
Levente Uzonyi -
Lukas Renggli -
Mariano Martinez Peck -
Martin Dias -
Nicolas Cellier -
Peter van Rooijen -
Sean Malloy -
Stéphane Ducasse -
Tudor Girba