[Pharo-project] STON Update
Hi, Based on feedback by Norbert Hartl, Stuart Herring and Dale Hendrichs (thank you) I improved the current STON version a bit. http://ss3.gemstone.com/ss/STON https://github.com/svenvc/ston Changelog: A nasty cycle bug was fixed Performance was improved, especially for large structures. Some implementation changes (recursion done using a stack, stonProcessSubObjects: optimized using stonContainsSubObjects) I did some stress/performance testing/tuning. Using STONTestMap classTreeExtended. to generate a structure mimicing the class hierarchy, with objects for all classes, about 7000 in my image and all methods, about 64000 in my image, with super/subclass and method owner links to create cycles. This results in a 7 Mb .ston file. The normal code, serializes this in 28 seconds and materializes it in 11.5 seconds. The 'fast' versions (using ZnBuffered[Write|Read]Streams and Fuel Large Datastructures) does it in 1.1 and 3.8 seconds respectively. New users are always welcome. Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 30 Nov 2012, at 23:39, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
Based on feedback by Norbert Hartl, Stuart Herring and Dale Hendrichs (thank you) I improved the current STON version a bit.
http://ss3.gemstone.com/ss/STON https://github.com/svenvc/ston
Changelog:
A nasty cycle bug was fixed Performance was improved, especially for large structures. Some implementation changes (recursion done using a stack, stonProcessSubObjects: optimized using stonContainsSubObjects)
I did some stress/performance testing/tuning. Using
STONTestMap classTreeExtended.
to generate a structure mimicing the class hierarchy, with objects for all classes, about 7000 in my image and all methods, about 64000 in my image, with super/subclass and method owner links to create cycles. This results in a 7 Mb .ston file.
The normal code, serializes this in 28 seconds and materializes it in 11.5 seconds. The 'fast' versions (using ZnBuffered[Write|Read]Streams and Fuel Large Datastructures) does it in 1.1 and 3.8 seconds respectively.
It occurred to me that I hadn't tried with Fuel (expecting it to be totally out of reach, since it is a binary protocol). Well, Fuel serializes my test structure in 5 seconds and materializes it in 1.2 seconds, the file being 6.4 Mb. Not bad for a textual protocol ;-) This does of course not proof a lot, results will depend on each specific use case.
New users are always welcome.
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On Sat, Dec 1, 2012 at 6:03 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 30 Nov 2012, at 23:39, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi,
Based on feedback by Norbert Hartl, Stuart Herring and Dale Hendrichs (thank you) I improved the current STON version a bit.
http://ss3.gemstone.com/ss/STON https://github.com/svenvc/ston
Changelog:
A nasty cycle bug was fixed Performance was improved, especially for large structures. Some implementation changes (recursion done using a stack, stonProcessSubObjects: optimized using stonContainsSubObjects)
I did some stress/performance testing/tuning. Using
STONTestMap classTreeExtended.
to generate a structure mimicing the class hierarchy, with objects for all classes, about 7000 in my image and all methods, about 64000 in my image, with super/subclass and method owner links to create cycles. This results in a 7 Mb .ston file.
The normal code, serializes this in 28 seconds and materializes it in 11.5 seconds. The 'fast' versions (using ZnBuffered[Write|Read]Streams and Fuel Large Datastructures) does it in 1.1 and 3.8 seconds respectively.
It occurred to me that I hadn't tried with Fuel (expecting it to be totally out of reach, since it is a binary protocol). Well, Fuel serializes my test structure in 5 seconds and materializes it in 1.2 seconds, the file being 6.4 Mb. Not bad for a textual protocol ;-)
So 5.6x for serializing and 9.5x for materializing. Similar stream result. So, yes, not bad :) SIXX was way slower from what I remember. Don't have time now but I will check your STONTestMap classTreeExtended ... BTW, which stream class you use for the bench?
This does of course not proof a lot, results will depend on each specific use case.
New users are always welcome.
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- Mariano http://marianopeck.wordpress.com
On 01 Dec 2012, at 19:04, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
BTW, which stream class you use for the bench?
Well, the 'fast' variants do two things: - use the Fuel datastructures - use ZnBuffered[Read|Write]Streams The underlying stream is a MulitByteFileStream. The buffering makes a huge difference, in both directions. I plan to write a separate mail message about that ;-)
On Fri, Nov 30, 2012 at 11:39 PM, Sven Van Caekenberghe <sven@stfx.eu>wrote:
Hi,
Based on feedback by Norbert Hartl, Stuart Herring and Dale Hendrichs (thank you) I improved the current STON version a bit.
http://ss3.gemstone.com/ss/STON https://github.com/svenvc/ston
Changelog:
A nasty cycle bug was fixed Performance was improved, especially for large structures. Some implementation changes (recursion done using a stack, stonProcessSubObjects: optimized using stonContainsSubObjects)
I did some stress/performance testing/tuning. Using
STONTestMap classTreeExtended.
to generate a structure mimicing the class hierarchy, with objects for all classes, about 7000 in my image and all methods, about 64000 in my image, with super/subclass and method owner links to create cycles. This results in a 7 Mb .ston file.
The normal code, serializes this in 28 seconds and materializes it in 11.5 seconds. The 'fast' versions (using ZnBuffered[Write|Read]Streams and Fuel Large Datastructures) does it in 1.1 and 3.8 seconds respectively.
You mean FLLargeHashedCollection and subclasses?
New users are always welcome.
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- Mariano http://marianopeck.wordpress.com
On 01 Dec 2012, at 18:58, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
You mean FLLargeHashedCollection and subclasses?
Yes: optimizeForLargeStructures self class environment at: #FLLargeIdentityDictionary ifPresent: [ :identityDictionaryClass | objects := identityDictionaryClass new ]. self class environment at: #FLSimpleStackstack ifPresent: [ :stackClass | stack := stackClass new ]
On Sat, Dec 1, 2012 at 7:02 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 01 Dec 2012, at 18:58, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
You mean FLLargeHashedCollection and subclasses?
Yes:
optimizeForLargeStructures self class environment at: #FLLargeIdentityDictionary ifPresent: [ :identityDictionaryClass | objects := identityDictionaryClass new ]. self class environment at: #FLSimpleStackstack ifPresent: [ :stackClass | stack := stackClass new ]
Excellent :) -- Mariano http://marianopeck.wordpress.com
On Fri, Nov 30, 2012 at 11:39 PM, Sven Van Caekenberghe <sven@stfx.eu>wrote:
Hi,
Based on feedback by Norbert Hartl, Stuart Herring and Dale Hendrichs (thank you) I improved the current STON version a bit.
http://ss3.gemstone.com/ss/STON https://github.com/svenvc/ston
Changelog:
A nasty cycle bug was fixed Performance was improved, especially for large structures. Some implementation changes (recursion done using a stack, stonProcessSubObjects: optimized using stonContainsSubObjects)
Yes, using a stack is also a good idea :) Good to see you are tacking some of the Fuel code/ideas to improve STON. It is awesome to have both, a fast binary and text serializers :)
I did some stress/performance testing/tuning. Using
STONTestMap classTreeExtended.
to generate a structure mimicing the class hierarchy, with objects for all classes, about 7000 in my image and all methods, about 64000 in my image, with super/subclass and method owner links to create cycles. This results in a 7 Mb .ston file.
The normal code, serializes this in 28 seconds and materializes it in 11.5 seconds. The 'fast' versions (using ZnBuffered[Write|Read]Streams and Fuel Large Datastructures) does it in 1.1 and 3.8 seconds respectively.
New users are always welcome.
Sven
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
-- Mariano http://marianopeck.wordpress.com
participants (2)
-
Mariano Martinez Peck -
Sven Van Caekenberghe