Storing test data in methods
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. Any ideas? Norbert
Have a look at InstanceEncoder http://www.smalltalkhub.com/#!/~hernan/InstanceEncoder Hernán El 23/07/2013 7:47, Norbert Hartl escribió:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
Am 23.07.2013 um 13:20 schrieb Hernán Morales Durand <hernan.morales@gmail.com>:
Have a look at InstanceEncoder
Thanks, how would you say it works compared to store an Fuel ByteArray? Norbert
El 23/07/2013 7:47, Norbert Hartl escribió:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
El 23/07/2013 9:01, Norbert Hartl escribió:
Am 23.07.2013 um 13:20 schrieb Hernán Morales Durand <hernan.morales@gmail.com>:
Have a look at InstanceEncoder
Thanks, how would you say it works compared to store an Fuel ByteArray?
This is not difficult to test. First save your image and download this file, is the chromosome 22 of a human reference genome (about 10mb of gzip file): | client ttr object fileName fStream | fileName := 'hs_alt_HuRef_chr22.fa.gz'. [ client := (FTPClient openOnHostNamed: 'ftp.ncbi.nlm.nih.gov') loginUser: 'anonymous' password: ''; binary; changeDirectoryTo: 'genomes/H_sapiens/CHR_22'. (FileStream newFileNamed: fileName) binary; nextPutAll: (client getFileNamed: fileName); close ] on: NetworkError, LoginFailedException do: [ : ex | self error: 'Connection failed' ]. Now storing with InstanceEncoder: fileName := 'hs_alt_HuRef_chr22.fa.gz'. ttr := [ Object store: fileName asFileReference contents asSelector: #storedObjectInstanceEncoder ] timeToRun. (Duration milliSeconds: ttr) asSeconds. And with Fuel Smalltalk garbageCollect. fileName := 'hs_alt_HuRef_chr22.fa.gz'. ttr := [ Object class compile: 'storedObjectFuel ^ ', (ByteArray streamContents: [ : stream | FLSerializer serialize: fileName asFileReference contents on: stream ]) storeString ] timeToRun. (Duration milliSeconds: ttr) asSeconds. Results for writing =================== InstanceEncoder avg: 8 sec. Fuel avg: 45 sec. And reading scripts: [ Object storedObjectInstanceEncoder ] timeToRun. [ FLMaterializer materializeFromByteArray: Object storedObjectFuel ] timeToRun Results for reading =================== InstanceEncoder avg: 3955 msec Fuel avg: 70 msec And the final cleaning of course :) Object class removeSelector: #storedObjectFuel. Object class removeSelector: #storedObjectInstanceEncoder. Cheers, Hernán
Norbert
El 23/07/2013 7:47, Norbert Hartl escribió:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
.
Hernán, Am 23.07.2013 um 16:13 schrieb Hernán Morales Durand <hernan.morales@gmail.com>:
El 23/07/2013 9:01, Norbert Hartl escribió:
Am 23.07.2013 um 13:20 schrieb Hernán Morales Durand <hernan.morales@gmail.com>:
Have a look at InstanceEncoder
Thanks, how would you say it works compared to store an Fuel ByteArray?
This is not difficult to test. First save your image and download this file, is the chromosome 22 of a human reference genome (about 10mb of gzip file):
| client ttr object fileName fStream | fileName := 'hs_alt_HuRef_chr22.fa.gz'. [ client := (FTPClient openOnHostNamed: 'ftp.ncbi.nlm.nih.gov') loginUser: 'anonymous' password: ''; binary; changeDirectoryTo: 'genomes/H_sapiens/CHR_22'. (FileStream newFileNamed: fileName) binary; nextPutAll: (client getFileNamed: fileName); close ] on: NetworkError, LoginFailedException do: [ : ex | self error: 'Connection failed' ].
Now storing with InstanceEncoder:
fileName := 'hs_alt_HuRef_chr22.fa.gz'. ttr := [ Object store: fileName asFileReference contents asSelector: #storedObjectInstanceEncoder ] timeToRun. (Duration milliSeconds: ttr) asSeconds.
And with Fuel
Smalltalk garbageCollect. fileName := 'hs_alt_HuRef_chr22.fa.gz'. ttr := [ Object class compile: 'storedObjectFuel ^ ', (ByteArray streamContents: [ : stream | FLSerializer serialize: fileName asFileReference contents on: stream ]) storeString ] timeToRun. (Duration milliSeconds: ttr) asSeconds.
Results for writing =================== InstanceEncoder avg: 8 sec. Fuel avg: 45 sec.
And reading scripts:
[ Object storedObjectInstanceEncoder ] timeToRun. [ FLMaterializer materializeFromByteArray: Object storedObjectFuel ] timeToRun
Results for reading =================== InstanceEncoder avg: 3955 msec Fuel avg: 70 msec
And the final cleaning of course :)
Object class removeSelector: #storedObjectFuel. Object class removeSelector: #storedObjectInstanceEncoder.
Thanks for the evaluation. I was more thinking about _how_ it works. But I take "similar" as an answer for now. :) Norbert
Norbert
El 23/07/2013 7:47, Norbert Hartl escribió:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
.
Norbert, for similar purposes (INI-File contents etc.), in order to keep them under version control in envy, I wrote methods that produced a String that could recreate the objects when evaluated. Such a piece of text makes for a great class or instance method that you can compile into some Smalltalk class and get it under version control together with the code that uses the objects. Example: IniFileEntry with iVars #key and #value instance method: creationStringOn: aStream aStream nextPutAll: self class name; nextPut: $=; nextPutAll: self value asString. Thus you can ask all your IniFileEntries to write their own creation string onto a Stream and compile that into a method. You can see me talking about this and other related stuff here: http://www.objektfabrik.de/Downloads/VASTForum2010/VFM10-04-GetBackToShapre-... (chapter "Get Everything under Version control, ca. 1/3 through the recording) HTH, Joachim m 23.07.13 12:47, schrieb Norbert Hartl:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
HI again, I just saw the link I posted doesn't work right now. The objektfabrik site moved to a new server a while agow and it seems a few files didn't make it. The upload is currently in process, so please check back in an hour or so. Joachim Am 23.07.13 13:25, schrieb jtuchel@objektfabrik.de:
Norbert,
for similar purposes (INI-File contents etc.), in order to keep them under version control in envy, I wrote methods that produced a String that could recreate the objects when evaluated. Such a piece of text makes for a great class or instance method that you can compile into some Smalltalk class and get it under version control together with the code that uses the objects.
Example: IniFileEntry with iVars #key and #value instance method:
creationStringOn: aStream aStream nextPutAll: self class name; nextPut: $=; nextPutAll: self value asString.
Thus you can ask all your IniFileEntries to write their own creation string onto a Stream and compile that into a method. You can see me talking about this and other related stuff here: http://www.objektfabrik.de/Downloads/VASTForum2010/VFM10-04-GetBackToShapre-... (chapter "Get Everything under Version control, ca. 1/3 through the recording)
HTH,
Joachim
m 23.07.13 12:47, schrieb Norbert Hartl:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
Joachim, Am 23.07.2013 um 13:25 schrieb "jtuchel@objektfabrik.de" <jtuchel@objektfabrik.de>:
Norbert,
for similar purposes (INI-File contents etc.), in order to keep them under version control in envy, I wrote methods that produced a String that could recreate the objects when evaluated. Such a piece of text makes for a great class or instance method that you can compile into some Smalltalk class and get it under version control together with the code that uses the objects.
Example: IniFileEntry with iVars #key and #value instance method:
creationStringOn: aStream aStream nextPutAll: self class name; nextPut: $=; nextPutAll: self value asString.
Thus you can ask all your IniFileEntries to write their own creation string onto a Stream and compile that into a method. You can see me talking about this and other related stuff here: http://www.objektfabrik.de/Downloads/VASTForum2010/VFM10-04-GetBackToShapre-... (chapter "Get Everything under Version control, ca. 1/3 through the recording)
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :) I think I have to options: 1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method Maybe I should try both. thanks, Norbert
m 23.07.13 12:47, schrieb Norbert Hartl:
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it.
Any ideas?
Norbert
Norbert, Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Joachim
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" <jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, the first test froze (hot freeze taking 100% CPU) my image. The model serialized to Fuel is > 280Kb and clicking anywhere in the image that might show the code of the compiled method just makes the image unresponsive. :) I'll report further findings, Norbert
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" <jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, I can't look at the method but I don't find ByteArrays not that interesting anyway. So the actual size of the serialized byte array is (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size is 237445 Bytes. Serializing [ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun took 647 ms but [FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text butâ¦. I think I stay with it for a while. Norbert
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster. Cheers, On Tue, Jul 23, 2013 at 9:40 AM, Norbert Hartl <norbert@hartl.name> wrote:
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" < jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, I can't look at the method but I don't find ByteArrays not that interesting anyway. So the actual size of the serialized byte array is
(ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size
is 237445 Bytes. Serializing
[ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun
took 647 ms
but
[FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun
took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text butâ¦.
I think I stay with it for a while.
Norbert
-- Mariano http://marianopeck.wordpress.com
Mariano, Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck <marianopeck@gmail.com>:
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster.
thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb. Thanks, I'll see but I am already satisfied with the current approach. Norbert
Cheers,
On Tue, Jul 23, 2013 at 9:40 AM, Norbert Hartl <norbert@hartl.name> wrote:
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" <jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, I can't look at the method but I don't find ByteArrays not that interesting anyway. So the actual size of the serialized byte array is
(ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size
is 237445 Bytes. Serializing
[ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun
took 647 ms
but
[FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun
took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text butâ¦.
I think I stay with it for a while.
Norbert
-- Mariano http://marianopeck.wordpress.com
On Tue, Jul 23, 2013 at 9:48 AM, Norbert Hartl <norbert@hartl.name> wrote:
Mariano,
Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck < marianopeck@gmail.com>:
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster.
thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb.
Thanks, I'll see but I am already satisfied with the current approach.
Mmmm I thought it was ready, sorry my bad. Seems it was not ready nor integrated: https://code.google.com/p/fuel/issues/detail?id=160 But I might be wrong. Max? Cheers,
Norbert
Cheers,
On Tue, Jul 23, 2013 at 9:40 AM, Norbert Hartl <norbert@hartl.name> wrote:
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" < jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, I can't look at the method but I don't find ByteArrays not that interesting anyway. So the actual size of the serialized byte array is
(ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size
is 237445 Bytes. Serializing
[ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun
took 647 ms
but
[FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun
took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text butâ¦.
I think I stay with it for a while.
Norbert
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 23.07.2013, at 15:32, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Jul 23, 2013 at 9:48 AM, Norbert Hartl <norbert@hartl.name> wrote: Mariano,
Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck <marianopeck@gmail.com>:
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster.
thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb.
Thanks, I'll see but I am already satisfied with the current approach.
Mmmm I thought it was ready, sorry my bad. Seems it was not ready nor integrated: https://code.google.com/p/fuel/issues/detail?id=160 But I might be wrong. Max?
No, that issue is still open. You're welcome to use the experimental version though (Fuel-MaxLeske.757.mcz in http://www.squeaksource.com/FuelExperiments) but keep in mind that this is a few months behind stable. The other idea however, that Mariano describes in the linked issue, is available but probably not desirable: simply pass a compressing stream to Fuel (see FLGZippedBasicSerializationTest for an example on how to do that). That will compress *all* data, not only strings (which is the idea outlined in the linked issue). Cheers, Max (Note: I don't read the users list so please put me on the CC explicitly if you want me to read on)
Cheers,
Norbert
Cheers,
On Tue, Jul 23, 2013 at 9:40 AM, Norbert Hartl <norbert@hartl.name> wrote:
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" <jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, I can't look at the method but I don't find ByteArrays not that interesting anyway. So the actual size of the serialized byte array is
(ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size
is 237445 Bytes. Serializing
[ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun
took 647 ms
but
[FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun
took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text butâ¦.
I think I stay with it for a while.
Norbert
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Max, Am 23.07.2013 um 17:27 schrieb Max Leske <maxleske@gmail.com>:
On 23.07.2013, at 15:32, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Jul 23, 2013 at 9:48 AM, Norbert Hartl <norbert@hartl.name> wrote: Mariano,
Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck <marianopeck@gmail.com>:
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster.
thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb.
Thanks, I'll see but I am already satisfied with the current approach.
Mmmm I thought it was ready, sorry my bad. Seems it was not ready nor integrated: https://code.google.com/p/fuel/issues/detail?id=160 But I might be wrong. Max?
No, that issue is still open. You're welcome to use the experimental version though (Fuel-MaxLeske.757.mcz in http://www.squeaksource.com/FuelExperiments) but keep in mind that this is a few months behind stable.
I'm not sure the priority is high enough to try.
The other idea however, that Mariano describes in the linked issue, is available but probably not desirable: simply pass a compressing stream to Fuel (see FLGZippedBasicSerializationTest for an example on how to do that). That will compress *all* data, not only strings (which is the idea outlined in the linked issue).
I don't think that will gain anything. A monticello package is written as zip. So I wouldn't expect much from an additional compression. thanks, Norbert
Cheers, Max
(Note: I don't read the users list so please put me on the CC explicitly if you want me to read on)
Cheers,
Norbert
Cheers,
On Tue, Jul 23, 2013 at 9:40 AM, Norbert Hartl <norbert@hartl.name> wrote:
Am 23.07.2013 um 14:23 schrieb "jtuchel@objektfabrik.de" <jtuchel@objektfabrik.de>:
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl:
Joachim,
I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :)
We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless.
I think I have to options:
1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method
Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such.
Well, I can't look at the method but I don't find ByteArrays not that interesting anyway. So the actual size of the serialized byte array is
(ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size
is 237445 Bytes. Serializing
[ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun
took 647 ms
but
[FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun
took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text butâ¦.
I think I stay with it for a while.
Norbert
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On Jul 23, 2013, at 5:33 , Norbert Hartl <norbert@hartl.name> wrote:
Max,
Am 23.07.2013 um 17:27 schrieb Max Leske <maxleske@gmail.com>:
On 23.07.2013, at 15:32, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Jul 23, 2013 at 9:48 AM, Norbert Hartl <norbert@hartl.name> wrote: Mariano,
Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck <marianopeck@gmail.com>:
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster.
thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb.
Thanks, I'll see but I am already satisfied with the current approach.
Mmmm I thought it was ready, sorry my bad. Seems it was not ready nor integrated: https://code.google.com/p/fuel/issues/detail?id=160 But I might be wrong. Max?
No, that issue is still open. You're welcome to use the experimental version though (Fuel-MaxLeske.757.mcz in http://www.squeaksource.com/FuelExperiments) but keep in mind that this is a few months behind stable.
I'm not sure the priority is high enough to try.
The other idea however, that Mariano describes in the linked issue, is available but probably not desirable: simply pass a compressing stream to Fuel (see FLGZippedBasicSerializationTest for an example on how to do that). That will compress *all* data, not only strings (which is the idea outlined in the linked issue).
I don't think that will gain anything. A monticello package is written as zip. So I wouldn't expect much from an additional compression.
thanks,
Norbert Unless MAPExampleModels tcap is something like ^'mySerializedTCAP.bin' asFile readStream binary contents instead of tcap ^#( 0 0 0 ⦠), compressing the fuel binary output would save you some runtime memory (method literal size) / .changes space (method source), though for test resources I guess that's a less relevant aspect than for other resources.
Cheers, Henry
Am 01.08.2013 um 15:33 schrieb Henrik Johansen <henrik.s.johansen@veloxit.no>:
On Jul 23, 2013, at 5:33 , Norbert Hartl <norbert@hartl.name> wrote:
Max,
Am 23.07.2013 um 17:27 schrieb Max Leske <maxleske@gmail.com>:
On 23.07.2013, at 15:32, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Tue, Jul 23, 2013 at 9:48 AM, Norbert Hartl <norbert@hartl.name> wrote:
Mariano,
Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck <marianopeck@gmail.com>:
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster. thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb.
Thanks, I'll see but I am already satisfied with the current approach.
Mmmm I thought it was ready, sorry my bad. Seems it was not ready nor integrated: https://code.google.com/p/fuel/issues/detail?id=160 But I might be wrong. Max?
No, that issue is still open. You're welcome to use the experimental version though (Fuel-MaxLeske.757.mcz in http://www.squeaksource.com/FuelExperiments) but keep in mind that this is a few months behind stable.
I'm not sure the priority is high enough to try.
The other idea however, that Mariano describes in the linked issue, is available but probably not desirable: simply pass a compressing stream to Fuel (see FLGZippedBasicSerializationTest for an example on how to do that). That will compress *all* data, not only strings (which is the idea outlined in the linked issue). I don't think that will gain anything. A monticello package is written as zip. So I wouldn't expect much from an additional compression.
thanks,
Norbert Unless MAPExampleModels tcap is something like ^'mySerializedTCAP.bin' asFile readStream binary contents instead of tcap ^#( 0 0 0 ⦠), compressing the fuel binary output would save you some runtime memory (method literal size) / .changes space (method source), though for test resources I guess that's a less relevant aspect than for other resources.
Right! They difference is in the image. I'll check that. It might come in handy when I have more of these methods. But then I need to try because every method is used multiple times and having it compressed in the image will add some time to every test run. And then it is measuring penalties (again) thanks, Norbert
participants (6)
-
Henrik Johansen -
Hernán Morales Durand -
jtuchel＠objektfabrik.de -
Mariano Martinez Peck -
Max Leske -
Norbert Hartl