Hi all, I'm just wondering. Would it work to have a package format based on Fuel? Would that make loading faster? Does it already exist? Thanks, Thierry
Thierry Goubier wrote:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
What is your use case for needing it faster? * Just every day developer use? * CI automation? * Something in production? * Other? cheers -ben
2014-12-04 23:43 GMT+01:00 Ben Coman <btc@openinworld.com>:
Thierry Goubier wrote:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
What is your use case for needing it faster? * Just every day developer use? * CI automation? * Something in production? * Other? cheers -ben
The three first ones. Anything which improves loading speed / reduces memory footprint, really.
Maybe the point to target is the version metadata. Thierry
Hi Thierry, On Thu, Dec 4, 2014 at 2:31 PM, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
It doesn't have to be based on Fuel. Monticello .mcz's are just zips that contain any number of files: Archive: trunkpackages/Kernel.spur-eem.866.mcz Length Date Time Name -------- ---- ---- ---- 15 08-07-14 09:12 package 294780 08-07-14 09:12 version 1370995 08-07-14 09:12 snapshot/source.st 1442870 08-07-14 09:12 snapshot.bin -------- ------- 3108660 4 files Right now package and version are juts textual, and you've been complaining about the parsing speed. It would be easy to extend the package to include e.g. version.fuel. It is also a huge advantage to keep both the .bin (a not-as-fast binary loading format similar in use to Fuel but more primitive) along side the source code, because the source can be extracted without a running system, and can be compiled if, for some reason, binary loading fails. So instead of changing the format you could simply augment it with a snapshot.fuel, modifying the writer to include it, and modify the reader to use the .fuel if it exists. Would that make loading faster?
Yes. But not hugely because the .bin already loads faster than the .st. You might do some profiling?
Does it already exist?
Much of what you need is there. Use the Pharo, Luke.
Thanks,
Thierry
-- best, Eliot
2014-12-05 0:08 GMT+01:00 Eliot Miranda <eliot.miranda@gmail.com>:
Hi Thierry,
On Thu, Dec 4, 2014 at 2:31 PM, Thierry Goubier <thierry.goubier@gmail.com
wrote:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
It doesn't have to be based on Fuel. Monticello .mcz's are just zips that contain any number of files:
Archive: trunkpackages/Kernel.spur-eem.866.mcz Length Date Time Name -------- ---- ---- ---- 15 08-07-14 09:12 package 294780 08-07-14 09:12 version 1370995 08-07-14 09:12 snapshot/source.st 1442870 08-07-14 09:12 snapshot.bin -------- ------- 3108660 4 files
Right now package and version are juts textual, and you've been complaining about the parsing speed. It would be easy to extend the package to include e.g. version.fuel. It is also a huge advantage to keep both the .bin (a not-as-fast binary loading format similar in use to Fuel but more primitive) along side the source code, because the source can be extracted without a running system, and can be compiled if, for some reason, binary loading fails. So instead of changing the format you could simply augment it with a snapshot.fuel, modifying the writer to include it, and modify the reader to use the .fuel if it exists.
Yes, or use an entirely different format. Whatever works. FileTree demonstrate that this can be done, and GitFileTree demonstrated that it could totally reproduce the mcz view on another tech repo. I'm more annoyed by the consequences of loading the full version file in the image.
Would that make loading faster?
Yes. But not hugely because the .bin already loads faster than the .st. You might do some profiling?
Good suggestion. This is what I wanted to know.
Does it already exist?
Much of what you need is there. Use the Pharo, Luke.
This is a part of Pharo where I know my way :) Thanks, Thierry
Thanks,
Thierry
-- best, Eliot
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc. Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel. @Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
2014-12-05 8:35 GMT+01:00 Max Leske <maxleske@gmail.com>:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isn't something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you don't strictly have to build the graph but you still need to read the entire file).
Hum. I was thinking of partial loading at the application level, not at the marshalling level. The process of cutting out Monticello packages versions ancestry in Pharo releases come to mind as a ... how should I say that ... We can/should do better (additionally, because if the mcz isn't available somewhere, having its ref its useless). Thierry
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
I will summon Martin and Mariano here :). Introducing fuel in the loading infrastructure had so far AFAIK two different experimental setups: - tanker: a package completely written in fuel - mixing tanker/fuel with monticello. However, the results were not so promising I remember. Apparently when loading a package, most of the time is spent not in the deserialization/recompilation but in the update of the system (system dictionary, categories, rpackages, update the corresponding subclass relationships). Just try the following: load a monticello package with (a) no Nautilus opened, and with (a) 10 Nautilus opened. I let Martin and Mariano give the details :). On Fri, Dec 5, 2014 at 9:28 AM, Thierry Goubier <thierry.goubier@gmail.com> wrote:
2014-12-05 8:35 GMT+01:00 Max Leske <maxleske@gmail.com>:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hum. I was thinking of partial loading at the application level, not at the marshalling level.
The process of cutting out Monticello packages versions ancestry in Pharo releases come to mind as a ... how should I say that ... We can/should do better (additionally, because if the mcz isn't available somewhere, having its ref its useless).
Thierry
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
2014-12-05 10:55 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com>:
I will summon Martin and Mariano here :).
Introducing fuel in the loading infrastructure had so far AFAIK two different experimental setups: - tanker: a package completely written in fuel
Oh, I missed that one, yes. I remember the name.
- mixing tanker/fuel with monticello.
Sort of what I'm thinking about.
However, the results were not so promising I remember. Apparently when loading a package, most of the time is spent not in the deserialization/recompilation but in the update of the system (system dictionary, categories, rpackages, update the corresponding subclass relationships).
Just try the following: load a monticello package with (a) no Nautilus opened, and with (a) 10 Nautilus opened.
I profiled AltBrowser for that use case... It's a bit more vulnerable to that than Nautilus because you may see more of the structure at a given time than Nautilus (such as having a single browser watching the methods of two classes). Still, once a bit optimised, MC loading time dominates. But I haven't looked too closely on exactly what; I was focused on getting the browser bit as small as possible. If anybody has things about the various costs, I'd be interested. Thierry
On Fri, Dec 5, 2014 at 6:55 AM, Guillermo Polito <guillermopolito@gmail.com> wrote:
I will summon Martin and Mariano here :).
Introducing fuel in the loading infrastructure had so far AFAIK two different experimental setups: - tanker: a package completely written in fuel
This one is the one I explain below.
- mixing tanker/fuel with monticello.
I think here you mean an experiment where we serialized the little thing MC serializes currently in the data file... We found that that serialization/materialization is INSIGNIFICANT in the export/import process. Ok... so.. Indeed: Tanker. Fuel is able to serialize methods, classes etc... (but not by default). So what Tanker does it kind of serialize a whole package (with internal classes, extension methods, etc) as a fuel file. And then, at the import, it materializes and then runs a lot of stuff besides the materialization itself. The idea was NOT to use the Compiler at all at import time, because that is/was one of the bottleneck when importing packages with Metacello. The compiler was not needed most of the times, so performance was much better. However, there were cases (like instVars changed in superclasses where we are importing) etc.. that causes that we needed to RE-Compile or stored methods because otherwise the instVar offsets were wrong. This, COULD be solved this the intermediate representation of Opal and to avoid a normal compilation. However, I don't remember us doing that. I think we ended up using the compiler for that scenario. But I am not sure.
However, the results were not so promising I remember. Apparently when loading a package, most of the time is spent not in the deserialization/recompilation but in the update of the system (system dictionary, categories, rpackages, update the corresponding subclass relationships).
Exactly. The ones we found that took most of the time was: 1) Compiling (could be solved with Fuel and even more if we can fix some scenarios with Opal) 2) Notification of created methods/classes. One way to solve this is by having bulk notifications where we notify a list rather than a single object. But we should also adapt the observers. 3) #become: when updating/migrating existing instances. But...just to see that there is a light in the tunnel, I showed at... mmm ESUG or PharoConf or.. how to load WHOLE seaside in 10 seconds...which in my machine takes like 20 minutes. Tanker was originally called "FuelPackageLoader" and was then renamed. Some useful links with details: http://marianopeck.wordpress.com/2012/08/11/tanker-transporting-packages-wit... http://marianopeck.wordpress.com/2012/09/28/new-tanker-current-status/ http://marianopeck.wordpress.com/2012/04/28/building-pharo-from-pharokernel-... http://marianopeck.wordpress.com/tag/tanker/ http://vimeo.com/63122424
Just try the following: load a monticello package with (a) no Nautilus opened, and with (a) 10 Nautilus opened.
Exactly!!! Sometimes the slowdown of the notifications is not in the observable but on the observers ;)
I let Martin and Mariano give the details :).
On Fri, Dec 5, 2014 at 9:28 AM, Thierry Goubier <thierry.goubier@gmail.com
wrote:
2014-12-05 8:35 GMT+01:00 Max Leske <maxleske@gmail.com>:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hum. I was thinking of partial loading at the application level, not at the marshalling level.
The process of cutting out Monticello packages versions ancestry in Pharo releases come to mind as a ... how should I say that ... We can/should do better (additionally, because if the mcz isn't available somewhere, having its ref its useless).
Thierry
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- Mariano http://marianopeck.wordpress.com
On Fri, Dec 5, 2014 at 10:07 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Fri, Dec 5, 2014 at 6:55 AM, Guillermo Polito < guillermopolito@gmail.com> wrote:
I will summon Martin and Mariano here :).
Introducing fuel in the loading infrastructure had so far AFAIK two different experimental setups: - tanker: a package completely written in fuel
This one is the one I explain below.
- mixing tanker/fuel with monticello.
I think here you mean an experiment where we serialized the little thing MC serializes currently in the data file... We found that that serialization/materialization is INSIGNIFICANT in the export/import process.
Ok... so..
Indeed: Tanker. Fuel is able to serialize methods, classes etc... (but not by default). So what Tanker does it kind of serialize a whole package (with internal classes, extension methods, etc) as a fuel file. And then, at the import, it materializes and then runs a lot of stuff besides the materialization itself. The idea was NOT to use the Compiler at all at import time, because that is/was one of the bottleneck when importing packages with Metacello. The compiler was not needed most of the times, so performance was much better. However, there were cases (like instVars changed in superclasses where we are importing) etc.. that causes that we needed to RE-Compile or stored methods because otherwise the instVar offsets were wrong. This, COULD be solved this the intermediate representation of Opal and to avoid a normal compilation. However, I don't remember us doing that. I think we ended up using the compiler for that scenario. But I am not sure.
However, the results were not so promising I remember. Apparently when loading a package, most of the time is spent not in the deserialization/recompilation but in the update of the system (system dictionary, categories, rpackages, update the corresponding subclass relationships).
Exactly. The ones we found that took most of the time was:
1) Compiling (could be solved with Fuel and even more if we can fix some scenarios with Opal) 2) Notification of created methods/classes. One way to solve this is by having bulk notifications where we notify a list rather than a single object. But we should also adapt the observers. 3) #become: when updating/migrating existing instances.
I forgot to said...this one could be improved once the new VM Eliot is doing has the lazy become.. So.... adapting Tanker to use Opal to avoid compilation in special cases, having a bulk notification and update a few observers (like test runner or nautilus), and a lazy become... that would bring a incredible change.
But...just to see that there is a light in the tunnel, I showed at... mmm ESUG or PharoConf or.. how to load WHOLE seaside in 10 seconds...which in my machine takes like 20 minutes.
Tanker was originally called "FuelPackageLoader" and was then renamed. Some useful links with details:
http://marianopeck.wordpress.com/2012/08/11/tanker-transporting-packages-wit... http://marianopeck.wordpress.com/2012/09/28/new-tanker-current-status/
http://marianopeck.wordpress.com/2012/04/28/building-pharo-from-pharokernel-... http://marianopeck.wordpress.com/tag/tanker/
Just try the following: load a monticello package with (a) no Nautilus opened, and with (a) 10 Nautilus opened.
Exactly!!! Sometimes the slowdown of the notifications is not in the observable but on the observers ;)
I let Martin and Mariano give the details :).
On Fri, Dec 5, 2014 at 9:28 AM, Thierry Goubier < thierry.goubier@gmail.com> wrote:
2014-12-05 8:35 GMT+01:00 Max Leske <maxleske@gmail.com>:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hum. I was thinking of partial loading at the application level, not at the marshalling level.
The process of cutting out Monticello packages versions ancestry in Pharo releases come to mind as a ... how should I say that ... We can/should do better (additionally, because if the mcz isn't available somewhere, having its ref its useless).
Thierry
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 05 Dec 2014, at 14:09, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
Mariano http://marianopeck.wordpress.com
Hey Mariano, Welcome back ! (Not that you left, but I read your latest blog entry) We miss(ed) your expertise and your work. Regards, Sven
2014-12-05 14:07 GMT+01:00 Mariano Martinez Peck <marianopeck@gmail.com>:
On Fri, Dec 5, 2014 at 6:55 AM, Guillermo Polito < guillermopolito@gmail.com> wrote:
I will summon Martin and Mariano here :).
Introducing fuel in the loading infrastructure had so far AFAIK two different experimental setups: - tanker: a package completely written in fuel
This one is the one I explain below.
- mixing tanker/fuel with monticello.
I think here you mean an experiment where we serialized the little thing MC serializes currently in the data file... We found that that serialization/materialization is INSIGNIFICANT in the export/import process.
Ok.
Ok... so..
Indeed: Tanker. Fuel is able to serialize methods, classes etc... (but not by default). So what Tanker does it kind of serialize a whole package (with internal classes, extension methods, etc) as a fuel file. And then, at the import, it materializes and then runs a lot of stuff besides the materialization itself. The idea was NOT to use the Compiler at all at import time, because that is/was one of the bottleneck when importing packages with Metacello. The compiler was not needed most of the times, so performance was much better. However, there were cases (like instVars changed in superclasses where we are importing) etc.. that causes that we needed to RE-Compile or stored methods because otherwise the instVar offsets were wrong. This, COULD be solved this the intermediate representation of Opal and to avoid a normal compilation. However, I don't remember us doing that. I think we ended up using the compiler for that scenario. But I am not sure.
What is the state of that for, say, Pharo 4?
However, the results were not so promising I remember. Apparently when loading a package, most of the time is spent not in the deserialization/recompilation but in the update of the system (system dictionary, categories, rpackages, update the corresponding subclass relationships).
Exactly. The ones we found that took most of the time was:
1) Compiling (could be solved with Fuel and even more if we can fix some scenarios with Opal) 2) Notification of created methods/classes. One way to solve this is by having bulk notifications where we notify a list rather than a single object. But we should also adapt the observers.
They can be adapted. You can also pile up notifications (well, you can't at the moment, but it would be nice) and have a way to merge them. Such optimisations are also usefull for heavy code generators such as SmaCC.
3) #become: when updating/migrating existing instances.
But...just to see that there is a light in the tunnel, I showed at... mmm ESUG or PharoConf or.. how to load WHOLE seaside in 10 seconds...which in my machine takes like 20 minutes.
Which is what I wanted to know :)
Tanker was originally called "FuelPackageLoader" and was then renamed. Some useful links with details:
http://marianopeck.wordpress.com/2012/08/11/tanker-transporting-packages-wit... http://marianopeck.wordpress.com/2012/09/28/new-tanker-current-status/
http://marianopeck.wordpress.com/2012/04/28/building-pharo-from-pharokernel-... http://marianopeck.wordpress.com/tag/tanker/
Thanks for the links!
Just try the following: load a monticello package with (a) no Nautilus opened, and with (a) 10 Nautilus opened.
Exactly!!! Sometimes the slowdown of the notifications is not in the observable but on the observers ;)
When, in the profile, the observers appear and you can optimize that. One of the way is to have a single central point (i.e. whatever the number of Nautilus instances opened, you send to a singleton), then memoize all queries you do on RPackage and friends before firing again notifications for the views (relevant package, class, etc..). In short, you have the thing. Thierry
Thanks Mariano. If we could optimize common case = no change in superclass shape change and get that one fast it would be so great. Stef
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments. Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple. Perhaps I should have called it partial installation, but you get the idea.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
On Sat, Dec 6, 2014 at 12:37 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Eliot, I do agree with all you said. But let me say a few bits please to reduce misunderstandings. We consider Fuel itself completely isolated from what is exporting/importing a package. In fact, as you know, by default Fuel serializes classes and methods "globally". So...all this discussion should actually be at another level... say Tanker level (or whatever tool we build on top of Fuel to manage packages). This is a difference to Parcels, which even if it was able to also serialize regular objects graphs, it was more tight to packages. In addition, as you said, Max is right in the sense that at FUEL level, the whole graph would have to be materialized. IN THE CASE OF TANKER, or whatever tool for packages, we may NOT install all parts, as you said. But they will indeed be materialized. So... to sum out, I think the clearer way of saying it is: - Fuel must have to materialize the whole graph anyway (at least as it is now). - Tanker or whatever tool for managing packages will also have to materialize all, but should be able to "install" only a part of it. The example you said about Fuel is very clear. Soo... I think we all agree. Just that we use a different naming. Best regards,
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
-- Mariano http://marianopeck.wordpress.com
On 06 Dec 2014, at 04:37, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Makes sense. Itâs just that I often get asked if it is possible to read / update only parts of a Fuel file, so I wanted to make it clear that that canât work.
Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Thanks for the explanation.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
Hi Max, On Sat, Dec 6, 2014 at 1:43 AM, Max Leske <maxleske@gmail.com> wrote:
On 06 Dec 2014, at 04:37, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier < thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Makes sense. Itâs just that I often get asked if it is possible to read / update only parts of a Fuel file, so I wanted to make it clear that that canât work.
Right, good point. But is there a use case? For example, in effect is there a difference between being able to patch a part of a Fuel file and loading that file into an image (possibly automatically), patching the object graph and saving a new version? Lets take an example like Fuel itself. This may have specialised
marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Thanks for the explanation.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
-- best, Eliot
On 07 Dec 2014, at 04:19, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Max,
Hi Eliot,
On Sat, Dec 6, 2014 at 1:43 AM, Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>> wrote:
On 06 Dec 2014, at 04:37, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Makes sense. Itâs just that I often get asked if it is possible to read / update only parts of a Fuel file, so I wanted to make it clear that that canât work.
Right, good point. But is there a use case? For example, in effect is there a difference between being able to patch a part of a Fuel file and loading that file into an image (possibly automatically), patching the object graph and saving a new version?
Yes, performance seems to be the main use case. Fuel files of hundreds of MB take a long time to load and save, even if you only change a single string. One instance where this is relevant is the Moose model. They currently use .mse files, which use a textual representation (if Iâm not mistaken). Usman asked me if Fuel were an option, especially for updating only parts of the model.
Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Thanks for the explanation.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
-- best, Eliot
Hi Max, On Dec 7, 2014, at 5:14 AM, Max Leske <maxleske@gmail.com> wrote:
On 07 Dec 2014, at 04:19, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Max,
Hi Eliot,
On Sat, Dec 6, 2014 at 1:43 AM, Max Leske <maxleske@gmail.com> wrote:
On 06 Dec 2014, at 04:37, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Makes sense. Itâs just that I often get asked if it is possible to read / update only parts of a Fuel file, so I wanted to make it clear that that canât work.
Right, good point. But is there a use case? For example, in effect is there a difference between being able to patch a part of a Fuel file and loading that file into an image (possibly automatically), patching the object graph and saving a new version?
Yes, performance seems to be the main use case. Fuel files of hundreds of MB take a long time to load and save, even if you only change a single string. One instance where this is relevant is the Moose model. They currently use .mse files, which use a textual representation (if Iâm not mistaken). Usman asked me if Fuel were an option, especially for updating only parts of the model.
Ok, but how could patching the file be much faster? The file has to be parsed into something; why not the objects themselves? To me loading, editing and red acing is patching and done as fast as Fuel can do things. Depending on syntax you may be able to scan for a string and replace using eg the sed stream editor. But that'll only work for simple data like strings. If you really need this to be faster have you considered - decomposing the Moose model into smaller individual components? - keeping the moose model to be saved in an image (possibly live running in a server) and patching the model in the image and saving it - using Spur [ ;-) ]. The Pharos tests now run and it should be significantly faster, at least at loading
Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Thanks for the explanation.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
-- best, Eliot
Yes, performance seems to be the main use case. Fuel files of hundreds of MB take a long time to load and save, even if you only change a single string. One instance where this is relevant is the Moose model. They currently use .mse files, which use a textual representation (if Iâm not mistaken). Usman asked me if Fuel were an option, especially for updating only parts of the model.
Ok, but how could patching the file be much faster? The file has to be parsed into something; why not the objects themselves? To me loading, editing and red acing is patching and done as fast as Fuel can do things.
Depending on syntax you may be able to scan for a string and replace using eg the sed stream editor. But that'll only work for simple data like strings.
If you really need this to be faster have you considered
- decomposing the Moose model into smaller individual components?
- keeping the moose model to be saved in an image (possibly live running in a server) and patching the model in the image and saving it
This is a classic problem; as models start to get large they get harder to update. If there were some way to update the file with just the parts in memory that changed since the last save, it would perform a lot faster for the end user. Of course whatever makes models large is also what tends to make them more interesting to more users, but moves inversely to the sharability. Some kind of server is needed to serve up the fuel file and use the above to dish out whatever portions of the file needed by user for the part of the model they're accessing. When they save, only the parts of the model they changed, nothing more, are sent back to the server which performs a random-access update to the fuel file in a controlled, synchronous manner. Although, even with that I guess users could still overlay other users work very easily. Some kind of concurrency control mechanism should look at the changes being made and signal a warning if that happens.
Hi Max, oops, I was being dense... On Dec 7, 2014, at 5:14 AM, Max Leske <maxleske@gmail.com> wrote:
On 07 Dec 2014, at 04:19, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Max,
Hi Eliot,
On Sat, Dec 6, 2014 at 1:43 AM, Max Leske <maxleske@gmail.com> wrote:
On 06 Dec 2014, at 04:37, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Makes sense. Itâs just that I often get asked if it is possible to read / update only parts of a Fuel file, so I wanted to make it clear that that canât work.
Right, good point. But is there a use case? For example, in effect is there a difference between being able to patch a part of a Fuel file and loading that file into an image (possibly automatically), patching the object graph and saving a new version?
Yes, performance seems to be the main use case. Fuel files of hundreds of MB take a long time to load and save, even if you only change a single string. One instance where this is relevant is the Moose model. They currently use .mse files, which use a textual representation (if Iâm not mistaken). Usman asked me if Fuel were an option, especially for updating only parts of the model.
Ah ok. Then yes, I suspect Fuel will be faster, but that patching is more complex because it can't be done in a text editor. How do they edit the .mse file? IME text editors suck on large files. What are the relative frequencies of writing, reading and editing these files?
Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Thanks for the explanation.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
-- best, Eliot
On 07 Dec 2014, at 19:24, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Max,
oops, I was being dense...
On Dec 7, 2014, at 5:14 AM, Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>> wrote:
On 07 Dec 2014, at 04:19, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote:
Hi Max,
Hi Eliot,
On Sat, Dec 6, 2014 at 1:43 AM, Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>> wrote:
On 06 Dec 2014, at 04:37, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote:
On Thu, Dec 4, 2014 at 11:35 PM, Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>> wrote:
On 05 Dec 2014, at 08:02, Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>> wrote:
Am 04.12.2014 um 23:31 schrieb Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>>:
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
I doubt it would work cross platform. I don't know how fuel serializes WideString, LargePositiveInteger, BoxedFloat64. These differ between smalltalk platforms. The source as string solves that. Strings are written as unicode string and numbers as certain number format etc.
True. Cross dialect loading isnât something we encourage to do with Fuel.
@Thierry You said something about partial loading: that will never be possible with Fuel because of its pickle format. To select any partial graph you have to first read the entire file and rebuild the graph first (ok, you donât strictly have to build the graph but you still need to read the entire file).
Hi Max, you misunderstand what partial loading means in this context. I designed and implemented partial loading for the Parcel system in VW, and Fuel is essentially a clean reimplementation of parcels. The idea is that one does indeed read the entire object graph, but then only the parts of the graph that mate with the current class hierarchy are installed, and the bits that don't fit are stored for later. Why? So that a component can define extensions on classes in components that may not be present. Why? So that one can maintain a single logical component in a single package instead of decomposing it into independently loadable fragments.
Makes sense. Itâs just that I often get asked if it is possible to read / update only parts of a Fuel file, so I wanted to make it clear that that canât work.
Right, good point. But is there a use case? For example, in effect is there a difference between being able to patch a part of a Fuel file and loading that file into an image (possibly automatically), patching the object graph and saving a new version?
Yes, performance seems to be the main use case. Fuel files of hundreds of MB take a long time to load and save, even if you only change a single string. One instance where this is relevant is the Moose model. They currently use .mse files, which use a textual representation (if Iâm not mistaken). Usman asked me if Fuel were an option, especially for updating only parts of the model.
Ah ok. Then yes, I suspect Fuel will be faster, but that patching is more complex because it can't be done in a text editor. How do they edit the .mse file? IME text editors suck on large files. What are the relative frequencies of writing, reading and editing these files?
I donât think they can patch the .mse files. Thatâs exactly why they are looking for alternatives. I have no clue about the read / write frenquencies they need. Iâm CCâing this to Usman, maybe can comment on that.
Lets take an example like Fuel itself. This may have specialised marshalling and unmarshalling extensions defined on may classes, some of which may be to do with the GUI. If we have partial loading we can load Fuel into a headless image. The extensions on the GUI classes will simply not be installed, *until* we load the GUI. Hence Fuel does not have to be decomposed every time we factor the system into subcomponents. Without partial loading Fuel must be decomposed into a series of fragments so that it can load that part that fits into the headless base, and we have to manage teh dependency to ensure the fragment that loads against the GUI is loaded when the GUI is loaded. Worse still, if we cut the GUI into two (e.g. development vs deployment tools) we have to visit the Fuel GUI component (and potentially many other component) and decompose it into two pieces. In practice this is extremely costly to maintain, verging on chaos. With partial loading things are simple.
Perhaps I should have called it partial installation, but you get the idea.
Thanks for the explanation.
Norbert
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
-- best, Eliot
-- best, Eliot
There was a Tanker prototype. Now Levente did long time ago an analysis and he mentioned that the compilation was not that slow. Now I would love to have a binary format (fuel) as part of the mcz. because when we started Fuel it was the goal (I remember being shocked by the speed of Parcels 30 min to load RB in VW25 and 3 seconds in VW 30) Stef
Hi all,
I'm just wondering.
Would it work to have a package format based on Fuel?
Would that make loading faster?
Does it already exist?
Thanks,
Thierry
participants (10)
-
Ben Coman -
Chris Muller -
Eliot Miranda -
Guillermo Polito -
Mariano Martinez Peck -
Max Leske -
Norbert Hartl -
stepharo -
Sven Van Caekenberghe -
Thierry Goubier