Hi, The current Pharo4 contains *a lot* of MCVersionInfo instances. MCVersionInfo allInstances size 11095 The hold on to strings, Date, UUID⦠a lot of stuff. All in all, this is around 10% of the current image. If you do MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ]. you image is a couple of MB smaller. In the past, when this information started to be >8MB, we did that. With the bad effect that we can not merge anymore across this boundary: we kill the past. Now this information is of course continained in the last MCZ file, too (all of them contain the complete historyâ¦) So would the following work? -> we set the âancestorsâ of MCVersionInfo to #reload -> the accessor, when it sees #reload, takes the name, deduces from that the package, and goes to the repo to load the ancestry info from the MCZFile. This means that e.g when saving a MCZFile, it would first re-create history info and then save it completely (like now), yet someone who just uses the system would never need to have this info in the image. Marcus
Chris, didn't you do something about this in Squeak? It would be nice to keep consistent if possible... Eliot (phone) On Feb 20, 2015, at 6:41 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
Hi,
The current Pharo4 contains *a lot* of MCVersionInfo instances.
MCVersionInfo allInstances size
11095
The hold on to strings, Date, UUID⦠a lot of stuff. All in all, this is around 10% of the current image.
If you do
MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].
you image is a couple of MB smaller.
In the past, when this information started to be >8MB, we did that. With the bad effect that we can not merge anymore across this boundary: we kill the past.
Now this information is of course continained in the last MCZ file, too (all of them contain the complete historyâ¦)
So would the following work?
-> we set the âancestorsâ of MCVersionInfo to #reload -> the accessor, when it sees #reload, takes the name, deduces from that the package, and goes to the repo to load the ancestry info from the MCZFile.
This means that e.g when saving a MCZFile, it would first re-create history info and then save it completely (like now), yet someone who just uses the system would never need to have this info in the image.
Marcus
Chris, didn't you do something about this in Squeak? It would be nice to keep consistent if possible...
Yes, but no one liked it because it employed the Proxy design-pattern and requires a become. Perhaps if I'd gotten it perfect the first time it'd have been better-received. But, I didn't, and the ensuing backlash wiped out any motivation for me to fix it. Marcus' suggestion is more explicit, and so might be better received by the community. There's a tradeoff between the two designs: In my design, you have to make sure you get it right or the image could lock up. In Marcus' design, you have to sprinkle multiple checks for #reload in the code and make sure you get it right, or you might end up saving corrupt MCZ packages (e.g., with #reload as the ancestry). Either way, its a wasteful part of the design of MC that should be addressed. I'm partial to fixing the Proxy solution, but I'd be in favor of ripping it out and adopting a stable alternative from Pharo if it becomes available.
Eliot (phone)
On Feb 20, 2015, at 6:41 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
Hi,
The current Pharo4 contains *a lot* of MCVersionInfo instances.
MCVersionInfo allInstances size
11095
The hold on to strings, Date, UUID⦠a lot of stuff. All in all, this is around 10% of the current image.
If you do
MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].
you image is a couple of MB smaller.
In the past, when this information started to be >8MB, we did that. With the bad effect that we can not merge anymore across this boundary: we kill the past.
Now this information is of course continained in the last MCZ file, too (all of them contain the complete historyâ¦)
So would the following work?
-> we set the âancestorsâ of MCVersionInfo to #reload -> the accessor, when it sees #reload, takes the name, deduces from that the package, and goes to the repo to load the ancestry info from the MCZFile.
This means that e.g when saving a MCZFile, it would first re-create history info and then save it completely (like now), yet someone who just uses the system would never need to have this info in the image.
Marcus
On 20 Feb 2015, at 18:11, Chris Muller <asqueaker@gmail.com> wrote:
Chris, didn't you do something about this in Squeak? It would be nice to keep consistent if possible...
Yes, but no one liked it because it employed the Proxy design-pattern and requires a become. Perhaps if I'd gotten it perfect the first time it'd have been better-received. But, I didn't, and the ensuing backlash wiped out any motivation for me to fix it.
This is always the danger⦠I start to be very careful in this regard: better make (and allow!) mistakes than doing nothing. Because mistakes can be fixed, while a discouraged person will just never do anything. (e.g. leave the project in case of open source). Of course #become: sadly get really slow when the image gets large⦠but this is fixed with Spur, which means it will possible.
Marcus' suggestion is more explicit, and so might be better received by the community. There's a tradeoff between the two designs: In my design, you have to make sure you get it right or the image could lock up. In Marcus' design, you have to sprinkle multiple checks for #reload in the code and make sure you get it right, or you might end up saving corrupt MCZ packages (e.g., with #reload as the ancestry).
Yes. I think the only reason I dared to let me think in this direction is that the #ancestors are accessed *only* via the accessor⦠but nevertheless: checks need to be added.
Either way, its a wasteful part of the design of MC that should be addressed. I'm partial to fixing the Proxy solution, but I'd be in favor of ripping it out and adopting a stable alternative from Pharo if it becomes available.
We could do it in two steps: adopt the #become: solution when we use spur 100%⦠I even want to have a default Proxy framework in the system to make these things easier. (something like Ghost). Marcus
Le 20/02/2015 15:41, Marcus Denker a écrit :
Hi,
The current Pharo4 contains *a lot* of MCVersionInfo instances.
MCVersionInfo allInstances size
11095
The hold on to strings, Date, UUID⦠a lot of stuff. All in all, this is around 10% of the current image.
If you do
MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].
you image is a couple of MB smaller.
In the past, when this information started to be >8MB, we did that. With the bad effect that we can not merge anymore across this boundary: we kill the past.
You can still merge across that boundary, but you need to do it in git, not in Monticello.
Now this information is of course continained in the last MCZ file, too (all of them contain the complete historyâ¦)
So would the following work?
-> we set the âancestorsâ of MCVersionInfo to #reload -> the accessor, when it sees #reload, takes the name, deduces from that the package, and goes to the repo to load the ancestry info from the MCZFile.
This means that e.g when saving a MCZFile, it would first re-create history info and then save it completely (like now), yet someone who just uses the system would never need to have this info in the image.
Yes, this could work. But I'll probably try a subclass of MCVersionInfo with a kind of lazy loading of the ancestry chain (in a weak array so that it gets eaten away if not used). And the weak array would ensure that you don't overuse memory without having to use become: Like that, on git, I could be faster and not write the whole ancestry at all. I just did something a bit reverse with MCDependencyVersion for GitFileTree: I added a MCResolvedDependencyVersion and it worked. Up to you. Thierry
Hi, I made a very simple experiment, with a MCLazyVersionInfo, which holds its ancestors and stepChildren in WeakArray(s), and loads it from repositories on demand. Installing it on Pharo 40511, and then flushing the Monticello package cache, reduces the image by around 5 MB: Before: memory 48,583,272 bytes old 40,721,508 bytes (83.80000000000001%) young 1,445,212 bytes (3.0%) used 42,166,720 bytes (86.80000000000001%) free 6,416,552 bytes (13.200000000000001%) After: memory 42,482,984 bytes old 35,484,236 bytes (83.5%) young 1,434,832 bytes (3.4000000000000004%) used 36,919,068 bytes (86.9%) free 5,563,916 bytes (13.100000000000001%) You can browse the history of any package and it will be reloaded on the fly :) But, but, but. When opening a repository inspector, MC will walk through all ancestors of all packages... reloading everything. Arrrg ! So I won't tell you where it is available, yet :( Thierry Le 20/02/2015 15:41, Marcus Denker a écrit :
Hi,
The current Pharo4 contains *a lot* of MCVersionInfo instances.
MCVersionInfo allInstances size
11095
The hold on to strings, Date, UUID⦠a lot of stuff. All in all, this is around 10% of the current image.
If you do
MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].
you image is a couple of MB smaller.
In the past, when this information started to be >8MB, we did that. With the bad effect that we can not merge anymore across this boundary: we kill the past.
Now this information is of course continained in the last MCZ file, too (all of them contain the complete historyâ¦)
So would the following work?
-> we set the âancestorsâ of MCVersionInfo to #reload -> the accessor, when it sees #reload, takes the name, deduces from that the package, and goes to the repo to load the ancestry info from the MCZFile.
This means that e.g when saving a MCZFile, it would first re-create history info and then save it completely (like now), yet someone who just uses the system would never need to have this info in the image.
Marcus
On 25 Feb 2015, at 23:28, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi,
I made a very simple experiment, with a MCLazyVersionInfo, which holds its ancestors and stepChildren in WeakArray(s), and loads it from repositories on demand.
Installing it on Pharo 40511, and then flushing the Monticello package cache, reduces the image by around 5 MB:
Nice! Marcus
2015-02-26 7:58 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 25 Feb 2015, at 23:28, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi,
I made a very simple experiment, with a MCLazyVersionInfo, which holds its ancestors and stepChildren in WeakArray(s), and loads it from repositories on demand.
Installing it on Pharo 40511, and then flushing the Monticello package cache, reduces the image by around 5 MB:
Nice!
Marcus
And it behaves as the non-lazy one. There are a few things to change GUI-side to avoid an unnecessary loading of the complete ancestry of all the packages in the image, but this is workable. I see two ways of using it: as an "install, flush package cache before release and gain 5MB of RAM". If you start regularly developping (loading packages, etc...) then new packages will revert to non-lazy version info (without: 'please wait, I'm loading ancestry' moments). as a permanent change in MC. The change is a bit more intrusive (need to change how working copies are created) but, well, as long as the worked on packages are inside the in-memory MC package cache, behavior is exactly the same as a non-lazy version info. As you prefer. I added an issue: https://pharo.fogbugz.com/default.asp?14997 Thierry
And it behaves as the non-lazy one.
There are a few things to change GUI-side to avoid an unnecessary loading of the complete ancestry of all the packages in the image, but this is workable.
I see two ways of using it:
as an "install, flush package cache before release and gain 5MB of RAM". If you start regularly developping (loading packages, etc...) then new packages will revert to non-lazy version info (without: 'please wait, I'm loading ancestry' moments).
as a permanent change in MC. The change is a bit more intrusive (need to change how working copies are created) but, well, as long as the worked on packages are inside the in-memory MC package cache, behavior is exactly the same as a non-lazy version info.
As you prefer.
My focus is that we ship an image that is not growing indefinitely⦠so this means it needs to be part of the release. I donât care too much that the image is large when I work, but it should not come with all this data in memory when it can be calculated (or lazyly loaded) when needed.
I added an issue:
https://pharo.fogbugz.com/default.asp?14997 <https://pharo.fogbugz.com/default.asp?14997>
Thanks! Marcus
Ok, a first slice is in, it's the first solution for now (second is just extending the scheme). Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?) Thierry 2015-02-26 9:33 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
And it behaves as the non-lazy one.
There are a few things to change GUI-side to avoid an unnecessary loading of the complete ancestry of all the packages in the image, but this is workable.
I see two ways of using it:
as an "install, flush package cache before release and gain 5MB of RAM". If you start regularly developping (loading packages, etc...) then new packages will revert to non-lazy version info (without: 'please wait, I'm loading ancestry' moments).
as a permanent change in MC. The change is a bit more intrusive (need to change how working copies are created) but, well, as long as the worked on packages are inside the in-memory MC package cache, behavior is exactly the same as a non-lazy version info.
As you prefer.
My focus is that we ship an image that is not growing indefinitely... so this means it needs to be part of the release.
I don't care too much that the image is large when I work, but it should not come with all this data in memory when it can be calculated (or lazyly loaded) when needed.
I added an issue:
https://pharo.fogbugz.com/default.asp?14997
Thanks!
Marcus
On 26 Feb 2015, at 10:15, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Ok,
a first slice is in, it's the first solution for now (second is just extending the scheme).
Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?)
Works fine with merge, too. The 511 image is 23.2MB after. This means with the Scriptloader fix from 510 together we would over 10MB⦠or nearly 1/3. Nice! Marcus
2015-02-26 10:44 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 26 Feb 2015, at 10:15, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Ok,
a first slice is in, it's the first solution for now (second is just extending the scheme).
Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?)
Works fine with merge, too.
The 511 image is 23.2MB after. This means with the Scriptloader fix from 510 together we would over 10MB... or nearly 1/3. Nice!
Ok. Second scheme (get it to work everywhere, everytime) seems to just need to change MCWorkingAncestry>>addAncestor: and MCWorkingCopy>> addStepChild: Thierry
Marcus
2015-02-26 10:51 GMT+01:00 Thierry Goubier <thierry.goubier@gmail.com>:
2015-02-26 10:44 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 26 Feb 2015, at 10:15, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Ok,
a first slice is in, it's the first solution for now (second is just extending the scheme).
Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?)
Works fine with merge, too.
The 511 image is 23.2MB after. This means with the Scriptloader fix from 510 together we would over 10MB... or nearly 1/3. Nice!
Ok. Second scheme (get it to work everywhere, everytime) seems to just need to change MCWorkingAncestry>>addAncestor: and MCWorkingCopy>> addStepChild:
Done. Slice uploaded.Tried with loading Roassal2, then flushing the Monticello package cache: MCVersionInfo allInstances size 2 Thierry
Thierry
Marcus
On 26 Feb 2015, at 11:11, Thierry Goubier <thierry.goubier@gmail.com> wrote:
2015-02-26 10:51 GMT+01:00 Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>>:
2015-02-26 10:44 GMT+01:00 Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>>:
On 26 Feb 2015, at 10:15, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Ok,
a first slice is in, it's the first solution for now (second is just extending the scheme).
Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?)
Works fine with merge, too.
The 511 image is 23.2MB after. This means with the Scriptloader fix from 510 together we would over 10MB⦠or nearly 1/3. Nice!
Ok. Second scheme (get it to work everywhere, everytime) seems to just need to change MCWorkingAncestry>>addAncestor: and MCWorkingCopy>> addStepChild:
Done. Slice uploaded.Tried with loading Roassal2, then flushing the Monticello package cache:
MCVersionInfo allInstances size 2
Thanks!! Marcus
Excellent! Le 26/2/15 11:11, Thierry Goubier a écrit :
2015-02-26 10:51 GMT+01:00 Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>>:
2015-02-26 10:44 GMT+01:00 Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>>:
> On 26 Feb 2015, at 10:15, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote: > > Ok, > > a first slice is in, it's the first solution for now (second is just extending the scheme). > > Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?) >
Works fine with merge, too.
The 511 image is 23.2MB after. This means with the Scriptloader fix from 510 together we would over 10MB⦠or nearly 1/3. Nice!
Ok. Second scheme (get it to work everywhere, everytime) seems to just need to change MCWorkingAncestry>>addAncestor: and MCWorkingCopy>> addStepChild:
Done. Slice uploaded.Tried with loading Roassal2, then flushing the Monticello package cache:
MCVersionInfo allInstances size 2
Thierry
Thierry
Marcus
Thierry this is really interesting. Could you add the logic in some class comments for the future? Stef Le 26/2/15 10:15, Thierry Goubier a écrit :
Ok,
a first slice is in, it's the first solution for now (second is just extending the scheme).
Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?)
Thierry
2015-02-26 9:33 GMT+01:00 Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>>:
And it behaves as the non-lazy one.
There are a few things to change GUI-side to avoid an unnecessary loading of the complete ancestry of all the packages in the image, but this is workable.
I see two ways of using it:
as an "install, flush package cache before release and gain 5MB of RAM". If you start regularly developping (loading packages, etc...) then new packages will revert to non-lazy version info (without: 'please wait, I'm loading ancestry' moments).
as a permanent change in MC. The change is a bit more intrusive (need to change how working copies are created) but, well, as long as the worked on packages are inside the in-memory MC package cache, behavior is exactly the same as a non-lazy version info.
As you prefer.
My focus is that we ship an image that is not growing indefinitely⦠so this means it needs to be part of the release.
I donât care too much that the image is large when I work, but it should not come with all this data in memory when it can be calculated (or lazyly loaded) when needed.
I added an issue:
Thanks!
Marcus
2015-02-26 14:04 GMT+01:00 stepharo <stepharo@free.fr>:
Thierry
this is really interesting. Could you add the logic in some class comments for the future?
Done in the v3 of the slice. A MCLazyVersionInfo puts ancestor and stepChilds data in WeakArrays to allow their memory to be reclaimed and reloaded on demand. For all purposes, a MCLazyVersionInfo behaves exactly like a MCVersionInfo with the same data, but: - let the ancestry (and step children) be garbage collected - and reload that ancestry (and stepChildren) from the MC repositories when accessed (MCLazyVersionInfo>>loadAncestorsAndStepChildren for the reload, MCLazyVersionInfo>>ancestors and MCLazyVersionInfo>>stepChildren for the access). MCLazyVersionInfo is installed just as the ancestor in a MCWorkingAncestry (the direct ancestry of a WorkingCopy, which is the state all packages take when they are loaded in an image). It allows a MCWorkingAncestry to still know which package it is coming from, and, of course, the MCLazyVersionInfo to be able to reload the relevant ancestry from the right package. Once installed, by flushing the Monticello package cache, all the stored ancestry is garbage collected, except the roots of those ancestry trees, which are the MCLazyVersionInfo instances inside the MCWorkingAncestry instances. Warnings: - Algorithms which traverse the entire ancestry chain of all packages will force a reload of all the packages contained in the image, which is a time and memory and network consuming process... - It should degrades gracefully when reloading becomes impossible (returns an empty ancestry, but keeps knowledge of the fact some ancestry is missing). But this has not been tested. Thierry
Stef
Le 26/2/15 10:15, Thierry Goubier a écrit :
Ok,
a first slice is in, it's the first solution for now (second is just extending the scheme).
Loading the slice should install the lazy version and flush the caches (does merging would initialize the class?)
Thierry
2015-02-26 9:33 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
And it behaves as the non-lazy one.
There are a few things to change GUI-side to avoid an unnecessary loading of the complete ancestry of all the packages in the image, but this is workable.
I see two ways of using it:
as an "install, flush package cache before release and gain 5MB of RAM". If you start regularly developping (loading packages, etc...) then new packages will revert to non-lazy version info (without: 'please wait, I'm loading ancestry' moments).
as a permanent change in MC. The change is a bit more intrusive (need to change how working copies are created) but, well, as long as the worked on packages are inside the in-memory MC package cache, behavior is exactly the same as a non-lazy version info.
As you prefer.
My focus is that we ship an image that is not growing indefinitely... so this means it needs to be part of the release.
I don't care too much that the image is large when I work, but it should not come with all this data in memory when it can be calculated (or lazyly loaded) when needed.
I added an issue:
https://pharo.fogbugz.com/default.asp?14997
Thanks!
Marcus
participants (5)
-
Chris Muller -
Eliot Miranda -
Marcus Denker -
stepharo -
Thierry Goubier