What Ben says that he thinks it might already be solved. Maybe not on all platforms.

On 16 Feb 2017, at 15:59, denker <marcus.denker@inria.fr> wrote:

Yes, this has to be solved on the VM level and will be read when it is ready. The release should bot be hold up due to this, therefore I removed the milestone.

...

On 16 Feb 2017, at 15:50, Ben Coman <btc@openinworld.com> wrote:

@Marcus, I was reviewing this issue before I noticed you flipped the milestone from Pharo 6.0 to Later.

But is it solved already? In a fresh image 47.7MB in size I did the following a bunch of times to grow the image file to 400MB o ifNil: [o := OrderedCollection new]. 100 timesRepeat: [ o add: (Array new: 100000) ]. Smalltalk saveSession

then I did this o := nil. Smalltalk saveSession.

and the image file returned to 47.7MB.

Version info... Linux Debian 8 Jessie 32-bit Latest update: #60391 CoInterpreter VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 StackToRegisterMappingCogit VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 https://github.com/pharo-project/pharo-vm.git Commit: b8ec25a570d7539653e1d793e97609adb509aaed Date: 2016-05-04 11:14:22 +0200 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #589

https://pharo.fogbugz.com/f/cases/17240/Spur-When-saving-an-image-it-never-s...

cheers -ben

On 16 Feb 2017, at 16:16, denker <marcus.denker@inria.fr> wrote:

No,

Our image is not 47 MB… more like 28.

My understanding is: -> Cog works with “pages” (is that the term?) of memory. -> if a page is empty, it gets cleaned up and does not cost space when saving. -> if there is some object still allocated, the whole page will be saved (partly empty)

Not sure if that is completely right, but it explains why the image with lots of objects allocated new can shrink again, while the 47MB never shrinks, even though the image for sure is much smaller.

=> what is missing is a “cleanup” phase that sweeps together all the half-empty pages.

What is fun is that we now ship a 47MB image and nobody ever complains. (or even asks)

Yeah, that proves it is not such a big deal, today. But still, we have to fix this discrepancy between 28 and 47. Is the 'release cleanup' done on this image each time ?

Marcus

...

On 16 Feb 2017, at 16:07, Sven Van Caekenberghe <sven@stfx.eu> wrote:

What Ben says that he thinks it might already be solved.

Maybe not on all platforms.

...

On 16 Feb 2017, at 15:59, denker <marcus.denker@inria.fr> wrote:

Yes, this has to be solved on the VM level and will be read when it is ready. The release should bot be hold up due to this, therefore I removed the milestone.

...

On 16 Feb 2017, at 15:50, Ben Coman <btc@openinworld.com> wrote:

@Marcus, I was reviewing this issue before I noticed you flipped the milestone from Pharo 6.0 to Later.

But is it solved already? In a fresh image 47.7MB in size I did the following a bunch of times to grow the image file to 400MB o ifNil: [o := OrderedCollection new]. 100 timesRepeat: [ o add: (Array new: 100000) ]. Smalltalk saveSession

then I did this o := nil. Smalltalk saveSession.

and the image file returned to 47.7MB.

Version info... Linux Debian 8 Jessie 32-bit Latest update: #60391 CoInterpreter VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 StackToRegisterMappingCogit VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 https://github.com/pharo-project/pharo-vm.git Commit: b8ec25a570d7539653e1d793e97609adb509aaed Date: 2016-05-04 11:14:22 +0200 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #589

https://pharo.fogbugz.com/f/cases/17240/Spur-When-saving-an-image-it-never-s...

cheers -ben

Hi all, A temporary solution had been integrated in the VM a year ago, but only part of the problem was solved. This issue has been entirely solved in the VM through the implementation of a new compactor for the full GC. The new compactor (called the planning compactor) has been marked as the default full GC compactor for further VM builds *yesterday,* replacing the existing Pig compactor. However it seems that the new compactor has not yet passed all the validation infrastructure (several builds are currently marked as 'error'). I would expect VM builds in the incoming days to feature the new compactor. This problem should therefore be solved very soon in the latest pharo VMs. In any case, this problem does not require any image-side changes. It will be in Pharo 6 if the Pharo 6 release VM features the new compactor. Best, Clement On Thu, Feb 16, 2017 at 4:26 PM, denker <marcus.denker@inria.fr> wrote:

...

On 16 Feb 2017, at 16:22, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 16 Feb 2017, at 16:16, denker <marcus.denker@inria.fr> wrote:

No,

Our image is not 47 MB… more like 28.

My understanding is: -> Cog works with “pages” (is that the term?) of memory. -> if a page is empty, it gets cleaned up and does not cost space when saving. -> if there is some object still allocated, the whole page will be saved (partly empty)

Not sure if that is completely right, but it explains why the image with lots of objects allocated new can shrink again, while the 47MB never shrinks, even though the image for sure is much smaller.

=> what is missing is a “cleanup” phase that sweeps together all the half-empty pages.

What is fun is that we now ship a 47MB image and nobody ever complains. (or even asks)

Yeah, that proves it is not such a big deal, today. But still, we have to fix this discrepancy between 28 and 47.

As I tried to explain: the VM is missing a “condenser” that cleans up half allocated pages.

...

Is the 'release cleanup' done on this image each time ?

Yes.

Marcus

On Thu, Feb 16, 2017 at 6:10 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

Thanks, Clément, for this update, really exciting news. Is the new GC faster too ?

The full GC is indeed much faster as the previous compaction algorithm was very slow. But unless you were executing memory intensive code in very large heaps, you won't see much difference as full GC are uncommon. We still need to make the GC incremental to avoid pauses ...

...

On 16 Feb 2017, at 17:16, Clément Bera <bera.clement@gmail.com> wrote:

Hi all,

A temporary solution had been integrated in the VM a year ago, but only part of the problem was solved.

This issue has been entirely solved in the VM through the implementation of a new compactor for the full GC. The new compactor (called the planning compactor) has been marked as the default full GC compactor for further VM builds yesterday, replacing the existing Pig compactor. However it seems that the new compactor has not yet passed all the validation infrastructure (several builds are currently marked as 'error').

I would expect VM builds in the incoming days to feature the new compactor. This problem should therefore be solved very soon in the latest pharo VMs.

In any case, this problem does not require any image-side changes. It will be in Pharo 6 if the Pharo 6 release VM features the new compactor.

Best,

Clement

On Thu, Feb 16, 2017 at 4:26 PM, denker <marcus.denker@inria.fr> wrote:

...

On 16 Feb 2017, at 16:22, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 16 Feb 2017, at 16:16, denker <marcus.denker@inria.fr> wrote:

No,

Our image is not 47 MB… more like 28.

My understanding is: -> Cog works with “pages” (is that the term?) of memory. -> if a page is empty, it gets cleaned up and does not cost space when saving. -> if there is some object still allocated, the whole page will be saved (partly empty)

Not sure if that is completely right, but it explains why the image with lots of objects allocated new can shrink again, while the 47MB never shrinks, even though the image for sure is much smaller.

=> what is missing is a “cleanup” phase that sweeps together all the half-empty pages.

What is fun is that we now ship a 47MB image and nobody ever complains. (or even asks)

Yeah, that proves it is not such a big deal, today. But still, we have to fix this discrepancy between 28 and 47.

As I tried to explain: the VM is missing a “condenser” that cleans up half allocated pages.

...

Is the 'release cleanup' done on this image each time ?

Yes.

Marcus

On Fri, Feb 17, 2017 at 12:05 PM, Cyril Ferlicot D. < cyril.ferlicot@gmail.com> wrote:

Le 17/02/2017 à 07:17, Clément Bera a écrit :

...

The full GC is indeed much faster as the previous compaction algorithm was very slow. But unless you were executing memory intensive code in very large heaps, you won't see much difference as full GC are uncommon. We still need to make the GC incremental to avoid pauses ...

Can this fix the problem that MSE materialization takes 4 time longer with spur memory management? It would be really cool :)

It may solve part of the problem. Another problem is that tenure heuristics needs to be changed when allocating lots of long-lived objects. One needs to implement a GCPolicy object in the image and use it from the application code...

Do you have an idea of when the first incr GC without pause will be available? (No rush, it's just for info to know if it's in months or years :) )

No one is working on this right now, but it's next step to improve the GC. When the development starts on this feature I can give you more information. If a paying customer requests it, it should be done in a couple months. --

Cyril Ferlicot

http://www.synectique.eu

2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On Fri, Feb 17, 2017 at 1:48 PM, Cyril Ferlicot D. <cyril.ferlicot@gmail.com

wrote:

Le 17/02/2017 à 13:31, Clément Bera a écrit :

...

It may solve part of the problem.

Another problem is that tenure heuristics needs to be changed when allocating lots of long-lived objects. One needs to implement a GCPolicy object in the image and use it from the application code...

What would be needed in order to make this work? I don't know the GC enough to know what this imply.

Basically you need to have a look at the GC heuristics and analyse how they impact GC performance. Based on that, you can have a GCPolicy instance to which you can send messages such as #changePolicyForLargeHeap, #changePolicyForSmallHeaps or #growingPolicyDuring: aBlock. Internally the GCPolicy object changes GC heuristics. Some heuristics are already present in the vmParameter primitive (Smalltalk vm parameterAt: 46 for example), but some of them currently require the VM to be restarted. A first step would be to investigate those parameters. For example, the GCPolicy could read those parameters to know what is the current status: *2 size of young/new-space (read-only)* * 3 size of heap (read-only)* * 6 survivor count tenuring threshold (read-write)* * 7 full GCs since startup (read-only)* * 8 total milliseconds in full GCs since startup (read-only)* * 9 scavenges since startup (read-only)* * ...* * 55 ratio of growth and image size at or above which a GC will be performed post scavenge* * 63 total milliseconds in compiled code compactions since startup (read-only; Cog only; otherwise nil)* Based on this and the message sent to the GCPolicy, one can set new GC heuristics through those parameters: *6 survivor count tenuring threshold (read-write)* * ...* * 45 desired size of eden, in bytes (stored in image file header)* * 55 ratio of growth and image size at or above which a GC will be performed post scavenge* What has to be done is: - create a GCPolicy object with methods such as #changePolicyForLargeHeap, #changePolicyForSmallHeaps or #growingPolicyDuring: aBlock. - use the VM parameters in those methods to analyse the current state and define new heuristics - define a range in which the parameters can be changed without crashing the VM - add new parameters on heuristics that are important but not configurable right now - add the possibility to change several heuristics without restarting the image (for example change of Eden size or existing memory segment size) For example, let's say you grow your image from 30 Mb to 8Gb (I think one of our larger user in 64 bits use 8 Gb images right now and the default Pharo image is around 30 Mb). An uncommonly high rate of objects are promoted from young space to old space and the memory is expected to grow by 26667%. So you may want to: - decrease the survivor count for tenures. This way your objects are promoted earlier to old objects. - increase the ratio of growth. Instead of doing a full GC each time the image size grow by 33% you may want to do it each time the image size is increased by 900%. This way instead of having 20 fullGC while growing you have only 3. You switch back the policy at the end and trigger manually a fourth fullGC and everything's fine. Maybe we don't need a GCPolicy object, but more a VMPolicy object, so we can conveniently tweak the stack (number and size of stack page) and machine code zone. If somebody wants to have fun :-)

...

No one is working on this right now, but it's next step to improve the

GC.

...

When the development starts on this feature I can give you more information. If a paying customer requests it, it should be done in a couple months.

-- Cyril Ferlicot

http://www.synectique.eu

2 rue Jacques Prévert 01, 59650 Villeneuve d'ascq France

On 16 Feb 2017, at 16:16, denker <marcus.denker@inria.fr> wrote:

No,

Our image is not 47 MB… more like 28.

With the newest VM being used on the CI, the 60410 is now 35,7MB instead of 47,7MB. The .zip is down to 17 from 20. Marcus