is there a way to know when a GC is happening?

SD
Stéphane Ducasse
Wed, Sep 9, 2020 11:38 AM

Hi

I would to be able to see when an incremental GC is happening.
is there a way to know when a GC is happening?

S.

Stéphane Ducasse
http://stephane.ducasse.free.fr / http://www.pharo.org
03 59 35 87 52
Assistant: Aurore Dalle
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley,
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France

Hi I would to be able to see when an incremental GC is happening. is there a way to know when a GC is happening? S. -------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr / http://www.pharo.org 03 59 35 87 52 Assistant: Aurore Dalle FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
EM
Eliot Miranda
Thu, Sep 10, 2020 2:58 AM

On Wed, Sep 9, 2020 at 5:49 AM Stéphane Ducasse stephane.ducasse@inria.fr
wrote:

Hi

I would to be able to see when an incremental GC is happening.

You can't.  All GC activity happens while the mutator (the rest of the
Smalltalk VM) has stopped executing.  i.e. GC does not occur in parallel
with the rest of the VM.

is there a way to know when a GC is happening?

It is impossible, because effectively GCs happen between sends, and even
between bytecodes.  But the VM collects statistics that show how many GCs
of the various kinds have occurred and how much time has been sent in them
The statistics are made available through the vm parameterAt: primitives.
They are typically displayed in a SystemReporter dialog.  Here's an example
from a Squeak image:

Virtual Machine Statistics

uptime 2d 5h 9m 56s (runtime 10h 27m 28s, idletime 1d 18h 42m 28s)
memory 238,690,304 bytes
old 229,738,272 bytes (96.2%)
young 7,190,528 bytes (3%)
used 198,213,464 bytes (83%)
free 32,454,736 bytes (13.6%)
GCs 110,155 (1737.5 ms between GCs 341.8 ms runtime between GCs)
full 115 totalling 17,644 ms (0.05% runtime), avg 153.4 ms
marking 6,600 ms (37.4%) avg 57.4 ms,
compacting 11,044 ms (62.6%) avg 96 ms
scavenges 110,040 totalling 68,213 ms (0.18% runtime), avg 0.6 ms
tenures 82,870,154 (avg 753 tenures per scavenge)
Code compactions
699 totalling 625 ms (0.002% runtime), avg 0.9 ms

These times were collected on a 2.9 GHz Intel Core i9 MacBook Pro.

A different question is can one be informed when a GC just happened?  The
current answer is no, but it would be very easy to add.  In VisualWorks,
for example, a WeakArray is primed with an Object instance, and this gets
collected every scavenge.  So the WeakArray is notified.  From this VW
builds a notification system.  I would engineer it differently.  If there
is a Semaphore in either of two slots in the specialObjectsArray the VM
could signal one Semaphore whenever a scavenge occurs and signal the other
when a full GC occurs.

P.S>  can we please stop using the term "incremental GC" for a young space
collection?  In Spur the young space GC is a classic Ungar-style scavenger,
so I like to call these GCs scavenges.  Soon I hope that the Spur VM will
have a proper incremental global GC. This will comprise a mark-sweep
collector and a compactor fort old space, but both will operate
incrementally.  The mark phase will be done in increments, allowing very
small pause times as marking proceeds.  Likewise sweeping to free objects
and especially compacting, to coalesce storage, will be done in small
increments.  So there will be pause times of the order of 2 or 3
millisecondes due to the incremental collector and overall the system will
avoid the relatively long pauses the global GC causes.

HTH
,,,^..^,,,
best, Eliot

On Wed, Sep 9, 2020 at 5:49 AM Stéphane Ducasse <stephane.ducasse@inria.fr> wrote: > > Hi > > I would to be able to see when an incremental GC is happening. You can't. All GC activity happens while the mutator (the rest of the Smalltalk VM) has stopped executing. i.e. GC does not occur in parallel with the rest of the VM. > is there a way to know when a GC is happening? It is impossible, because effectively GCs happen between sends, and even between bytecodes. But the VM collects statistics that show how many GCs of the various kinds have occurred and how much time has been sent in them The statistics are made available through the vm parameterAt: primitives. They are typically displayed in a SystemReporter dialog. Here's an example from a Squeak image: Virtual Machine Statistics -------------------------- uptime 2d 5h 9m 56s (runtime 10h 27m 28s, idletime 1d 18h 42m 28s) memory 238,690,304 bytes old 229,738,272 bytes (96.2%) young 7,190,528 bytes (3%) used 198,213,464 bytes (83%) free 32,454,736 bytes (13.6%) GCs 110,155 (1737.5 ms between GCs 341.8 ms runtime between GCs) full 115 totalling 17,644 ms (0.05% runtime), avg 153.4 ms marking 6,600 ms (37.4%) avg 57.4 ms, compacting 11,044 ms (62.6%) avg 96 ms scavenges 110,040 totalling 68,213 ms (0.18% runtime), avg 0.6 ms tenures 82,870,154 (avg 753 tenures per scavenge) Code compactions 699 totalling 625 ms (0.002% runtime), avg 0.9 ms These times were collected on a 2.9 GHz Intel Core i9 MacBook Pro. A different question is can one be informed when a GC just happened? The current answer is no, but it would be very easy to add. In VisualWorks, for example, a WeakArray is primed with an Object instance, and this gets collected every scavenge. So the WeakArray is notified. From this VW builds a notification system. I would engineer it differently. If there is a Semaphore in either of two slots in the specialObjectsArray the VM could signal one Semaphore whenever a scavenge occurs and signal the other when a full GC occurs. P.S> can we please stop using the term "incremental GC" for a young space collection? In Spur the young space GC is a classic Ungar-style scavenger, so I like to call these GCs scavenges. Soon I hope that the Spur VM will have a proper incremental global GC. This will comprise a mark-sweep collector and a compactor fort old space, but both will operate incrementally. The mark phase will be done in increments, allowing very small pause times as marking proceeds. Likewise sweeping to free objects and especially compacting, to coalesce storage, will be done in small increments. So there will be pause times of the order of 2 or 3 millisecondes due to the incremental collector and overall the system will avoid the relatively long pauses the global GC causes. HTH _,,,^..^,,,_ best, Eliot
EM
Esteban Maringolo
Thu, Sep 10, 2020 3:43 AM

On Wed, Sep 9, 2020 at 11:58 PM Eliot Miranda eliot.miranda@gmail.com wrote:

In VisualWorks, for example, a WeakArray is primed with an Object instance, and this gets collected every scavenge.
So the WeakArray is notified.  From this VW builds a notification system.

I guess this is how VisualWorks changes its cursor from a regular
pointer to the, sometimes dreadful, "GC" icon during garbage
collection.
Isn't it?

Regards!

Esteban A. Maringolo

On Wed, Sep 9, 2020 at 11:58 PM Eliot Miranda <eliot.miranda@gmail.com> wrote: > In VisualWorks, for example, a WeakArray is primed with an Object instance, and this gets collected every scavenge. > So the WeakArray is notified. From this VW builds a notification system. I guess this is how VisualWorks changes its cursor from a regular pointer to the, sometimes dreadful, "GC" icon during garbage collection. Isn't it? Regards! Esteban A. Maringolo
EM
Eliot Miranda
Thu, Sep 10, 2020 5:02 AM

On Wed, Sep 9, 2020 at 9:49 PM Esteban Maringolo emaringolo@gmail.com
wrote:

On Wed, Sep 9, 2020 at 11:58 PM Eliot Miranda eliot.miranda@gmail.com
wrote:

In VisualWorks, for example, a WeakArray is primed with an Object

instance, and this gets collected every scavenge.

So the WeakArray is notified.  From this VW builds a notification system.

I guess this is how VisualWorks changes its cursor from a regular
pointer to the, sometimes dreadful, "GC" icon during garbage
collection.
Isn't it?

I think so :-)

Regards!

Esteban A. Maringolo

and to you, Esteban!

,,,^..^,,,
best, Eliot

On Wed, Sep 9, 2020 at 9:49 PM Esteban Maringolo <emaringolo@gmail.com> wrote: > On Wed, Sep 9, 2020 at 11:58 PM Eliot Miranda <eliot.miranda@gmail.com> > wrote: > > > In VisualWorks, for example, a WeakArray is primed with an Object > instance, and this gets collected every scavenge. > > So the WeakArray is notified. From this VW builds a notification system. > > I guess this is how VisualWorks changes its cursor from a regular > pointer to the, sometimes dreadful, "GC" icon during garbage > collection. > Isn't it? > I think so :-) > > Regards! > > Esteban A. Maringolo > and to you, Esteban! _,,,^..^,,,_ best, Eliot
MM
Martin McClure
Thu, Sep 10, 2020 6:53 AM

On 9/9/20 8:43 PM, Esteban Maringolo wrote:

On Wed, Sep 9, 2020 at 11:58 PM Eliot Miranda eliot.miranda@gmail.com wrote:

In VisualWorks, for example, a WeakArray is primed with an Object instance, and this gets collected every scavenge.
So the WeakArray is notified.  From this VW builds a notification system.

I guess this is how VisualWorks changes its cursor from a regular
pointer to the, sometimes dreadful, "GC" icon during garbage
collection.
Isn't it?

No, that wouldn't work. Notification based on a weak array can only
notify after the scavenge (or other GC) has happened, at which point
it's too late to put up the GC cursor.

And a weak array doesn't seem like it could be reliable for notification
of scavenge -- what if the Object in the weak array gets tenured? Then
only an incremental or full GC could collect it and notify. Scavenge
notification is done by the VM signaling a semaphore that a scavenge
notification thread is waiting on.

And the GC cursor is simple -- it's explicitly set by Smalltalk code
before a compacting GC, and set back afterward.

Regards,
-Martin

On 9/9/20 8:43 PM, Esteban Maringolo wrote: > On Wed, Sep 9, 2020 at 11:58 PM Eliot Miranda <eliot.miranda@gmail.com> wrote: > >> In VisualWorks, for example, a WeakArray is primed with an Object instance, and this gets collected every scavenge. >> So the WeakArray is notified. From this VW builds a notification system. > I guess this is how VisualWorks changes its cursor from a regular > pointer to the, sometimes dreadful, "GC" icon during garbage > collection. > Isn't it? > No, that wouldn't work. Notification based on a weak array can only notify after the scavenge (or other GC) has happened, at which point it's too late to put up the GC cursor. And a weak array doesn't seem like it could be reliable for notification of scavenge -- what if the Object in the weak array gets tenured? Then only an incremental or full GC could collect it and notify. Scavenge notification is done by the VM signaling a semaphore that a scavenge notification thread is waiting on. And the GC cursor is simple -- it's explicitly set by Smalltalk code before a compacting GC, and set back afterward. Regards, -Martin
DG
Davide Grandi
Thu, Sep 10, 2020 7:21 AM

If Pharo's gc algorithm is concurrent => maybe yes,
otherwise surely no : if I'm not wrong gc starts inside VM
when checking allocation spaces during a basicNew or alike.
And the VM seems stuck, at that moment.

One can try to create an ephemeron and check it every 100-500 ms,
refreshing it if it's still here (so it will stay in new space).
But it shift the question to the definitive target group :
the VM guys !!!

Chers from Lombardy,
    Davide
(PS : but creating the ephemeron could start an invisible gc !!!)

On 09/09/2020 13:38, Stéphane Ducasse wrote:

Hi

I would to be able to see when an incremental GC is happening.
is there a way to know when a GC is happening?

S.

Stéphane Ducasse
http://stephane.ducasse.free.fr/ http://www.pharo.org
03 59 35 87 52
Assistant: Aurore Dalle
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley,
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France

--
Ing. Davide Grandi
email    : davide.grandi@email.it
mobile  : +39 339 7468 778
linkedin : http://linkedin.com/in/davidegrandi

If Pharo's gc algorithm is concurrent => maybe yes, otherwise surely no : if I'm not wrong gc starts inside VM when checking allocation spaces during a basicNew or alike. And the VM seems stuck, at that moment. One can try to create an ephemeron and check it every 100-500 ms, refreshing it if it's still here (so it will stay in new space). But it shift the question to the definitive target group : the VM guys !!! Chers from Lombardy,     Davide (PS : but creating the ephemeron could start an invisible gc !!!) On 09/09/2020 13:38, Stéphane Ducasse wrote: > Hi > > I would to be able to see when an incremental GC is happening. > is there a way to know when a GC is happening? > > S. > -------------------------------------------- > Stéphane Ducasse > http://stephane.ducasse.free.fr/ http://www.pharo.org > 03 59 35 87 52 > Assistant: Aurore Dalle > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley, > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France > -- Ing. Davide Grandi email : davide.grandi@email.it mobile : +39 339 7468 778 linkedin : http://linkedin.com/in/davidegrandi
AS
Aliaksei Syrel
Thu, Sep 10, 2020 8:41 AM

Hi

I wish there was a way to control when full GC is allowed to happen. For
example, I would prefer GC to occur while the UI rendering loop is paused
and not in the middle of the frame resulting in sometimes sluggish
animations in the case of large images.

Thank you

On Wed, 9 Sep 2020 at 14:39, Stéphane Ducasse stephane.ducasse@inria.fr
wrote:

Hi

I would to be able to see when an incremental GC is happening.
is there a way to know when a GC is happening?

S.


Stéphane Ducasse
http://stephane.ducasse.free.fr / http://www.pharo.org
03 59 35 87 52
Assistant: Aurore Dalle
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley
https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g
,
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France

--

Cheers,
Alex

Hi I wish there was a way to control *when* full GC is allowed to happen. For example, I would prefer GC to occur while the UI rendering loop is paused and not in the middle of the frame resulting in sometimes sluggish animations in the case of large images. Thank you On Wed, 9 Sep 2020 at 14:39, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote: > Hi > > I would to be able to see when an incremental GC is happening. > is there a way to know when a GC is happening? > > S. > > > -------------------------------------------- > Stéphane Ducasse > http://stephane.ducasse.free.fr / http://www.pharo.org > 03 59 35 87 52 > Assistant: Aurore Dalle > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley > <https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g> > , > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France > > > > > > > > -- Cheers, Alex
DG
Davide Grandi
Thu, Sep 10, 2020 9:13 AM

Maybe it's undecidable.
Now it happens when a memory space is full, and I
think that memory spaces aren't dynamically configurable.

    Davide

On 10/09/2020 10:41, Aliaksei Syrel wrote:

Hi

I wish there was a way to control when full GC is allowed to happen.
For example, I would prefer GC to occur while the UI rendering loop is
paused and not in the middle of the frame resulting in sometimes
sluggish animations in the case of large images.

Thank you

On Wed, 9 Sep 2020 at 14:39, Stéphane Ducasse
<stephane.ducasse@inria.fr mailto:stephane.ducasse@inria.fr> wrote:

 Hi

 I would to be able to see when an incremental GC is happening.
 is there a way to know when a GC is happening?

 S.

 --------------------------------------------
 Stéphane Ducasse
 http://stephane.ducasse.free.fr/ http://www.pharo.org
 03 59 35 87 52
 Assistant: Aurore Dalle
 FAX 03 59 57 78 50
 TEL 03 59 35 86 16
 S. Ducasse - Inria
 40, avenue Halley
 <https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g>,

 Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
 Villeneuve d'Ascq 59650
 France

--
Cheers,
Alex

--
Ing. Davide Grandi
email    : davide.grandi@email.it
mobile  : +39 339 7468 778
linkedin : http://linkedin.com/in/davidegrandi

Maybe it's undecidable. Now it happens when a memory space is full, and I think that memory spaces aren't dynamically configurable.     Davide On 10/09/2020 10:41, Aliaksei Syrel wrote: > Hi > > I wish there was a way to control *when* full GC is allowed to happen. > For example, I would prefer GC to occur while the UI rendering loop is > paused and not in the middle of the frame resulting in sometimes > sluggish animations in the case of large images. > > Thank you > > On Wed, 9 Sep 2020 at 14:39, Stéphane Ducasse > <stephane.ducasse@inria.fr <mailto:stephane.ducasse@inria.fr>> wrote: > > Hi > > I would to be able to see when an incremental GC is happening. > is there a way to know when a GC is happening? > > S. > > -------------------------------------------- > Stéphane Ducasse > http://stephane.ducasse.free.fr/ http://www.pharo.org > 03 59 35 87 52 > Assistant: Aurore Dalle > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley > <https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g>, > > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France > > -- > Cheers, > Alex -- Ing. Davide Grandi email : davide.grandi@email.it mobile : +39 339 7468 778 linkedin : http://linkedin.com/in/davidegrandi
MM
Martin McClure
Fri, Sep 11, 2020 3:25 AM

Hi Aliaksei,

In VW, scavenges are done automatically by the VM, but all
heavier-weight GC is controlled by the image, through a MemoryPolicy
object. In the GemStone client for VW we provide a memory policy that
discourages GC during times when our mapping dictionaries are strong,
deferring it to just after we make the dictionaries weak so the GC can
actually collect things that are mapped but no longer used.

It sounds like you'd like something very much like this, to tend to
defer GC to a good point in the rendering cycle. And "tend to" is
important -- when space is getting really low, it's a good idea to go
ahead and do the GC regardless of whether it's a "good" time.

Regards,
-Martin

On 9/10/20 1:41 AM, Aliaksei Syrel wrote:

Hi

I wish there was a way to control when full GC is allowed to happen.
For example, I would prefer GC to occur while the UI rendering loop is
paused and not in the middle of the frame resulting in sometimes
sluggish animations in the case of large images.

Thank you

On Wed, 9 Sep 2020 at 14:39, Stéphane Ducasse
<stephane.ducasse@inria.fr mailto:stephane.ducasse@inria.fr> wrote:

 Hi

 I would to be able to see when an incremental GC is happening.
 is there a way to know when a GC is happening?

 S.


 --------------------------------------------
 Stéphane Ducasse
 http://stephane.ducasse.free.fr / http://www.pharo.org
 03 59 35 87 52
 Assistant: Aurore Dalle
 FAX 03 59 57 78 50
 TEL 03 59 35 86 16
 S. Ducasse - Inria
 40, avenue Halley
 <https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g>,

 Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
 Villeneuve d'Ascq 59650
 France

--
Cheers,
Alex

Hi Aliaksei, In VW, scavenges are done automatically by the VM, but all heavier-weight GC is controlled by the image, through a MemoryPolicy object. In the GemStone client for VW we provide a memory policy that discourages GC during times when our mapping dictionaries are strong, deferring it to just after we make the dictionaries weak so the GC can actually collect things that are mapped but no longer used. It sounds like you'd like something very much like this, to tend to defer GC to a good point in the rendering cycle. And "tend to" is important -- when space is getting *really* low, it's a good idea to go ahead and do the GC regardless of whether it's a "good" time. Regards, -Martin On 9/10/20 1:41 AM, Aliaksei Syrel wrote: > Hi > > I wish there was a way to control *when* full GC is allowed to happen. > For example, I would prefer GC to occur while the UI rendering loop is > paused and not in the middle of the frame resulting in sometimes > sluggish animations in the case of large images. > > Thank you > > On Wed, 9 Sep 2020 at 14:39, Stéphane Ducasse > <stephane.ducasse@inria.fr <mailto:stephane.ducasse@inria.fr>> wrote: > > Hi > > I would to be able to see when an incremental GC is happening. > is there a way to know when a GC is happening? > > S. > > > -------------------------------------------- > Stéphane Ducasse > http://stephane.ducasse.free.fr / http://www.pharo.org > 03 59 35 87 52 > Assistant: Aurore Dalle > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley > <https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g>, > > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France > > > > > > > > -- > Cheers, > Alex