Hi Guille, On Wed, Jun 8, 2016 at 12:36 AM, Guille Polito <guillermopolito@gmail.com> wrote:

Hi Eliot,

Pharo does AFAIK the same with the source files when you're navigating source code.

Now, I remember that while adapting the finalization scheme that you sent me to Pharo (because there are indeed subtle differences), I noticed that there was a missing loop.

[looking for the code... found!]

finalizationProcess "The finalization process arranges to send mourn to each element of the VM's finalization queue, which is accessed via primitiveFetchMourner. The VM signals FinalizationSemaphore whenever the queue is non-empty. This process loops, waiting on the semaphore, fetches the first element of the queue and then spawns a process at a higher priority to acually send the mourn messages. If an error occurs in the higher priority mourn loop process then this process will simply spawn another process, hence ensuring that errors in finalization methods don't break finalization.

In addition this process also runs the old finalization scheme, supporting clients of the older, WeakRegistry based scheme. Hopefully this will go away when all cleints have moved over." | throttle firstMourner | throttle := Semaphore new. [FinalizationSemaphore wait; initSignals. "Support the old scheme until things have changed over..." self doOldFinalization. [firstMourner := self primitiveFetchMourner. firstMourner notNil] whileTrue: [[throttle signal. self mournLoopWith: firstMourner] forkAt: Processor activePriority + 1. throttle wait]]

Doh! So I wrote a naked block, not a loop !! So it did nothing. Thank you!! I think the compiler should warn about naked blocks. At first I was using that code that you sent me and I noticed that the

finalization process in there is a loop that is never evaluated! So I updated it to the following using a [true] whileTrue:

finalizationProcess "The finalization process arranges to send mourn to each element of the VM's finalization queue, which is accessed via primitiveFetchMourner. The VM signals FinalizationSemaphore whenever the queue is non-empty. This process loops, waiting on the semaphore, fetches the first element of the queue and then spawns a process at a higher priority to acually send the mourn messages. If an error occurs in the higher priority mourn loop process then this process will simply spawn another process, hence ensuring that errors in finalization methods don't break finalization.

In addition this process also runs the old finalization scheme, supporting clients of the older, WeakRegistry based scheme. Hopefully this will go away when all cleints have moved over." | throttle firstMourner | throttle := Semaphore new. [true] whileTrue: [FinalizationSemaphore wait; initSignals. "Support the old scheme until things have changed over..." self doOldFinalization. [firstMourner := self primitiveFetchMourner. firstMourner notNil] whileTrue: [[throttle signal. self mournLoopWith: firstMourner] forkAt: Processor activePriority + 1. throttle wait]]

Maybe that's the reason of weak arrays not being finalized in squeak?

Guille

-------- Original Message --------

Hi Guille,

good news! But I'm seeing something wrong with finalisation of weak arrays using the new scheme. In Squeak method source access is done by default by opening a new read-only file for each method's source read (crazy, but that's not the issue). So by default something that accesses lots of source ends up running out of file descriptors, which causes primOpen:writable: to fail. The surrounding code then uses retryWithGC:until:forFileNamed: to do a GC to try and reclaim non-longer referenced files, close file descriptors and continue:

StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName.

But in my tests I'm not seeing any files reclaimed. I wonder whether the new finalisation code is failing to finalise weak arrays properly. I wonder whether your weak tests work properly with the new scheme or not.

Anyway, I think I can reproduce the pathology in the simulator if I modify it to implement a small limit on the number of file descriptors. I'm going to try that to shed light on the problem. I can't easily debug in a running image because...I run out of file descriptors ;-)

On Tue, Jun 7, 2016 at 2:32 AM, Guille Polito <guillermopolito@gmail.com> wrote:

...

Hi All,

Since this morning, in Pharo #60065, Ephemeron support is in the image. Most of the changes are infrastructural, so far transparent for the users. It is important to notice that even while the support is there, it is not enabled by default. Also, this required changes in the virtual machine that are not yet distributed everywhere. For the ones that would like more detail, I invite you to read the following :)

* On the infrastructure side

- There is support to create Ephemeric classes and load them from monticello

- There is a new finalization mechanism (by default disabled) that will process Ephemerons using a finalization queue. This will avoid scanning collections in look for weak objects to finalize ,as it happens now with the WeakDependent mechanism in WeakArray.

- System-Finalization features two new classes *Ephemeron* and *EphemeronRegistry*. For the ones that want more details on Ephemerons, you can read the associated paper [1], or the class comment of Ephemeron:

I represent ephemeric key-value objects. Ephemerons are key-value objects (subclasses of Association) with special semantics during garbage collection. My special behavior can resumed as follows:

- The garbage collection will iterate my instances only if the key is not referenced strongly by another object.

- Then, if no strong references to the key are found, then the values of this ephemeron are hold weakly.

- Otherwise, the values are hold strongly.

In this implementation, an Ephemeron can hold more than one value, which are all treated in the same manner. This ephemeron instance knows its container, which allows the ephemeron to remove itself from a container (such as a Dictionary) upon finalization.

!! Example usages

In general terms, do not use myself directly. Use instead an Ephemeric container like EphemeronRegistry. An Ephemeron registry will guarantee the collection of keys and values of the object inside the Ephemeron.

Otherwise, if you want to use it, you can create an Ephemeron as any association:

ephemeron := Ephemeron key: aKey value: aValue.

ephemeron container: aContainer.

!! Ephemeron Finalization

When an ephemeron's key is hold strongly just by the ephemeron itself, the Ephemeron will be mourned (finalized). That means that the VM will:

- put the Ephemeron in the mourning queue waiting for the image to take care of mourning

- make the Ephemeron non ephemeric. That is, the ephemeron instance cannot be reused.

On the image side, the finalization process will send the message #mourn to an Ephemeron. #mourn will #finalize the Ephemeron's key, and remove the Ephemeron from it's container to allow its collection during a subsequent garbage collection.

!! More Documentation

You can read the associated paper to understand better the semantics of ephemerons:

[1]Ephemerons: A New Finalization Mechanism. Barry Hayes. OOPSLA '97

- WARNING: to be able to use ephemerons, you need to use the *latestVm* that has several fixes for making ephemerons work, and you need to enable ephemerons on the image side by evaluating:

Smalltalk supportsQueueingFinalization: true.

- With latest vm and ephemerons enabled, tests should be green, otherwise they are skipped

* From the user point of view:

- The Weak registries were not yet migrated to the new finalization mechanism.

- We expect nothing will change from the user point of view. Just less memory leaks.

* Next steps (in order)

1) Bless the latest vm as stable

2) Enable queueing finalization by default

3) Replace Weak Registry by Ephemeron Registry.

Informed by: Guille

-- _,,,^..^,,,_ best, Eliot

-- _,,,^..^,,,_ best, Eliot

Maybe I will find some time to hack the tree-matching rules to highlight the matched piece of code. It’s so frustrating that they don’t do it… Uko

On 30 Jun 2016, at 10:17, Marcus Denker <marcus.denker@inria.fr> wrote:

finalizationProcess "The finalization process arranges to send mourn to each element of the VM's finalization queue, which is accessed via primitiveFetchMourner. The VM signals FinalizationSemaphore whenever the queue is non-empty. This process loops, waiting on the semaphore, fetches the first element of the queue and then spawns a process at a higher priority to acually send the mourn messages. If an error occurs in the higher priority mourn loop process then this process will simply spawn another process, hence ensuring that errors in finalization methods don't break finalization.

In addition this process also runs the old finalization scheme, supporting clients of the older, WeakRegistry based scheme. Hopefully this will go away when all cleints have moved over." | throttle firstMourner | throttle := Semaphore new. [FinalizationSemaphore wait; initSignals. "Support the old scheme until things have changed over..." self doOldFinalization. [firstMourner := self primitiveFetchMourner. firstMourner notNil] whileTrue: [[throttle signal. self mournLoopWith: firstMourner] forkAt: Processor activePriority + 1. throttle wait]]

Doh! So I wrote a naked block, not a loop !! So it did nothing. Thank you!! I think the compiler should warn about naked blocks.

I added a new Code-Critic rule "RBDeadBlockRule", with

initialize super initialize. self matcher matches: '`{:node | node isBlock and: [node parent isSequence and: [ node isLastStatementInBlock not ]]}' do: [ :node :answer | node ]

This now shows a warning in the tools (the explanation is shown when clicking on the ? button): It does not yet highlight the block and has no automatic transformation to fix the problem, but as a first step it should be nice to have.

<Dead.jpg>

This is in 60 update 126.

(I did not read the mail again before doing it, so now it is called "dead", not "naked", we can change that... dead somehow makes it look to be related a "dead context"...).

Marcus

On 30 Jun 2016, at 14:57, John Brant <brant@refactoryworkers.com> wrote:

On 06/30/2016 03:17 AM, Marcus Denker wrote:

...

I added a new Code-Critic rule "RBDeadBlockRule", with

initialize super initialize. self matcher matches: '`{:node | node isBlock and: [node parent isSequence and: [ node isLastStatementInBlock not ]]}' do: [ :node :answer | node ]

Instead of "node parent isSequence and: [ node isLastStatementInBlock not ]", you can use "node isUsed not".

Nice! I have submitted a patch to change it. Marcus