Inline below... Cheers, Alistair (on phone) On Wed., 18 Sep. 2019, 15:05 Guillermo Polito, <guillermopolito@gmail.com> wrote:

About bug1) I’ve found that Calypso query scopes use sometimes #methods and sometimes #localMethods

ClyLocalClassScope >> methodsDo: aBlock

self classesDo: [ :eachClass | self metaLevelsOf: eachClass do: [ :concreteMetaLevelClass | concreteMetaLevelClass methods do: aBlock ] ]

ClyClassHierarchyScope >> methodsDo: aBlock

self classesDo: [ :eachClass | self metaLevelsOf: eachClass do: [ :concreteMetaLevelClass | concreteMetaLevelClass methods do: aBlock ] ]

ClyPackageScope >> methodsDo: aBlock self classesDo: [:class | class localMethods do: aBlock. class classSide localMethods do: aBlock].

self packagesDo: [ :package | package extensionMethods do: aBlock ]

ClySystemEnvironmentScope >> methodsDo: aBlock

self classesDo: [:class | class instanceSide localMethods do: aBlock. class classSide localMethods do: aBlock]

We should get an homogeneous version of this.

Pablo is checking the UFFI compilation now :)

Right. I wrote that it shouldn't matter whether methodDict or local methods is used, but I agree that there should be a single accessor that is used everywhere. Thanks! Alistair

...

El 18 sept 2019, a las 14:20, Alistair Grant <akgrant0710@gmail.com> escribió:

P.S. Two implications of this:

1. The code you're looking at may not be the code that is executed. If in doubt, use the standard System Browser (not senders, implementers, etc.). 2. Iceberg may not be loading extension methods that override an existing method correctly, so the old version is left as the executable code. There may be happening in other situations as well, I haven't checked.

Thanks, Alistair

On Wed, 18 Sep 2019 at 11:04, Alistair Grant <akgrant0710@gmail.com> wrote:

...

Hi Everyone,

TraitedMetaclass stores methods in two different instance variables:

- methodDict (inherited from Behavior) - localMethods

And the same method can be different in each dictionary.

Different browsers show different versions, i.e.:

- The System Browser uses the method in methodDict. - Calypso's "Senders of" browser uses localMethods. - The Iceberg versions browser uses localMethods.

It appears that the method in methodDict is the one that is actually used when a message is sent, but it would be nice to get confirmation.

To reproduce this in a clean Pharo 8 image:

- Open the System Browser on LGitBuf class>>signature_free: (the methodDict version). - Open the system browser on TLGitCalloutTrait>>call:options: and then press "Senders". The resulting window will include LGitBuf class>>signature_free:, but the localMethods version.

In this example, #signature_free: isn't defined by the trait, and isn't overwritten, so I would expect that the method in both dictionaries should be the same or it should only be in one dictionary.

Can someone say what the intended behaviour is?

Thanks, Alistair

Hi Guille, Replies inline... On Wed, 18 Sep 2019 at 14:44, Guillermo Polito <guillermopolito@gmail.com> wrote:

Hi Alistair,

we are checking this with Pablo here, some questions inline

...

El 18 sept 2019, a las 11:04, Alistair Grant <akgrant0710@gmail.com> escribió:

Hi Everyone,

TraitedMetaclass stores methods in two different instance variables:

- methodDict (inherited from Behavior) - localMethods

And the same method can be different in each dictionary.

Yes, trais actually work by flattening the methods of traits into the user class. - localMethods defines the methods actually defined in the class - methodDict has the raw low-level flattened version the VM uses for the lookup

So methodDict and localMethods should always contain the same instance of CompiledMethod for a given selector, right?

...

Different browsers show different versions, i.e.:

- The System Browser uses the method in methodDict.

Which is this System browser? Calypso system browser?

Yep, as in World Menu -> Tools -> System Browser

...

- Calypso's "Senders of" browser uses localMethods.

and this is Calypso senders, right?

Yes, pressing the "Senders" button from the System Browser (Calypso).

...

- The Iceberg versions browser uses localMethods.

It appears that the method in methodDict is the one that is actually used when a message is sent, but it would be nice to get confirmation.

To reproduce this in a clean Pharo 8 image:

- Open the System Browser on LGitBuf class>>signature_free: (the methodDict version). - Open the system browser on TLGitCalloutTrait>>call:options: and then press "Senders". The resulting window will include LGitBuf class>>signature_free:, but the localMethods version.

In this example, #signature_free: isn't defined by the trait, and isn't overwritten, so I would expect that the method in both dictionaries should be the same or it should only be in one dictionary.

Can someone say what the intended behaviour is?

We are tracking this, it seems you have found two bugs :D.

1st bug) Calypso and the senders/implementors should be in sync and show the same (always the local methods for traited classes)

From what you've written above (and below), the two dictionaries should always have the same instance of CompiledMethod so it shouldn't matter where it comes from.

Moreover, (I’m confirming this with Pablo on line right now) the local methods dictionary should be just a subset of the method dictionary. In other words, the invariant is that methods in localMethods should be included in the methoddict, they should not differ as they are differing there.

2nd bug) UFFI? seems to be messing with this invariant :)

When a UFFI binding is executed for the very first time, the ffi call definition gets compiled and a new compiled method is generated performing that call. When a session finishes (the image stops/restarts) ffi bindings are reset (because their compilation is platform dependent).

What we are seeing there is that somehow UFFI managed to reset a binding method and put a wrong one in the local method

I’ve seen this bug before but never got the insight of what was producing it (neither put too much time to dig on it either :P). Another possibility is that the culprit are the LibGit bindings, which extend UFFI binding compilation.

I haven't looked at this, but what I saw was the localMethods was correct and the methodDict was old: Redefine a base system method (LGitCommit>>commit_author:) using an extension method resulted in methodDict containing the original definition and the localMethods containing the overridden method. What I would expect is that the overridden method is what is used everywhere. I'm not sure if this is related to UFFI or not. I'm on Discord if you'd like to chat real-time (back around 15:25). Thanks! Alistair

Hi Guille, On Wed, 18 Sep 2019 at 15:22, Guillermo Polito <guillermopolito@gmail.com> wrote:

El 18 sept 2019, a las 15:05, Alistair Grant <akgrant0710@gmail.com> escribió:

[snip/]

From what you've written above (and below), the two dictionaries should always have the same instance of CompiledMethod so it shouldn't matter where it comes from.

Just for the record, yes and no :).

With the new trait implementation, we have actually a modular MOP. All classes (whether traited or not) understand: - #localMethods => returning the methods as they are implemented in the class - #methods => returning the actual methods executed by the VM

For normal classes, #localMethods is defined as follows:

localMethods ^ self methods

And methods understand - #origin => where that method is actually defined For example, in a traited class a lookup should be done in the trait composition, otherwise is the class where the method is installed.

Tools should work against #localMethods and #origin, which hide how the actual implementation stores the things :). We already had in the past a lot of tools that hardcoded/duplicated the internals of Traits for example.

The other advantage of this is that, imagine you want to extend Pharo with: - multimethods => localMethods will have a single multi-method implementation, and the implementation side will contain maybe an expanded version with type checks or even one method per type combination - optional parameters => a single method with many parameters should be expanded to many methods with less parameters + default values

The idea is that tools do not hardcode the implementation details of these extensions, so the public API is #localMethods and #origin :) Of course, one could design a tool showing #methods, which will show the low-level view of classes.

Thanks for the clarification, that makes sense and will help with tracking this further.

...

I haven't looked at this, but what I saw was the localMethods was correct and the methodDict was old:

Redefine a base system method (LGitCommit>>commit_author:) using an extension method resulted in methodDict containing the original definition and the localMethods containing the overridden method. What I would expect is that the overridden method is what is used everywhere.

I'm not sure if this is related to UFFI or not.

Pablo tracked this down to possibly

FFIMethodRegistry >> removeMethod: aMethod

aMethod methodClass methodDict at: aMethod selector put: (aMethod propertyValueAt: #ffiNonCompiledMethod)

That does look bad :-) Using the test class included below: MethodDictionaryConsistencyTest new inconsistentMethodDefinitions answers a collection of 240 methods where methodDict and localMethods aren't the same instance. All of the methods class names begin with 'LGit' :-) There may still be one more bug involved since the scenario I mentioned above regarding Iceberg and extension methods doesn't seem to fit the FFI description (it calls an ffi method, but isn't one itself). Thanks, Alistair