Hi Pierce, I can't determine from DemoFFI how the autorelease works. I see memory is allocated by the C library here.... DemoFFIAutoThing>>allocByReturnedPointer super allocByReturnedPointer. handle autoRelease where super is... DemoFFIThing>>allocByReturnedPointer ((handle := DemoFFILibrary uniqueInstance apiAllocByReturnedPointer) isNull) ifTrue: [ DemoFFIError signal: 'bummer' ] and... DemoFFILibrary>>apiAllocByReturnedPointer "demo_thing* alloc_by_returned_pointer(void)" ^ self ffiCall: #(DemoFFIExternalObject alloc_by_returned_pointer ()) which I guess is to be released by... DemoFFILibrary>>apiFree: handle "int free_thing(demo_thing*)" ^ self ffiCall: #(int free_thing (DemoFFIExternalObject handle)) but this has only one sender... DemoFFIThing>>letGo handle isNull ifFalse: [ DemoFFILibrary uniqueInstance apiFree: handle ] and the only sender of #letGo is... DEMOFFIThingTest >> tearDown

From this I can't follow how #apiFree: ultimately gets automatically invoked by the object finalization. Are you able to fill me in on the details?

cheers -ben On Sun, Sep 4, 2016 at 10:40 PM, Ben Coman <btc@openinworld.com> wrote:

On Fri, Sep 2, 2016 at 10:04 PM, Pierce Ng <pierce@samadhiweb.com> wrote:

...

On Fri, Sep 02, 2016 at 12:40:06PM +0800, Ben Coman wrote:

...

Referring to the example show_clang_version() below, cleanup is done with clang_disposeString(). How does UFFI hook into the object finalisation mechanism to call this when the object is garage collected.

Ben, see if this addresses your query:

http://www.samadhiweb.com/blog/2016.03.12.demoffi.html

Pierce

Thanks Pierce. I've loaded DemoFFI and browsing it now. autorelease looks like part of what I need. I guess it is #finalize where I should call the libraryAPI_free() function?

cheers -ben

On Tue, Sep 6, 2016 at 8:08 PM, Esteban Lorenzano <estebanlm@gmail.com> wrote:

Hi,

sorry for arriving so late to this, but I was on holidays :) this is how autoRelease works:

1) #autoRelease of an object registers object for finalisation with a particular executor. Then behaviour is divided:

2.1.1) for ExternalAddresses, it just registers in regular way, who will call #finalize on GC 2.1.2) finalize will just call a free assuming ExternalAddress was allocated (which is a malloc)

2.2.1) for all FFIExternalReference, it will register for finalisation what #resourceData answers (normally, the handle of the object) 2.2.2) finalisation process will call the object class>>#finalizeResourceData: method, with the #resourceData result as parameter 2.2.3) each kind of external reference can decide how to free that data (by default is also just freeing).

An example of this is how CairoFontFace works (or AthensCairoSurface).

In you case, you will have something like:

FFIExternalObject subclass: #CXString

Thanks Esteban for the detailed description. However I've come to the conclusion that the C-library dynamically allocating memory for a field inside the struct of CXString doesn't exactly fit any of the pre-existing FFI types. When CXString subclasses FFIExternalObject the call to clang_getClangVersion () crashes the VM since its handle ==> ExternalAddress of 4 bytes but 8 bytes are required for a pointer + uint, https://github.com/llvm-mirror/clang/blob/google/stable/include/clang-c/CXSt... /* The CXString type is used to return strings from the interface when * the ownership of that string might differ from one call to the next. * Use clang_getCString() to retrieve the string data and, once finished * with the string data, call clang_disposeString() to free the string. */ typedef struct { const void *data; unsigned private_flags; } CXString; In my limited experience so far it seems FFIExternalStructure is most appropriate So what I've got working is... FFIExternalStructure subclass: #CXString instanceVariableNames: '' classVariableNames: '' poolDictionaries: 'CXStringFlag' package: 'Libclang' CXString class>>fieldsDesc ^ #( void *data; uint private_flags; ) Thus handle of CXString new ==> ByteArray of 8 bytes. However ByteArray and FFIExternalStructure doesn't have any of the smarts of ExternalAddress and FFIExternalReference. So... CXString>>autoRelease self class finalizationRegistry add: self CXString>>finalize self dispose. Transcript crShow: 'debug CXString disposed'. CXString>>dispose self private_flags = CXS_Malloc value "as set by C library" ifTrue: [ self private_flags: CXS_Disposed value. self unguardedDispose. ] ifFalse: [ Error signal: 'Guard prevents double-dispose' ]. CXString>>unguardedDispose self ffiCall: #( void clang_disposeString ( CXString self ) ) module: Libclang

CXString class>>#finalizeResourceData: version self ffiCall: #(void clang_disposeString(void *version))

This only seems applicable to FFIExternalReference subclasses. I guess I could copy FFIExternalReference>>autoRelease to CXString. Do you think there would be much advantage to doing this? I see FFIExternalResourceExecutor>>finalize has some session tracking, presumably to avoid freeing dangling references. but I'm not sure about the semantics of #addResource:'s call to #resourceData, and what CXString>>resourceData would return. cheers -ben

On 07 Sep 2016, at 14:56, Ben Coman <btc@openInWorld.com> wrote:

On Tue, Sep 6, 2016 at 8:08 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com>> wrote:

...

Hi,

sorry for arriving so late to this, but I was on holidays :) this is how autoRelease works:

1) #autoRelease of an object registers object for finalisation with a particular executor. Then behaviour is divided:

2.1.1) for ExternalAddresses, it just registers in regular way, who will call #finalize on GC 2.1.2) finalize will just call a free assuming ExternalAddress was allocated (which is a malloc)

2.2.1) for all FFIExternalReference, it will register for finalisation what #resourceData answers (normally, the handle of the object) 2.2.2) finalisation process will call the object class>>#finalizeResourceData: method, with the #resourceData result as parameter 2.2.3) each kind of external reference can decide how to free that data (by default is also just freeing).

An example of this is how CairoFontFace works (or AthensCairoSurface).

At the bottom of FFIExternalResourceExecutor class comment I read... "Note that in #finalizeResourceData: you cannot access any other properties of your instance, since it is already garbage collected."

But in my experiments it seems okay to access instance variables in #finalize. For example...

CXString >> autoRelease self class finalizationRegistry add: self

CXString >> finalize Transcript crShow: 'Finalizing CXString ' ; show: self private_flags. self dispose. Transcript show: ', done!'.

CXString >>private_flags "This method was automatically generated" ^handle unsignedLongAt: 5

Libclang getClangVersion autoRelease. Smalltalk garbageCollect. "==> Finalizing CXString 1, done! "

Is this an unlucky coincidence? Or maybe something changed from NB to UFFI? (There is a reference to NB there)

yes, is a coincidence. the idea of using #finalizeResourceData: is that you keep minimal information (in general, just the handle)… this way we ensure instances will be collected because we will not have circular references (preventing the weakregistry to work). In general, you can always implement as you did it, but I would prefer the #finalizeResourceData: approach, even for structures. The only reason it is not implemented is because I didn’t reach the necessity, then I just skipped it (not in purpose, it was not in my head :P), but now is a good moment to implement it… if you want it :) Esteban

I am wary of believing my results contrary to the comment. That old engineering principle "When things work, it may be only reinforcing your misconceptions." :)

cheers -ben

...

In you case, you will have something like:

FFIExternalObject subclass: #CXString

CXString class>>#finalizeResourceData: version self ffiCall: #(void clang_disposeString(void *version))

notice that here I casted to void*… this is because actually resourceData answers a pointer, not the object.

cheers, Esteban

On Wed, Sep 7, 2016 at 10:42 PM, Ben Coman <btc@openinworld.com> wrote:

On Wed, Sep 7, 2016 at 9:09 PM, Esteban Lorenzano <estebanlm@gmail.com> wrote:

...

On 07 Sep 2016, at 14:56, Ben Coman <btc@openInWorld.com> wrote:

On Tue, Sep 6, 2016 at 8:08 PM, Esteban Lorenzano <estebanlm@gmail.com> wrote:

Hi,

sorry for arriving so late to this, but I was on holidays :) this is how autoRelease works:

1) #autoRelease of an object registers object for finalisation with a particular executor. Then behaviour is divided:

2.1.1) for ExternalAddresses, it just registers in regular way, who will call #finalize on GC 2.1.2) finalize will just call a free assuming ExternalAddress was allocated (which is a malloc)

2.2.1) for all FFIExternalReference, it will register for finalisation what #resourceData answers (normally, the handle of the object) 2.2.2) finalisation process will call the object class>>#finalizeResourceData: method, with the #resourceData result as parameter 2.2.3) each kind of external reference can decide how to free that data (by default is also just freeing).

An example of this is how CairoFontFace works (or AthensCairoSurface).

At the bottom of FFIExternalResourceExecutor class comment I read... "Note that in #finalizeResourceData: you cannot access any other properties of your instance, since it is already garbage collected."

But in my experiments it seems okay to access instance variables in #finalize. For example...

CXString >> autoRelease self class finalizationRegistry add: self

CXString >> finalize Transcript crShow: 'Finalizing CXString ' ; show: self private_flags. self dispose. Transcript show: ', done!'.

CXString >>private_flags "This method was automatically generated" ^handle unsignedLongAt: 5

Libclang getClangVersion autoRelease. Smalltalk garbageCollect. "==> Finalizing CXString 1, done! "

Is this an unlucky coincidence? Or maybe something changed from NB to UFFI? (There is a reference to NB there)

yes, is a coincidence. the idea of using #finalizeResourceData: is that you keep minimal information (in general, just the handle)… this way we ensure instances will be collected because we will not have circular references (preventing the weakregistry to work).

In general, you can always implement as you did it, but I would prefer the #finalizeResourceData: approach, even for structures. The only reason it is not implemented is because I didn’t reach the necessity, then I just skipped it (not in purpose, it was not in my head :P), but now is a good moment to implement it… if you want it :)

Thanks for the offer, but hold off for the moment. I think I actually need more than just finalization session management. To get a real displayable string requires calling the clang_getCString() library function. I imagine I'd like to call this from CXString>>printOn: -- but this of course this would break after restarting the image.

extern "C" { const char *clang_getCString(CXString string) { .... return static_cast<const char *>(string.data); }}

typedef struct { const void *data; unsigned private_flags; } CXString;

So I think I need to register CXString with SessionManager to set *data <-- 0 on startup.

An alternative might be adding a session variable to CXString, FFIExternalStructure subclass: #CXString instanceVariableNames: 'session' classVariableNames: '' poolDictionaries: 'CXStringFlag' package: 'Libclang'

except doing so crashes when calling.... getClangVersion ^ self ffiCall: #( CXString clang_getClangVersion () ) module: Libclang

So this worked... CXString >> resetData handle unsignedLongAt: 1 put: 0. CXString class >> startUp: resuming resuming ifTrue: [ self allInstances do: [ :cxs | cxs resetData ] ]. CXString class >> initialize "self initialize" SessionManager default registerSystemClassNamed: self name Now immediate after a save/restart, doing... CXString allInstances first getString "==> UndefinedObject(nil)" instead of crashing the VM. Can you see/guess any traps hidden from me? Now I wonder early it is practical to prioritise such a reset. My first thought is at least before #printString starts getting called ?? cheers -ben

On Mon, Sep 12, 2016 at 6:08 PM, Henrik Johansen < henrik.s.johansen@veloxit.no> wrote:

On 07 Sep 2016, at 5:38 , Ben Coman <btc@openInWorld.com <btc@openinworld.com>> wrote:

On Wed, Sep 7, 2016 at 10:42 PM, Ben Coman <btc@openinworld.com> wrote:

On Wed, Sep 7, 2016 at 9:09 PM, Esteban Lorenzano <estebanlm@gmail.com> wrote:

On 07 Sep 2016, at 14:56, Ben Coman <btc@openInWorld.com <btc@openinworld.com>> wrote:

On Tue, Sep 6, 2016 at 8:08 PM, Esteban Lorenzano <estebanlm@gmail.com> wrote:

Hi,

sorry for arriving so late to this, but I was on holidays :) this is how autoRelease works:

1) #autoRelease of an object registers object for finalisation with a particular executor. Then behaviour is divided:

2.1.1) for ExternalAddresses, it just registers in regular way, who will call #finalize on GC 2.1.2) finalize will just call a free assuming ExternalAddress was allocated (which is a malloc)

2.2.1) for all FFIExternalReference, it will register for finalisation what #resourceData answers (normally, the handle of the object) 2.2.2) finalisation process will call the object class>>#finalizeResourceData: method, with the #resourceData result as parameter 2.2.3) each kind of external reference can decide how to free that data (by default is also just freeing).

An example of this is how CairoFontFace works (or AthensCairoSurface).

At the bottom of FFIExternalResourceExecutor class comment I read... "Note that in #finalizeResourceData: you cannot access any other properties of your instance, since it is already garbage collected."

But in my experiments it seems okay to access instance variables in #finalize. For example...

CXString >> autoRelease self class finalizationRegistry add: self

CXString >> finalize Transcript crShow: 'Finalizing CXString ' ; show: self private_flags. self dispose. Transcript show: ', done!'.

CXString >>private_flags "This method was automatically generated" ^handle unsignedLongAt: 5

Libclang getClangVersion autoRelease. Smalltalk garbageCollect. "==> Finalizing CXString 1, done! "

Is this an unlucky coincidence? Or maybe something changed from NB to UFFI? (There is a reference to NB there)

yes, is a coincidence. the idea of using #finalizeResourceData: is that you keep minimal information (in general, just the handle)… this way we ensure instances will be collected because we will not have circular references (preventing the weakregistry to work).

In general, you can always implement as you did it, but I would prefer the #finalizeResourceData: approach, even for structures. The only reason it is not implemented is because I didn’t reach the necessity, then I just skipped it (not in purpose, it was not in my head :P), but now is a good moment to implement it… if you want it :)

Thanks for the offer, but hold off for the moment. I think I actually need more than just finalization session management. To get a real displayable string requires calling the clang_getCString() library function. I imagine I'd like to call this from CXString>>printOn: -- but this of course this would break after restarting the image.

extern "C" { const char *clang_getCString(CXString string) { .... return static_cast<const char *>(string.data); }}

typedef struct { const void *data; unsigned private_flags; } CXString;

So I think I need to register CXString with SessionManager to set *data <-- 0 on startup.

An alternative might be adding a session variable to CXString, FFIExternalStructure subclass: #CXString instanceVariableNames: 'session' classVariableNames: '' poolDictionaries: 'CXStringFlag' package: 'Libclang'

except doing so crashes when calling.... getClangVersion ^ self ffiCall: #( CXString clang_getClangVersion () ) module: Libclang

So this worked...

CXString >> resetData handle unsignedLongAt: 1 put: 0.

CXString class >> startUp: resuming resuming ifTrue: [ self allInstances do: [ :cxs | cxs resetData ] ].

CXString class >> initialize "self initialize" SessionManager default registerSystemClassNamed: self name

Now immediate after a save/restart, doing... CXString allInstances first getString "==> UndefinedObject(nil)" instead of crashing the VM.

Can you see/guess any traps hidden from me? Now I wonder early it is practical to prioritise such a reset. My first thought is at least before #printString starts getting called ??

For traps, there's the assumption in resetData that pointer stored in handle is unsignedLong-sized. Shouldn't there be auto-generated accessors for #data you can use instead to change the value to Pointer void?

Cheers, Henry

Good point. This is the auto-generated accessor... data: anObject handle pointerAt: 1 put: anObject getHandle. However getHandle is only understood by ExternalObject and FFIExternalArray, and it seems a bit awkward to create a null on of those just to zero the handle. Actually your comment lead me to dicover ExternalAddress>>beNull, which looked promising. However the /handle/ of a FFIExternalStructure is a ByteArray rather than an ExternalAddress (which I guess makes sense). So... 1. maybe push beNull up to ByteArray, and maybe also isNull 2. #beNull implementation is 'self atAllPut: 0' which is understood by ByteArray. In my case I don't care if private_flags is also cleared, so maybe... CXString >> beNull handle atAllPut: 0. cheers -ben