Re: [Pharo-dev] strange FFI behaviour? (or lack of understanding?)

Sept. 5, 2016

      On Mon, Sep 5, 2016 at 9:08 PM, Ben Coman <btc@openinworld.com> wrote:
...
Is it a **requirement** that FFI calls have a method all to themselves
like a primitive call?
I have this definition...
FFIExternalStructure subclass: #CXString
CXString class>>fieldsDesc
    ^ #(
    void *data;
    uint private_flags;
    )
The instance returned by this...
    Xxxx>>getClangVersion
        ^ self ffiCall: #( CXString clang_getClangVersion () ) module: Libclang
has private_flags=1 indicating the library allocated external memory
for *data and to release this I need to call  clang_disposeString(
CXString string).  However this doesn't guard against being called
twice and double-free()'ing CXString crashing the VM, so I defined the
following...
CXString>>dispose
   self inform: 'debug_dispose1'.
   self private_flags = 1
       ifTrue:
            [ self inform: 'debug_dispose2'.
              self private_flags: 0.
               self ffiCall: #( void clang_disposeString ( CXString
self ) ) module: Libclang
            ]
       ifFalse:
           [  self inform: 'debug_dispose3'.
              Error signal: 'Cannot dispose twice' ].
However something strange, dispose will only execute once.  For example for...
s := Libclang getClangVersion.
s dispose.   "==>debug_dispose1, debug_dispose2"
s dispose.   "==>nothing at all"
I would expect the second call to dispose
to result it "==>debug_dispose1, debug_dispose3"
but it does nothing at all.  Debugging into "S dispose" displays the
source of dispose and then returns without executing anything.  The
image seems fine and displays the updated zero in private_flags -- at
least for a few minutes, then it might crash indicating the
clang_disposeString() has been called twice.
However it works if I separate out the FFI call to a separate method...
CXString>>unsafeDispose
    self ffiCall: #( void clang_disposeString ( CXString self ) )
module: Libclang
CXString>>dispose
   self inform: 'dispose1'.
   self private_flags = 1
       ifTrue:
            [ self inform: 'dispose2'.
              self private_flags: 0.
              self unsafeDispose.
            ]
       ifFalse:
           [  self inform: 'dispose3'.
              Error signal: 'Cannot dispose twice' ].
s := Libclang getClangVersion.
s dispose.   "==>debug_dispose1, debug_dispose2"
s dispose.   "==>debug_dispose1, debug_dispose3"
s dispose.   "==>debug_dispose1, debug_dispose3"
s dispose.   "==>debug_dispose1, debug_dispose3"
Can someone hazard a guess what is behind this behaviour?  I would
prefer not to require the unguarded #unsafeDispose.
cheers -ben
I'm also trying to understand the FFI memory footprint.
A freshly started image uses this memory (kb)...
    VIRT    RES    SHR
118132 108644   5920

Creation and GC of the FFIExternalStructure is no problem...
   all := OrderedCollection new.
   10 000 000 timesRepeat: [ all add: CXString new. ].
   all := nil.
   Smalltalk garbageCollect.
        VIRT     RES     SHR
 122,268 115,244   5,788

And performing the FFI calls is not too bad (particularly that
repeating this doesn't grow memory further)...
    10 000 000 timesRepeat: [ Libclang getClangVersion unsafeDispose ].
    Smalltalk garbageCollect.
        VIRT    RES    SHR
 170,404 123,252  12,084

But combining those two...
    all := OrderedCollection new.
    1000000 timesRepeat: [ all add: Libclang getClangVersion ].
    all do: [  :s | s unsafeDispose. ].
    all := nil.
    Smalltalk garbageCollect.
    CXString allInstances size "==> 0".
increases memory considerably...
         VIRT          RES        SHR
1,029,836    982,336    11,740

But again, repeating that a dozen times doesn't grow memory further.

Save/quiting the image and reopening it returns to original memory usage...
        VIRT        RES       SHR
118,132     108,620     5,896
.

So I'm curious what might be holding onto the memory.

cheers -ben

P.S. a bit more background info on the library...

extern "C" {
  CXString clang_getClangVersion() {
    return cxstring::createDup(getClangFullVersion());
  }
} // https://github.com/llvm-mirror/clang/blob/google/stable/tools/libclang/CInde...

CXString createDup(const char *String) {
  if (!String)
    return createNull();
  if (String[0] == '\0')
    return createEmpty();
  CXString Str;
  Str.data = strdup(String);
  Str.private_flags = CXS_Malloc;
  return Str;
}

void clang_disposeString(CXString string) {
  switch ((CXStringFlag) string.private_flags) {
    case CXS_Unmanaged:
       break;
    case CXS_Malloc:
      if (string.data)
        free(const_cast<void *>(string.data));
        break;
     case CXS_StringBuf:
        static_cast<cxstring::CXStringBuf *>(
        const_cast<void *>(string.data))->dispose();
        break;
    }
}