[Pharo-project] NativeBoostFFI out parameters passing pointers back and forth is confusing...
Hi: I don't really understand how NativeBoostFFI handles pointers. I have two functions like this: self call: #(int git_repository_open(git_repository * * handle, char * path)) and self call: #(int git_repository_head(git_reference * * refHandle, git_repository * repo)) They use a type mapping like this: #('git_repository' 'NBVoid' 'git_reference' 'NBVoid' ) So, nothing special going on here, just pointers to pointers, and pointers to opaque data structures. Now, the call to the _open function is done with a `repoHandle := NBExternalHandle new`. This works, and I get the result value of the c lib inside my repoHandle. However, if I want to use it in the _head call, I first have to convert it into a NBExternalAddress (`NBExternalAddress value: repoHandle value`). Took me a while to figure out that passing in the repoHandle to _head does actually do the wrong thing. The _head function then gets the address of repoHandle instead of its value, which is wrong for my usecase. However, just passing in `repoHandle value` results in some strange _nil_ error. So, the main question is, why do I need to convert between Handle and Address? For the library I am trying to use, that becomes a somehow recurring pattern: | repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'. LG2RepositoryH uniqueInstance repository_open: repo path: repoPath. repo := NBExternalAddress value: repo value. "<-- Why is this necessary?" headHandle := NBExternalHandle new. LG2RepositoryH uniqueInstance repository_head: headHandle repo: repo. ^ (LG2RefsH uniqueInstance reference_name: (NBExternalAddress value: headHandle value)) "<-- Why is this necessary?" readString Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
Hi Stefan, First of all I want to thank you for this mail, which showed me a bug that I had in TalkFFI (the git_repository 'NBVoid *') which introduces one more indirection level, which is wrong. On Fri, Jan 25, 2013 at 2:13 AM, Stefan Marr <smalltalk@stefan-marr.de>wrote:
Hi:
I don't really understand how NativeBoostFFI handles pointers.
probably Igor is the best to answer this question... in details
I have two functions like this:
self call: #(int git_repository_open(git_repository * * handle, char * path))
and
self call: #(int git_repository_head(git_reference * * refHandle, git_repository * repo))
They use a type mapping like this: #('git_repository' 'NBVoid' 'git_reference' 'NBVoid' )
So, nothing special going on here, just pointers to pointers, and pointers to opaque data structures.
Now, the call to the _open function is done with a `repoHandle := NBExternalHandle new`. This works, and I get the result value of the c lib inside my repoHandle.
However, if I want to use it in the _head call, I first have to convert it into a NBExternalAddress (`NBExternalAddress value: repoHandle value`).
Took me a while to figure out that passing in the repoHandle to _head does actually do the wrong thing. The _head function then gets the address of repoHandle instead of its value, which is wrong for my usecase.
However, just passing in `repoHandle value` results in some strange _nil_ error.
So, the main question is, why do I need to convert between Handle and Address?
For the library I am trying to use, that becomes a somehow recurring pattern:
| repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'.
LG2RepositoryH uniqueInstance repository_open: repo path: repoPath. repo := NBExternalAddress value: repo value. "<-- Why is this necessary?"
headHandle := NBExternalHandle new. LG2RepositoryH uniqueInstance repository_head: headHandle repo: repo.
^ (LG2RefsH uniqueInstance reference_name: (NBExternalAddress value: headHandle value)) "<-- Why is this necessary?" readString
Another solution which removes this recurring pattern is to use the NBExternalObject, either directly, or by subclassing it. Here is a snippet showing how: |headHandle path repo name| repo := NBExternalObject new. path := NBExternalAddress fromString: '/Users/ciprian/github/local/enso'. LG2RepositoryH uniqueInstance repository_open: repo path: path. headHandle := NBExternalObject new. LG2RepositoryH uniqueInstance repository_head: headHandle repo: repo. name := LG2RefsH uniqueInstance reference_name: headHandle. name readString. but now FFI methods need to be changed to: repository_open: out path: path ^self call: #(NBInt32 git_repository_open(NBExternalObject * out, NBCharacterType * path)) options: #( ) repository_head: out repo: repo ^self call: #(NBInt32 git_repository_head(NBExternalObject * out, NBExternalObject repo)) options: #( ) reference_name: ref ^self call: #(NBCharacterType * git_reference_name(NBExternalObject ref)) options: #( ) Notice that when using NBExternalObject one level of indirection is gone so git_repository ** becomes NBExternalObject * Even nicer, if you subclass the NBExternalObject to have LG2Repository and LG2Reference, the signatures become clearer... I will look this weekend at TalkFFI to see how can I make use of this NBExternalObject, at least in the simple cases... to prevent the need for (NBExternalAddress value: repo value) all over the place. Cheers, Ciprian
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Dr. Ciprian TEODOROV Ingénieur Développement CAO tél : 06 08 54 73 48 mail : ciprian.teodorov@gmail.com www.teodorov.ro
Hi Ciprian: On 25 Jan 2013, at 20:58, Ciprian Teodorov wrote:
repo := NBExternalObject new. [...]
repository_open: out path: path ^self call: #(NBInt32 git_repository_open(NBExternalObject * out, NBCharacterType * path)) options: #( )
Notice that when using NBExternalObject one level of indirection is gone so git_repository ** becomes NBExternalObject *
Indeed, much nicer. I am now using type aliases such as: #(git_repository_ptr NBExternalObject git_reference_ptr NBExternalObject) With a signature such as: ^self call: #(int git_repository_open(git_repository_ptr * handle, char * path)) That stays close to the C, and tells me what the pointer are. And with the NBExternalObject it works nicely. Thanks a lot! Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
Hi Stefan, I've just committed a little modification to TalkFFI to automatically generate subclasses of NBExternalObject for definitions like: typedef struct my_struct my_struct_t; void fct(my_struct_t *inArg) the ffi call is void fct(MyStructTPtr inArg). This gives us type-checking and simplifies the usage pattern, that u have presented, to: |headHandle path repo name| repo := LG2GitRepositoryPtr new. path := NBExternalAddress fromString: '/Users/ciprian/github/local/enso'. LG2RepositoryH uniqueInstance repository_open: repo path: path. headHandle := LG2GitReferencePtr new. LG2RepositoryH uniqueInstance repository_head: headHandle repo: repo. name := LG2RefsH uniqueInstance reference_name: headHandle. name readString. Cheers, On Fri, Jan 25, 2013 at 9:45 PM, Stefan Marr <smalltalk@stefan-marr.de>wrote:
Hi Ciprian:
On 25 Jan 2013, at 20:58, Ciprian Teodorov wrote:
repo := NBExternalObject new. [...]
repository_open: out path: path ^self call: #(NBInt32 git_repository_open(NBExternalObject * out, NBCharacterType * path)) options: #( )
Notice that when using NBExternalObject one level of indirection is gone so git_repository ** becomes NBExternalObject *
Indeed, much nicer.
I am now using type aliases such as: #(git_repository_ptr NBExternalObject git_reference_ptr NBExternalObject)
With a signature such as: ^self call: #(int git_repository_open(git_repository_ptr * handle, char * path))
That stays close to the C, and tells me what the pointer are.
And with the NBExternalObject it works nicely.
Thanks a lot! Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Dr. Ciprian TEODOROV Ingénieur Développement CAO tél : 06 08 54 73 48 mail : ciprian.teodorov@gmail.com www.teodorov.ro
On 25 January 2013 02:13, Stefan Marr <smalltalk@stefan-marr.de> wrote:
Hi:
I don't really understand how NativeBoostFFI handles pointers.
I have two functions like this:
self call: #(int git_repository_open(git_repository * * handle, char * path))
and
self call: #(int git_repository_head(git_reference * * refHandle, git_repository * repo))
They use a type mapping like this: #('git_repository' 'NBVoid' 'git_reference' 'NBVoid' )
So, nothing special going on here, just pointers to pointers, and pointers to opaque data structures.
Now, the call to the _open function is done with a `repoHandle := NBExternalHandle new`. This works, and I get the result value of the c lib inside my repoHandle.
However, if I want to use it in the _head call, I first have to convert it into a NBExternalAddress (`NBExternalAddress value: repoHandle value`).
Took me a while to figure out that passing in the repoHandle to _head does actually do the wrong thing. The _head function then gets the address of repoHandle instead of its value, which is wrong for my usecase.
However, just passing in `repoHandle value` results in some strange _nil_ error.
So, the main question is, why do I need to convert between Handle and Address?
For the library I am trying to use, that becomes a somehow recurring pattern:
| repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'.
why not just pass the string?
LG2RepositoryH uniqueInstance repository_open: repo path: repoPath. repo := NBExternalAddress value: repo value. "<-- Why is this necessary?"
headHandle := NBExternalHandle new. LG2RepositoryH uniqueInstance repository_head: headHandle repo: repo.
^ (LG2RefsH uniqueInstance reference_name: (NBExternalAddress value: headHandle value)) "<-- Why is this necessary?" readString
yeah.. that double indirection is often source of confusion. but you can make a subclass of NBExternalObject , say 'GitReference' to be equal to 'git_reference *' and 'GitRepository' to 'git_repository *' like that, then in signature you need to rewrite to: self call: #(int git_repository_head(GitReference * refHandle, GitRepository repo)) this should work, because converter will push a pointer to the location where your handle value is stored (if with *) or push its value without.
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Best regards, Igor Stasenko.
Hi Igor: On 29 Jan 2013, at 11:26, Igor Stasenko wrote:
| repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'.
why not just pass the string?
I had the feeling (i.e., I had crashes, and was checking with GDB) that plain Smalltalk strings ala 'test' are not actually passed properly into the function. What is passed in is a reference to the begin of the string, but not a properly null terminated string. Do I miss something? Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On 29 January 2013 23:56, Stefan Marr <smalltalk@stefan-marr.de> wrote:
Hi Igor:
On 29 Jan 2013, at 11:26, Igor Stasenko wrote:
| repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'.
why not just pass the string?
I had the feeling (i.e., I had crashes, and was checking with GDB) that plain Smalltalk strings ala 'test' are not actually passed properly into the function. What is passed in is a reference to the begin of the string, but not a properly null terminated string.
Do I miss something?
Yes, to ease work with strings there is a special 'String' type it will automatically make a null-terminated copy of string on stack and pass it to function. So, you don't need to manually do 'NBExternalAddress fromString:' and then symmetrical #free, to free used memory, ( which your code snippet does not includes btw ;) ).
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Best regards, Igor Stasenko.
Hi Igor: On 30 Jan 2013, at 09:53, Igor Stasenko wrote:
On 29 January 2013 23:56, Stefan Marr <smalltalk@stefan-marr.de> wrote:
Hi Igor:
On 29 Jan 2013, at 11:26, Igor Stasenko wrote:
| repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'.
why not just pass the string?
I had the feeling (i.e., I had crashes, and was checking with GDB) that plain Smalltalk strings ala 'test' are not actually passed properly into the function. What is passed in is a reference to the begin of the string, but not a properly null terminated string.
Do I miss something?
Yes, to ease work with strings there is a special 'String' type it will automatically make a null-terminated copy of string on stack and pass it to function. So, you don't need to manually do 'NBExternalAddress fromString:' and then symmetrical #free, to free used memory, ( which your code snippet does not includes btw ;) ).
Right, I haven't worried about memory at all so far... libgit2 isn't entirely trivial in that regard. Ok, now that I saw NBExternalString, I added a type mapping to my list (char_ptr NBExternalString). How does it work for return types? You assume that you will have the responsibility to free the returned char* if I use NBExternalString as return type for functions that return char*? Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On 30 January 2013 10:47, Stefan Marr <smalltalk@stefan-marr.de> wrote:
Hi Igor:
On 30 Jan 2013, at 09:53, Igor Stasenko wrote:
On 29 January 2013 23:56, Stefan Marr <smalltalk@stefan-marr.de> wrote:
Hi Igor:
On 29 Jan 2013, at 11:26, Igor Stasenko wrote:
| repo repoPath cb branches headHandle | repo := NBExternalHandle new. repoPath := NBExternalAddress fromString: '/Users/smarr/tmp/filetree'.
why not just pass the string?
I had the feeling (i.e., I had crashes, and was checking with GDB) that plain Smalltalk strings ala 'test' are not actually passed properly into the function. What is passed in is a reference to the begin of the string, but not a properly null terminated string.
Do I miss something?
Yes, to ease work with strings there is a special 'String' type it will automatically make a null-terminated copy of string on stack and pass it to function. So, you don't need to manually do 'NBExternalAddress fromString:' and then symmetrical #free, to free used memory, ( which your code snippet does not includes btw ;) ).
Right, I haven't worried about memory at all so far... libgit2 isn't entirely trivial in that regard.
Ok, now that I saw NBExternalString, I added a type mapping to my list (char_ptr NBExternalString).
How does it work for return types? You assume that you will have the responsibility to free the returned char* if I use NBExternalString as return type for functions that return char*?
it depends.. usually, functions which return string do not demand from you to free them explicitly.. in case if its needed, then of course, you may need to free them explicitly.. but not using NBExternalAddress>>free , because this will work only for memory allocated via Nativeboost>>allocate, but not malloc() or any other allocation procedure, used by external libs.
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Best regards, Igor Stasenko.
participants (3)
-
Ciprian Teodorov -
Igor Stasenko -
Stefan Marr