[Pharo-project] Cog/FFI Return Values
For kicks, I tried running the GemStone tools in a Cog VM. It *almost* works. Apparently this sort of call:
apiGciPerform: anOopType a: aString a: anArray a: anInteger
<apicall: OopType64 'GciPerform' (OopType64 char* ulong long) > ^self externalCallFailed
... works differently now. In the old (non-Cog) VM, it return an instance of OopType64. In Cog, it returns a SmallInteger. The function itself returns a uint64_t, so apparently the OopType64 class (a subclass of ExternalStructure) is being used to trick the FFI into automatically wrapping the return value in the way we want. 1. Is this trick no longer supported/legit (assuming it ever was)? 2. Is there a better way to accomplish the same thing, or do we just have to do the wrapping ourselves now? I'm pretty new to the FFI, so forgive me if the premise is wrong. All I know is that it doesn't work the way it used to. :) -- Ken Treis Miriam Technologies, Inc.
Hi Ken, On Tue, Mar 15, 2011 at 3:19 PM, Ken Treis <ken@miriamtech.com> wrote:
For kicks, I tried running the GemStone tools in a Cog VM. It *almost* works. Apparently this sort of call:
apiGciPerform: anOopType a: aString a: anArray a: anInteger
<apicall: OopType64 'GciPerform' (OopType64 char* ulong long) > ^self externalCallFailed
... works differently now. In the old (non-Cog) VM, it return an instance of OopType64. In Cog, it returns a SmallInteger.
The function itself returns a uint64_t, so apparently the OopType64 class (a subclass of ExternalStructure) is being used to trick the FFI into automatically wrapping the return value in the way we want.
1. Is this trick no longer supported/legit (assuming it ever was)?
I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin. If you're brave you'll have a look at both plugins and try and verify that that's the case. But I have to say that the FFI plugins are not simple.
2. Is there a better way to accomplish the same thing, or do we just have to do the wrapping ourselves now?
Difficult to answer without answering 1. I'm guessing that 2. is a non-issue and that the real answer is to fix the bug in ReentrantFFIPlugin. For that I'd like to enlist your help. You have the patient on the table as it were and I don't have the time to attempt to reproduce the example and fix it now.
I'm pretty new to the FFI, so forgive me if the premise is wrong. All I know is that it doesn't work the way it used to. :)
The premise is right. The ReentrantFFIPlugin is supposed to be a plug-and-play replacement for the old FFIPlugin (at least in Cog; a little more work may be necessary to get it to work in the standard VM). let me know how you get on. best, Eliot
-- Ken Treis Miriam Technologies, Inc.
On Mar 15, 2011, at 3:36 PM, Eliot Miranda wrote:
I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin. If you're brave you'll have a look at both plugins and try and verify that that's the case. But I have to say that the FFI plugins are not simple.
Brave, yes. Smart, no. Here's what it looks like to my untrained eye: 1. When the ExternalFunction is compiled, the return argType is not flagged as a structure. So the primitives, old and new, see this as a regular non-struct callout, but with return class = OopType64. So far, so good. 2. The old plugin had this one-slot "wrapping" capability at the bottom of #ffiCreateReturn: (which is called only for non-pointer, non-structure functions): ffiRetClass := interpreterProxy popRemappableOop. ffiRetClass == interpreterProxy nilObject ifTrue:[ "Just return oop" ^interpreterProxy push: retOop]. "Otherwise create an instance of external structure and store the return oop" interpreterProxy pushRemappableOop: retOop. retOop := interpreterProxy instantiateClass: ffiRetClass indexableSize: 0. oop := interpreterProxy popRemappableOop. interpreterProxy storePointer: 0 ofObject: retOop withValue: oop. ^interpreterProxy push: retOop. 3. There's some very similar code in FFIPlugin>>ffiCreateReturnStruct and ReentrantIA32FFIPlugin>>ffiReturnStruct:ofType:in: -- the main difference is a memcpy(). Am I on the right track? If so, how would you like to see the code structured to get this feature restored? We could add a new wrap-if-necessary return method, or we could try to reuse #ffiReturnStruct:ofType:in: if we guarded the memcpy() with a flag check for FFIFlagStructure. Or maybe there's a cleaner solution that's beyond my hazy view at the moment. -- Ken Treis Miriam Technologies, Inc.
On Wed, Mar 16, 2011 at 4:25 PM, Ken Treis <ken@miriamtech.com> wrote:
On Mar 15, 2011, at 3:36 PM, Eliot Miranda wrote:
I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin. If you're brave you'll have a look at both plugins and try and verify that that's the case. But I have to say that the FFI plugins are not simple.
Brave, yes. Smart, no. Here's what it looks like to my untrained eye:
1. When the ExternalFunction is compiled, the return argType is not flagged as a structure. So the primitives, old and new, see this as a regular non-struct callout, but with return class = OopType64. So far, so good.
2. The old plugin had this one-slot "wrapping" capability at the bottom of #ffiCreateReturn: (which is called only for non-pointer, non-structure functions):
ffiRetClass := interpreterProxy popRemappableOop. ffiRetClass == interpreterProxy nilObject ifTrue:[ "Just return oop" ^interpreterProxy push: retOop]. "Otherwise create an instance of external structure and store the return oop" interpreterProxy pushRemappableOop: retOop. retOop := interpreterProxy instantiateClass: ffiRetClass indexableSize: 0. oop := interpreterProxy popRemappableOop. interpreterProxy storePointer: 0 ofObject: retOop withValue: oop. ^interpreterProxy push: retOop.
3. There's some very similar code in FFIPlugin>>ffiCreateReturnStruct and ReentrantIA32FFIPlugin>>ffiReturnStruct:ofType:in: -- the main difference is a memcpy().
Am I on the right track? If so, how would you like to see the code structured to get this feature restored? We could add a new wrap-if-necessary return method, or we could try to reuse #ffiReturnStruct:ofType:in: if we guarded the memcpy() with a flag check for FFIFlagStructure.
If you're happy producing a fix I'll take whatever works, a test case and the I can always muck it up if I disapprove :)
Or maybe there's a cleaner solution that's beyond my hazy view at the moment.
I think you're on the right track. I probably misread when I wrote my first cut. Apologies. best Eliot
-- Ken Treis Miriam Technologies, Inc.
See attached for changes vs VMMaker-oscog.40. I added a couple of simple test methods to the preamble and used them in my test cases. The push/popRemappableOop stuff is over my head. I basically cloned the code quoted below, but I had to introduce a temporary variable. I'm guessing that it doesn't need the same treatment, but no doubt you can clean that up for me if needs be. With this change, GemTools work in Cog for me. Thanks for the guidance and encouragement. Ken On Mar 16, 2011, at 4:45 PM, Eliot Miranda wrote:
On Wed, Mar 16, 2011 at 4:25 PM, Ken Treis <ken@miriamtech.com> wrote: On Mar 15, 2011, at 3:36 PM, Eliot Miranda wrote:
I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin. If you're brave you'll have a look at both plugins and try and verify that that's the case. But I have to say that the FFI plugins are not simple.
Brave, yes. Smart, no. Here's what it looks like to my untrained eye:
1. When the ExternalFunction is compiled, the return argType is not flagged as a structure. So the primitives, old and new, see this as a regular non-struct callout, but with return class = OopType64. So far, so good.
2. The old plugin had this one-slot "wrapping" capability at the bottom of #ffiCreateReturn: (which is called only for non-pointer, non-structure functions):
ffiRetClass := interpreterProxy popRemappableOop. ffiRetClass == interpreterProxy nilObject ifTrue:[ "Just return oop" ^interpreterProxy push: retOop]. "Otherwise create an instance of external structure and store the return oop" interpreterProxy pushRemappableOop: retOop. retOop := interpreterProxy instantiateClass: ffiRetClass indexableSize: 0. oop := interpreterProxy popRemappableOop. interpreterProxy storePointer: 0 ofObject: retOop withValue: oop. ^interpreterProxy push: retOop.
3. There's some very similar code in FFIPlugin>>ffiCreateReturnStruct and ReentrantIA32FFIPlugin>>ffiReturnStruct:ofType:in: -- the main difference is a memcpy().
Am I on the right track? If so, how would you like to see the code structured to get this feature restored? We could add a new wrap-if-necessary return method, or we could try to reuse #ffiReturnStruct:ofType:in: if we guarded the memcpy() with a flag check for FFIFlagStructure.
If you're happy producing a fix I'll take whatever works, a test case and the I can always muck it up if I disapprove :)
Or maybe there's a cleaner solution that's beyond my hazy view at the moment.
I think you're on the right track. I probably misread when I wrote my first cut. Apologies.
best Eliot
-- Ken Treis Miriam Technologies, Inc.
-- Ken Treis Miriam Technologies, Inc.
Hi Ken, On Thu, Mar 17, 2011 at 5:38 PM, Ken Treis <ken@miriamtech.com> wrote:
See attached for changes vs VMMaker-oscog.40. I added a couple of simple test methods to the preamble and used them in my test cases.
The push/popRemappableOop stuff is over my head. I basically cloned the code quoted below, but I had to introduce a temporary variable. I'm guessing that it doesn't need the same treatment, but no doubt you can clean that up for me if needs be.
With this change, GemTools work in Cog for me. Thanks for the guidance and encouragement.
woot! Thanks so much for this Ken! Welcome to the Cog team!!
Ken
On Mar 16, 2011, at 4:45 PM, Eliot Miranda wrote:
On Wed, Mar 16, 2011 at 4:25 PM, Ken Treis <ken@miriamtech.com> wrote:
On Mar 15, 2011, at 3:36 PM, Eliot Miranda wrote:
I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin. If you're brave you'll have a look at both plugins and try and verify that that's the case. But I have to say that the FFI plugins are not simple.
Brave, yes. Smart, no. Here's what it looks like to my untrained eye:
1. When the ExternalFunction is compiled, the return argType is not flagged as a structure. So the primitives, old and new, see this as a regular non-struct callout, but with return class = OopType64. So far, so good.
2. The old plugin had this one-slot "wrapping" capability at the bottom of #ffiCreateReturn: (which is called only for non-pointer, non-structure functions):
ffiRetClass := interpreterProxy popRemappableOop. ffiRetClass == interpreterProxy nilObject ifTrue:[ "Just return oop" ^interpreterProxy push: retOop]. "Otherwise create an instance of external structure and store the return oop" interpreterProxy pushRemappableOop: retOop. retOop := interpreterProxy instantiateClass: ffiRetClass indexableSize: 0. oop := interpreterProxy popRemappableOop. interpreterProxy storePointer: 0 ofObject: retOop withValue: oop. ^interpreterProxy push: retOop.
3. There's some very similar code in FFIPlugin>>ffiCreateReturnStruct and ReentrantIA32FFIPlugin>>ffiReturnStruct:ofType:in: -- the main difference is a memcpy().
Am I on the right track? If so, how would you like to see the code structured to get this feature restored? We could add a new wrap-if-necessary return method, or we could try to reuse #ffiReturnStruct:ofType:in: if we guarded the memcpy() with a flag check for FFIFlagStructure.
If you're happy producing a fix I'll take whatever works, a test case and the I can always muck it up if I disapprove :)
Or maybe there's a cleaner solution that's beyond my hazy view at the moment.
I think you're on the right track. I probably misread when I wrote my first cut. Apologies.
best Eliot
-- Ken Treis Miriam Technologies, Inc.
-- Ken Treis Miriam Technologies, Inc.
ken it would be good to enter a bug entry on the cog vm bug tracker. Stef On Mar 15, 2011, at 11:19 PM, Ken Treis wrote:
For kicks, I tried running the GemStone tools in a Cog VM. It *almost* works. Apparently this sort of call:
apiGciPerform: anOopType a: aString a: anArray a: anInteger
<apicall: OopType64 'GciPerform' (OopType64 char* ulong long) > ^self externalCallFailed
... works differently now. In the old (non-Cog) VM, it return an instance of OopType64. In Cog, it returns a SmallInteger.
The function itself returns a uint64_t, so apparently the OopType64 class (a subclass of ExternalStructure) is being used to trick the FFI into automatically wrapping the return value in the way we want.
1. Is this trick no longer supported/legit (assuming it ever was)? 2. Is there a better way to accomplish the same thing, or do we just have to do the wrapping ourselves now?
I'm pretty new to the FFI, so forgive me if the premise is wrong. All I know is that it doesn't work the way it used to. :)
-- Ken Treis Miriam Technologies, Inc.
http://code.google.com/p/cog/issues/list On Wed, Mar 16, 2011 at 8:03 AM, Stéphane Ducasse <stephane.ducasse@inria.fr
wrote:
ken
it would be good to enter a bug entry on the cog vm bug tracker.
Stef
On Mar 15, 2011, at 11:19 PM, Ken Treis wrote:
For kicks, I tried running the GemStone tools in a Cog VM. It *almost* works. Apparently this sort of call:
apiGciPerform: anOopType a: aString a: anArray a: anInteger
<apicall: OopType64 'GciPerform' (OopType64 char* ulong long) > ^self externalCallFailed
... works differently now. In the old (non-Cog) VM, it return an instance of OopType64. In Cog, it returns a SmallInteger.
The function itself returns a uint64_t, so apparently the OopType64 class (a subclass of ExternalStructure) is being used to trick the FFI into automatically wrapping the return value in the way we want.
1. Is this trick no longer supported/legit (assuming it ever was)? 2. Is there a better way to accomplish the same thing, or do we just have to do the wrapping ourselves now?
I'm pretty new to the FFI, so forgive me if the premise is wrong. All I know is that it doesn't work the way it used to. :)
-- Ken Treis Miriam Technologies, Inc.
participants (4)
-
Eliot Miranda -
Ken Treis -
Mariano Martinez Peck -
Stéphane Ducasse