[Pharo-project] FFI Documentation
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists. I compiled every piece of info I could find. The focus in on FFI, with stubs for other strategies. Please read this *very* rough outline for missing info and inaccuracies: FFI What is FFI and what is it used for? Calling functions in libraries outside of the image... FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1] How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags. How do I use it? 1. make a method (whose structure is similar to a named primitive method) Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)" <apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification" ^self externalCallFailed. Let's take it piece by piece: system: aString Method name - by convention named 'apiXxx' <apicall: long 'system' (char*) module: 'libSystem.dylib'> Function specification - should be the first line - enclosed in angle brackets: < > containing: 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention) - Mac - use either one - Unix - use cdecl - Windows - use apical 2. Return Type (see types) 3. External Function Name (literal string) 4. Argument Types (a literal array) 5. Module - "module: " + [filename of the external library (literal string)] (see below). self externalCallFailed. Failure handler - normal smalltalk code - executed if the linking to or calling the external function fails - API calls don't know how to communicate failure like Squeak primitives do, so: - it does not tell you whether the external function succeeded - the most common code is simply '^self externalCallFailed.' Argument Types - must be names of ExternalTypes, either: - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes): void bool byte (unsigned) sbyte (signed) ushort (16-bit unsigned) short (16-bit signed) ulong (32-bit unsigned) long (32-bit signed) ulonglong (64-bit unsigned) longlong (64-bit signed) char (unsigned) schar (signed) float (single-precision float) double (double-precision float) Structure Types [4] - subclass of ExternalStructure - class>>fields that returns an array of field descriptions (see below) - Example: fields ^#((red 'byte')(green 'byte')(blue 'byte')) - class>>initialize which includes "self defineFields" (which must be called before using the class) - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure) Field description [4] - 2-element array (or three but that does something else, I'm not sure what): - first element is the field name - second is the type Mac Memory Allocation Issues [4] (not sure about this) If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different. Module Name - depends on the platform - Mac - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c' - Snow Leopard - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below) - With pre-mach-o VMs - For Classic applications, use 'InterfaceLib' - For Carbon libs, use 'CarbonLib' Module Location - where the external library file lives - depends on the platform - Mac - pre Snow Leopard - checks VM path and common library paths - Snow Leopard - only looks in VM bundle's Resources file, you must either [5]: - store all external libraries there - ln -s path/to/library path/to/VM/Resources/library_name - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats - security - malicious users could call arbitrary functions in the OS e.g. "format c:" from "system.dll" [7] - VMs do not protect against buffer overflow from bad parameters [8]: "this would require an attacker to execute arbitrary Smalltalk code on your server. Of course if they can do that they own you anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh * difficulty - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2] - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2] What do I need to use FFI with Squeak? You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so. You can also build the plugin yourself. See VMMaker. References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5] http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635: Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence. * if you need callbacks * mac-only? Plugins - write external code and dynamically link it to the VM Pros: * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm / * fast - faster than FFI * you can create new functionality that doesn't exist in any library Cons: * harder to write * plugin must be distributed with the Smalltalk code - there can be complications with platforms Primitive method - invokes behavior in the VM or a plugin [3] Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
thanks sean this is cool to see such kind of documentation emerging. Stef On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. The focus in on FFI, with stubs for other strategies. Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
<apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification" ^self externalCallFailed.
Let's take it piece by piece:
system: aString Method name - by convention named 'apiXxx'
<apicall: long 'system' (char*) module: 'libSystem.dylib'> Function specification - should be the first line - enclosed in angle brackets: < > containing: 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention) - Mac - use either one - Unix - use cdecl - Windows - use apical 2. Return Type (see types) 3. External Function Name (literal string) 4. Argument Types (a literal array) 5. Module - "module: " + [filename of the external library (literal string)] (see below).
self externalCallFailed. Failure handler - normal smalltalk code - executed if the linking to or calling the external function fails - API calls don't know how to communicate failure like Squeak primitives do, so: - it does not tell you whether the external function succeeded - the most common code is simply '^self externalCallFailed.'
Argument Types - must be names of ExternalTypes, either: - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes): void bool byte (unsigned) sbyte (signed) ushort (16-bit unsigned) short (16-bit signed) ulong (32-bit unsigned) long (32-bit signed) ulonglong (64-bit unsigned) longlong (64-bit signed) char (unsigned) schar (signed) float (single-precision float) double (double-precision float)
Structure Types [4] - subclass of ExternalStructure - class>>fields that returns an array of field descriptions (see below) - Example: fields ^#((red 'byte')(green 'byte')(blue 'byte')) - class>>initialize which includes "self defineFields" (which must be called before using the class) - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4] - 2-element array (or three but that does something else, I'm not sure what): - first element is the field name - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform - Mac - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c' - Snow Leopard - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below) - With pre-mach-o VMs - For Classic applications, use 'InterfaceLib' - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives - depends on the platform - Mac - pre Snow Leopard - checks VM path and common library paths - Snow Leopard - only looks in VM bundle's Resources file, you must either [5]: - store all external libraries there - ln -s path/to/library path/to/VM/Resources/library_name - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats - security - malicious users could call arbitrary functions in the OS e.g. "format c:" from "system.dll" [7] - VMs do not protect against buffer overflow from bad parameters [8]: "this would require an attacker to execute arbitrary Smalltalk code on your server. Of course if they can do that they own you anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2] - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5] http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence. * if you need callbacks * mac-only?
Plugins - write external code and dynamically link it to the VM Pros: * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm / * fast - faster than FFI * you can create new functionality that doesn't exist in any library Cons: * harder to write * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Very nice topic coverage. I hope, eventually NB will also have something like this :) But it having a little different pros/cons table: NB Pros: * as well as for plugin, you can create new functionality that doesn't exist in any library * no need to recompile plugin/VM when you making a changes * all your code is distributed with the Smalltalk code, but there can be complications with platforms * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster Cons: * ?unsafe? - you have to provide a safety layers (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there) * harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals. So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell. On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. Â The focus in on FFI, with stubs for other strategies. Â Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Â Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. Â FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
   <apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification"    ^self externalCallFailed.
Let's take it piece by piece:
   system: aString        Method name - by convention named 'apiXxx'
   <apicall: long 'system' (char*) module: 'libSystem.dylib'>        Function specification            - should be the first line            - enclosed in angle brackets: < > containing:                1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention)                    - Mac - use either one                    - Unix - use cdecl                    - Windows - use apical                2. Return Type (see types)                3. External Function Name (literal string)                4. Argument Types (a literal array)                5. Module - "module: " + [filename of the external library (literal string)] (see below).
   self externalCallFailed.        Failure handler            - normal smalltalk code            - executed if the linking to or calling the external function fails            - API calls don't know how to communicate failure like Squeak primitives do, so:                - it does not tell you whether the external function succeeded                - the most common code is simply '^self externalCallFailed.'
Argument Types    - must be names of ExternalTypes, either:        - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes):            void            bool            byte (unsigned)            sbyte (signed)            ushort (16-bit unsigned)            short (16-bit signed)            ulong (32-bit unsigned)            long (32-bit signed)            ulonglong (64-bit unsigned)            longlong (64-bit signed)            char (unsigned)            schar (signed)            float (single-precision float)            double (double-precision float)
Structure Types [4]    - subclass of ExternalStructure        - class>>fields that returns an array of field descriptions (see below)            - Example:                fields                    ^#((red  'byte')(green 'byte')(blue  'byte'))        - class>>initialize which includes "self defineFields" (which must be called before using the class)    - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4]    - 2-element array (or three but that does something else, I'm not sure what):        - first element is the field name        - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform    - Mac        - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c'        - Snow Leopard            - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below)        - With pre-mach-o VMs            - For Classic applications, use 'InterfaceLib'            - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives    - depends on the platform        - Mac            - pre Snow Leopard                - checks VM path and common library paths            - Snow Leopard                - only looks in VM bundle's Resources file, you must either [5]:                    - store all external libraries there                    - ln -s path/to/library path/to/VM/Resources/library_name                    - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats    - security        - malicious users could call arbitrary functions in the OS e.g.  "format c:" from "system.dll" [7]        - VMs do not protect against buffer overflow from bad parameters [8]:            "this would require an attacker to execute arbitrary Smalltalk            code on your server. Of course if they can do that they own you            anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty    - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2]    - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5] http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence.    * if you need callbacks    * mac-only?
Plugins - write external code and dynamically link it to the VM    Pros:        * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm /        * fast - faster than FFI        * you can create new functionality that doesn't exist in any library    Cons:        * harder to write        * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
Igor Stasenko wrote:
NB Pros: * as well as for plugin, you can create new functionality that doesn't exist in any library * no need to recompile plugin/VM when you making a changes * all your code is distributed with the Smalltalk code, but there can be complications with platforms * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster
Cons: * ?unsafe? - you have to provide a safety layers (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there)
* harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals.
So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell.
Cool, I'm glad you wrote that. I've been following your posts about NB, and from that quick summary, I already have a much better understanding of where it fits into the "external code" picture. Sean -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225585.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
Hi. Igor Can you compare NB callout functionality with FFI and Alien functionality. Can NB replace FFI, Alien? What absent in NB callout? You implement callbacks with NB. FFI not support it. For me NB foreign interface seems more attractive than something else. 2010/5/21 Igor Stasenko <siguctua@gmail.com>
Very nice topic coverage. I hope, eventually NB will also have something like this :)
But it having a little different pros/cons table:
NB Pros: * as well as for plugin, you can create new functionality that doesn't exist in any library * no need to recompile plugin/VM when you making a changes * all your code is distributed with the Smalltalk code, but there can be complications with platforms * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster
Cons: * ?unsafe? - you have to provide a safety layers (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there)
* harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals.
So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell.
On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book,
as
similar questions pop up regularly on the lists.
I compiled every piece of info I could find. The focus in on FFI, with stubs for other strategies. Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
<apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification" ^self externalCallFailed.
Let's take it piece by piece:
system: aString Method name - by convention named 'apiXxx'
<apicall: long 'system' (char*) module: 'libSystem.dylib'> Function specification - should be the first line - enclosed in angle brackets: < > containing: 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention) - Mac - use either one - Unix - use cdecl - Windows - use apical 2. Return Type (see types) 3. External Function Name (literal string) 4. Argument Types (a literal array) 5. Module - "module: " + [filename of the external library (literal string)] (see below).
self externalCallFailed. Failure handler - normal smalltalk code - executed if the linking to or calling the external function fails - API calls don't know how to communicate failure like Squeak primitives do, so: - it does not tell you whether the external function succeeded - the most common code is simply '^self externalCallFailed.'
Argument Types - must be names of ExternalTypes, either: - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes): void bool byte (unsigned) sbyte (signed) ushort (16-bit unsigned) short (16-bit signed) ulong (32-bit unsigned) long (32-bit signed) ulonglong (64-bit unsigned) longlong (64-bit signed) char (unsigned) schar (signed) float (single-precision float) double (double-precision float)
Structure Types [4] - subclass of ExternalStructure - class>>fields that returns an array of field descriptions (see below) - Example: fields ^#((red 'byte')(green 'byte')(blue 'byte')) - class>>initialize which includes "self defineFields" (which must be called before using the class) - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4] - 2-element array (or three but that does something else, I'm not sure what): - first element is the field name - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform - Mac - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c' - Snow Leopard - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below) - With pre-mach-o VMs - For Classic applications, use 'InterfaceLib' - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives - depends on the platform - Mac - pre Snow Leopard - checks VM path and common library paths - Snow Leopard - only looks in VM bundle's Resources file, you must either [5]: - store all external libraries there - ln -s path/to/library path/to/VM/Resources/library_name - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats - security - malicious users could call arbitrary functions in the OS e.g. "format c:" from "system.dll" [7] - VMs do not protect against buffer overflow from bad parameters [8]: "this would require an attacker to execute arbitrary Smalltalk code on your server. Of course if they can do that they own you anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2] - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5]
http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8...
[6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence. * if you need callbacks * mac-only?
Plugins - write external code and dynamically link it to the VM Pros: * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm / * fast - faster than FFI * you can create new functionality that doesn't exist in any library Cons: * harder to write * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
I too am waiting to find something about NB that we should not like. No offense to Sig, there has to be _something_ wrong with it, right? :) A draft list of stuff we should check: (1) many examples of calls with int, float/double, pointers, structs, etc. (2) mac, Linux, Windows (3) callbacks Bonus: (4) calls on separate OS threads so the image can defend itself against something that does not return in a "reasonable" time, with reasonable being up to either the programmer or the end user (which the programmer can hopefully enforce if the image is not locked). Bill ________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Denis Kudriashov Sent: Tuesday, May 25, 2010 1:08 PM To: Pharo-project Subject: Re: [Pharo-project] FFI Documentation Hi. Igor Can you compare NB callout functionality with FFI and Alien functionality. Can NB replace FFI, Alien? What absent in NB callout? You implement callbacks with NB. FFI not support it. For me NB foreign interface seems more attractive than something else. 2010/5/21 Igor Stasenko <siguctua@gmail.com<mailto:siguctua@gmail.com>> Very nice topic coverage. I hope, eventually NB will also have something like this :) But it having a little different pros/cons table: NB Pros: * as well as for plugin, you can create new functionality that doesn't exist in any library * no need to recompile plugin/VM when you making a changes * all your code is distributed with the Smalltalk code, but there can be complications with platforms * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster Cons: * ?unsafe? - you have to provide a safety layers (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there) * harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals. So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell. On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr<mailto:stephane.ducasse@inria.fr>> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. The focus in on FFI, with stubs for other strategies. Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
<apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification" ^self externalCallFailed.
Let's take it piece by piece:
system: aString Method name - by convention named 'apiXxx'
<apicall: long 'system' (char*) module: 'libSystem.dylib'> Function specification - should be the first line - enclosed in angle brackets: < > containing: 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention) - Mac - use either one - Unix - use cdecl - Windows - use apical 2. Return Type (see types) 3. External Function Name (literal string) 4. Argument Types (a literal array) 5. Module - "module: " + [filename of the external library (literal string)] (see below).
self externalCallFailed. Failure handler - normal smalltalk code - executed if the linking to or calling the external function fails - API calls don't know how to communicate failure like Squeak primitives do, so: - it does not tell you whether the external function succeeded - the most common code is simply '^self externalCallFailed.'
Argument Types - must be names of ExternalTypes, either: - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes): void bool byte (unsigned) sbyte (signed) ushort (16-bit unsigned) short (16-bit signed) ulong (32-bit unsigned) long (32-bit signed) ulonglong (64-bit unsigned) longlong (64-bit signed) char (unsigned) schar (signed) float (single-precision float) double (double-precision float)
Structure Types [4] - subclass of ExternalStructure - class>>fields that returns an array of field descriptions (see below) - Example: fields ^#((red 'byte')(green 'byte')(blue 'byte')) - class>>initialize which includes "self defineFields" (which must be called before using the class) - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4] - 2-element array (or three but that does something else, I'm not sure what): - first element is the field name - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform - Mac - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c' - Snow Leopard - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below) - With pre-mach-o VMs - For Classic applications, use 'InterfaceLib' - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives - depends on the platform - Mac - pre Snow Leopard - checks VM path and common library paths - Snow Leopard - only looks in VM bundle's Resources file, you must either [5]: - store all external libraries there - ln -s path/to/library path/to/VM/Resources/library_name - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats - security - malicious users could call arbitrary functions in the OS e.g. "format c:" from "system.dll" [7] - VMs do not protect against buffer overflow from bad parameters [8]: "this would require an attacker to execute arbitrary Smalltalk code on your server. Of course if they can do that they own you anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2] - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5] http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence. * if you need callbacks * mac-only?
Plugins - write external code and dynamically link it to the VM Pros: * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm / * fast - faster than FFI * you can create new functionality that doesn't exist in any library Cons: * harder to write * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr<mailto:Pharo-project@lists.gforge.inria.fr> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr<mailto:Pharo-project@lists.gforge.inria.fr> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig. _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr<mailto:Pharo-project@lists.gforge.inria.fr> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
2010/5/25 Schwab,Wilhelm K <bschwab@anest.ufl.edu>:
I too am waiting to find something about NB that we should not like. No offense to Sig, there has to be _something_ wrong with it, right? :)
Right. :) Wrong, is lack of Unix and mac support. It should be written. Examples is missing too.
A draft list of stuff we should check:
(1)Â many examples of calls with int, float/double, pointers, structs, etc. (2) mac, Linux, Windows (3) callbacks
Bonus: (4) calls on separate OS threads so the image can defend itself against something that does not return in a "reasonable" time, with reasonable being up to either the programmer or the end user (which the programmer can hopefully enforce if the image is not locked).
Bill
________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Denis Kudriashov Sent: Tuesday, May 25, 2010 1:08 PM To: Pharo-project Subject: Re: [Pharo-project] FFI Documentation
Hi. Igor
Can you compare NB callout functionality with FFI and Alien functionality. Can NB replace FFI, Alien?
What absent in NB callout?
You implement callbacks with NB. FFI not support it. For me NB foreign interface seems more attractive than something else.
2010/5/21 Igor Stasenko <siguctua@gmail.com>
Very nice topic coverage. I hope, eventually NB will also have something like this :)
But it having a little different pros/cons table:
NB Pros:        * as well as for plugin, you can create new functionality that doesn't exist in any library        * no need to recompile plugin/VM when you making a changes        * all your code is distributed with the Smalltalk code, but there can be complications with platforms        * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster
   Cons:        * ?unsafe? - you have to provide a safety layers        (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there)
       * harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals.
So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell.
On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. Â The focus in on FFI, with stubs for other strategies. Â Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Â Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. Â FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
   <apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification"    ^self externalCallFailed.
Let's take it piece by piece:
   system: aString        Method name - by convention named 'apiXxx'
   <apicall: long 'system' (char*) module: 'libSystem.dylib'>        Function specification            - should be the first line            - enclosed in angle brackets: < > containing:                1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention)                    - Mac - use either one                    - Unix - use cdecl                    - Windows - use apical                2. Return Type (see types)                3. External Function Name (literal string)                4. Argument Types (a literal array)                5. Module - "module: " + [filename of the external library (literal string)] (see below).
   self externalCallFailed.        Failure handler            - normal smalltalk code            - executed if the linking to or calling the external function fails            - API calls don't know how to communicate failure like Squeak primitives do, so:                - it does not tell you whether the external function succeeded                - the most common code is simply '^self externalCallFailed.'
Argument Types    - must be names of ExternalTypes, either:        - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes):            void            bool            byte (unsigned)            sbyte (signed)            ushort (16-bit unsigned)            short (16-bit signed)            ulong (32-bit unsigned)            long (32-bit signed)            ulonglong (64-bit unsigned)            longlong (64-bit signed)            char (unsigned)            schar (signed)            float (single-precision float)            double (double-precision float)
Structure Types [4]    - subclass of ExternalStructure        - class>>fields that returns an array of field descriptions (see below)            - Example:                fields                    ^#((red  'byte')(green 'byte')(blue  'byte'))        - class>>initialize which includes "self defineFields" (which must be called before using the class)    - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4]    - 2-element array (or three but that does something else, I'm not sure what):        - first element is the field name        - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform    - Mac        - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c'        - Snow Leopard            - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below)        - With pre-mach-o VMs            - For Classic applications, use 'InterfaceLib'            - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives    - depends on the platform        - Mac            - pre Snow Leopard                - checks VM path and common library paths            - Snow Leopard                - only looks in VM bundle's Resources file, you must either [5]:                    - store all external libraries there                    - ln -s path/to/library path/to/VM/Resources/library_name                    - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats    - security        - malicious users could call arbitrary functions in the OS e.g.  "format c:" from "system.dll" [7]        - VMs do not protect against buffer overflow from bad parameters [8]:            "this would require an attacker to execute arbitrary Smalltalk            code on your server. Of course if they can do that they own you            anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty    - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2]    - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5]
http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence.    * if you need callbacks    * mac-only?
Plugins - write external code and dynamically link it to the VM    Pros:        * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm /        * fast - faster than FFI        * you can create new functionality that doesn't exist in any library    Cons:        * harder to write        * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
Sig, *nix/mac is **essential** - otherwise, we are not cross-platform. What is involved in writing the interfaces? I fear it is a little out of my league, but you have my attention :) You mentioned that anything can be done because one can use native code: true enough and a strength, but I don't want to be forced to resort to it, especially for simple things. In fact, therein might lurk a weakness of Alien; IIRC, the author(s) of its documentation declined to boast of its call-out capabilities, leaving me wondering whether one is left to manage the stack rather than having the callout mechanism do it. I am working on a wrapper for the Gnu Scientific Library. Given the unix code, I could take a shot at transforming my code to use it. If you have callbacks, I would have a big reason to dive into it. As it is, I am working around the need for callbacks, but it will eventually become unavoidable. One thing I am considering for curve fitting is to create a .so with a large number of suitable functions whose addresses can be obtained and given to the library. Ultimately, we need callbacks though. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko Sent: Tuesday, May 25, 2010 4:51 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] FFI Documentation 2010/5/25 Schwab,Wilhelm K <bschwab@anest.ufl.edu>:
I too am waiting to find something about NB that we should not like. No offense to Sig, there has to be _something_ wrong with it, right? :)
Right. :) Wrong, is lack of Unix and mac support. It should be written. Examples is missing too.
A draft list of stuff we should check:
(1) many examples of calls with int, float/double, pointers, structs, etc. (2) mac, Linux, Windows (3) callbacks
Bonus: (4) calls on separate OS threads so the image can defend itself against something that does not return in a "reasonable" time, with reasonable being up to either the programmer or the end user (which the programmer can hopefully enforce if the image is not locked).
Bill
________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Denis Kudriashov Sent: Tuesday, May 25, 2010 1:08 PM To: Pharo-project Subject: Re: [Pharo-project] FFI Documentation
Hi. Igor
Can you compare NB callout functionality with FFI and Alien functionality. Can NB replace FFI, Alien?
What absent in NB callout?
You implement callbacks with NB. FFI not support it. For me NB foreign interface seems more attractive than something else.
2010/5/21 Igor Stasenko <siguctua@gmail.com>
Very nice topic coverage. I hope, eventually NB will also have something like this :)
But it having a little different pros/cons table:
NB Pros: * as well as for plugin, you can create new functionality that doesn't exist in any library * no need to recompile plugin/VM when you making a changes * all your code is distributed with the Smalltalk code, but there can be complications with platforms * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster
Cons: * ?unsafe? - you have to provide a safety layers (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there)
* harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals.
So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell.
On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. The focus in on FFI, with stubs for other strategies. Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
<apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification" ^self externalCallFailed.
Let's take it piece by piece:
system: aString Method name - by convention named 'apiXxx'
<apicall: long 'system' (char*) module: 'libSystem.dylib'> Function specification - should be the first line - enclosed in angle brackets: < > containing: 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention) - Mac - use either one - Unix - use cdecl - Windows - use apical 2. Return Type (see types) 3. External Function Name (literal string) 4. Argument Types (a literal array) 5. Module - "module: " + [filename of the external library (literal string)] (see below).
self externalCallFailed. Failure handler - normal smalltalk code - executed if the linking to or calling the external function fails - API calls don't know how to communicate failure like Squeak primitives do, so: - it does not tell you whether the external function succeeded - the most common code is simply '^self externalCallFailed.'
Argument Types - must be names of ExternalTypes, either: - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes): void bool byte (unsigned) sbyte (signed) ushort (16-bit unsigned) short (16-bit signed) ulong (32-bit unsigned) long (32-bit signed) ulonglong (64-bit unsigned) longlong (64-bit signed) char (unsigned) schar (signed) float (single-precision float) double (double-precision float)
Structure Types [4] - subclass of ExternalStructure - class>>fields that returns an array of field descriptions (see below) - Example: fields ^#((red 'byte')(green 'byte')(blue 'byte')) - class>>initialize which includes "self defineFields" (which must be called before using the class) - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4] - 2-element array (or three but that does something else, I'm not sure what): - first element is the field name - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform - Mac - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c' - Snow Leopard - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below) - With pre-mach-o VMs - For Classic applications, use 'InterfaceLib' - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives - depends on the platform - Mac - pre Snow Leopard - checks VM path and common library paths - Snow Leopard - only looks in VM bundle's Resources file, you must either [5]: - store all external libraries there - ln -s path/to/library path/to/VM/Resources/library_name - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats - security - malicious users could call arbitrary functions in the OS e.g. "format c:" from "system.dll" [7] - VMs do not protect against buffer overflow from bad parameters [8]: "this would require an attacker to execute arbitrary Smalltalk code on your server. Of course if they can do that they own you anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2] - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5]
http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence. * if you need callbacks * mac-only?
Plugins - write external code and dynamically link it to the VM Pros: * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm / * fast - faster than FFI * you can create new functionality that doesn't exist in any library Cons: * harder to write * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig. _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On 26 May 2010 01:58, Schwab,Wilhelm K <bschwab@anest.ufl.edu> wrote:
Sig,
*nix/mac is **essential** - otherwise, we are not cross-platform. Â What is involved in writing the interfaces? Â I fear it is a little out of my league, but you have my attention :)
You mentioned that anything can be done because one can use native code: true enough and a strength, but I don't want to be forced to resort to it, especially for simple things. Â In fact, therein might lurk a weakness of Alien; IIRC, the author(s) of its documentation declined to boast of its call-out capabilities, leaving me wondering whether one is left to manage the stack rather than having the callout mechanism do it.
I am working on a wrapper for the Gnu Scientific Library. Â Given the unix code, I could take a shot at transforming my code to use it. Â If you have callbacks, I would have a big reason to dive into it. Â As it is, I am working around the need for callbacks, but it will eventually become unavoidable. Â One thing I am considering for curve fitting is to create a .so with a large number of suitable functions whose addresses can be obtained and given to the library. Â Ultimately, we need callbacks though.
For porting it onto other platform, the basic procedure is following: - subclass NativeBoost class - implement a bootstrap routines: #pointerSize #newAssembler #initializeExternalHeap #generateCallgateCode #basicAllocate: aSize on Win32 i am using a HeapCreate/HeapAlloc kernel functions for working with extenal heap. On unixes same functionality could be provided by mmap + own heap management code, or, link with some library, which provides a basic heap operations (allocate/free) on a heap with execution enabled. The main feature, what a platform-specific code should provide is to enable putting a native code into external heap so, other NB parts could be able to make calls to it. This means that memory, which allocated by calls to NativeBoost>>allocate: should always have an execution enabled. http://linux.die.net/man/2/mmap http://linux.die.net/man/2/mprotect
Bill
-----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Igor Stasenko Sent: Tuesday, May 25, 2010 4:51 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] FFI Documentation
2010/5/25 Schwab,Wilhelm K <bschwab@anest.ufl.edu>:
I too am waiting to find something about NB that we should not like. No offense to Sig, there has to be _something_ wrong with it, right? :)
Right. :) Wrong, is lack of Unix and mac support. It should be written.
Examples is missing too.
A draft list of stuff we should check:
(1) many examples of calls with int, float/double, pointers, structs, etc. (2) mac, Linux, Windows (3) callbacks
Bonus: (4) calls on separate OS threads so the image can defend itself against something that does not return in a "reasonable" time, with reasonable being up to either the programmer or the end user (which the programmer can hopefully enforce if the image is not locked).
Bill
________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Denis Kudriashov Sent: Tuesday, May 25, 2010 1:08 PM To: Pharo-project Subject: Re: [Pharo-project] FFI Documentation
Hi. Igor
Can you compare NB callout functionality with FFI and Alien functionality. Can NB replace FFI, Alien?
What absent in NB callout?
You implement callbacks with NB. FFI not support it. For me NB foreign interface seems more attractive than something else.
2010/5/21 Igor Stasenko <siguctua@gmail.com>
Very nice topic coverage. I hope, eventually NB will also have something like this :)
But it having a little different pros/cons table:
NB Pros:        * as well as for plugin, you can create new functionality that doesn't exist in any library        * no need to recompile plugin/VM when you making a changes        * all your code is distributed with the Smalltalk code, but there can be complications with platforms        * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster
   Cons:        * ?unsafe? - you have to provide a safety layers        (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there)
       * harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals.
So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell.
On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. Â The focus in on FFI, with stubs for other strategies. Â Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Â Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. Â FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
   <apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification"    ^self externalCallFailed.
Let's take it piece by piece:
   system: aString        Method name - by convention named 'apiXxx'
   <apicall: long 'system' (char*) module: 'libSystem.dylib'>        Function specification            - should be the first line            - enclosed in angle brackets: < > containing:                1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention)                    - Mac - use either one                    - Unix - use cdecl                    - Windows - use apical                2. Return Type (see types)                3. External Function Name (literal string)                4. Argument Types (a literal array)                5. Module - "module: " + [filename of the external library (literal string)] (see below).
   self externalCallFailed.        Failure handler            - normal smalltalk code            - executed if the linking to or calling the external function fails            - API calls don't know how to communicate failure like Squeak primitives do, so:                - it does not tell you whether the external function succeeded                - the most common code is simply '^self externalCallFailed.'
Argument Types    - must be names of ExternalTypes, either:        - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes):            void            bool            byte (unsigned)            sbyte (signed)            ushort (16-bit unsigned)            short (16-bit signed)            ulong (32-bit unsigned)            long (32-bit signed)            ulonglong (64-bit unsigned)            longlong (64-bit signed)            char (unsigned)            schar (signed)            float (single-precision float)            double (double-precision float)
Structure Types [4]    - subclass of ExternalStructure        - class>>fields that returns an array of field descriptions (see below)            - Example:                fields                    ^#((red  'byte')(green 'byte')(blue  'byte'))        - class>>initialize which includes "self defineFields" (which must be called before using the class)    - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4]    - 2-element array (or three but that does something else, I'm not sure what):        - first element is the field name        - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform    - Mac        - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c'        - Snow Leopard            - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below)        - With pre-mach-o VMs            - For Classic applications, use 'InterfaceLib'            - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives    - depends on the platform        - Mac            - pre Snow Leopard                - checks VM path and common library paths            - Snow Leopard                - only looks in VM bundle's Resources file, you must either [5]:                    - store all external libraries there                    - ln -s path/to/library path/to/VM/Resources/library_name                    - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats    - security        - malicious users could call arbitrary functions in the OS e.g.  "format c:" from "system.dll" [7]        - VMs do not protect against buffer overflow from bad parameters [8]:            "this would require an attacker to execute arbitrary Smalltalk            code on your server. Of course if they can do that they own you            anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty    - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2]    - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5]
http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence.    * if you need callbacks    * mac-only?
Plugins - write external code and dynamically link it to the VM    Pros:        * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm /        * fast - faster than FFI        * you can create new functionality that doesn't exist in any library    Cons:        * harder to write        * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
2010/5/25 Denis Kudriashov <dionisiydk@gmail.com>:
Hi. Igor
Can you compare NB callout functionality with FFI and Alien functionality. Can NB replace FFI, Alien?
Yes, it can.
What absent in NB callout?
I don't know. Its hard to imagine what can't be done in it, since you having a direct access to native code generation.
You implement callbacks with NB. FFI not support it. For me NB foreign interface seems more attractive than something else.
2010/5/21 Igor Stasenko <siguctua@gmail.com>
Very nice topic coverage. I hope, eventually NB will also have something like this :)
But it having a little different pros/cons table:
NB Pros:        * as well as for plugin, you can create new functionality that doesn't exist in any library        * no need to recompile plugin/VM when you making a changes        * all your code is distributed with the Smalltalk code, but there can be complications with platforms        * fast - faster than any FFI implementation written in C, and as fast as plugin primitive or even faster
   Cons:        * ?unsafe? - you have to provide a safety layers        (But hey, you have to deal with same sorts of stuff, when writing plugin. No magician workers there)
       * harder to write Yes, its harder than plain smalltalk - true. But i can't say, that writing an assembler is harder than writing a plugin's slang code. If you writing a plugin, you should have an expertise, in VM internals and how to build VM , etc etc and if you writing an native code, you should have an expertise in assembler as well as VM internals.
So, they are different, and definitely much harder comparing to plain smalltalk code, but which one is easier is hard to tell.
On 20 May 2010 22:25, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
thanks sean this is cool to see such kind of documentation emerging.
Stef
On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote:
I want to get this info into the help system and/or collaborative book, as similar questions pop up regularly on the lists.
I compiled every piece of info I could find. Â The focus in on FFI, with stubs for other strategies. Â Please read this *very* rough outline for missing info and inaccuracies:
FFI
What is FFI and what is it used for? Â Calling functions in libraries outside of the image...
FFI, the Squeak Foreign Function Interface, is used to call functions located in shared libraries that are not part of the Squeak VM nor its plugins. It also provides means to read and write memory structures that are associated with the use of those shared libraries. A typical use is to directly invoke operating system APIs. As such, applications that use FFI can only be used on the platform(s) that support the particular API being used. C conventions are used throughout, though the external function could have been written by any language capable of generating object code that follows C conventions. Â FFI is probably the easiest way to do the things it does. FFI is pretty fast too. Croquet uses FFI calls to OpenGL for all it's drawing routines.[1]
How does FFI work? Technically what happens is: * you define what the interface is - the parameters, types etc. * when you make the call, the FFI logic assembles the data from the Squeak Objects into the proper structures according to the routine calling conventions for your architecture, and of course manages the return values. So no magic but perhaps just a little assembler in the plugin to properly deal with all the registers and condition flags.
How do I use it?
1. make a method (whose structure is similar to a named primitive method)
Example: system: aString "name (by convention is apiXxx: e.g. apiSystem:)"
   <apicall: long 'system' (char*) module: 'libSystem.dylib'> "first line should be the external function specification"    ^self externalCallFailed.
Let's take it piece by piece:
   system: aString        Method name - by convention named 'apiXxx'
   <apicall: long 'system' (char*) module: 'libSystem.dylib'>        Function specification            - should be the first line            - enclosed in angle brackets: < > containing:                1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention)                    - Mac - use either one                    - Unix - use cdecl                    - Windows - use apical                2. Return Type (see types)                3. External Function Name (literal string)                4. Argument Types (a literal array)                5. Module - "module: " + [filename of the external library (literal string)] (see below).
   self externalCallFailed.        Failure handler            - normal smalltalk code            - executed if the linking to or calling the external function fails            - API calls don't know how to communicate failure like Squeak primitives do, so:                - it does not tell you whether the external function succeeded                - the most common code is simply '^self externalCallFailed.'
Argument Types    - must be names of ExternalTypes, either:        - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes):            void            bool            byte (unsigned)            sbyte (signed)            ushort (16-bit unsigned)            short (16-bit signed)            ulong (32-bit unsigned)            long (32-bit signed)            ulonglong (64-bit unsigned)            longlong (64-bit signed)            char (unsigned)            schar (signed)            float (single-precision float)            double (double-precision float)
Structure Types [4]    - subclass of ExternalStructure        - class>>fields that returns an array of field descriptions (see below)            - Example:                fields                    ^#((red  'byte')(green 'byte')(blue  'byte'))        - class>>initialize which includes "self defineFields" (which must be called before using the class)    - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
Field description [4]    - 2-element array (or three but that does something else, I'm not sure what):        - first element is the field name        - second is the type
Mac Memory Allocation Issues [4] (not sure about this)
If you allocate external structures, those with memory outside the Squeak process space, you may need to increase the amount of memory that is reserved outside the object heap for such use. The Mac OS (9 and previous) needs this, other platforms may be able to dynamically get more memory from the OS. To see how much memory is currently reserved printIt 'Smalltalk extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: someNumberofBytes' Then save your image and quit. When you next start up, the amount of memory you requested will be reserved. (JMM) Note the OSX versions of the VM ignore extraVMMemory because the memory model for OS-X/unix applications is quite different.
Module Name - depends on the platform    - Mac        - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c'        - Snow Leopard            - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below)        - With pre-mach-o VMs            - For Classic applications, use 'InterfaceLib'            - For Carbon libs, use 'CarbonLib'
Module Location - where the external library file lives    - depends on the platform        - Mac            - pre Snow Leopard                - checks VM path and common library paths            - Snow Leopard                - only looks in VM bundle's Resources file, you must either [5]:                    - store all external libraries there                    - ln -s path/to/library path/to/VM/Resources/library_name                    - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false." Caveats    - security        - malicious users could call arbitrary functions in the OS e.g.  "format c:" from "system.dll" [7]        - VMs do not protect against buffer overflow from bad parameters [8]:            "this would require an attacker to execute arbitrary Smalltalk            code on your server. Of course if they can do that they own you            anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
* difficulty    - if you make a mistake you'll not drop into the debugger but Squeak will just crash [2]    - If you crash Squeak when it is running the garbage collector, then you know your FFI code is leaking bits into object memory [2]
What do I need to use FFI with Squeak?
You need the FFI plugin, which is included with most VM's as of Squeak 3.6 or so.
You can also build the plugin yourself. See VMMaker.
References: [1] http://wiki.squeak.org/squeak/1414 [2] http://wiki.squeak.org/squeak/2424 [3] http://wiki.squeak.org/squeak/5716 [4] http://wiki.squeak.org/squeak/2426 [5]
http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td8... [6] http://wiki.squeak.org/squeak/5846 [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 [8] http://forum.world.st/Security-td99624.html#a99635:
Other choices: In the fall of 2008, Alien the FFI interface (written by Cadence Design Systems, Inc.) was put into squeaksource: http://www.squeaksource.com/Alien.html. This API allows the primitive to call back to Smalltalk Code, and return error code information, and apparently is much faster due to a less complex call sequence.    * if you need callbacks    * mac-only?
Plugins - write external code and dynamically link it to the VM    Pros:        * safest - users are limited to using the functionality you provide and can not call other external functions e.g. system rm /        * fast - faster than FFI        * you can create new functionality that doesn't exist in any library    Cons:        * harder to write        * plugin must be distributed with the Smalltalk code - there can be complications with platforms
Primitive method - invokes behavior in the VM or a plugin [3]
Questions: * why would you want to build the FFI plugin yourself? * api prefix or no for method names? * not sure about pre Snow Leopard library search * OSProcess - how does this fit into the bigger picture? * "^self externalCallFailed." or "self externalCallFailed." i.e. return self or return the result, or doesn't matter? * why would a field description have three elements? * Mac Memory Allocation Issues? -- View this message in context: http://forum.world.st/FFI-Documentation-tp2225148p2225148.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Best regards, Igor Stasenko AKA sig.
2010/5/20 Sean P. DeNigris <sean@clipperadams.com>:
    <apicall: long 'system' (char*) module: 'libSystem.dylib'>         Function specification             - should be the first line             - enclosed in angle brackets: < > containing:                 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention)                     - Mac - use either one                     - Unix - use cdecl                     - Windows - use apical
In Windows the cdecl calling convention may be used when you want your code interfacing the external library to be portable to other platforms.
Questions: * why would you want to build the FFI plugin yourself?
To provide better failure handlers. Most of the time writing an interface is lost decoding the type of failure description when calling external functions, specially the conversion of types from Smalltalk to C (coercion).
* why would a field description have three elements?
I think this is the case when you have in C char myArray [8] then you would write in the fields description: (myArray 'char' 8) Cheers, Hernán
participants (6)
-
Denis Kudriashov -
Hernán Morales Durand -
Igor Stasenko -
Schwab,Wilhelm K -
Sean P. DeNigris -
Stéphane Ducasse