Lukas sad sad. May be eliot was not aware of your work. may be we should ressurect what you did and now we do not care about the FFI maintainer :) We can have PharoFFI if needed. Stef

...

I'd like to discuss, what changes to compiler we may introduce to allow FFI to handle pragmas in non-intrusive manner. I suppose something, like registration mechanism for compiler, where any external package can register and provide own handler(s) for pragmas. Then, of course, an Encoder should have a public API, which would allow such handlers to add/change literals and method header.

We already having at least another use of pragmas - preferences. It's also could use same mechanism which can be used to notify a preferences system about appearance of new or removal existing preferences as a result of method compilation.

In fact we had that in Squeak 3.9 and early Pharo versions. I even wrote code that adapted this mechanism for FFI, but the FFI maintainer insisted on sticking with his compiler hacks.

The idea was that you could add methods with a specific pragma to one of the compiler classes (I don't remember which one). There was also one such method for each of the Smalltalk primitives currently supported (#primitive:, 3primitive:module:, ...). When encountering a pragma in the source code the compiler would check if this was a compiler-pragma with the same name defined and perform that method. In the case of the primitive the compiler would change the compile method to use the configured primitive. A very strait forward use of pragmas.

However the functionality that made compiler pragmas and primitives extensible is gone today even in Pharo. For reasons unknown to me Eliot removed all the code when he added the closure support. Back to hacking and patching the compiler.

Lukas

-- Lukas Renggli http://www.lukas-renggli.ch

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Yes Alien is the way to go. Stef On Feb 14, 2010, at 10:28 AM, Lukas Renggli wrote:

...

sad sad. May be eliot was not aware of your work. may be we should ressurect what you did and now we do not care about the FFI maintainer :) We can have PharoFFI if needed.

I don't have a problem with that. I am just saying that we've been there and we've done that :-)

I don't care about FFI. According to Fernando Alien seems the way to go nowadays anyway. As far as I know it does not even depend on the compiler. What is way better than anything else.

Lukas

-- Lukas Renggli http://www.lukas-renggli.ch

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

...

Yes Alien is the way to go.

I wonder why you think that. Alien is - mac only (you may say it's not true, because one could build a unix or windows vm that supports it, but that didn't happen in the past 1.5 years) - x86 only

I'm not knowledgeable enough but I refer here to some discussions I got with eliot when we were at amsterdam. The comments I heard on the FFI implementation were not that positive. Now may be there is a road for FFI and Alien and it would be good to have the extension lukas did in 3.9. Stef

Adding a multi-threaded FFI interface is always feasible. What I'd suggest is some get their corporate sponsor to pay for it as part of building the Alien FFI plugin for Unix and Windows. Then gift it back to the community, otherwise you'll see zero movement. On 2010-02-14, at 4:42 AM, Mariano Martinez Peck wrote:

I have a simple question: Does someone know if AlienFFI still locks the complete VM while executing a function ? I mean, it behaves exactly as normal FFI in that aspect ? I would love a FFI that doesn't lock the VM. Or even better, let you choose, like VW FFI.

Cheers

Mariano

On Sun, Feb 14, 2010 at 11:44 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:

...

...

Yes Alien is the way to go.

I wonder why you think that. Alien is - mac only (you may say it's not true, because one could build a unix or windows vm that supports it, but that didn't happen in the past 1.5 years) - x86 only

I'm not knowledgeable enough but I refer here to some discussions I got with eliot when we were at amsterdam. The comments I heard on the FFI implementation were not that positive.

Now may be there is a road for FFI and Alien and it would be good to have the extension lukas did in 3.9.

Stef

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

-- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

FFI is weak (nearly useless??) on callbacks, where Alien appears to provide good support. I agree the neither system is comlete w/o the ability to direct calls to an os thread to avoid blocking the entire vm. Bill ________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Mariano Martinez Peck Sent: Sunday, February 14, 2010 5:36 PM To: Pharo-project@lists.gforge.inria.fr; johnmci@smalltalkconsulting.com Subject: Re: [Pharo-project] Making FFI non-intrusive (Was: Re: Pharo changing the game) Ok...if AlienFFI doesn't support multi-threaded, then, I think the same as Levente. Why should I use AlienFFI instead of the normal FFI ? Which advantages does Alien have? Does someone know ? (I ask just because of my ignorance) Because for the moment it has two disadvantages, the ones Levente said: only mac, only x86. Thanks Mariano 2010/2/14 John M McIntosh <johnmci@smalltalkconsulting.com<mailto:johnmci@smalltalkconsulting.com>> Adding a multi-threaded FFI interface is always feasible. What I'd suggest is some get their corporate sponsor to pay for it as part of building the Alien FFI plugin for Unix and Windows. Then gift it back to the community, otherwise you'll see zero movement. On 2010-02-14, at 4:42 AM, Mariano Martinez Peck wrote: I have a simple question: Does someone know if AlienFFI still locks the complete VM while executing a function ? I mean, it behaves exactly as normal FFI in that aspect ? I would love a FFI that doesn't lock the VM. Or even better, let you choose, like VW FFI. Cheers Mariano On Sun, Feb 14, 2010 at 11:44 AM, Stéphane Ducasse <stephane.ducasse@inria.fr<mailto:stephane.ducasse@inria.fr>> wrote:

...

Yes Alien is the way to go.

I wonder why you think that. Alien is - mac only (you may say it's not true, because one could build a unix or windows vm that supports it, but that didn't happen in the past 1.5 years) - x86 only

I'm not knowledgeable enough but I refer here to some discussions I got with eliot when we were at amsterdam. The comments I heard on the FFI implementation were not that positive. Now may be there is a road for FFI and Alien and it would be good to have the extension lukas did in 3.9. Stef _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr<mailto:Pharo-project@lists.gforge.inria.fr> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr<mailto:Pharo-project@lists.gforge.inria.fr> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project -- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com<mailto:johnmci@smalltalkconsulting.com>> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com =========================================================================== _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr<mailto:Pharo-project@lists.gforge.inria.fr> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Sorry, Lucas!  This was because I started the closure work in a Qwaq^H^H^H^HTeleplace image that lacked the pragma extensibility.  So when it came to port the Closure compiler to other dialects I a) didn't notice the extensibility stuff and b) overwrite it.

No problem, I thought that there was a particular reason that it got removed.

So what to do?  Please point me to/email me a change set of the pragma extensibility code and I will try to integrate it back into the closure compiler as soon as time allows.

These are the methods #primitive: and #primitive:module: that carry the <primitive> pragma to tell the compiler that these pragmas have to be considered during compilation: Parser>>primitive: anIntegerOrString "Create indexed primitive." <primitive> ^ anIntegerOrString isInteger ifTrue: [ anIntegerOrString ] ifFalse: [ anIntegerOrString isString ifTrue: [ self primitive: anIntegerOrString module: nil ] ifFalse: [ self expected: 'Indexed primitive' ] ] Parser>>primitive: aNameString module: aModuleStringOrNil "Create named primitive." <primitive> (aNameString isString and: [ aModuleStringOrNil isNil or: [ aModuleStringOrNil isString ] ]) ifFalse: [ ^ self expected: 'Named primitive' ]. self allocateLiteral: (Array with: (aModuleStringOrNil isNil ifFalse: [ aModuleStringOrNil asSymbol ]) with: aNameString asSymbol with: 0 with: 0). ^ 117 Then the primitive are setup from ... Parser>>pragmaPrimitives | pragmas primitives | self properties pragmas isEmpty ifTrue: [ ^ 0 ]. pragmas := Pragma allNamed: #primitive from: self class to: Parser. primitives := self properties pragmas select: [ :prim | pragmas anySatisfy: [ :prag | prag selector = prim keyword ] ]. primitives isEmpty ifTrue: [ ^ 0 ]. primitives size = 1 ifFalse: [ ^ self notify: 'Ambigous primitives' ]. ^ primitives first message sendTo: self .. which in turn is called from Parser>>method: doit context: ctxt encoder: encoderToUse to calculate the primitive number. Maybe this wasn't the smartest idea after all? I made this so that FFI (and other libraries) could easily extend the range of possible primitive types, but since it was never used it is probably sort of pointless? Lukas -- Lukas Renggli http://www.lukas-renggli.ch

Ok thanks for the historical refresh. Some people may now understand why we finally decided to do Pharo. Now let us stop ranting and think about the blue sky of the new horizon. What do we do in Pharo 1.1? :) I'm full ears open. Stef

...

...

...

It is interesting to note that of course, after this was added to 3.9, there was someone *very hard* arguing that adding Pragmas was a very dumb idea...

Who that person was is left as an exercise to the reader of course ;-)

Hint: FFI has its own proprietary hardcoded pragma format. It unnecessary complicates the compiler even when not loaded. Not to mention that it breaks all tools that have their own parser, such as RB for example.

I'd like to discuss, what changes to compiler we may introduce to allow FFI to handle pragmas in non-intrusive manner.

Very simple: FFI just needs to use the pragmas as introduces in 3.9, than remove all knowledge about FFI from the Compiler. Problem solved. Isn't that beautiful?

< cdecl: void 'InvalRect' ( MacRect ) module: 'InterfaceLib'>

of that would be changed so that the stuff inside the <> would be a standard Smalltalk Message expression, problem solved.

to illustrate... using the SMaCC grammar of the NewCompiler (a bit simplified) as extendend for Pragmas, Primitives and FFI by Mathiue Suen.

A method in Squeak has Pragmas:

Method: | MethodPattern Temporaries Pragmas Statements {#methodTempsPragma:};

So they are difined like this (we can have multiple ones):

Pragmas: "<" PragmaMessage ">" {#pragma:} | Pragmas "<" PragmaMessage ">" {#pragmas:}

So it's quite simple: there is one message expression inside... simple. But there is FFI. Primitives are there, too, but that's not really a problem: they fit in the Pragmas. primitive: 15 is a valid message in Smalltalk. (Math did the gramar for primitives to have nice error messages, but that's not needed)

PragmaMessage: Apicall {#messagePragma:} | Primitive {#messagePragma:} | MessagePragma {#messagePragma:};

How Primitive and MessagePragma are done is not important. But we need Gramar rules just to be able to parse FFI:

Apicall: TypeCall ExternalType IndexName "(" ParameterApicall <rightParentheses> {#externalCall:} | TypeCall ExternalType IndexName "(" ParameterApicall <rightParentheses> "module:" <string> {#externalModuleCall:};

IndexName: <string> {#externalFunction:} | <number> {#externalIndex:};

TypeCall: "apicall:" {#callConvention:} | "cdecl:" {#callConvention:};

ParameterApicall: ExternalType {#parameterExtCall:} | ParameterApicall ExternalType {#parametersExtCall:};

ExternalType: <name> {#externalType:} | <name> "*" {#externalTypePointer:};

And this is *always* in the Grammar. Even when FFI is not loaded! (a nice example where our simple notion of modularity brakes down completely).

No the solution here is simple: a small trivial modification to FFI to use the Evil Pragmas as introduced by the Dumb People (tm) in 3.9:

< cdecl: 'void InvalRect ( MacRect )' module: 'InterfaceLib'>

or *whatever*. just something that is a valid Smalltalk message expression... can't be hard, can it?

Ah. And and about "notion of modularity"... of course the problem "how can I extend the language from my package without having to spread code over the parser, compiler, debugger, browser..." is intellectually an interesting one. And, oh, we are payed for finding these kind of Problems and especially solving these Problems in a way that goes beyond of just doing a pragmatic and simple solution (aka what I described here). (They call it "Academic Research". Involves mostly a programming language called TeX and can be very boring at times but it's complementary to the practical world... in a very interesting way).

And int his context the result is, of course, tataaa....: Lukas' Helvetia:

http://scg.unibe.ch/research/helvetia/languageboxes

See? So *everything* fits together... research and non-research, problems and solutions (on multiple levels), progress in the small and the large!

Marcus

-- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Feb 14, 2010, at 4:01 PM, Schwab,Wilhelm K wrote:

Stef,

I've always understood why you launched Pharo.

On FFI/Alien and blocking the entire VM, I injected Dolphin's overlapped calls into a discussion of the (rather serious) problems that can arise when something does not return as soon as expected (or at all), and somebody with good VM awareness (I do not recall who[*]) described a way to get around it. I recall thinking what a loss it is that good ideas often don't get packaged for general use.

FWIW, here are my suggestions for ideas that should be given high priority:

I would like to be able to do it myself but I don't know how to do it.

(1) Alien plugins for Linux and Windows; Alien as a mac-only entity is far less attractive than a cross-platform alternative. yes we should get the one of linux running since there was a fix.

(2) For FFI, we need a generic callback mechanism. It sounds as though one can be built using a plugin and some indirection. (3) For FFI and Alien, we need something like Dolphin's overlapped calls. Hopefully a generic solution that starts a thread and waits on a semaphore can be created once, and perhaps eventually enhanced with some thread pooling options. For now, I'd take one os thread per call, or (perhaps the ideal) one os thread per Smalltalk process blocked on such a call, as I think Dolphin does now, in part due to Windows stupidities.

I also recall a lengthy discussion about efficiency in making calls to COM objects from Dolphin, with someone arguing very hard to make calls a certain way because the alternative (while much simpler) was too slow. The flaw in that argument was that calls in question were by nature few and infrequent; the overhead of making the call was dwarfed by the work that resulted. Much of my use of FFI is similar. I am not arguing in favor of inefficient calls, but a sub-optimal but flexible way to make a callback (FFI) and to make a call on an os thread (FFI/Alien) would be much appreciated in most situations,

I see

even if it takes a few extra milliseconds compared to something optimally coded if I had time and knowledge to do so. I do not make millions of external calls; I make a few that do millions of floating point operations. A little extra time to put a socket connect (thinking OpenSSL) call on a background thread is trivial next to the cost of having a direct call go into limbo and hang a server process.

Bill

[*] seriously, it might have been "the FFI maintainer" or somebody else. I do recall that whoever it is knows a lot more about the vm than I do.

________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Sunday, February 14, 2010 4:30 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] Making FFI non-intrusive (Was: Re: Pharo changing the game)

Ok thanks for the historical refresh. Some people may now understand why we finally decided to do Pharo. Now let us stop ranting and think about the blue sky of the new horizon. What do we do in Pharo 1.1? :) I'm full ears open.

Stef

...

...

...

...

It is interesting to note that of course, after this was added to 3.9, there was someone *very hard* arguing that adding Pragmas was a very dumb idea...

Who that person was is left as an exercise to the reader of course ;-)

Hint: FFI has its own proprietary hardcoded pragma format. It unnecessary complicates the compiler even when not loaded. Not to mention that it breaks all tools that have their own parser, such as RB for example.

I'd like to discuss, what changes to compiler we may introduce to allow FFI to handle pragmas in non-intrusive manner.

Very simple: FFI just needs to use the pragmas as introduces in 3.9, than remove all knowledge about FFI from the Compiler. Problem solved. Isn't that beautiful?

< cdecl: void 'InvalRect' ( MacRect ) module: 'InterfaceLib'>

of that would be changed so that the stuff inside the <> would be a standard Smalltalk Message expression, problem solved.

to illustrate... using the SMaCC grammar of the NewCompiler (a bit simplified) as extendend for Pragmas, Primitives and FFI by Mathiue Suen.

A method in Squeak has Pragmas:

Method: | MethodPattern Temporaries Pragmas Statements {#methodTempsPragma:};

So they are difined like this (we can have multiple ones):

Pragmas: "<" PragmaMessage ">" {#pragma:} | Pragmas "<" PragmaMessage ">" {#pragmas:}

So it's quite simple: there is one message expression inside... simple. But there is FFI. Primitives are there, too, but that's not really a problem: they fit in the Pragmas. primitive: 15 is a valid message in Smalltalk. (Math did the gramar for primitives to have nice error messages, but that's not needed)

PragmaMessage: Apicall {#messagePragma:} | Primitive {#messagePragma:} | MessagePragma {#messagePragma:};

How Primitive and MessagePragma are done is not important. But we need Gramar rules just to be able to parse FFI:

Apicall: TypeCall ExternalType IndexName "(" ParameterApicall <rightParentheses> {#externalCall:} | TypeCall ExternalType IndexName "(" ParameterApicall <rightParentheses> "module:" <string> {#externalModuleCall:};

IndexName: <string> {#externalFunction:} | <number> {#externalIndex:};

TypeCall: "apicall:" {#callConvention:} | "cdecl:" {#callConvention:};

ParameterApicall: ExternalType {#parameterExtCall:} | ParameterApicall ExternalType {#parametersExtCall:};

ExternalType: <name> {#externalType:} | <name> "*" {#externalTypePointer:};

And this is *always* in the Grammar. Even when FFI is not loaded! (a nice example where our simple notion of modularity brakes down completely).

No the solution here is simple: a small trivial modification to FFI to use the Evil Pragmas as introduced by the Dumb People (tm) in 3.9:

< cdecl: 'void InvalRect ( MacRect )' module: 'InterfaceLib'>

or *whatever*. just something that is a valid Smalltalk message expression... can't be hard, can it?

Ah. And and about "notion of modularity"... of course the problem "how can I extend the language from my package without having to spread code over the parser, compiler, debugger, browser..." is intellectually an interesting one. And, oh, we are payed for finding these kind of Problems and especially solving these Problems in a way that goes beyond of just doing a pragmatic and simple solution (aka what I described here). (They call it "Academic Research". Involves mostly a programming language called TeX and can be very boring at times but it's complementary to the practical world... in a very interesting way).

And int his context the result is, of course, tataaa....: Lukas' Helvetia:

http://scg.unibe.ch/research/helvetia/languageboxes

See? So *everything* fits together... research and non-research, problems and solutions (on multiple levels), progress in the small and the large!

Marcus

-- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project