July 2013 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] NativeBoost and lacking string argument/return handling

by Henrik Johansen

On Jul 5, 2013, at 4:59 , Igor Stasenko wrote: > Well, to my thinking , in C 'string' defined as 'a sequence of > characters (bytes) terminated by zero byte'. Yeah, except, you know, most/all libraries do not really deal with theoretically pure C strings, and we certainly when passing in "a string" from Smalltalk. In practice, we're dealing with passing/converting Smalltalk strings to functions expecting a pointer to bytes in a certain encoding. (either null-terminated or with an additional size arg) > There is nothing about encoding, because it is related to text, but C > 'string' can be something else than human-readable text and contain > anything and used for different purpose. > > From that perspective, if strictly following such definition, i do not > see how you can force encoding to be a mandatory part of it. (except > from having special encoding, named "NO_ENCODING" ;) And from that perspective, any user will *never* create safe bindings to/from anything but ASCII ByteStrings by default. FFI, meet 1988. And even those who do, have to jump through an excessive amount of hoops, ref https://dl.dropboxusercontent.com/u/6751081/Environment.zip > > But sure thing, i am not opposed to having a really nice > encoding/decoding schemes, which you can easily control. > > The caveat is that options is applicable to whole callout's generated code. > What if i have function which takes 2 arguments: > - string with utf-8 encoding > - string with utf-16 encoding > > so, you cannot put that in options, because you need to control it on > a per-argument basis. > > The only way i see how to do that is same as in NBUTF8StringExample: > you explicitly designate the type of argument e.g. > > void foo(utf8String x, utf16String y) No, not for FFI you don't. Any sane external library will never use different encodings for string parameters to the same method. In fact, I challenge you to find *any* which doesn't. Cheers, Henry

July 5, 2013

Re: [Pharo-dev] NativeBoost with no access to sources

by Igor Stasenko

Here is another quick and dirty recipe 4U: - make a extension method in CompiledMethod nbEnsureTempNames self propertyValueAt: #tempNames ifAbsent: [ self propertyValueAt: #tempNames put: self tempNames ] nbTempNames ^ self propertyValueAt: #tempNames NativeBoost class>>prepareForProduction CompiledMethod allInstancesDo: [:aMethod | ((aMethod primitive = 117) and: [ | lit | lit := aMethod literalAt: 1. lit first = #NativeBoostPlugin and: [ lit second == #primitiveNativeCall]]) ifTrue: [ aMethod nbEnsureTempNames ]. ] (and then sure thing, make sure to use #nbTempNames instead of tempNames). but for the future, i want a clear protocol in CompiledMethod to do that, and do not make it depend on system changes. Because if i do, the stuff which works in 2.0 (or 3.0) will suddenly stop working in next version, because of changes in system.. that's why i against even thinking that NB having something to do with method's source code. It needs method's argument names, and nothing else, period. -- Best regards, Igor Stasenko.

July 5, 2013

Re: [Pharo-dev] NativeBoost with no access to sources

by Igor Stasenko

On 5 July 2013 16:55, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: > > > Le 05/07/2013 16:36, Igor Stasenko a Ã©crit : > >> On 5 July 2013 16:09, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>> >>> >>> >>> Le 05/07/2013 15:31, Igor Stasenko a Ã©crit : >>>> >>>> >>>> On 5 July 2013 15:22, Igor Stasenko <siguctua(a)gmail.com> wrote: >>>>> >>>>> >>>>> On 5 July 2013 11:03, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>>>>> >>>>>> >>>>>> Could you rather test for unability to get the source of the method >>>>>> and >>>>>> raise a NB-defined error? So that I could write a fallback on non-NB >>>>>> code >>>>>> with a on: do:. Please :) >>>>>> >>>>>> (it would then both handle startup related errors and unreadable >>>>>> sources >>>>>> as >>>>>> well). >>>>>> >>>>>> And with a backport to 2.0, because the odds of seeing an Opal based >>>>>> solution on 2.0 are fairly low. >>>>>> >>>>> the problem is that you will get same error when you put wrong name >>>>> into signature >>>>> or when no sources avail e.g.: >>>>> >>>>> foo: bar >>>>> ^ self nbCall: #( void foo (int baz) ) >>>>> >>>>> will give same error (unable to find binding for name 'baz' in given >>>>> context). >>> >>> >>> >>> Of course. I could do with having the same error for both and knowing >>> that >>> when developping and testing I would get it for stupid syntax errors and >>> in >>> production to handle a real situation. >>> >>> >>>>> If its fine to you, i can do it. >>> >>> >>> >>> Maybe I can try myself to see if I can write, before entering that loader >>> stuff, a correct test which raises an error if the method source isn't >>> available. >>> >>> >> well, all you need is basicaly replace this: >> >> Error signal: 'Could not find accessor for variable named "' , >> argName , '" in ' , method name >> >> with: >> >> NBArgumentNotFound signal: argName >> >> .. or NBCannotBindName.. >> >> what name for exception? > > > I believe the proper way would be to have : > > NBCannotBindName (for the errors you already handle at the moment) > > and, somewhere in there, a > > NBMethodSourceNotAvailable > > triggered in NBFFICallout>>sender: if the source code is decompiledCode > (but, as far as I can see, there is no method to test a CompiledMethod for > that :)) > > Something like > > CompiledMethod>>hasSourceCode > "Say false if we have to reconstruct the code." > > | trailer source | > trailer := self trailer. > trailer sourceCode > ifNotNil: [:code | ^ true ]. > trailer hasSourcePointer > ifFalse: [ ^ false]. > "Situation normal; read the sourceCode from the file" > source := [self getSourceFromFile] > on: Error > do: [ :ex | ex return: nil]. > source isEmptyOrNil > ifTrue: [ ^ false ]. > ^ true > > and, in sender: > > aSenderContext method hasSourceCode > ifFalse: [NBMethodSourceNotAvailable signal: method printString] > > What do you think ? > I think it is not responsibility of NB to deal with availability of source code. NB simply using reflection protocol of system to get what it wants to get, and where it come (from source code, or from something else) is completely irrelevant. Yes, with current implementation, it ends up with getting & parsing source code, but it is not what NB is asking for: the intent is to simply ask a compiled method to provide the argument names. That's why i totally agree with introduction of NBCannotBindName exception, but i cannot see place for NBMethodSourceNotAvailable. Otherwise similar tools with similar requirements will need to (re)implement same? This should be a responsibility of kernel/system.. but not NB itself. > > Thierry > -- > Thierry Goubier > CEA list > Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s > 91191 Gif sur Yvette Cedex > France > Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 > -- Best regards, Igor Stasenko.

July 5, 2013

Re: [Pharo-dev] NativeBoost and lacking string argument/return handling

by Igor Stasenko

Well, to my thinking , in C 'string' defined as 'a sequence of characters (bytes) terminated by zero byte'. There is nothing about encoding, because it is related to text, but C 'string' can be something else than human-readable text and contain anything and used for different purpose. >From that perspective, if strictly following such definition, i do not see how you can force encoding to be a mandatory part of it. (except from having special encoding, named "NO_ENCODING" ;) But sure thing, i am not opposed to having a really nice encoding/decoding schemes, which you can easily control. On 5 July 2013 15:45, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: > I just had a glance at Nix/Win32Environment NB calls, and nearly cried... (context: http://www.youtube.com/watch?v=Um41DPPs5ZA&list=SP843D1D545F9F52B6&index=15… ) > > Which got me thinking, maybe an extra opt (possibly non-static, ref database libraries string encodings) if Strings are expected as parameters should be mandatory, for specifying the expected encoding of the library being called? > (oh yes, MANDATORY, if you don't know what is required, you shouldn't be passing in Smalltalk strings in the first place) > > You'd then be able to cut out ALL the manual cruft in the video's getEnvironmentVariableW: wrapper call basically needed for any current call to a method accepting Strings. > (assuming you're not ok with mysterious failures once parameter/return is outside ascii range) > > Might it work with a few choice assembly implementations of encoders (utf8 + utf16) along the lines of NBUTF8StringExample (expanded to handle both Wide- and ByteString arguments/returns), as well as a fallback (or just) using callback which employs in-image converters? > (Such a preamble would also be a good place to do embedded 0 character checks for args expected to be c strings: http://www.hakipedia.com/index.php/Poison_Null_Byte) > > This have some other implications as well of course; > -NBExternalAddress readString/writeString should be removed or made encoding aware. In the current form, they are dangerous/useless outside of pure asciiâ¦ > - How to handle char * parameters. Perhaps parameter-specific validity checks? > a) NBExternalAddress - Assume correctly manually encoded contents if allowExternalAddressOpt or whatever its called is set? > b) ByteArray - Assume correctly manually encoded contents? > c) String - Same as above, either encoding opt set, or return error > The caveat is that options is applicable to whole callout's generated code. What if i have function which takes 2 arguments: - string with utf-8 encoding - string with utf-16 encoding so, you cannot put that in options, because you need to control it on a per-argument basis. The only way i see how to do that is same as in NBUTF8StringExample: you explicitly designate the type of argument e.g. void foo(utf8String x, utf16String y) > Cheers, > Henry > > -- Best regards, Igor Stasenko.

July 5, 2013

Re: [Pharo-dev] NativeBoost with no access to sources

by Goubier Thierry

Le 05/07/2013 16:36, Igor Stasenko a Ã©crit : > On 5 July 2013 16:09, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >> >> >> Le 05/07/2013 15:31, Igor Stasenko a Ã©crit : >>> >>> On 5 July 2013 15:22, Igor Stasenko <siguctua(a)gmail.com> wrote: >>>> >>>> On 5 July 2013 11:03, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>>>> >>>>> Could you rather test for unability to get the source of the method and >>>>> raise a NB-defined error? So that I could write a fallback on non-NB >>>>> code >>>>> with a on: do:. Please :) >>>>> >>>>> (it would then both handle startup related errors and unreadable sources >>>>> as >>>>> well). >>>>> >>>>> And with a backport to 2.0, because the odds of seeing an Opal based >>>>> solution on 2.0 are fairly low. >>>>> >>>> the problem is that you will get same error when you put wrong name >>>> into signature >>>> or when no sources avail e.g.: >>>> >>>> foo: bar >>>> ^ self nbCall: #( void foo (int baz) ) >>>> >>>> will give same error (unable to find binding for name 'baz' in given >>>> context). >> >> >> Of course. I could do with having the same error for both and knowing that >> when developping and testing I would get it for stupid syntax errors and in >> production to handle a real situation. >> >> >>>> If its fine to you, i can do it. >> >> >> Maybe I can try myself to see if I can write, before entering that loader >> stuff, a correct test which raises an error if the method source isn't >> available. >> >> > well, all you need is basicaly replace this: > > Error signal: 'Could not find accessor for variable named "' , > argName , '" in ' , method name > > with: > > NBArgumentNotFound signal: argName > > .. or NBCannotBindName.. > > what name for exception? I believe the proper way would be to have : NBCannotBindName (for the errors you already handle at the moment) and, somewhere in there, a NBMethodSourceNotAvailable triggered in NBFFICallout>>sender: if the source code is decompiledCode (but, as far as I can see, there is no method to test a CompiledMethod for that :)) Something like CompiledMethod>>hasSourceCode "Say false if we have to reconstruct the code." | trailer source | trailer := self trailer. trailer sourceCode ifNotNil: [:code | ^ true ]. trailer hasSourcePointer ifFalse: [ ^ false]. "Situation normal; read the sourceCode from the file" source := [self getSourceFromFile] on: Error do: [ :ex | ex return: nil]. source isEmptyOrNil ifTrue: [ ^ false ]. ^ true and, in sender: aSenderContext method hasSourceCode ifFalse: [NBMethodSourceNotAvailable signal: method printString] What do you think ? Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

July 5, 2013

[update 3.0] #30257/#30258

by Marcus Denker

30257/30258 ----- 11143 Update Opal to latest version https://pharo.fogbugz.com/f/cases/11143 11095 Array >> #elementsForwardIdentityTo:copyHash: More explicit tests and comments https://pharo.fogbugz.com/f/cases/11095 11007 Pharo 3 now uses mustBeBooleanInMagic: by default https://pharo.fogbugz.com/f/cases/11007 Diff information: http://smalltalkhub.com/mc/Pharo/Pharo30/main/Tests-MarcusDenker.569.diff http://smalltalkhub.com/mc/Pharo/Pharo30/main/OpalCompiler-Tests-MarcusDenk… http://smalltalkhub.com/mc/Pharo/Pharo30/main/Kernel-MarcusDenker.1513.diff http://smalltalkhub.com/mc/Pharo/Pharo30/main/Collections-Sequenceable-Marc…

July 5, 2013

Re: [Pharo-dev] NativeBoost with no access to sources

by Igor Stasenko

On 5 July 2013 16:09, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: > > > Le 05/07/2013 15:31, Igor Stasenko a Ã©crit : >> >> On 5 July 2013 15:22, Igor Stasenko <siguctua(a)gmail.com> wrote: >>> >>> On 5 July 2013 11:03, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>>> >>>> Could you rather test for unability to get the source of the method and >>>> raise a NB-defined error? So that I could write a fallback on non-NB >>>> code >>>> with a on: do:. Please :) >>>> >>>> (it would then both handle startup related errors and unreadable sources >>>> as >>>> well). >>>> >>>> And with a backport to 2.0, because the odds of seeing an Opal based >>>> solution on 2.0 are fairly low. >>>> >>> the problem is that you will get same error when you put wrong name >>> into signature >>> or when no sources avail e.g.: >>> >>> foo: bar >>> ^ self nbCall: #( void foo (int baz) ) >>> >>> will give same error (unable to find binding for name 'baz' in given >>> context). > > > Of course. I could do with having the same error for both and knowing that > when developping and testing I would get it for stupid syntax errors and in > production to handle a real situation. > > >>> If its fine to you, i can do it. > > > Maybe I can try myself to see if I can write, before entering that loader > stuff, a correct test which raises an error if the method source isn't > available. > > well, all you need is basicaly replace this: Error signal: 'Could not find accessor for variable named "' , argName , '" in ' , method name with: NBArgumentNotFound signal: argName .. or NBCannotBindName.. what name for exception? -- Best regards, Igor Stasenko.

July 5, 2013

Re: [Pharo-dev] NativeBoost with no access to sources

by Igor Stasenko

On 5 July 2013 16:09, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: > > > Le 05/07/2013 15:31, Igor Stasenko a Ã©crit : >> >> On 5 July 2013 15:22, Igor Stasenko <siguctua(a)gmail.com> wrote: >>> >>> On 5 July 2013 11:03, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>>> >>>> Could you rather test for unability to get the source of the method and >>>> raise a NB-defined error? So that I could write a fallback on non-NB >>>> code >>>> with a on: do:. Please :) >>>> >>>> (it would then both handle startup related errors and unreadable sources >>>> as >>>> well). >>>> >>>> And with a backport to 2.0, because the odds of seeing an Opal based >>>> solution on 2.0 are fairly low. >>>> >>> the problem is that you will get same error when you put wrong name >>> into signature >>> or when no sources avail e.g.: >>> >>> foo: bar >>> ^ self nbCall: #( void foo (int baz) ) >>> >>> will give same error (unable to find binding for name 'baz' in given >>> context). > > > Of course. I could do with having the same error for both and knowing that > when developping and testing I would get it for stupid syntax errors and in > production to handle a real situation. > > >>> If its fine to you, i can do it. > > > Maybe I can try myself to see if I can write, before entering that loader > stuff, a correct test which raises an error if the method source isn't > available. > > >>>> >>>> Is there a real, practical difference between "at compilation time" and >>>> "when receiving MethodAdded event"? i.e. is the complexity added in the >>>> compilation process worthwhile? >>>> >>> the problem is that you may not always receive MethodAdded event, >>> because different tools putting different meaning into "this should be >>> compiled silently". >>> This is outstanding issue in Pharo.. and that's the main reason why i >>> would go with compiler (like >>> that i have guarantee that instance of CompiledMethod is well-formed, >>> and don't need to rely on >>> 3-rd party toolchain to deliver notification to me. > > > You mean that hacking support into the third party compiler isn't 3rd party > toolchain :) > well, to me the compiler's behavior is much much more predictable comparing to chances that when you loading code with broken Monticello/RPackage or running test, you won't receive the event you expecting. > >>> and besides, we need such hook mechanism in Compiler anyways (for many >>> other use cases) >>> and so, it will be not "renting a Boeing for single passenger" >>> >> >> oh.. and i forgot about most important aspect of it: >> i don't wanna see >> <primitive: #primitiveNativeCall module: #NativeBoostPlugin error: >> errorCode > >> in code, instead if wanna use >> >> <native> >> (or similar), >> which admit, much more concise, elegant and user-friendly, and hides >> unnecessary implementation detail. > > > I remember seeing people introducing syntax extensions inside smalltalk > methods in VisualWorks, would that work as well ? Customize on a per class / > per method which compiler we should use. > yes. But i don't wanna extend syntax in case of NB , obviously :) > > Thierry > -- > Thierry Goubier > CEA list > Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s > 91191 Gif sur Yvette Cedex > France > Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 > -- Best regards, Igor Stasenko.

July 5, 2013

Re: [Pharo-dev] [Moose-dev] SourceCity works

by Alexandre Bergel

> NBXLib means it will be available only for linux ? No idea. Roassal 3d works on Windows and OSX so far. No idea about Linux. Alexandre > > > abergel wrote >> Hi Erwan and all, >> >> Just to make a few things clear about Roassal 3d vs SourceCity. >> >> We are currently working on Roassal 3d, which intends to be a nice layer >> at the top of NBOpenGL and NBXLib to easily create 3d scenes with cubes, >> lights, camera, interactions, zooming, translucent coloring, >> clicking/rotating/translating individual elements and so on. Everything >> that is good for Roassal is equally good for 3D Scene. >> >> Having a city-like visualization will be one of our first objectives since >> (i) it is a compelling use of 3D and (ii) it is easy to implement (at >> least the bases). We started to work on a CityBuilder, to easily create >> cities from objects and provided metrics. Ideally, it would be great to >> have SourceCity based on our effort. Not now since we have just started >> Roassal 3D, but on some point we should merge (this is similar in the 2d >> World with Mondrian and EyeSee moving from plain Morphic to Roassal). >> >> Our objective is to offer a great 3d framework to make a truly flying >> SourceCity. >> Does this make sense to you? >> >> Cheers, >> Alexandre >> >> >> >> On Jul 4, 2013, at 10:47 AM, Erwan Douaille < > >> douailleerwan@ > >> > wrote: >> >>> https://ci.inria.fr/pharo-contribution/job/SourceCity/ >>> >>> done >>> >>> >>> 2013/7/4 Camillo Bruni < > >> camillobruni@ > >> > >>> >>> On 2013-07-04, at 15:35, Erwan Douaille < > >> douailleerwan@ > >> > wrote: >>> >>>> I updated the ConfigurationOfSourceCity. >>>> >>>> Gofer new >>>> smalltalkhubUser: 'ErwanDouaille' project: 'SourceCity'; >>>> configuration; >>>> load. >>>> ConfigurationOfSourceCity loadBleedingEdge >>> >>> would it make sense to make a jenkins job for this? >>> >>> >>> >>> >>> -- >>> Best regards, >>> >>> Douaille Erwan < > >> douaille.erwan@ > >> > >> >> -- >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >> Alexandre Bergel http://www.bergel.eu >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > -- > View this message in context: http://forum.world.st/Pharo-dev-OpenGL-NBMacGlAPI-missing-tp4692287p4697523… > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

July 5, 2013

Re: [Pharo-dev] NativeBoost with no access to sources

by Goubier Thierry

Le 05/07/2013 15:31, Igor Stasenko a Ã©crit : > On 5 July 2013 15:22, Igor Stasenko <siguctua(a)gmail.com> wrote: >> On 5 July 2013 11:03, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>> Could you rather test for unability to get the source of the method and >>> raise a NB-defined error? So that I could write a fallback on non-NB code >>> with a on: do:. Please :) >>> >>> (it would then both handle startup related errors and unreadable sources as >>> well). >>> >>> And with a backport to 2.0, because the odds of seeing an Opal based >>> solution on 2.0 are fairly low. >>> >> the problem is that you will get same error when you put wrong name >> into signature >> or when no sources avail e.g.: >> >> foo: bar >> ^ self nbCall: #( void foo (int baz) ) >> >> will give same error (unable to find binding for name 'baz' in given context). Of course. I could do with having the same error for both and knowing that when developping and testing I would get it for stupid syntax errors and in production to handle a real situation. >> If its fine to you, i can do it. Maybe I can try myself to see if I can write, before entering that loader stuff, a correct test which raises an error if the method source isn't available. >>> >>> Is there a real, practical difference between "at compilation time" and >>> "when receiving MethodAdded event"? i.e. is the complexity added in the >>> compilation process worthwhile? >>> >> the problem is that you may not always receive MethodAdded event, >> because different tools putting different meaning into "this should be >> compiled silently". >> This is outstanding issue in Pharo.. and that's the main reason why i >> would go with compiler (like >> that i have guarantee that instance of CompiledMethod is well-formed, >> and don't need to rely on >> 3-rd party toolchain to deliver notification to me. You mean that hacking support into the third party compiler isn't 3rd party toolchain :) >> and besides, we need such hook mechanism in Compiler anyways (for many >> other use cases) >> and so, it will be not "renting a Boeing for single passenger" >> > > oh.. and i forgot about most important aspect of it: > i don't wanna see > <primitive: #primitiveNativeCall module: #NativeBoostPlugin error: > errorCode > > in code, instead if wanna use > > <native> > (or similar), > which admit, much more concise, elegant and user-friendly, and hides > unnecessary implementation detail. I remember seeing people introducing syntax extensions inside smalltalk methods in VisualWorks, would that work as well ? Customize on a per class / per method which compiler we should use. Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

July 5, 2013