- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] AsmJit

by Jan Vrany

Hi Eliot, > What's the use case?Â What do you mean "copy to VM code space"?Â I > want to know because I want to give you a viable direction, but I > need to know more about your intended use first. I really appreciate it, but to make it clear: I'm not going to use Cog/Spur VMs for these experiments.Â By "copy to VM code space" I mean installing a code intoÂ the area in which VM keeps machine code. I think you call this "code zone" in Cog. This effectively allows me toÂ generate a code for a method from the "image" itself.Â Jan > Â > > Â > > Thanks! Jan > > > > > > > > On Tue, 2015-12-15 at 18:34 +0100, ClÃ©ment Bera wrote: > > > We're getting away from Jan initial question here ... > > > > > > I think the point here is that Cog has: > > > - 2 stable back ends: ARMv5 and x86Â > > > - 2 backends stable in the simulator with production planned for > > ~ > > > April: x64 and MIPS > > > and Tim is willing to do the ARMv8 backend. > > > > > > All the 4, and soon 5, cog back ends are maintained: > > > - x86 and x64 maintained by Eliot > > > - MIPS maintained by Ryan > > > - ARMv5 and soon ARMv8 maintained by Tim > > > > > > Each backend requires months of work. > > > > > > Do you want to maintain 10 backends instead of 5 ? Do want to > > spend > > > time to implement 5 backends or 10 ? > > > > > > I don't. Only the x86 backend is duplicated right now in AsmJIT. > > > > > > So one idea is to reuse Cog's backends from the image instead of > > > AsmJIT. To do so, we can use encodings available in the sista > > > extended instruction set to tell Cog what machine code to > > generate. A > > > project named uFFI aims, among other things, at doing this, by > > > providing a unified interface between the Cog backends and > > AsmJIT. > > > > > > > > > 2015-12-15 16:43 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com > > >: > > > > Hi Jan, > > > > > > > > > On Dec 15, 2015, at 3:06 AM, Jan Vrany <jan.vrany(a)fit.cvut.cz > > > > > > > wrote: > > > > > > > > > > Hi guys, > > > > > > > > > > two queations: > > > > > > > > > > (i) Is AsmJit going to be developed any more or it's > > abandoned > > > > >Â Â Â as well as native boost? > > > > > > > > AsmJIT is effectively being abandoned but NativeBoost is not. > > > > > > > > The key limitation of AsmJIT is that it was not designed to be > > > > cross platform; it is effectively an x86 assembler.Â As such > > it's > > > > use gets in the way of ARM and x86_64 (I am currently getting > > the C > > > > version of 64-bit Cog Spur working on x86_64, given that it is > > > > working in the simulator). > > > > > > > > Another limitation is that it doesn't play that well with the > > VM's > > > > JIT.Â Igor and I never managed to work on integrating it > > better.Â > > > > The VM's job is managing code and Igor's approach was to hack; > > > > eliminating execution protection in the entire heap, instead of > > > > extending the support that either the Alien plugin's callback > > > > support or the JIT's executable method zone provides.Â Making > > the > > > > entire heap executable is /not/ a sensible approach. > > > > > > > > But there is a better way!Â A key component of the Sista > > adaptive > > > > optimizer/speculative inlined that ClÃ©ment is currently > > stabilizing > > > > (!!) is a set of bytecodes that encode unsafe operations like > > > > at:put: without bounds, type or store checks.Â For example, the > > > > normal at:put: is about a hundred instructions, checks for > > > > smallinteger indices, differentiates between byte, 32-but long > > and > > > > pointer objects and does a store check. But one of the Sista > > codes > > > > for at:put: generates about two instructions, one to adjust the > > > > index, the other to do the store.Â Distaste job is to analyze > > code > > > > and inline methods using these unsafe bytecodes where they are > > > > proven to be safe, hence increasing performance. > > > > > > > > Unlike AsmJIT, Sista's unsafe bytecodes are cross platform, > > and, > > > > being executed by the VM, can work on an interpreter VM or be > > > > converted to machine code by the JIT. > > > > > > > > So our plan is to extend these bytecodes with ones that support > > > > marshaling arguments for NativeBoost calls.Â Ronie Salgado has > > > > already extended his lowcode scheme to define these > > instructions > > > > and sometime soon (hopefully 2016) we shall rewrite NativeBoost > > to > > > > target these bytecodes. > > > > > > > > HTH > > > > > > > > > (ii)Where can I find latest AsmJit? I'm properly confused: > > > > > > > > > >Â Â Â * Is is the one in latest Pharo 4.0 (5.0) image? > > > > >Â Â Â * Is it the one here: http://smalltalkhub.com/#!/~Pharo/A > > smJi > > > > t ? > > > > >Â Â Â Â (the one in the image seem to be based on completely > > > > disjunct > > > > >Â Â Â Â set of .mcz than those in the repo above). > > > > > > > > > > Best, Jan > > > > > > > > _,,,^..^,,,_ (phone) > > > > > > > > > > > -- > _,,,^..^,,,_ > best,Â Eliot

Dec. 15, 2015

Re: [Pharo-dev] Question about errno and NB FFI

by Eliot Miranda

On Tue, Dec 15, 2015 at 1:34 PM, Mariano Martinez Peck < marianopeck(a)gmail.com> wrote: > Hi guys, > > I am using NB FFI and I am calling a function that could answer a NULL* in > case of error and that it should have set the number on *errno.* > > So now I have some questions: > > Say the code is this: > > run > | whatever | > whatever := self *primitiveWhatever*. > ^ popenFile ifNil: [ > self *primitivePerror*. > > Imagine that the C lib called from #primitiveWhatever was the one setting > *errno.* > My question is... the following #primitivePerror is a DIFFERENT system > call. Couldn't that happened that another error happened before the first > system call and time I am executing the second one and so the "last error" > (errno) is actually the wrong one I am thinking of? > Yes, which is why it's really important that the FFI be able to be told to answer errno on error. See e.g. the VW FFI where IIRC one can specify that if a call answers -1, the FFI call can fail with a primitive error code that includes errno. > > Even worst, I may think that the scope of the "global variable errno" may > existed at the scope of the invoked lib only? In this case I don't have > above problem, but then....how can I get such number once the NB primitive > finished (so that I can call, for example, strerror() ? > Well, there are two issues here. One is that one should use waterer the underlying platform defines for getting "errno", for example some platforms will say something like #define errno get_errno(), to get the errno specific to the current thread. Second, all one's libraries need to use the same approach so that errno is shared between the VM and the libraries, otherwise your library may not set the errno seen by the VM and the FFI. > Thanks in advance, > > -- > Mariano > http://marianopeck.wordpress.com > -- _,,,^..^,,,_ best, Eliot

Dec. 15, 2015

Re: [Pharo-dev] [squeak-dev] Re: [Cuis] Sorting Unicode strings (Re: [Unicode] collation sequences (Re: Unicode Support))

by Eliot Miranda

Hi Dale, On Thu, Dec 10, 2015 at 12:27 PM, Dale Henrichs < dale.henrichs(a)gemtalksystems.com> wrote: > > > On 12/09/2015 04:31 PM, Levente Uzonyi wrote: > >> On Wed, 9 Dec 2015, Dale Henrichs wrote: >> >> >>> >>> On 12/09/2015 12:44 AM, Stephan Eggermont wrote: >>> >>>> On 08-12-15 22:35, Dale Henrichs wrote: >>>> >>>>> What I meant is that you can't _always_ use the code point for >>>>> collation, i.e., sorting based on the value of code points is not >>>>> always >>>>> correct[1]. >>>>> >>>> >>>> I have given up on universal sorting when I learned that dutch >>>> libraries sorting of author names depends on the country of origin of the >>>> author. So if Jan van Beek is dutch he will be sorted under B, while if >>>> he's belgian under V. I haven't checked what happens if the author >>>> emigrates, or changes nationality... >>>> >>>> Stephan >>>> >>>> Well, with ICU (and GemStone's implementation) you can choose which >>> collator to use (Country specific) at the image level or on a comparison by >>> comparison bases ... for example for an indexed collection (Unicode) >>> Strings, you can choose the collator to use for that particular index ... >>> so while it's true that universal sorter is not possible, it is possible to >>> choose a collator that will satisfy a particlar customer .... >>> >> >> I expect my image to compare strings using the codepoint-based (+language >> tags) lexicographical method, because it's simple, deterministic and fast. >> Imagine having failing tests just because your image uses different >> default comparison methods based on some (external) parameter... >> It's also a nightmare to find out why your program is slow on some >> machine, while it's fast on another. >> > > When we implemented the Unicode support in GemStone we preserved the > legacy string classes and their legacy behavior ... We added new Unicode* > classes with the new collator-based behavior for sorting and comparison ... > That way legacy applications (and legacy) tests were not impacted by the > choice of collator ... And folks could choose whether or not their > application would benefit by the use of the new Unicode* classes.... > > The ICU library performance is actually comparable to our original > implementations, so there isn't a noticeable performance difference - we > built the support into our vm and if folks are interested in some of the > gory technical details, we'd be willing to share our experience, as there > are several things that we did to minimize potential performance impacts --- > Just so you know, I will dig my heels in as deeply as I am able to prevent the use of C++ libraries in the VM. It destroys the simulator, which is the most important thing we have for VM development productivity. As far as I'm concerned any use of external libraries to implement core functionality kills the VM-in-Smalltalk concept that Squeak (and Pharo) are built upon. So for me it's a non-starter. I hope others agree. _,,,^..^,,,_ best, Eliot

Dec. 15, 2015

Question about errno and NB FFI

by Mariano Martinez Peck

Hi guys, I am using NB FFI and I am calling a function that could answer a NULL* in case of error and that it should have set the number on *errno.* So now I have some questions: Say the code is this: run | whatever | whatever := self *primitiveWhatever*. ^ popenFile ifNil: [ self *primitivePerror*. Imagine that the C lib called from #primitiveWhatever was the one setting *errno.* My question is... the following #primitivePerror is a DIFFERENT system call. Couldn't that happened that another error happened before the first system call and time I am executing the second one and so the "last error" (errno) is actually the wrong one I am thinking of? Even worst, I may think that the scope of the "global variable errno" may existed at the scope of the invoked lib only? In this case I don't have above problem, but then....how can I get such number once the NB primitive finished (so that I can call, for example, strerror() ? Thanks in advance, -- Mariano http://marianopeck.wordpress.com

Dec. 15, 2015

Re: [Pharo-dev] Compact representation of source code history

by Stephan Eggermont

On 15-12-15 21:35, Stephan Eggermont wrote: > Pharo/Pharo50 on smalltalkhub contains > > 2502 package versions (the separate mczs) Oops 3512, of which I had issues with 10 Stephan

Dec. 15, 2015

Re: [Pharo-dev] Compact representation of source code history

by Stephan Eggermont

Pharo/Pharo50 on smalltalkhub contains 2502 package versions (the separate mczs) in 389 different packages containing 96390 unique MCDefinitions (nearly all MCmethodDefinition) That whole repo can be represented in a 221.9 MB fuel file, compressed as tar.gz: 80.6 MB I wonder if I might be able to fit all smalltalkhub source incl history in ram on a modern pc Stephan

Dec. 15, 2015

Re: [Pharo-dev] potential to include cryptography in base image

by Robert Withers

Thanks for the links. I am ready to copy my ConfigurationOfCryptography to the MetaRepoForPharo50, though I did my work in Pharo 4, if that is ok. Could you add me so I can publish there, please? robert On 12/15/2015 12:55 PM, Peter UhnÃ¡k wrote: >> Alright, please let me know. If someone shows me how to get this into >> configurations and the Catalog, we'll get it done. > You just need to commit your configuration to MetaRepoForPharo50 (this > should be public repo; if you can't commit ask on the mailing list and > someone will add you). > > But before you do that, open Versionner (World > Tools > Versionner), > right click your configuration and select "add catalog methods". > This will add class-side methods to the Configuration class which you > can modify to write things like description, website, tags, etc. > > You can also use the Veresionner to create your ConfigurationOf if you > don't have one already, more info on Configurations can be found in > DeepIntoPharo (Metacello chapter) > https://ci.inria.fr/pharo-contribution/view/Books/job/DeepIntoPharo/lastSuc… > or look through other configurations in the image or search through > mailing list archives, there should be plenty of information. > > Peter > -- . .. .. ^,^ best, robert

Dec. 15, 2015

Re: [Pharo-dev] AsmJit

by Eliot Miranda

Hi Jan, On Tue, Dec 15, 2015 at 11:22 AM, Jan Vrany <jan.vrany(a)fit.cvut.cz> wrote: > Hi Esteban, Eliot, Clemont > > thanks very much for elaborate answer. I'm aware of all what you said > (well, most :-), but my actual needs are very low and primitive. > > All I need now is something that allows me to turn x86_64 assembly > into a machine code using a sane Smalltalk API, which I can later copy > to VM code space. Nothing more, nothing less :-) I do not fancy > spending time writing yet-another-x86-assembler hence my interest in > asmjit. > What's the use case? What do you mean "copy to VM code space"? I want to know because I want to give you a viable direction, but I need to know more about your intended use first. > > Thanks! Jan > > > > On Tue, 2015-12-15 at 18:34 +0100, ClÃ©ment Bera wrote: > > We're getting away from Jan initial question here ... > > > > I think the point here is that Cog has: > > - 2 stable back ends: ARMv5 and x86 > > - 2 backends stable in the simulator with production planned for ~ > > April: x64 and MIPS > > and Tim is willing to do the ARMv8 backend. > > > > All the 4, and soon 5, cog back ends are maintained: > > - x86 and x64 maintained by Eliot > > - MIPS maintained by Ryan > > - ARMv5 and soon ARMv8 maintained by Tim > > > > Each backend requires months of work. > > > > Do you want to maintain 10 backends instead of 5 ? Do want to spend > > time to implement 5 backends or 10 ? > > > > I don't. Only the x86 backend is duplicated right now in AsmJIT. > > > > So one idea is to reuse Cog's backends from the image instead of > > AsmJIT. To do so, we can use encodings available in the sista > > extended instruction set to tell Cog what machine code to generate. A > > project named uFFI aims, among other things, at doing this, by > > providing a unified interface between the Cog backends and AsmJIT. > > > > > > 2015-12-15 16:43 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > > > Hi Jan, > > > > > > > On Dec 15, 2015, at 3:06 AM, Jan Vrany <jan.vrany(a)fit.cvut.cz> > > > wrote: > > > > > > > > Hi guys, > > > > > > > > two queations: > > > > > > > > (i) Is AsmJit going to be developed any more or it's abandoned > > > > as well as native boost? > > > > > > AsmJIT is effectively being abandoned but NativeBoost is not. > > > > > > The key limitation of AsmJIT is that it was not designed to be > > > cross platform; it is effectively an x86 assembler. As such it's > > > use gets in the way of ARM and x86_64 (I am currently getting the C > > > version of 64-bit Cog Spur working on x86_64, given that it is > > > working in the simulator). > > > > > > Another limitation is that it doesn't play that well with the VM's > > > JIT. Igor and I never managed to work on integrating it better. > > > The VM's job is managing code and Igor's approach was to hack; > > > eliminating execution protection in the entire heap, instead of > > > extending the support that either the Alien plugin's callback > > > support or the JIT's executable method zone provides. Making the > > > entire heap executable is /not/ a sensible approach. > > > > > > But there is a better way! A key component of the Sista adaptive > > > optimizer/speculative inlined that ClÃ©ment is currently stabilizing > > > (!!) is a set of bytecodes that encode unsafe operations like > > > at:put: without bounds, type or store checks. For example, the > > > normal at:put: is about a hundred instructions, checks for > > > smallinteger indices, differentiates between byte, 32-but long and > > > pointer objects and does a store check. But one of the Sista codes > > > for at:put: generates about two instructions, one to adjust the > > > index, the other to do the store. Distaste job is to analyze code > > > and inline methods using these unsafe bytecodes where they are > > > proven to be safe, hence increasing performance. > > > > > > Unlike AsmJIT, Sista's unsafe bytecodes are cross platform, and, > > > being executed by the VM, can work on an interpreter VM or be > > > converted to machine code by the JIT. > > > > > > So our plan is to extend these bytecodes with ones that support > > > marshaling arguments for NativeBoost calls. Ronie Salgado has > > > already extended his lowcode scheme to define these instructions > > > and sometime soon (hopefully 2016) we shall rewrite NativeBoost to > > > target these bytecodes. > > > > > > HTH > > > > > > > (ii)Where can I find latest AsmJit? I'm properly confused: > > > > > > > > * Is is the one in latest Pharo 4.0 (5.0) image? > > > > * Is it the one here: http://smalltalkhub.com/#!/~Pharo/AsmJi > > > t ? > > > > (the one in the image seem to be based on completely > > > disjunct > > > > set of .mcz than those in the repo above). > > > > > > > > Best, Jan > > > > > > _,,,^..^,,,_ (phone) > > > > > -- _,,,^..^,,,_ best, Eliot

Dec. 15, 2015

Re: [Pharo-dev] AsmJit

by Jan Vrany

Hi Esteban, Eliot, Clemont thanks very much for elaborate answer. I'm aware of all what you said (well, most :-), but my actual needs are very low and primitive.Â All I need now is something that allows me to turn x86_64 assemblyÂ into a machine code using a sane Smalltalk API, which I can later copy to VM code space. Nothing more, nothing less :-) I do not fancy spending time writing yet-another-x86-assembler hence my interest inÂ asmjit.Â Thanks! Jan On Tue, 2015-12-15 at 18:34 +0100, ClÃ©ment Bera wrote: > We're getting away from Jan initial question here ... > > I think the point here is that Cog has: > - 2 stable back ends: ARMv5 and x86Â > - 2 backends stable in the simulator with production planned for ~ > April: x64 and MIPS > and Tim is willing to do the ARMv8 backend. > > All the 4, and soon 5, cog back ends are maintained: > - x86 and x64 maintained by Eliot > - MIPS maintained by Ryan > - ARMv5 and soon ARMv8 maintained by Tim > > Each backend requires months of work. > > Do you want to maintain 10 backends instead of 5 ? Do want to spend > time to implement 5 backends or 10 ? > > I don't. Only the x86 backend is duplicated right now in AsmJIT. > > So one idea is to reuse Cog's backends from the image instead of > AsmJIT. To do so, we can use encodings available in the sista > extended instruction set to tell Cog what machine code to generate. A > project named uFFI aims, among other things, at doing this, by > providing a unified interface between the Cog backends and AsmJIT. > > > 2015-12-15 16:43 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > > Hi Jan, > > > > > On Dec 15, 2015, at 3:06 AM, Jan Vrany <jan.vrany(a)fit.cvut.cz> > > wrote: > > > > > > Hi guys, > > > > > > two queations: > > > > > > (i) Is AsmJit going to be developed any more or it's abandoned > > >Â Â Â as well as native boost? > > > > AsmJIT is effectively being abandoned but NativeBoost is not. > > > > The key limitation of AsmJIT is that it was not designed to be > > cross platform; it is effectively an x86 assembler.Â As such it's > > use gets in the way of ARM and x86_64 (I am currently getting the C > > version of 64-bit Cog Spur working on x86_64, given that it is > > working in the simulator). > > > > Another limitation is that it doesn't play that well with the VM's > > JIT.Â Igor and I never managed to work on integrating it better.Â > > The VM's job is managing code and Igor's approach was to hack; > > eliminating execution protection in the entire heap, instead of > > extending the support that either the Alien plugin's callback > > support or the JIT's executable method zone provides.Â Making the > > entire heap executable is /not/ a sensible approach. > > > > But there is a better way!Â A key component of the Sista adaptive > > optimizer/speculative inlined that ClÃ©ment is currently stabilizing > > (!!) is a set of bytecodes that encode unsafe operations like > > at:put: without bounds, type or store checks.Â For example, the > > normal at:put: is about a hundred instructions, checks for > > smallinteger indices, differentiates between byte, 32-but long and > > pointer objects and does a store check. But one of the Sista codes > > for at:put: generates about two instructions, one to adjust the > > index, the other to do the store.Â Distaste job is to analyze code > > and inline methods using these unsafe bytecodes where they are > > proven to be safe, hence increasing performance. > > > > Unlike AsmJIT, Sista's unsafe bytecodes are cross platform, and, > > being executed by the VM, can work on an interpreter VM or be > > converted to machine code by the JIT. > > > > So our plan is to extend these bytecodes with ones that support > > marshaling arguments for NativeBoost calls.Â Ronie Salgado has > > already extended his lowcode scheme to define these instructions > > and sometime soon (hopefully 2016) we shall rewrite NativeBoost to > > target these bytecodes. > > > > HTH > > > > > (ii)Where can I find latest AsmJit? I'm properly confused: > > > > > >Â Â Â * Is is the one in latest Pharo 4.0 (5.0) image? > > >Â Â Â * Is it the one here: http://smalltalkhub.com/#!/~Pharo/AsmJi > > t ? > > >Â Â Â Â (the one in the image seem to be based on completely > > disjunct > > >Â Â Â Â set of .mcz than those in the repo above). > > > > > > Best, Jan > > > > _,,,^..^,,,_ (phone) > >

Dec. 15, 2015

Re: [Pharo-dev] potential to include cryptography in base image

by Ben Coman

On Wed, Dec 16, 2015 at 1:36 AM, Robert Withers <robert.w.withers(a)gmail.com> wrote: > > > On 12/15/2015 05:20 AM, Sven Van Caekenberghe wrote: >> >> Robert, >> >> I think that the Pharo community, part of which is more business oriented, >> is absolutely interested in more and better Crypto code. In any case, I am. >> >> What we absolutely want, if it is not already the case (I did not check), >> is that the Crypto code can be loaded using 1 single action (through our >> validated Monticello configurations and Catalog mechanism) - I am sure you >> will find help to achieve and maintain (through a CI process) that goal. > > > Alright, please let me know. If someone shows me how to get this into > configurations and the Catalog, we'll get it done. Versionneer... http://forum.world.st/Versioneer-nice-mmMmmm-td4779483.html Catalog... http://catalog.pharo.org/catalog/note-for-developers?_k=1Ur7YrxY2p1TaD4k Repository for Configuration... http://smalltalkhub.com/#!/~Pharo/MetaRepoForPharo50 cheers -ben > > robert > >> >> Whether it should be a base part of the image is another question. >> Modularity is a huge goal for Pharo. This is a much harder discussion (as >> the same can be said of or asked for for many packages that are generally >> useful: XML, CSV, JSON, SQL, ...). In any the case, the first step is the >> one described in the previous paragraph. Then you need traction, usage, and >> maybe demand for full inclusion. >> >> Regards, >> >> Sven >> >>> On 15 Dec 2015, at 11:00, Robert Withers <robert.w.withers(a)gmail.com> >>> wrote: >>> >>> It was suggested to me that I write to the list and raise the question >>> about cryptography being included in the base image. Really I have 3 >>> questions I would ask you all: >>> >>> â¢ is it desirable to include cryptography? >>> â¢ is it feasible to include cryptography? >>> â¢ what is the time frame for including cryptography? >>> Given the thread on password hashing (and salting and so on), there are >>> good, solid implementations in the cryptography package. Looking in the >>> Cryptography repository, there is a Pharo 5.0 compatible Cryptography >>> package. >>> >>> In light of another recent thread discussing random number generation, >>> discussion about the best approach to random algorithms in cryptography >>> ought be engaged. For instance, the SecureRandom algorithm evidently >>> provides some level of guarantee. >>> >>> To underline the solidity I am attaching a profile of all 102 >>> cryptography tests passing green. This profile demonstrates that there are >>> no areas of particular inefficiency - nothing stands out to be improved - >>> means that the entire library is maximally efficient. >>> >>> And so I please ask that we have these discussions, for there is a lot of >>> value in this package for general and basic use. >>> >>> >>> -- >>> . .. ... ^,^ best, robert >>> <Cryptography Spy Results.text.gz> > > > -- > . .. .. ^,^ best, robert >

Dec. 15, 2015