- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

by Eliot Miranda

On Sat, Jan 16, 2016 at 4:37 PM, Levente Uzonyi <leves(a)caesar.elte.hu> wrote: > > On Sat, 16 Jan 2016, Mariano Martinez Peck wrote: > > (Still no quote.) > > How will you read the output of the process without having your image's > process blocked in the FFI callout? > How will you make sure that writes to input of the process won't block the > FFI callout? This presupposes the threaded FFI. The threaded FFI allows the VM to make any number of blocking calls, adding a new thread to run the VM whenever the VM is stalled when the heartbeat beats. hence one can freely read and write to/from i/o blocking i/o streams (including pipes and sockets) or blocking database connexions, etc, all without stating that the FFI call must be done in a special way, since all calls through the FFI can block without blocking the VM. Note that the scheme is also amenable to plugins, but the plugins must be rewritten to include the release vm/acquire vm calls around a blocking call. With the threaded VM the FFI includes these calls around every FFI call. HTH _,,,^..^,,,_ best, Eliot

Jan. 17, 2016

Re: [Pharo-dev] [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

by Eliot Miranda

On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck < marianopeck(a)gmail.com> wrote: > > Hi all, > > Sorry for reviving an old thread but I thought it was better to continue > the discussion here because of the context. > As you may have read, the other day I released a first approeach to a > subset of OSProcess based on FFI (posix_spwan() family of functions): > > https://github.com/marianopeck/OSSubprocess > > And with that in mind, I wanted to share a few things with you. The main > 2 problems I found with implementing this with FFI was: > > 1) We have all already agree and discussed that fork+exec cannot be done > in separate FFI calls. So at the very min you need either a plugin method > that does the fork()+exec() OR wrapping a lib like posix_spwan() > > 2) The other main problem, is, as you all said (and mostly Nicolas), is > the problems with the preprocessor (constants, macros, etc). > > With all that said, I was able to get my stuff working. However, I am > still using some primitives of OSProcess plugin because of 2). > > I read Eliot idea and what I don't like is the need of a C compiler in the > user machine. I think that's a high constrain. Then Igor suggested that WE > (developers and maintainers of a certain tool) are the ones that compiles > the little C program to extract constant values etc and then WE provide as > part of our source code, some packages with some SharedPool depending on > the platform/OS. And Igor approach looked a bit better to me. > You misunderstand the proposal. The C compiler is needed /only when changing the set of constants/, i.e. when /developing/ the interface. The C compiler is /not/ needed when deploying. The idea is to a) at development time, e.g. when a new variable is added to a SharedPool containing platform constants, a C program is autogenerated that outputs in some format a description of the names and values of all the constants defined in the pool. One convenient notation is e.g. STON. For the purposes of this discussion let's assume we're using ston, but any format the image an parse (or indeed a shared object the image can load on teh current pkatform) will do. The output of the autogenerated C program would be called something like <SharedPoolName>.<PlatformName>.ston, e.g. UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston. The ston files can easily be parsed by facilities in the Smalltalk image. b) when deploying the system to a set of platforms one includes all the relevant platform-specific ston files. c) at startup the image checks its current platform. If the platform is the same that it was saved on, no action is taken. But if the platform as changed then the relevant ston file is selected, parsed, and the values for the variables in the shared pool updated to reflect the values of the current platform. So the C compiler is only needed when developing the interface, not when deploying it. > Then Nicolas made a point that if we plan to manage all that complexity at > the image level it may become a hell too. > > So.... what if we take a simpler (probably not better) approach and we > consider the "c program that exports constants and sizes" a VM Plugin? > Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux > and other's Unix I imagine for the time being) which provides a function > (that is exported) which answers an array of arrays. For each constant, we > include the name of the constant, the value, and the sizeof(). Then from > image side, we simply do one FFI call, we get the large array and we adapt > it to a SharedPool or whatever kind of object representing that info. > This is what I suggestred in teh first place. That what is autogenerated is a shared object (be it a plgin or a dll doesn't matter, it is machine code generated by a C compiler form an autogenerated C program compiled with the platform's C compiler) that can be loaded at run-time and interrogated to fetch the values of a set of variables. But I think that the textual notation suggested above is simpler. The test files are easier to distribute and change. Shared objects and plugins have a habit of going stale, and there needs to be metadata in there to describe the set of constants etc, which is tricky to generate and parse because it is binary (pointer sizes, etc, etc). Instead a simple textual format should be much more robust. One could even edit by hand to add new constants. It would be easy to make the textual file a versioned file. Etc, etc. > > I know that different users will need different constants. But let's say > the infrastructure (plugin etc) is already done. And let's say I am a user > that I want to build something with FFI and I need some constants that I > see are not defined. Then I can simply add the ones I need in the plugin, > and next VM release will have those. If Cog gets moved to Github, then this > is even easier. Everybody can do a PR with the constants he needs. And in > fact, if we have the infrastructure in place, I think that we each of us > spend half an hour, we may have almost everything we need. > > For example, I can add myself all those for signals (to use kill() from > FFI), all those from fcntl (to make none blocking pipes), all those from > wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc > etc etc. > > I know it's not the best approach but it's something that could be done > very easily and would allow A LOT of stuff to be moved to FFI just because > we have no access to preprocess constants or sizeof() (to know how to > allocate). I also know this won't cover macros and other stuff. But still. > > If you think this is a good idea, I can spend the time to do it. > > Cheers, > > > > > > > > > > On Thu, May 10, 2012 at 10:09 AM, Nick Ager <nick.ager(a)gmail.com> wrote: > >> <snip> >> Well, like opendbx, maybe because opengl has quite standard interface... >> </snip> >> >> and >> >> <snip> >> It's not that it's not doable, it's that we gonna reinvent gaz plant >> and it gonna be so boring... >> I'd like to see a proof of concept, even if we restrict to libc, libm, >> kernel.dll, msvcrt.dll ... >> </snip> >> >> <snip> >> Is the unix style select() >> ubiquitous or should I use WaitForMultipleObject() on Windows? Are >> specification of read/write streams implementation machine independant >> (bsd/sysv/others...) >> </snip> >> >> Perhaps *a* way forward is to try to find existing projects which have >> already created cross-platform abstractions for platform specific >> functionality. Then we can use FFI to access that interface in a similar >> way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps >> they have a useful cross-platform abstraction, boost has abstractions of >> IPC etc >> >> Nick >> > > > > -- > Mariano > http://marianopeck.wordpress.com > > -- _,,,^..^,,,_ best, Eliot

Jan. 17, 2016

Re: [Pharo-dev] gtdebugger in pharo 5.0

by Cyril Ferlicot D.

Le 09/01/2016 10:32, Tudor Girba a Ã©crit : > Hi, > > Could you be specific? What exactly do you find less intuitive? To be able to react, it is useful to get concrete points. > > Cheers, > Doru > > > -- > www.tudorgirba.com > www.feenk.com > > "Value is always contextual." > > Hi, After more reflection I find another thing that was, for me, better in the SpecDebugger. The fact that the methods and the classes was in two columns in the stack made the debugger clearer. Most of the time we know the methods and we do not need to check the name of the class. It was easier to check it with the two columns. And more, it make the debugger "ventilated" (I don't know if the term work in English too). I mean that with more space between the classes and the methods it's better for the eye of the user. -- Cyril Ferlicot http://www.synectique.eu 165 Avenue Bretagne Lille 59000 France

Jan. 17, 2016

Re: [Pharo-dev] Color yzw

by Nicolai Hess

2016-01-16 13:46 GMT+01:00 Marcus Denker <marcus.denker(a)inria.fr>: > > On 16 Jan 2016, at 13:26, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2016-01-16 10:11 GMT+01:00 Marcus Denker <marcus.denker(a)inria.fr>: > >> >> anyone an idea? >> >> I'm trying to think. >> Looks like it would be good to be able to make a distinction between >> symbols and selectors (selectors installed >> in method dictionaries) >> >> >> yes, this would improve code completion a bit, too, as there are more >> symbols than symbols that are actually names >> of methods. >> >> The data could be created per environment (Smalltalk globals) the first >> time and then cached till shutdown. >> (it would be an array of size 46451, pointing to existing symbols). >> >> The environment could cache sent selectors, tooâ¦ pablo did an experiment >> with that. >> >> Marcus >> >> > How about a new SymbolTable "SelectorTable" that gets new entries on every > method compilation ? > > > maybe better when a method is added to a Method Dictionary? This way even > adding things the strange > way would work... > I tried some methods from TBehavior basicAddSelector:withMethod: but for example Nautilus, directly modifies method dict > > > I hacked a simple experimental version. > But I don't yet understand the workflow for initialisation of the Symbol > class (with rehash, interned, and compacting). > > > If we hook into adding methods, then compactSymbolTable and rehash could > just reset the SelectorTable and fill it after by going over all methodsâ¦ > see attached changesets. load with: DangerousClassNotifier disableDuring:[ String subclass: #Symbol instanceVariableNames: '' classVariableNames: 'NewSymbols OneCharacterSymbols SymbolTable SelectorTable' package: 'Collections-Strings' ]. DangerousClassNotifier disableDuring:[ 'selectortable.1.cs' asFileReference fileIn. ]. Symbol initSelectorTable. 'use_selector_table.cs' asFileReference fileIn. what do you think ? (selectortable.cs defines SelectorTable and some methods in Symbol and String) (use_selector_table adds selectors as internedSelectorSymbol if this selector is set as a selector on CompiledMethod and uses findInternedSelector .... in RubStyler visitMessageNode) > > > Marcus > >

Jan. 16, 2016

Re: [Pharo-dev] [Vm-dev] Unable to read Utf8 ?

by Eliot Miranda

Hi Esteban, > On Jan 16, 2016, at 2:51 AM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > > >> On 16 Jan 2016, at 11:39, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: >> >> >>> On 16 Jan 2016, at 10:45, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: >>> >>> >>> Hi Thierry, >>> >>> up until a few hours ago the latest VM sources were indeed broken by work I was doing on 64 bits. My sources are AFAIA now correct and I suppose you're waiting on Esteban doing a merge. But I'm responding since there seems to be a process error here. >> >> is still broken :) > > btwâ¦ Iâm not merging anymore VMMaker packages, they are taken and built âas isâ (I worked a lot to get back to this, heh) :) > this is what âlatestâ vm is. And I think this is the right thing to do. Can I get a stream of email from the ci build/test server which shows whether the VM is good or not? > what happens is that time to time I still have to merge platform sourcesâ¦ but I made that Thursday so we are ok (just verified). > > Anyway, it is still broken :( I'm working on it :) >>> I need the freedom to break the VM, otherwise I can only extend it with difficulty. This assumes that there are currently servers that build and test the VM, and implies that we maintain the notion of the most recent good VM that has passed all the tests, and that this is the "latest" VM everyone is using. >>> >>> If there isn't this distinction then every time I extend the VM and make inevitable mistakes I affect users. That's clearly not sensible. >> >> yes of course, what we have is two versions of the VM: >> >> - âlatest" build is a build with latest sourcesâ¦ this might be broken. People willing to help us find bugs should use this. >> - âstableâ build is the VM after it passed all testsâ¦ and thatâs what people willing some stability should use :) >> >> what actually is happening now is that there is a lot of people helping in the migration of spur (something I thank, a lot) and then they use latest vm versionâ¦ who is usually very stable (but not always, as is the case). >> >> cheers, >> Esteban >> >> >>> >>> _,,,^..^,,,_ (phone) >>> >>>> On Jan 16, 2016, at 1:21 AM, Thierry Goubier <thierry.goubier(a)gmail.com> wrote: >>>> >>>> Hi Guys, >>>> >>>> has something changed in string handling? I'm unable to get the latest Pharo to read files containing utf8 data. >>>> >>>> I tracked it down to a primitive (at:put:) in WideString and to the latest vm for Pharo 5, so, maybe there is something wrong in the latest vm. >>>> >>>> Thierry >

Jan. 16, 2016

Re: [Pharo-dev] Contributing to Pharo

by Sven Van Caekenberghe

> On 16 Jan 2016, at 14:44, Ben Coman <btc(a)openinworld.com> wrote: > > On Sat, Jan 16, 2016 at 7:25 PM, David Allouche <david(a)allouche.net> wrote: >> >>> On 16 Jan 2016, at 12:14, stepharo <stepharo(a)free.fr> wrote: >>> >>> Le 16/1/16 09:59, David Allouche a Ã©crit : >>>> >>>> But, really, this "list of dirty packages" should really only list dirty packages. What is the use case for selecting a non-dirty package? >>> >>> because you never know >>> Sometimes you save a package then you do a slice and you should be able to decide. >> >> That does not make sense to me. I guess it will, when I understand Monticello better. >> >> > > Its a very corner case. I've had to do it on occasion but I can't > exactly remember the circumstance. Now what might be useful is a > checkbox for displaying non-dirty packages - so we only display the > short list in the usual case. (??) +100 it is very confusing right now > cheers -ben >

Jan. 16, 2016

Re: [Pharo-dev] Git repository and branch for spur enable pharo-vm

by Mariano Martinez Peck

Hi, Today I was also needed to compile Spur VM and just by chance by this email I found out I should be using Esteban fork. So maybe at least a sentence in the original repo saying that for Spur / Pharo 5.0 you should use Esteban fork would help. At least until you merge. Cheers, On Fri, Jan 15, 2016 at 5:23 PM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > Hi, > > > On 15 Jan 2016, at 18:29, Holger Freyther <holger(a)freyther.de> wrote: > > > > Hi, > > > > what is the official pharo-vm git repository? It looks like > github.com/pharo-project/pharo-vm is not being updated right now. It > looks like esteban's fork is currently being used and the spur64 branch. Is > that correct? Can it it to be used for spur32 and spure64? Is there any > plan to merge this back into a repositpory in pharo-project/pharo-vm? > > yes to everything :) > > - right now is the one Iâm updating > - it will be merged with pharo-project (I didnât have the time to do it, > but it will be soon) > - you can do a spur32 and theoretically a stack 64bits, but since we still > do not have a 64bits image (one of my soon (â¢) next steps) it doesnât looks > meaningful right now :) > > cheers, > Esteban > > > > > kind regards > > holger > > > -- Mariano http://marianopeck.wordpress.com

Jan. 16, 2016

Running processes at startup vs startUpList, which one first?

by Mariano Martinez Peck

Hi, I have a funny question. Let's say I did: [ something ] fork. and `something` never finishes and does not get GC. And I have a class that I registered in startup list. Which gets done first? the resume of the above process of the processing of the startup list? Thanks! -- Mariano http://marianopeck.wordpress.com

Jan. 16, 2016

Re: [Pharo-dev] How do you normally merge Git branches with Metadata-less GitFileTree?

by Mariano Martinez Peck

OK, thanks Thierry. BTW, thanks for all the help you have been giving me in the last weeks and for your great GitFileTree :) On Sat, Jan 16, 2016 at 11:14 AM, Thierry Goubier <thierry.goubier(a)gmail.com > wrote: > Le 16/01/2016 15:06, Mariano Martinez Peck a Ã©crit : > >> >> >> On Sat, Jan 16, 2016 at 5:15 AM, Thierry Goubier >> <thierry.goubier(a)gmail.com <mailto:thierry.goubier@gmail.com>> wrote: >> >> Le 16/01/2016 03:23, Mariano Martinez Peck a Ã©crit : >> >> Hi guys, >> >> First, let me say that I found very cool that I can do a "git >> checkout >> X" from command line, and from Pharo, opening the MC browser >> detects I >> am in another branch and everything seems to work. So I guess >> that's the >> way I manage branches? Simply "git checkout X" and then go to MC >> , and >> do a "load" of the last version of the repo? (or another image, >> whatever). >> >> >> Yes, exactly. >> >> >> OK. >> >> >> >> The problem is now with merging. Not necessary about the >> metadata ( I >> guess we have less metadata conflicts with Metadata-less >> GitFileTree >> right???) , but real code changes conflicts between branches. >> How do you >> manage this? You manage everything at Git level using git and >> text editors? >> >> >> yes, or with git gui tools, or with the github interface (if there >> is no conflict). The only thing a bit problematic are the eventual >> conflicts, but, in that metadata-less format, they are less frequent >> and easier to solve. >> >> >> OK... but let me confirm... with metadata-less gitfiletree, would I >> still benefit from >> https://github.com/ThierryGoubier/GitFileTree-MergeDriver >> to minimize conflicts? >> Or that was when you were having filetree with metadata? >> > > The merge driver does three things: > - merge metadata version files > - merge method properties json files > - merge class definition json files (merge instances variables from both > branches) > > Items one and two do not exist anymore in metadata-less format. Third one > is not allways seen as a good thing. > > So the merge driver is rarely usefull in metadata-less mode. > > I cannot think how to do that from MC browser "Merge" because MC >> sees >> only one repo associated to one current branch. >> >> >> It is possible to do the merge in MC (think of merging your current >> working copy and the top of the branch) but they won't be recorded >> in the git log as a merge. >> >> >> OK. I prefer git to see it as a merge. But thanks anyway. >> > > I understand and do the same. Moreover, git is better than MC in my > opinion to do the merge properly. > > Thierry > > -- Mariano http://marianopeck.wordpress.com

Jan. 16, 2016

Re: [Pharo-dev] How do you normally merge Git branches with Metadata-less GitFileTree?

by Thierry Goubier

Le 16/01/2016 15:06, Mariano Martinez Peck a Ã©crit : > > > On Sat, Jan 16, 2016 at 5:15 AM, Thierry Goubier > <thierry.goubier(a)gmail.com <mailto:thierry.goubier@gmail.com>> wrote: > > Le 16/01/2016 03:23, Mariano Martinez Peck a Ã©crit : > > Hi guys, > > First, let me say that I found very cool that I can do a "git > checkout > X" from command line, and from Pharo, opening the MC browser > detects I > am in another branch and everything seems to work. So I guess > that's the > way I manage branches? Simply "git checkout X" and then go to MC > , and > do a "load" of the last version of the repo? (or another image, > whatever). > > > Yes, exactly. > > > OK. > > > > The problem is now with merging. Not necessary about the > metadata ( I > guess we have less metadata conflicts with Metadata-less GitFileTree > right???) , but real code changes conflicts between branches. > How do you > manage this? You manage everything at Git level using git and > text editors? > > > yes, or with git gui tools, or with the github interface (if there > is no conflict). The only thing a bit problematic are the eventual > conflicts, but, in that metadata-less format, they are less frequent > and easier to solve. > > > OK... but let me confirm... with metadata-less gitfiletree, would I > still benefit from > https://github.com/ThierryGoubier/GitFileTree-MergeDriver > to minimize conflicts? > Or that was when you were having filetree with metadata? The merge driver does three things: - merge metadata version files - merge method properties json files - merge class definition json files (merge instances variables from both branches) Items one and two do not exist anymore in metadata-less format. Third one is not allways seen as a good thing. So the merge driver is rarely usefull in metadata-less mode. > I cannot think how to do that from MC browser "Merge" because MC > sees > only one repo associated to one current branch. > > > It is possible to do the merge in MC (think of merging your current > working copy and the top of the branch) but they won't be recorded > in the git log as a merge. > > > OK. I prefer git to see it as a merge. But thanks anyway. I understand and do the same. Moreover, git is better than MC in my opinion to do the merge properly. Thierry

Jan. 16, 2016