Mariano, I am really happy to see this and I look forword to helping you in any way I can. Dimitris, It is really interesting that you should suggest this. I think it may be a bit off topic for Mariano's initial goals, but Mariano is the driving force behind Fuel, which is the serializer that I use for RemoteTask in OSProcess/CommandShell. This attempts to do much of what you suggest, using Fuel to return the results of processing executed in a forked image (in the unix sense of "forked", i.e. extremely memory efficient). So maybe if I can help Mariano with his project (which of course I plan to do anyway), then he might be able to help me turn RemoteTask into something reliable enough for general purpose use. Another nice connection here is that Eliot's Spur image model will make #forkSqueak (the primitive behind RemoteTask) more efficient in cases where the cooperating images do a lot of processing, because memory changes are more localized in the Spur memory model. In principle it should be possible to run very large object memories in 64-bit Spur, while forking cooperating images that consume very little additional real memory. Dave On Sat, Dec 19, 2015 at 01:33:07PM +0000, Dimitris Chloupis wrote:
I am very excited for such a project.
I hope you manage to fix the stdout problem on macos.
One thing I would like to see is an expansion of the ability of OSProcess to launch multiple instances of pharo and make them communicate with each other. Thats a way to have pharo running on multiple cores. Maybe establish a framework of communication of images that will be based on the existing design of the fork mechanism in Pharo.
Python is doing something similar with multiprocessing module which launches multiple processes of cpython executable and makes them communication with each other through a similar api to its threading module. This way python takes advantage of multiple cores. Python threads like pharo are green threads and pharo like python has a GIL like mechanism.
Taking into account that current computers have at least 4 cores, its not a bad idea to speed up pharo like this at least 400% ;)
On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck <marianopeck@gmail.com> wrote:
Dear all,
I am tremendously happy to announce you that Pharo Consortium will sponsor yet another development effort. In this particular case, it's my honor to carry on such an effort and I will be developing and improving a few things and ideas. All the contract and paperwork has already been done so it's time to start working.
Regarding the developments itself, we discussed about these topics:
1) Experiment with a simpler yet more limited OSProcess alternative to execute OS commands. 2) Improving FileSystem in order to better deal with some POSIX stuff like symbolic links, unix file permissions, etc etc.
I have already started with 1). OSProcess is super complete and it involves some packages (OSProcess / CommandShell) as well as some VM plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of features (like forking the running image) but we would like to focus only in executing OS commands. In addition, OSProcess dates from quite some years ago when many of the current infrastructure features did not exist yet.
So the idea is to think a simpler alternative to execute OS commands, using new features such as threaded FFI (for example, for reading async from pipes), pinned objects, etc. We want to use FFI as much as possible rather than VM plugins.
From the image side, we are thinking about an API which would look like a builder-like API (FileSystem, XStream, etc).
We have been discussing with David Lewis (OSProcess author) as well as with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc in order to agree in a project that would be worth doing (otherwise we would continue using existing tools). And they have all been very kind with me providing positive discussions.
I will soon do a survey to measure the most common use cases of OSProcess and related tools.
If you have any thought, question, feedback, or whatever, please let us know.
Finally, note that I will be doing this project together with another client's project (Quuve) and my life does not have much free time these days...so please be patient!!!
Best,
-- Mariano http://marianopeck.wordpress.com