Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort
On Tue, Dec 22, 2015 at 3:20 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Sun, Dec 20, 2015 at 5:21 PM, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Mariano,
Le 20/12/2015 19:54, Mariano Martinez Peck a écrit :
Hi Dimitri, Dave et all.
First let me do a little disclaimer. Regarding the developments itself of this sponsorship, we discussed about the topics I said in the first email. However, those are not written in stone and that's why I also asked for feedback. As you know, Pharo has relatively little resources so those should be well spent. If we decide to take a spin into a project X, it doesn't mean other projects are not worth. It simply means that we think X will be the most useful for the time being.
One of the final goals of the discusses topics, is to improve a bit the infrastructure for using Pharo as scripting (imagine Coral). For such a thing, we believe that improving FileSystem and OSProcess-like is a good first step to improve the underlying infrastructure.
Would we have access then to a command-line oriented minimal Pharo image?
Well, just for putting perspective my sponsorship is about 3 full months of work effort. It doesn't mean I won't work beyond that. But to give you an idea. I think we all want minimal images , command line etc. And Pharo has already done a huge effort in that direction.
That being said, I of course like the idea of distributing processing across multiple images. No wonder, my PhD tool (Marea) and related sub-projects (Fuel, Ghost proxies, etc) have something to do with the mentioned topic. Dave already mentioned about RemoteTask, which was something I was about to mention. There is also Seamless project (now used by Remote Debugger stuff) and quite other tools. Finally, I think this project may have a big percentage that could be considered as "research". So having a PhD / master student working on this would also be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he is now in RMOD team).
Also, not that forking the image is not my main short-term goal. As said, I will experiment first with a simple tool to execute OS commands that would be FFI based, trying to make async operations, and that would help in most cases. Maybe for those that are not supported, then fallback in OSProcess. Note that Dave (OSProcess author) is very helpful with me and we are discussing together about this.
My guess is that, at least on Linux/Unix, that FFI will be exactly the same as the one used/expressed by the OSProcess plugin, that is popen/system.
No, wait. You are getting down to the details so I want to discuss :)
OSProcess does not use popen()/system() but rather fork()+exec(). That means that OSProcess has implementing the very very low building blocks that would allow you do to almost everything. But then it has it's complicated matters. fork() + exec() cannot be implemented via FFI (this was already discussed in another thread).
Can you link that thread? I'm curious to learn this. Is it multi-thread related? I read [a]... ..."The biggest problem with using fork+exec in this way is that you can't guarantee *nothing* happens between the fork call and the exec call. If, for example, the JVM decides to GC or move memory around, you can have a fatal crash at the JVM process level. Because of that, I don't recommend using fork + exec via FFI in JRuby, even though it's pretty cool. " ..."However, since this post I've learned of the "posix_spawn" [b] function available on most Unix variants. It's basically fork + exec in a single function, plus most of the typical security and IO tweaks you might do after forking and before execing. It's definitely my recommended alternative to fork+exec for JRuby." And posix_spawn also seems recommended for use with the JVM [d] and that Windows and POSIX might have similar usage, and even better match Windows CreateProcess than fork/exec [f] [g] [h] And since with 64-bit Spur we can expect very large images, spawn may be preferred over fork/exec to minimize memory usage for creating application subprocesses [i] Fork versus threads section of [c] or should we be making use of pthread_atfork [e] ? Overall I learnt something new about forking in a multi-threaded environment "When fork() is called, only the calling thread is duplicated in the child process." [a] http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html [b] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf [e] http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-f... [f] http://stackoverflow.com/questions/33249253/what-is-the-difference-between-s... [g] http://stackoverflow.com/questions/5883462/linux-createprocess [h] http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix... [i] http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439...
However, there is a lot of other stuff that we believe it could. So... what we wanted to see if it is possible is to have one only (or very few) primitives (like forkAndExec) and all the rest via FFI. And then all the features I listed in the first email of this thread.
With that in mind, we believe that:
1) Before starting dealing with forkAndExec, we could try a first prototype using popen(). That would allow us test a 100% FFI based approeach, deal with writing and reading from pipes via stream, face the blocking issue, see a prototype on how to read async from pipes etc etc etc. If we see this works nicely, on a second step we can build another approach that instead of popen() uses forkAndExec primitive.
Is popen() being restricted to unidirectional part of this consideration [j] ? Also I read [j] that a problem with popen is that you do not get pid of child so you cannot wait on command to complete. Could posix_spawn be preferrable to popen? and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l] [j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipe... [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-fun...
2) We believe that there is a large number of OSProcess users that use OSProcess only for executing OS commands AND popen() approach would be more than enough (we will do a survey about this). So, a very simply solution via FFI for these users rather than having all OSProcess, Command Shell, the plugins etc, may be worth.
What do you think?
Now, given some of the difficulties seen when dealing with external processes, I'd be wary of the "async" side of things. Random failures seemingly linked to signal loss are certainly not something I'd like to debug.
Indeed, it won't be easy I guess. But sometimes the process you execute may take quite some time and blocking the VM for that period is not a good choice. So... what would you prefer to solve that? Do you prefer image-based pooling as OSProcess does when IOPlugin is not used?
How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement." cheers -ben P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)... http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-usin...
No, wait. You are getting down to the details so I want to discuss :)
OSProcess does not use popen()/system() but rather fork()+exec(). That
means
that OSProcess has implementing the very very low building blocks that would allow you do to almost everything. But then it has it's complicated matters. fork() + exec() cannot be implemented via FFI (this was already discussed in another thread).
Can you link that thread? I'm curious to learn this.
It's a bit long discussion, but here it is: http://forum.world.st/ANN-Limbo-a-simple-shell-wrapper-for-NativeBoost-td465... Is it
multi-thread related? I read [a]... ..."The biggest problem with using fork+exec in this way is that you can't guarantee *nothing* happens between the fork call and the exec call. If, for example, the JVM decides to GC or move memory around, you can have a fatal crash at the JVM process level. Because of that, I don't recommend using fork + exec via FFI in JRuby, even though it's pretty cool. "
Yes, exactly. Pretty much the same. Since fork() clones and both parent and child execute same code (smalltalk vm in our case) until you call exec(), for that period of time, you will have the child running too, and that can screw up everything. ..."However, since this post I've learned of the "posix_spawn" [b]
function available on most Unix variants. It's basically fork + exec in a single function, plus most of the typical security and IO tweaks you might do after forking and before execing. It's definitely my recommended alternative to fork+exec for JRuby."
*WOW. This seems to be one of the most useful recommendations I got. Thank you VERY MUCH.* I was thinking myself "how could I be the only one trying to call a fork+exec kind of function from FFI". Here is the answer. BTW...I have already started coding a FFI wrapper for such a function. I have very basic first command working :)
And posix_spawn also seems recommended for use with the JVM [d]
Nice.
and that Windows and POSIX might have similar usage, and even better match Windows CreateProcess than fork/exec [f] [g] [h]
And since with 64-bit Spur we can expect very large images, spawn may be preferred over fork/exec to minimize memory usage for creating application subprocesses [i]
Interesting.
Fork versus threads section of [c]
or should we be making use of pthread_atfork [e] ?
Overall I learnt something new about forking in a multi-threaded environment "When fork() is called, only the calling thread is duplicated in the child process."
I learned that today too :)
[a] http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html [b] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf [e] http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-f... [f] http://stackoverflow.com/questions/33249253/what-is-the-difference-between-s... [g] http://stackoverflow.com/questions/5883462/linux-createprocess [h] http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix... [i] http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439...
Thanks for all the links, very useful!
However, there is a lot of other stuff that we believe it could. So... what we wanted to see if it is possible is to have one only (or very few) primitives (like forkAndExec) and all the rest via FFI. And then all the features I listed in the first email of this thread.
With that in mind, we believe that:
1) Before starting dealing with forkAndExec, we could try a first prototype using popen(). That would allow us test a 100% FFI based approeach, deal with writing and reading from pipes via stream, face the blocking issue, see a prototype on how to read async from pipes etc etc etc. If we see this works nicely, on a second step we can build another approach that instead of popen() uses forkAndExec primitive.
Is popen() being restricted to unidirectional part of this consideration [j] ?
Yep. You can either define the read more or the write mode (stdout vs stdin) but not both, since popen() opens ONE pipe.
Also I read [j] that a problem with popen is that you do not get pid of child so you cannot wait on command to complete. Could posix_spawn be preferrable to popen?
Exactly. There is no standard way to get child pid from popen() so you can't do polling in image side to wait it's completion. So yeah, posix_spawn() does fix this issue as well as many many other.
and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l]
[j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipe... [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-fun...
Thanks. I will check that on a second step ;)
2) We believe that there is a large number of OSProcess users that use OSProcess only for executing OS commands AND popen() approach would be
more
than enough (we will do a survey about this). So, a very simply solution via FFI for these users rather than having all OSProcess, Command Shell, the plugins etc, may be worth.
What do you think?
Now, given some of the difficulties seen when dealing with external processes, I'd be wary of the "async" side of things. Random failures seemingly linked to signal loss are certainly not something I'd like to debug.
Indeed, it won't be easy I guess. But sometimes the process you execute may take quite some time and blocking the VM for that period is not a good choice. So... what would you prefer to solve that? Do you prefer image-based pooling as OSProcess does when IOPlugin is not used?
How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement."
Well, Eliot said I may need to give a hand to them in order to finish the Threaded FFI. However, I am still not 100% certain if the threaded FFI calls to read from pipes would be worth. Eliot said that yes. Anyway, once I have the first version working I will digest this and goes to the next level and see how I can make things better than polling, if polling proves to be a problem.
cheers -ben
P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)...
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-usin...
Thanks for the reference. Cheers, -- Mariano http://marianopeck.wordpress.com
On Wed, Dec 23, 2015 at 4:08 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
No, wait. You are getting down to the details so I want to discuss :)
OSProcess does not use popen()/system() but rather fork()+exec(). That
means
that OSProcess has implementing the very very low building blocks that would allow you do to almost everything. But then it has it's complicated matters. fork() + exec() cannot be implemented via FFI (this was already discussed in another thread).
Can you link that thread? I'm curious to learn this.
It's a bit long discussion, but here it is: http://forum.world.st/ANN-Limbo-a-simple-shell-wrapper-for-NativeBoost-td465...
Is it
multi-thread related? I read [a]... ..."The biggest problem with using fork+exec in this way is that you can't guarantee *nothing* happens between the fork call and the exec call. If, for example, the JVM decides to GC or move memory around, you can have a fatal crash at the JVM process level. Because of that, I don't recommend using fork + exec via FFI in JRuby, even though it's pretty cool. "
Yes, exactly. Pretty much the same. Since fork() clones and both parent and child execute same code (smalltalk vm in our case) until you call exec(), for that period of time, you will have the child running too, and that can screw up everything.
..."However, since this post I've learned of the "posix_spawn" [b]
function available on most Unix variants. It's basically fork + exec in a single function, plus most of the typical security and IO tweaks you might do after forking and before execing. It's definitely my recommended alternative to fork+exec for JRuby."
*WOW. This seems to be one of the most useful recommendations I got. Thank you VERY MUCH.* I was thinking myself "how could I be the only one trying to call a fork+exec kind of function from FFI". Here is the answer. BTW...I have already started coding a FFI wrapper for such a function. I have very basic first command working :)
And posix_spawn also seems recommended for use with the JVM [d]
Nice.
and that Windows and POSIX might have similar usage, and even better match Windows CreateProcess than fork/exec [f] [g] [h]
And since with 64-bit Spur we can expect very large images, spawn may be preferred over fork/exec to minimize memory usage for creating application subprocesses [i]
Interesting.
Fork versus threads section of [c]
or should we be making use of pthread_atfork [e] ?
Overall I learnt something new about forking in a multi-threaded environment "When fork() is called, only the calling thread is duplicated in the child process."
I learned that today too :)
[a] http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html [b] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf [e] http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-f... [f] http://stackoverflow.com/questions/33249253/what-is-the-difference-between-s... [g] http://stackoverflow.com/questions/5883462/linux-createprocess [h] http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix... [i] http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439...
Thanks for all the links, very useful!
However, there is a lot of other stuff that we believe it could. So... what we wanted to see if it is possible is to have one only (or very few) primitives (like forkAndExec) and all the rest via FFI. And then all the features I listed in the first email of this thread.
With that in mind, we believe that:
1) Before starting dealing with forkAndExec, we could try a first prototype using popen(). That would allow us test a 100% FFI based approeach, deal with writing and reading from pipes via stream, face the blocking issue, see a prototype on how to read async from pipes etc etc etc. If we see this works nicely, on a second step we can build another approach that instead of popen() uses forkAndExec primitive.
Is popen() being restricted to unidirectional part of this consideration [j] ?
Yep. You can either define the read more or the write mode (stdout vs stdin) but not both, since popen() opens ONE pipe.
Also I read [j] that a problem with popen is that you do not get pid of child so you cannot wait on command to complete. Could posix_spawn be preferrable to popen?
Exactly. There is no standard way to get child pid from popen() so you can't do polling in image side to wait it's completion. So yeah, posix_spawn() does fix this issue as well as many many other.
Looks nice and at the same time, popen support is useful/wanted. A generic way to deal with pids (in Linux at least, is to have the process write its pid in /var/run/ or a subdir of it. That's how monit checks for processes for example. http://unix.stackexchange.com/questions/12815/what-are-pid-and-lock-files-fo... We need as many options as we can here. Mariano, I added you to a Trello board that captured a couple of items about the topic you deal with. https://trello.com/b/Hji4F0Zp/clipharo We also have a card that says the following: Pharo follows Smalltalk which uses a global object named Smalltalk which in fact is an instance of SmalltalkImage, confusing if you ask me. I would add a simple class named System which have class-side accessors for static global things like - image => links to the old Smalltalk - stdin - stdout - session - environment for ENV vars - workingDirectory accessing the process working directory $PWD - 'pid' That's a very easy thing to do, but IMO it will help anybody coming from outside smalltalk to easily find these things. Probably there are more things that would fit here. I'd like to have that. The discussion involved Camillo who was very into these topics. His Pinocchio work seemed to have contributions that can be useful of the CLI support for Pharo. Phil
and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l]
[j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipe... [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-fun...
Thanks. I will check that on a second step ;)
2) We believe that there is a large number of OSProcess users that use OSProcess only for executing OS commands AND popen() approach would be
more
than enough (we will do a survey about this). So, a very simply solution via FFI for these users rather than having all OSProcess, Command Shell, the plugins etc, may be worth.
What do you think?
Now, given some of the difficulties seen when dealing with external processes, I'd be wary of the "async" side of things. Random failures seemingly linked to signal loss are certainly not something I'd like to debug.
Indeed, it won't be easy I guess. But sometimes the process you execute may take quite some time and blocking the VM for that period is not a good choice. So... what would you prefer to solve that? Do you prefer image-based pooling as OSProcess does when IOPlugin is not used?
How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement."
Well, Eliot said I may need to give a hand to them in order to finish the Threaded FFI. However, I am still not 100% certain if the threaded FFI calls to read from pipes would be worth. Eliot said that yes. Anyway, once I have the first version working I will digest this and goes to the next level and see how I can make things better than polling, if polling proves to be a problem.
cheers -ben
P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)...
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-usin...
Thanks for the reference.
Cheers,
-- Mariano http://marianopeck.wordpress.com
Also I read [j] that a problem with popen is that you do not get pid of child so you cannot wait on command to complete. Could posix_spawn be preferrable to popen?
Exactly. There is no standard way to get child pid from popen() so you can't do polling in image side to wait it's completion. So yeah, posix_spawn() does fix this issue as well as many many other.
Looks nice and at the same time, popen support is useful/wanted.
Yes, but I wonder if it would still make sense if I finish the posix_spawn flavor. Anyway, we can decide later since the Popen wrapper is almost written (a very very first prototype)
A generic way to deal with pids (in Linux at least, is to have the process write its pid in /var/run/ or a subdir of it.
That's right, but that only works for a certain number of cases, and polling with checking files does not seem good. I think that you if you expect to run a a command that takes some time, then use the posix_spawn wrapper and use popen only in "blocking" mode.
That's how monit checks for processes for example.
http://unix.stackexchange.com/questions/12815/what-are-pid-and-lock-files-fo...
We need as many options as we can here.
I not fully convinced with that statement.
Mariano, I added you to a Trello board that captured a couple of items about the topic you deal with. https://trello.com/b/Hji4F0Zp/clipharo
Thanks!
We also have a card that says the following:
Pharo follows Smalltalk which uses a global object named Smalltalk which in fact is an instance of SmalltalkImage, confusing if you ask me.
I would add a simple class named System which have class-side accessors for static global things like
My answers below. Just as a matter of how that System class could be written:
- image => links to the old Smalltalk
Well..we have now "Smalltalk image"
- stdin - stdout - session
OK about those.
- environment for ENV vars
we have "Smalltalk os environment"
- workingDirectory accessing the process working directory $PWD
We have "FileSystem workingDirectory"
- 'pid'
Well, we need OSProcess right now, but it is also very very easy to do via FFI.
That's a very easy thing to do, but IMO it will help anybody coming from outside smalltalk to easily find these things. Probably there are more things that would fit here.
I'd like to have that.
The discussion involved Camillo who was very into these topics. His Pinocchio work seemed to have contributions that can be useful of the CLI support for Pharo.
Phil
and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l]
[j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipe... [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-fun...
Thanks. I will check that on a second step ;)
2) We believe that there is a large number of OSProcess users that use OSProcess only for executing OS commands AND popen() approach would be
more
than enough (we will do a survey about this). So, a very simply solution via FFI for these users rather than having all OSProcess, Command Shell, the plugins etc, may be worth.
What do you think?
Now, given some of the difficulties seen when dealing with external processes, I'd be wary of the "async" side of things. Random failures seemingly linked to signal loss are certainly not something I'd like
to
debug.
Indeed, it won't be easy I guess. But sometimes the process you execute may take quite some time and blocking the VM for that period is not a good choice. So... what would you prefer to solve that? Do you prefer image-based pooling as OSProcess does when IOPlugin is not used?
How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement."
Well, Eliot said I may need to give a hand to them in order to finish the Threaded FFI. However, I am still not 100% certain if the threaded FFI calls to read from pipes would be worth. Eliot said that yes. Anyway, once I have the first version working I will digest this and goes to the next level and see how I can make things better than polling, if polling proves to be a problem.
cheers -ben
P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)...
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-usin...
Thanks for the reference.
Cheers,
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On Wed, Dec 23, 2015 at 1:54 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
Also I read [j] that a problem with popen is that you do not get pid of child so you cannot wait on command to complete. Could posix_spawn be preferrable to popen?
Exactly. There is no standard way to get child pid from popen() so you can't do polling in image side to wait it's completion. So yeah, posix_spawn() does fix this issue as well as many many other.
Looks nice and at the same time, popen support is useful/wanted.
Yes, but I wonder if it would still make sense if I finish the posix_spawn flavor. Anyway, we can decide later since the Popen wrapper is almost written (a very very first prototype)
A generic way to deal with pids (in Linux at least, is to have the process write its pid in /var/run/ or a subdir of it.
That's right, but that only works for a certain number of cases, and polling with checking files does not seem good.
Lots of software already works that way. In the name of proper integration of external processes/"things", it is a must have. Well, I need it in production.
I think that you if you expect to run a a command that takes some time, then use the posix_spawn wrapper and use popen only in "blocking" mode.
That's how monit checks for processes for example.
http://unix.stackexchange.com/questions/12815/what-are-pid-and-lock-files-fo...
We need as many options as we can here.
I not fully convinced with that statement.
Anything making Pharo less of an island is a step in the right direction. popen is how it is done in one of the most successful platforms. I am using it too, so, being able to port knowledge from point A to B is useful in my view. https://docs.python.org/2/library/subprocess.html You'll see spawn and popen. And Popen2,3, 4.
Mariano, I added you to a Trello board that captured a couple of items about the topic you deal with. https://trello.com/b/Hji4F0Zp/clipharo
Thanks!
We also have a card that says the following:
Pharo follows Smalltalk which uses a global object named Smalltalk which in fact is an instance of SmalltalkImage, confusing if you ask me.
I would add a simple class named System which have class-side accessors for static global things like
My answers below. Just as a matter of how that System class could be written:
- image => links to the old Smalltalk
Well..we have now "Smalltalk image"
- stdin - stdout - session
OK about those.
- environment for ENV vars
we have "Smalltalk os environment"
- workingDirectory accessing the process working directory $PWD
We have "FileSystem workingDirectory"
Look above, one has to peek all around to find them (not to mention being aware. And I'd even say, foreget FileSystem, why not FileLocator, which is meant for that). FWIW, https://nodejs.org/api/os.html
- 'pid'
Well, we need OSProcess right now, but it is also very very easy to do via FFI.
That's a very easy thing to do, but IMO it will help anybody coming from outside smalltalk to easily find these things. Probably there are more things that would fit here.
I'd like to have that.
The discussion involved Camillo who was very into these topics. His Pinocchio work seemed to have contributions that can be useful of the CLI support for Pharo.
Phil
and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l]
[j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipe... [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-fun...
Thanks. I will check that on a second step ;)
2) We believe that there is a large number of OSProcess users that use OSProcess only for executing OS commands AND popen() approach would
be more
than enough (we will do a survey about this). So, a very simply solution via FFI for these users rather than having all OSProcess, Command Shell, the plugins etc, may be worth.
What do you think?
Now, given some of the difficulties seen when dealing with external processes, I'd be wary of the "async" side of things. Random failures seemingly linked to signal loss are certainly not something I'd like
to
debug.
Indeed, it won't be easy I guess. But sometimes the process you execute may take quite some time and blocking the VM for that period is not a good choice. So... what would you prefer to solve that? Do you prefer image-based pooling as OSProcess does when IOPlugin is not used?
How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement."
Well, Eliot said I may need to give a hand to them in order to finish the Threaded FFI. However, I am still not 100% certain if the threaded FFI calls to read from pipes would be worth. Eliot said that yes. Anyway, once I have the first version working I will digest this and goes to the next level and see how I can make things better than polling, if polling proves to be a problem.
cheers -ben
P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)...
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-usin...
Thanks for the reference.
Cheers,
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On Wed, Dec 23, 2015 at 7:46 PM, phil@highoctane.be <phil@highoctane.be> wrote:
Pharo follows Smalltalk which uses a global object named Smalltalk which in fact is an instance of SmalltalkImage, confusing if you ask me.
I also have felt that same impedance mismatch in the naming, but I thought the accessors were deliberately moved to the instance side to help (maybe later) to minimise global state. So I am guessing that adding class-side accessors is a step backwards - but maybe System class could provide a singleton instance of itself. cheers -ben
I would add a simple class named System which have class-side accessors for static global things like
image => links to the old Smalltalk stdin stdout session environment for ENV vars workingDirectory accessing the process working directory $PWD 'pid'
That's a very easy thing to do, but IMO it will help anybody coming from outside smalltalk to easily find these things. Probably there are more things that would fit here.
I'd like to have that.
The discussion involved Camillo who was very into these topics. His Pinocchio work seemed to have contributions that can be useful of the CLI support for Pharo.
Phil
and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l]
[j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipe... [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-fun...
Thanks. I will check that on a second step ;)
2) We believe that there is a large number of OSProcess users that use OSProcess only for executing OS commands AND popen() approach would be more than enough (we will do a survey about this). So, a very simply solution via FFI for these users rather than having all OSProcess, Command Shell, the plugins etc, may be worth.
What do you think?
Now, given some of the difficulties seen when dealing with external processes, I'd be wary of the "async" side of things. Random failures seemingly linked to signal loss are certainly not something I'd like to debug.
Indeed, it won't be easy I guess. But sometimes the process you execute may take quite some time and blocking the VM for that period is not a good choice. So... what would you prefer to solve that? Do you prefer image-based pooling as OSProcess does when IOPlugin is not used?
How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement."
Well, Eliot said I may need to give a hand to them in order to finish the Threaded FFI. However, I am still not 100% certain if the threaded FFI calls to read from pipes would be worth. Eliot said that yes. Anyway, once I have the first version working I will digest this and goes to the next level and see how I can make things better than polling, if polling proves to be a problem.
cheers -ben
P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)...
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-usin...
Thanks for the reference.
Cheers,
-- Mariano http://marianopeck.wordpress.com
participants (3)
-
Ben Coman -
Mariano Martinez Peck -
phil＠highoctane.be