>
>
> No, wait. You are getting down to the details so I want to discuss :)
>
> OSProcess does not use popen()/system() but rather fork()+exec(). That means
> that OSProcess has implementing the very very low building blocks that would
> allow you do to almost everything. But then it has it's complicated matters.
> fork() + exec() cannot be implemented via FFI (this was already discussed in
> another thread).

Can you link that thread? I'm curious to learn this.

�� Is it
multi-thread related?�� ��I read [a]...
..."The biggest problem with using fork+exec in this way is that you
can't guarantee *nothing* happens between the fork call and the exec
call. If, for example, the JVM decides to GC or move memory around,
you can have a fatal crash at the JVM process level. Because of that,
I don't recommend using fork + exec via FFI in JRuby, even though it's
pretty cool. "

..."However, since this post I've learned of the "posix_spawn" [b]
function available on most Unix variants. It's basically fork + exec
in a single function, plus most of the typical security and IO tweaks
you might do after forking and before execing. It's definitely my
recommended alternative to fork+exec for JRuby."

And posix_spawn also seems recommended for use with the JVM [d]

and that Windows and POSIX might have similar usage, and even better
match Windows CreateProcess than fork/exec [f] [g] [h]

And since with 64-bit Spur we can expect very large images, spawn may
be preferred over fork/exec to minimize memory usage for creating
application subprocesses [i]

Fork versus threads section of [c]

or should we be making use of pthread_atfork [e] ?

Overall I learnt something new about forking in a multi-threaded environment
"When fork() is called, only the calling thread is duplicated in the
child process."

[a] http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html
[b] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html
[d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html
[c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf
[e] http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-follow-fork-in-a-multi-threaded-process
[f] http://stackoverflow.com/questions/33249253/what-is-the-difference-between-spawn-and-fork
[g] http://stackoverflow.com/questions/5883462/linux-createprocess
[h] http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix_spawn-835385/
[i] http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html

> However, there is a lot of other stuff that we believe it
> could. So... what we wanted to see if it is possible is to have one only (or
> very few) primitives (like forkAndExec) and all the rest via FFI. And then
> all the features I listed in the first email of this thread.
>
> With that in mind, we believe that:
>
> 1) Before starting dealing with forkAndExec, we could try a first prototype
> using popen(). That would allow us test a 100% FFI based approeach, deal
> with writing and reading from pipes via stream, face the blocking issue, see
> a prototype on how to read async from pipes etc etc etc. If we see this
> works nicely, on a second step we can build another approach that instead of
> popen() uses forkAndExec primitive.

Is popen() being restricted to unidirectional part of this consideration [j] ?

Also I read [j] that a problem with popen is that you do not get pid
of child so you cannot wait on command to complete.�� Could posix_spawn
be preferrable to popen?

and just btw, mildly interesting is using sockets rather than pipes
for interprocess communication [l]

[j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipes-like-pipe-fork
[k] http://stackoverflow.com/questions/6743771/popen-alternative
[l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-functionality-i-o/25177958#25177958

>
> 2) We believe that there is a large number of OSProcess users that use
> OSProcess only for executing OS commands AND popen() approach would be more
> than enough (we will do a survey about this). So, a very simply solution via
> FFI for these users rather than having all OSProcess, Command Shell, the
> plugins etc, may be worth.
>
> What do you think?
>
>>
>> Now, given some of the difficulties seen when dealing with external
>> processes, I'd be wary of the "async" side of things. Random failures
>> seemingly linked to signal loss are certainly not something I'd like to
>> debug.
>>
>
> Indeed, it won't be easy I guess. But sometimes the process you execute may
> take quite some time and blocking the VM for that period is not a good
> choice. So... what would you prefer to solve that? Do you prefer image-based
> pooling as OSProcess does when IOPlugin is not used?


How much might your work overlap with the Threaded FFI project for Cog?...
..."The Cog VM is single-threaded, providing green threads to the
Smalltalk level, as do most Smalltalk VMs. There is a prototype
subclass of the CoInterpreter, CoInterpreterMT, which provides a
Python-like multi-threaded VM where any number of threads can share
the VM with only one running the VM at any one time.�� This uses an
extremely simple low-level scheme for releasing and re-acquiring
ownership of the VM designed by David Simmons.�� This results in any
and all FFI calls being non-blocking since there is such a low
overhead to making a non-blocking FFI call that all FFI calls can be
non-blocking.�� Eliot has worked on this, implementing the prototype
and the reentrant FFI plugin it requires.�� But this needs to be
revived.�� It would be a really powerful enhancement."

cheers -ben

P.S. Just another article warning against mixing fork and
multiple-threads (which it seems posix_spawn may avoid)...
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them