[Ann] OSSubprocess 1.0.0
Hi all, News from the OSSubprocess side. For those who do not know it, OSSubprocess is the library to call external processes from Pharo. These are the main points of this release. - 64bits support - Issues Working on both Pharo 6 and 7 - Issue #34: Adding tests to validate that the return code is correctly handled - Issue #27: Cleanup: remove OSSubprocess-Tests package - PR #26: Add stress tests This version works for Pharo >=6. Users of Pharo5 will need to stick with v0.2.5, because of non compatible changes in uFFI (required mostly for 64 bits). Additionally: - We moved the repository to pharo-contribution to remove pressure from Mariano for releases and integrations - We updated the configuration in the catalog - We decided to move to a 1.0.0 versioning schema because 0.* does look like the library is not stable and it actually is (We have been using it in pillar and iceberg for quite some time). You can load this version from the catalog or using with: Metacello new baseline: 'OSSubprocess'; repository: 'github://pharo-contributions/OSSubprocess:v1.0.0/repository'; load. Detailed changes log in - https://github.com/pharo-contributions/OSSubprocess/releases/tag/v1.0.0 - https://github.com/pharo-contributions/OSSubprocess/compare/v0.2.5...v1.0.0 If you have any issues (or PR!), we wait for your issue entry in https://github.com/pharo-contributions/OSSubprocess/issues Have a nice weekend, -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Guillermo Polito wrote
If you have any issuesâ¦
IMHO the biggest issue is incompatibility with OSProcess. This seems not to matter when one is loading Project A depending on OSSP in a clean image, but the second one tries to load Project A into an image where Project B already depends on OSP - KABOOM! Game over. On a more philosophical note, I never understood why OSSP was created in the first place since we have OSP; and especially why the API is different, making switching more difficult. I'm sure there was a good reason to do OSSP, but it might be helpful to clarify in the docs (so tradeoffs between OSP and OSSP can be understood by prospective users). ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
Hi Sean On Fri, May 18, 2018 at 9:49 PM, Sean P. DeNigris <sean@clipperadams.com> wrote:
Guillermo Polito wrote
If you have any issuesâ¦
IMHO the biggest issue is incompatibility with OSProcess. This seems not to matter when one is loading Project A depending on OSSP in a clean image, but the second one tries to load Project A into an image where Project B already depends on OSP - KABOOM! Game over.
Well, I cannot say much about why this happens. I've seen it happening, but I try not to depend on projects using OSProcess in that case. Can we also say the same the other way around? If we have OSSubprocess loaded and I download a project using OSProcess, does it break the same? I understand that the incompatibilities bother, but putting all the fault in OSSubprocess and not in OSProcess seems not fair :). Maybe both should be modified to be compatible? Dunno...
On a more philosophical note, I never understood why OSSP was created in the first place since we have OSP;
I cannot answer this, I'll let Mariano or somebody from the Pharo board on this one.
and especially why the API is different, making switching more difficult.
I found OSProcess API particularly bad. It's never clear to me whether to use OSProcess, PipeableSomething or CommandWhatever (I don't even remember the names of the things that **do work** and I have to look them every time). To be fair, to me even sometimes OSProcess is too verbose. Most people only want to do output := OSProcess executeCommand: 'ls'. Blocking. Recovering stdout. And that's all. So in that sense, I think that letting us move on and rethink the API is not bad at all... Then, we can think on providing some compatibility layer to help people in migrating. So, if anyone want to take action on this point... I'm sure there was a good reason to do
OSSP, but it might be helpful to clarify in the docs (so tradeoffs between OSP and OSSP can be understood by prospective users).
Well from a technical point of view, if you want to choose from one or another today I'd say: - OSSP is written mostly in FFI, what means that it is easier to modify and ship. - I'm sure OSSP works: it has way more tests and we are using it for several projects ourselves. (Please notice I do not say that OSP does not work, I say I have no way to be sure that it works because tests do not run). So we are eating our own food: - we use them to run iceberg tests - pillar is using it to compile pdfs I've put myself quite some effort to have a working CI, with stress tests, 64bits. I've just tried to run OSP tests and they do not even run in Pharo because they use FileDirectory (and even latest version is updated to work on Cuis)... Having a compatibility layer to make such a project run on squeak, cuis and Pharo looks like a gigantic task for a project that seems so core to me.
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Guillermo Polito wrote
I try not to depend on projects using OSProcess in that case.
That is not exactly a robust solution! What is the only project providing the functionality you need depends on OSP?! Guillermo Polito wrote
I understand that the incompatibilities bother, but putting all the fault in OSSubprocess and not in OSProcess seems not fair :)
Ha ha, well OSProcess was born in 2005, and then OSSP appeared reusing some of its functionality just recently, so I'd say it's fair indeed! Guillermo Polito wrote
Maybe both should be modified to be compatible?
Maybe. I don't really understand the source of the incompatibility well enough. Guillermo Polito wrote
I found OSProcess API particularly bad. It's never clear to me whether to use OSProcess, PipeableSomething or CommandWhatever (I don't even remember the names of the things that **do work** and I have to look them every time).
The things you're describing are not accidental complexity, but the fact that OSP (and its companion CommandShell) actually have a far wider feature set than just simple commands. CommandShell makes many shell-like things possible in-image, but one doesn't have to use or even know anything about all that to do simple things. That said, every time I did into CS and see all the things that are possible I am amazed. Guillermo Polito wrote
Most people only want to do output := OSProcess executeCommand: 'ls'. Blocking. Recovering stdout. And that's all.
(PipeableOSProcess command: 'ls') output That doesn't seem horribly complex to do in OSP ;) Guillermo Polito wrote
Well from a technical point of view, if you want to choose from one or another today I'd sayâ¦
Ah, interesting and informative. Thank you. ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
On Wed, May 23, 2018 at 07:11:29AM -0700, Sean P. DeNigris wrote:
Guillermo Polito wrote
I try not to depend on projects using OSProcess in that case.
That is not exactly a robust solution! What is the only project providing the functionality you need depends on OSP?!
Guillermo Polito wrote
I understand that the incompatibilities bother, but putting all the fault in OSSubprocess and not in OSProcess seems not fair :)
Ha ha, well OSProcess was born in 2005, and then OSSP appeared reusing some of its functionality just recently, so I'd say it's fair indeed!
Actually, OSProcess was born in 1999 :-) First announcement: http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-June/016553.html And this appears to be where I first starting thinking about the problem: http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-January/025236.h... http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-May/023712.html Kudos to Mariano for working closely with me as he developed the FFI based functionality in OSSP, he did a really good job of working out issues wherever possible. FFI based solutions work at a different level of abstraction than VM plugins, and there is a role for both. Dave
David T. Lewis wrote
FFI based solutions work at a different level of abstraction than VM plugins, and there is a role for both.
Thanks for the context, David. Now that we understand the different niches, the main problem is that they can not be loaded in the same image without breaking :/ ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris <sean@clipperadams.com> wrote:
David T. Lewis wrote
FFI based solutions work at a different level of abstraction than VM plugins, and there is a role for both.
Thanks for the context, David. Now that we understand the different niches, the main problem is that they can not be loaded in the same image without breaking :/
The problem is to find a solution that works for both, Squeak and Pharo as well as for OSProcess and OSSubprocess. I guess one possibility is to modify both, OSProcess and OSSubprocess initialization of the child reaper so that they install the same "generic" child reapear. Aside from initializating the child repear, they should be added into an "observer list". When, when it comes the second project to get loaded it which check that a child reaper is already registered..in which case he just register itself as "observer". This child reaper should be generic enough (cannot be coupled to WHAT to do when the semaphore is signaled). Using Announcements (or other mechanisim that would work for Pharo and Squeak) we could simply "notify" the registered observers and each observer would do whatever is needed (what is actually now hardcoded in each child reaper) Maybe that works...just an idea.
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
-- Mariano http://marianopeck.wordpress.com
Hi Mariano, 2018-05-23 19:57 GMT+02:00 Mariano Martinez Peck <marianopeck@gmail.com>:
On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris <sean@clipperadams.com> wrote:
David T. Lewis wrote
FFI based solutions work at a different level of abstraction than VM plugins, and there is a role for both.
Thanks for the context, David. Now that we understand the different niches, the main problem is that they can not be loaded in the same image without breaking :/
The problem is to find a solution that works for both, Squeak and Pharo as well as for OSProcess and OSSubprocess.
I guess one possibility is to modify both, OSProcess and OSSubprocess initialization of the child reaper so that they install the same "generic" child reapear. Aside from initializating the child repear, they should be added into an "observer list". When, when it comes the second project to get loaded it which check that a child reaper is already registered..in which case he just register itself as "observer".
This child reaper should be generic enough (cannot be coupled to WHAT to do when the semaphore is signaled). Using Announcements (or other mechanisim that would work for Pharo and Squeak) we could simply "notify" the registered observers and each observer would do whatever is needed (what is actually now hardcoded in each child reaper)
Maybe that works...just an idea.
I think that looks like a nice design. Just one question: do we need to track down whether the child reaper event should be directed to OSProcess or to OSSubprocess? Wondering if there is an issue with one of them trying to close the other one's resource (but I haven't checked, so I may be wrong). Thierry
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
-- Mariano http://marianopeck.wordpress.com
Hi Thierry. Excellent question. For OSSubprocess this is not an issue because I look up by pid in my list of forked children... So if not found I do nothing... At least that's what I remember (away from my machine now). So at worst it would be a performance problem when both loaded. As for osprocess I don't recall if this would be a problem but I imagine it won't. Maybe David can tell. On Wed, May 23, 2018, 5:31 PM Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Mariano,
2018-05-23 19:57 GMT+02:00 Mariano Martinez Peck <marianopeck@gmail.com>:
On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris <sean@clipperadams.com> wrote:
David T. Lewis wrote
FFI based solutions work at a different level of abstraction than VM plugins, and there is a role for both.
Thanks for the context, David. Now that we understand the different niches, the main problem is that they can not be loaded in the same image
without
breaking :/
The problem is to find a solution that works for both, Squeak and Pharo as well as for OSProcess and OSSubprocess.
I guess one possibility is to modify both, OSProcess and OSSubprocess initialization of the child reaper so that they install the same "generic" child reapear. Aside from initializating the child repear, they should be added into an "observer list". When, when it comes the second project to get loaded it which check that a child reaper is already registered..in which case he just register itself as "observer".
This child reaper should be generic enough (cannot be coupled to WHAT to do when the semaphore is signaled). Using Announcements (or other mechanisim that would work for Pharo and Squeak) we could simply "notify" the registered observers and each observer would do whatever is needed (what is actually now hardcoded in each child reaper)
Maybe that works...just an idea.
I think that looks like a nice design. Just one question: do we need to track down whether the child reaper event should be directed to OSProcess or to OSSubprocess? Wondering if there is an issue with one of them trying to close the other one's resource (but I haven't checked, so I may be wrong).
Thierry
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
-- Mariano http://marianopeck.wordpress.com
Hi Mariano, hi Thierry, I cannot follow up for a few days but I like Mariano's proposal and I think that it will work. Maybe move the child reaper process to a small OSChildWatcher class and have interested clients (OSP, OSSP, others) subscribe to update notifications. Dave
Hi Thierry. Excellent question. For OSSubprocess this is not an issue because I look up by pid in my list of forked children... So if not found I do nothing... At least that's what I remember (away from my machine now). So at worst it would be a performance problem when both loaded.
As for osprocess I don't recall if this would be a problem but I imagine it won't. Maybe David can tell.
On Wed, May 23, 2018, 5:31 PM Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Mariano,
2018-05-23 19:57 GMT+02:00 Mariano Martinez Peck <marianopeck@gmail.com>:
On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris
<sean@clipperadams.com>
wrote:
David T. Lewis wrote
FFI based solutions work at a different level of abstraction than VM plugins, and there is a role for both.
Thanks for the context, David. Now that we understand the different niches, the main problem is that they can not be loaded in the same image
without
breaking :/
The problem is to find a solution that works for both, Squeak and Pharo as well as for OSProcess and OSSubprocess.
I guess one possibility is to modify both, OSProcess and OSSubprocess initialization of the child reaper so that they install the same "generic" child reapear. Aside from initializating the child repear, they should be added into an "observer list". When, when it comes the second project to get loaded it which check that a child reaper is already registered..in which case he just register itself as "observer".
This child reaper should be generic enough (cannot be coupled to WHAT to do when the semaphore is signaled). Using Announcements (or other mechanisim that would work for Pharo and Squeak) we could simply "notify" the registered observers and each observer would do whatever is needed (what is actually now hardcoded in each child reaper)
Maybe that works...just an idea.
I think that looks like a nice design. Just one question: do we need to track down whether the child reaper event should be directed to OSProcess or to OSSubprocess? Wondering if there is an issue with one of them trying to close the other one's resource (but I haven't checked, so I may be wrong).
Thierry
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
-- Mariano http://marianopeck.wordpress.com
Hi guys, First, I would like to say that David was always very helpful, positive and we have discussed and worked together a lot for this. When I was sponsored to build OSSubprocess I already got the "we would like this have this", so maybe some additional opinion from the ESUG board may help clarify. From what I understand/remember/believe, the main reasons for building OSSubprocess were: 1) *FFI-based tool*. Low the barrier of having to modify the VM to make a fix. Despite all our efforts, we still have way more people able to deal with FFI than with the VM. Even more if the VM is now shared under a common project, in which is not always easy to plug Pharo-specific stuff. 2) *Cleaner/nicer API*. You may or may not agree with this and that's perfect (we all have different opinions). When I was using OSProcess I always thought the API was not as I would like to, so my goal was to provide a API closer to my wishes. Again, not necessary better. 3) *Smaller functionality than OSProcess*: OSProcess is huge...it allows you to fork squeak images, fork() + exec*() whatever you want, runs on Windows, etc. We wanted something smaller...just enough to be able to fork processes. That is... to capture the most common/easy process forking.. But if you really need something very advance, you may still go to bare metals and use OSProcess approach (fork + exec). 4) *Added infrastructure*. We wanted the project to be on Git, with a very detailed documentation, very high test coverage, and CI integration. ----- So...that is about the reasons... as to why it breaks when OSProcess is loaded to, it's because for OSSubprocess I took the wonderful idea from OSProcess of the child reaper. This is a process that runs on the Smalltalk side and handles exception (via semaphore) when the child has died. This is (currently) only possible thanks to OSProcess primitive. That means OSSubprocess still needs (one or very little) OSProcess primitives down in the VM. When you load either of the projects, they launch the child reaper process, register the semaphore etc etc. So the problem is when you loaded one and you then try to load the second one, the initialization of the second one you plug its own semaphore etc etc and so the previous gets broken as it doesn't have a working child reaper process. If someone has a better idea on how to solve this, then please speak up. Cheers, On Wed, May 23, 2018 at 11:12 AM Sean P. DeNigris <sean@clipperadams.com> wrote:
Guillermo Polito wrote
I try not to depend on projects using OSProcess in that case.
That is not exactly a robust solution! What is the only project providing the functionality you need depends on OSP?!
Guillermo Polito wrote
I understand that the incompatibilities bother, but putting all the fault in OSSubprocess and not in OSProcess seems not fair :)
Ha ha, well OSProcess was born in 2005, and then OSSP appeared reusing some of its functionality just recently, so I'd say it's fair indeed!
Guillermo Polito wrote
Maybe both should be modified to be compatible?
Maybe. I don't really understand the source of the incompatibility well enough.
Guillermo Polito wrote
I found OSProcess API particularly bad. It's never clear to me whether to use OSProcess, PipeableSomething or CommandWhatever (I don't even remember the names of the things that **do work** and I have to look them every time).
The things you're describing are not accidental complexity, but the fact that OSP (and its companion CommandShell) actually have a far wider feature set than just simple commands. CommandShell makes many shell-like things possible in-image, but one doesn't have to use or even know anything about all that to do simple things. That said, every time I did into CS and see all the things that are possible I am amazed.
Guillermo Polito wrote
Most people only want to do output := OSProcess executeCommand: 'ls'. Blocking. Recovering stdout. And that's all.
(PipeableOSProcess command: 'ls') output
That doesn't seem horribly complex to do in OSP ;)
Guillermo Polito wrote
Well from a technical point of view, if you want to choose from one or another today I'd sayâ¦
Ah, interesting and informative. Thank you.
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
-- Mariano http://marianopeck.wordpress.com
Mariano Martinez Peck wrote
From what I understand/remember/believe, the main reasons for building OSSubprocess were:
Thanks, Mariano!! Very informative :) ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
participants (5)
-
David T. Lewis -
Guillermo Polito -
Mariano Martinez Peck -
Sean P. DeNigris -
Thierry Goubier