[Pharo-project] OSProcess - working on gnuplot
Dave, I am slowly gathering ideas on how to write an interface to gnuplot. It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs. Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work. For a single graph, think I see how to make that go. For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :) Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so? I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC?? Bill
Hi, You don't have to reinvent the wheel all the time: http://www.squeaksource.com/GNUPlot.html Cheers, Levente On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:
Dave,
I am slowly gathering ideas on how to write an interface to gnuplot. It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs.
Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work. For a single graph, think I see how to make that go.
For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :)
Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so? I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC??
Bill
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
It makes explicit reference to win32, so I excluded it on that basis. Are you saying it runs elsewhere? -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Levente Uzonyi Sent: Sunday, November 15, 2009 3:26 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot Hi, You don't have to reinvent the wheel all the time: http://www.squeaksource.com/GNUPlot.html Cheers, Levente On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:
Dave,
I am slowly gathering ideas on how to write an interface to gnuplot. It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs.
Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work. For a single graph, think I see how to make that go.
For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :)
Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so? I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC??
Bill
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
It runs on all platforms, including win32. Levente On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:
It makes explicit reference to win32, so I excluded it on that basis. Are you saying it runs elsewhere?
-----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Levente Uzonyi Sent: Sunday, November 15, 2009 3:26 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot
Hi,
You don't have to reinvent the wheel all the time: http://www.squeaksource.com/GNUPlot.html
Cheers, Levente
On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:
Dave,
I am slowly gathering ideas on how to write an interface to gnuplot. It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs.
Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work. For a single graph, think I see how to make that go.
For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :)
Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so? I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC??
Bill
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Sun, Nov 15, 2009 at 03:04:49PM -0500, Schwab,Wilhelm K wrote:
I am slowly gathering ideas on how to write an interface to gnuplot. It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs.
Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work. For a single graph, think I see how to make that go.
For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :)
Hi Bill, You should probably first check out the suggestion from Levente:
You don't have to reinvent the wheel all the time: http://www.squeaksource.com/GNUPlot.html
But your question is interesting to me, because the first free software thing I ever wrote was gl_plot, which appeared in comp.sources.unix volume 18, and was used for generating graphics to support gnuplot on PC unix (Microport System V/AT) back before PC unix had graphics support. It's still out there in case you want to run gnuplot on your 286 unix pc: http://ftp.isc.org/pub/usenet/comp.sources.unix/volume18/gl_plot Anyway, back to the new millenium. Yes you should definitely be able to control a gnuplot process with a connected PipeableOSProcess. That's what it is designed for.
Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so?
Sure, see PipeableOSProcess class>>tkExample for an example of how to do this. Use /usr/bin/gnuplot instead of /usr/bin/wish of course. However (and this is a big caveat at the moment), OSProcess and CommandShell currently do not work properly on Pharo for reasons that I don't yet understand. So definitely check out Levente's suggestion as a first step.
I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC??
CommandShell has user interfaces for both Morphic and MVC. Pharo does not have MVC, but that should not effect the Morphic side, other than the annoyance of having to skip over the errors when you load it. Dave
Dave, ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix. Please let me know if I am missing something. It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work. I am having some success with code such as gp := PipeableOSProcess command:'gnuplot'. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. but I cannot get any output without shutting down the process. Are any of the problems you describe relevant to that, or am I asking too much, or perhaps just going about this incorrectly? Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Sunday, November 15, 2009 6:57 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot On Sun, Nov 15, 2009 at 03:04:49PM -0500, Schwab,Wilhelm K wrote:
I am slowly gathering ideas on how to write an interface to gnuplot. It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs.
Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work. For a single graph, think I see how to make that go.
For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :)
Hi Bill, You should probably first check out the suggestion from Levente:
You don't have to reinvent the wheel all the time: http://www.squeaksource.com/GNUPlot.html
But your question is interesting to me, because the first free software thing I ever wrote was gl_plot, which appeared in comp.sources.unix volume 18, and was used for generating graphics to support gnuplot on PC unix (Microport System V/AT) back before PC unix had graphics support. It's still out there in case you want to run gnuplot on your 286 unix pc: http://ftp.isc.org/pub/usenet/comp.sources.unix/volume18/gl_plot Anyway, back to the new millenium. Yes you should definitely be able to control a gnuplot process with a connected PipeableOSProcess. That's what it is designed for.
Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so?
Sure, see PipeableOSProcess class>>tkExample for an example of how to do this. Use /usr/bin/gnuplot instead of /usr/bin/wish of course. However (and this is a big caveat at the moment), OSProcess and CommandShell currently do not work properly on Pharo for reasons that I don't yet understand. So definitely check out Levente's suggestion as a first step.
I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC??
CommandShell has user interfaces for both Morphic and MVC. Pharo does not have MVC, but that should not effect the Morphic side, other than the annoyance of having to skip over the errors when you load it. Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Hi! On Mon, 16 Nov 2009, Schwab,Wilhelm K wrote:
Dave,
ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix. Please let me know if I am missing something.
Though I'm not Dave, I can tell you that ProcessWrapper is for win32 only. It exists because PipeableOSProcess doesn't work under win32.
It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work. I am having some success with code such as
gp := PipeableOSProcess command:'gnuplot'. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'.
You still don't have to reinvent the wheel. If you don't want to use Igor's project - which you should IMO - you can still check out his code and see how he uses PipeableOSProcess to do the thing you want to do. Levente
On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
Dave,
ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix. Please let me know if I am missing something.
It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work. I am having some success with code such as
gp := PipeableOSProcess command:'gnuplot'. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'.
but I cannot get any output without shutting down the process. Are any of the problems you describe relevant to that, or am I asking too much, or perhaps just going about this incorrectly?
Bill
You may be hanging up trying to read a blocking stream from the gnuplot program. Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image). Your original example, but with #setNonBlockingOutput: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. gp close. "close stdin to gnuplot, like <ctl>D" gp upToEndOfFile inspect. "stdout, the graphics output" gp errorUpToEndOfFile inspect. "stderr output if any" If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data" gp upToEnd inspect. "all available stdout data" gp errorUpToEnd inspect. "all available stderr data" gp close. You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'; exec: 'exit'. "exit gnuplot, similar effect to gp close" gp upToEndOfFile inspect. gp errorUpToEndOfFile inspect. If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ). HTH, Dave
Dave, Thanks for the examples, and the reminder about Squeak/Pharo differences. I'll see what I can figure out. You might have already explained why one of my tests failed. I do want the option to keep gnuplot running. It might turn out that the cases in which that matters are precisely the ones in which gnuplot should be allowed to write the output files, but I'd rather not start out with that approach. Re the problems in Pharo, one perhaps completely worthless point: I ran some of the unit tests with Pharo launched from a terminal, and got Ipv6 related messages. It made no sense to me, but (I think) that code is whacked, and if there is any relationship, it *might* be the problem. Usual caveats about this being a potential red herring apply. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Monday, November 16, 2009 7:40 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
Dave,
ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix. Please let me know if I am missing something.
It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work. I am having some success with code such as
gp := PipeableOSProcess command:'gnuplot'. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'.
but I cannot get any output without shutting down the process. Are any of the problems you describe relevant to that, or am I asking too much, or perhaps just going about this incorrectly?
Bill
You may be hanging up trying to read a blocking stream from the gnuplot program. Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image). Your original example, but with #setNonBlockingOutput: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. gp close. "close stdin to gnuplot, like <ctl>D" gp upToEndOfFile inspect. "stdout, the graphics output" gp errorUpToEndOfFile inspect. "stderr output if any" If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data" gp upToEnd inspect. "all available stdout data" gp errorUpToEnd inspect. "all available stderr data" gp close. You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'; exec: 'exit'. "exit gnuplot, similar effect to gp close" gp upToEndOfFile inspect. gp errorUpToEndOfFile inspect. If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ). HTH, Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Dave, That works - thanks! Knowing how much data to read will be a problem. My assumption was that I would have an image reader pull from the stream, taking the hint when it finds a complete image. Sadly, they do bad things to the stream, and I am not sure how to correct that. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Schwab,Wilhelm K Sent: Monday, November 16, 2009 7:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot Dave, Thanks for the examples, and the reminder about Squeak/Pharo differences. I'll see what I can figure out. You might have already explained why one of my tests failed. I do want the option to keep gnuplot running. It might turn out that the cases in which that matters are precisely the ones in which gnuplot should be allowed to write the output files, but I'd rather not start out with that approach. Re the problems in Pharo, one perhaps completely worthless point: I ran some of the unit tests with Pharo launched from a terminal, and got Ipv6 related messages. It made no sense to me, but (I think) that code is whacked, and if there is any relationship, it *might* be the problem. Usual caveats about this being a potential red herring apply. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Monday, November 16, 2009 7:40 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
Dave,
ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix. Please let me know if I am missing something.
It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work. I am having some success with code such as
gp := PipeableOSProcess command:'gnuplot'. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'.
but I cannot get any output without shutting down the process. Are any of the problems you describe relevant to that, or am I asking too much, or perhaps just going about this incorrectly?
Bill
You may be hanging up trying to read a blocking stream from the gnuplot program. Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image). Your original example, but with #setNonBlockingOutput: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. gp close. "close stdin to gnuplot, like <ctl>D" gp upToEndOfFile inspect. "stdout, the graphics output" gp errorUpToEndOfFile inspect. "stderr output if any" If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data" gp upToEnd inspect. "all available stdout data" gp errorUpToEnd inspect. "all available stderr data" gp close. You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'; exec: 'exit'. "exit gnuplot, similar effect to gp close" gp upToEndOfFile inspect. gp errorUpToEndOfFile inspect. If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ). HTH, Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
The image readers heap a lot of abuse on the streams given to them (reset vs. restoring position, sending #binary, #close, etc.). With PNG, I was able to create a modified reader that uses the IEND chunk to stop the read rather than the end of stream, which is hideous for some uses (such as mine). My "fix" could be applied to the existing code. Any interest? My wrapper around the stdout pipe is still failing me, but I think I see why. With a proper page-reading trick, I will hopefully be able to get it to work, which I define here as having the PNG read stop at the end of the data. Not now though: it's getting too late for this nonsense :) Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Schwab,Wilhelm K Sent: Monday, November 16, 2009 9:28 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot Dave, That works - thanks! Knowing how much data to read will be a problem. My assumption was that I would have an image reader pull from the stream, taking the hint when it finds a complete image. Sadly, they do bad things to the stream, and I am not sure how to correct that. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Schwab,Wilhelm K Sent: Monday, November 16, 2009 7:59 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot Dave, Thanks for the examples, and the reminder about Squeak/Pharo differences. I'll see what I can figure out. You might have already explained why one of my tests failed. I do want the option to keep gnuplot running. It might turn out that the cases in which that matters are precisely the ones in which gnuplot should be allowed to write the output files, but I'd rather not start out with that approach. Re the problems in Pharo, one perhaps completely worthless point: I ran some of the unit tests with Pharo launched from a terminal, and got Ipv6 related messages. It made no sense to me, but (I think) that code is whacked, and if there is any relationship, it *might* be the problem. Usual caveats about this being a potential red herring apply. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Monday, November 16, 2009 7:40 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
Dave,
ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix. Please let me know if I am missing something.
It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work. I am having some success with code such as
gp := PipeableOSProcess command:'gnuplot'. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'.
but I cannot get any output without shutting down the process. Are any of the problems you describe relevant to that, or am I asking too much, or perhaps just going about this incorrectly?
Bill
You may be hanging up trying to read a blocking stream from the gnuplot program. Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image). Your original example, but with #setNonBlockingOutput: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. gp close. "close stdin to gnuplot, like <ctl>D" gp upToEndOfFile inspect. "stdout, the graphics output" gp errorUpToEndOfFile inspect. "stderr output if any" If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'. (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data" gp upToEnd inspect. "all available stdout data" gp errorUpToEnd inspect. "all available stderr data" gp close. You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above: gp := PipeableOSProcess command:'gnuplot'. gp setNonBlockingOutput. gp exec:'set output ""'; exec:'set terminal png'; exec:'plot sin(x)'; exec: 'exit'. "exit gnuplot, similar effect to gp close" gp upToEndOfFile inspect. gp errorUpToEndOfFile inspect. If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ). HTH, Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Hi Dave, On Nov 17, 2009, at 01:40 , David T. Lewis wrote: [...]
If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).
What is the simplest thing to do to reproduce the problem? Do we know if it is on all platforms? Does it work in a Squeak Trunk image? Maybe I or other people can help to get more insights about what goes wrong. OSProcess is certainly a very important package for Pharo, so we should try to get it working again. Thanks, Adrian ___________________ http://www.adrian-lienhard.ch/
2 Bill & others. I started gnuplot to make an abstract layer for its functionality. Its not really matter, how it invokes the OSProcess, so feel free to replace/improve that part of code. And of course, its not for a single OS. There are GPWrapper which should deal with platform-specific crap, while rest of API should simply reflect the gnuplot api in convenient way. It was tested & run on windoze & unix(es). Potentially, it could be split on 2 packages (feel free to do it): - basic classes - platform-specific / OS-specific stuff Also, if i remember, the API is not fully covered, so feel free to add new protocol(s). Repository is writable, so do not hesitate to put your hands on it. -- Best regards, Igor Stasenko AKA sig.
Yes! What are the problems? I read the mail three times but I could not get it. Stef On Nov 17, 2009, at 9:30 AM, Adrian Lienhard wrote:
Hi Dave,
On Nov 17, 2009, at 01:40 , David T. Lewis wrote:
[...]
If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).
What is the simplest thing to do to reproduce the problem?
Do we know if it is on all platforms? Does it work in a Squeak Trunk image?
Maybe I or other people can help to get more insights about what goes wrong. OSProcess is certainly a very important package for Pharo, so we should try to get it working again.
Thanks, Adrian ___________________ http://www.adrian-lienhard.ch/
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Stef, The only problems I have found are in the unit tests, which somewhere along the line cause a vm crash. Otherwise, it seems to work. Until it either proves unstable or otherwise lets me down, I'm trying to press forward with it, since I agree that it is REALLY important that this and other other FFI-like features work and work well. In general, I find the Linux vm to be too willing to crash w/o leaving a trace, or at least a trace that I know how to find(??). Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Stéphane Ducasse Sent: Tuesday, November 17, 2009 8:02 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot Yes! What are the problems? I read the mail three times but I could not get it. Stef On Nov 17, 2009, at 9:30 AM, Adrian Lienhard wrote:
Hi Dave,
On Nov 17, 2009, at 01:40 , David T. Lewis wrote:
[...]
If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).
What is the simplest thing to do to reproduce the problem?
Do we know if it is on all platforms? Does it work in a Squeak Trunk image?
Maybe I or other people can help to get more insights about what goes wrong. OSProcess is certainly a very important package for Pharo, so we should try to get it working again.
Thanks, Adrian ___________________ http://www.adrian-lienhard.ch/
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Tue, Nov 17, 2009 at 03:20:41PM -0500, Schwab,Wilhelm K wrote:
Stef,
The only problems I have found are in the unit tests, which somewhere along the line cause a vm crash. Otherwise, it seems to work. Until it either proves unstable or otherwise lets me down, I'm trying to press forward with it, since I agree that it is REALLY important that this and other other FFI-like features work and work well.
In general, I find the Linux vm to be too willing to crash w/o leaving a trace, or at least a trace that I know how to find(??).
Hi Bill, I am not sure what was causing the crashes (I have seen this too, although not on the latest Pharo that I tried tonight). But I am quite sure that you should not blame the VM, and you should not blame Pharo either. In such a case, the fault lies with the person who wrote the plugin that crashed (that would be me). I expect that the unit tests ended up passing the "wrong" object as a parameter to some primitive in OSProcessPlugin, which in turn led to a crash. Most likely the symptoms (VM crash) will go away whenever we figure out the issues with OSProcess on Pharo, but only correct solution is to write the primitives with enough parameter checking that a VM crash cannot happen. If I see this problem again, I'll try running the VM under a gdb debugger to find out which primitive is causing the crash. That's probably the only way to find it, because the VM cannot prevent a sloppy plugin from doing a segfault and exiting. Dave
Dave, It is nice to see a programmer accept blame for something. Too many hide behind the euphemism du jour ("issue") as if they had nothing to do with it. A long time ago, I read an article that objected to the term "bug," suggesting that it was too easy to pass of as something relatively harmless; the author favored the term "defect." It is even more applicable now. If you want to have fun, get across a table with a programmer who measures his/her worth by listing their MS certifications, and insist on using "defect" every time they pass something off as an "issue." The looks of horror can be quite entertaining. But I digress. Ok, you blew it<g>. Feel better? You're man enough to take the heat (not that we're going to give you any). However, the vm is partially at fault, because (AFICT), it fails to trap crashes. IMHO, the worst thing a program can do is lock up. The next worst thing it can do is disappear w/o leaving a reason for it. So, I *do* blame the vm for not doing more to help us find this. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Tuesday, November 17, 2009 11:09 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot On Tue, Nov 17, 2009 at 03:20:41PM -0500, Schwab,Wilhelm K wrote:
Stef,
The only problems I have found are in the unit tests, which somewhere along the line cause a vm crash. Otherwise, it seems to work. Until it either proves unstable or otherwise lets me down, I'm trying to press forward with it, since I agree that it is REALLY important that this and other other FFI-like features work and work well.
In general, I find the Linux vm to be too willing to crash w/o leaving a trace, or at least a trace that I know how to find(??).
Hi Bill, I am not sure what was causing the crashes (I have seen this too, although not on the latest Pharo that I tried tonight). But I am quite sure that you should not blame the VM, and you should not blame Pharo either. In such a case, the fault lies with the person who wrote the plugin that crashed (that would be me). I expect that the unit tests ended up passing the "wrong" object as a parameter to some primitive in OSProcessPlugin, which in turn led to a crash. Most likely the symptoms (VM crash) will go away whenever we figure out the issues with OSProcess on Pharo, but only correct solution is to write the primitives with enough parameter checking that a VM crash cannot happen. If I see this problem again, I'll try running the VM under a gdb debugger to find out which primitive is causing the crash. That's probably the only way to find it, because the VM cannot prevent a sloppy plugin from doing a segfault and exiting. Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Tue, Nov 17, 2009 at 09:30:16AM +0100, Adrian Lienhard wrote:
Hi Dave,
On Nov 17, 2009, at 01:40 , David T. Lewis wrote:
[...]
If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).
What is the simplest thing to do to reproduce the problem?
Do we know if it is on all platforms? Does it work in a Squeak Trunk image?
Hi Adrian, There are unit tests for OSProcess as well as for CommandShell, and these should all be green. Yes, all tests pass on Squeak trunk (see attached). I get inconsistent results on a Pharo image. I just tried again tonight on pharo1.0-10495-rc1dev09.11.3, and things look better than the last time I tried (which might be coincidental, I'm running various VMs and maybe I got myself confused the other evening). I'm not sure, but I think I am seeing two types of problem. The easy one is that I'm getting test failures in AioEventHandlerTestCase that are probably caused by changes in the socket code for Pharo compared to Squeak. I have not found the cause yet, but I'm not worried about it because it's probably just a change in the socket code for Pharo, and I probably will just need to adjust the tests for this. The harder problem is that I am getting intermittent failures, most often showing up in CommandShellTestCase (which does a lot of pipes, event handling, and OS process handling). I think that I am seeing failures in reading from pipes, which in turn results in failure to find end of file and clean up exiting processes. This causes the tests to get "stuck", and if I look in a terminal window, I can see that many of the processes spawned by the unit tests are in zombie mode, indicating that the grimReaperProcess running in Pharo/Squeak is failing to clean up processes that are ready to exit. I think that when I ran this on earlier Pharo images I was getting errors that had something to do with the fileID of a file stream (sorry I did not save the specific errors), so I am wondering if anything may have changed that would affect access to the fileID handle for a file stream. I have also seen VM crashes when running the tests, which suggests possibly passing the wrong kind of object to a primitive that expects a fileID byte array (and of course it also means that I need to do a better job of validating parameters passed to primitives in OSPP, but that's a separate topic ;)
Maybe I or other people can help to get more insights about what goes wrong. OSProcess is certainly a very important package for Pharo, so we should try to get it working again.
Thanks for offering to help. I suspect that this will turn out to be one or two simple problems, and I just have not found them yet. Dave
Hi Dave, Just to make sure I'm looking at the right code: The latest version (for Pharo) are in the projects OSProcess and CommandShell on SqueakSource.com? MC complains about missing classes when loading; I just ignored them (probably they are not relevant as they seem related to the GUI). Running all tests crashes the Mac VM. But even the simple test of chdir fails: (UnixProcessAccessorTestCase selector: #testChDir) debug It does not seem to invoke any Pharo-specific code. So I wonder if I miss something trivial. UnixOSProcessPlugin.bundle exists in my VM's Contents/Resources/ directory. Adrian On Nov 18, 2009, at 04:55 , David T. Lewis wrote:
On Tue, Nov 17, 2009 at 09:30:16AM +0100, Adrian Lienhard wrote:
Hi Dave,
On Nov 17, 2009, at 01:40 , David T. Lewis wrote:
[...]
If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).
What is the simplest thing to do to reproduce the problem?
Do we know if it is on all platforms? Does it work in a Squeak Trunk image?
Hi Adrian,
There are unit tests for OSProcess as well as for CommandShell, and these should all be green. Yes, all tests pass on Squeak trunk (see attached).
I get inconsistent results on a Pharo image. I just tried again tonight on pharo1.0-10495-rc1dev09.11.3, and things look better than the last time I tried (which might be coincidental, I'm running various VMs and maybe I got myself confused the other evening).
I'm not sure, but I think I am seeing two types of problem. The easy one is that I'm getting test failures in AioEventHandlerTestCase that are probably caused by changes in the socket code for Pharo compared to Squeak. I have not found the cause yet, but I'm not worried about it because it's probably just a change in the socket code for Pharo, and I probably will just need to adjust the tests for this.
The harder problem is that I am getting intermittent failures, most often showing up in CommandShellTestCase (which does a lot of pipes, event handling, and OS process handling). I think that I am seeing failures in reading from pipes, which in turn results in failure to find end of file and clean up exiting processes. This causes the tests to get "stuck", and if I look in a terminal window, I can see that many of the processes spawned by the unit tests are in zombie mode, indicating that the grimReaperProcess running in Pharo/Squeak is failing to clean up processes that are ready to exit.
I think that when I ran this on earlier Pharo images I was getting errors that had something to do with the fileID of a file stream (sorry I did not save the specific errors), so I am wondering if anything may have changed that would affect access to the fileID handle for a file stream. I have also seen VM crashes when running the tests, which suggests possibly passing the wrong kind of object to a primitive that expects a fileID byte array (and of course it also means that I need to do a better job of validating parameters passed to primitives in OSPP, but that's a separate topic ;)
Maybe I or other people can help to get more insights about what goes wrong. OSProcess is certainly a very important package for Pharo, so we should try to get it working again.
Thanks for offering to help. I suspect that this will turn out to be one or two simple problems, and I just have not found them yet.
Dave
<OSP-Tests-Squeak.png>_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Wed, Nov 18, 2009 at 09:51:24AM +0100, Adrian Lienhard wrote:
Hi Dave,
Just to make sure I'm looking at the right code: The latest version (for Pharo) are in the projects OSProcess and CommandShell on SqueakSource.com?
Yes, these are the correct versions to use.
MC complains about missing classes when loading; I just ignored them (probably they are not relevant as they seem related to the GUI).
These are the MVC user interface classes for CommandShell, and it is safe to ignore them. The tests do not required these classes.
Running all tests crashes the Mac VM.
I have seen crashes on my Linux VM also (testing with an earlier Pharo image), although it was not happening yesterday when I was testing. Presumably there is inadequate parameter checking in one or more of the OSPP primitives, and whatever is going wrong in the tests has exposed this.
But even the simple test of chdir fails:
(UnixProcessAccessorTestCase selector: #testChDir) debug
It does not seem to invoke any Pharo-specific code. So I wonder if I miss something trivial. UnixOSProcessPlugin.bundle exists in my VM's Contents/Resources/ directory.
I do not have a Mac to test with. I would think that the fact that you have gotten the VM to crash is an indication that you have located the plugin bundle though ;) Just to double check, try doing this in a workspace: OSProcess accessor osppModuleVersionString ==> '4.3.3' The #osppModuleVersionString method will make a call to the plugin without doing much else, so that is a way to verify that you have loaded the bundle. Also, if the version is not fairly close to the latest (4.3.3) it might be an issue. On OS X, I don't think that you have the AioPlugin or XDisplayControlPlugin available, so some of the tests will fail as a result of this. I don't have time right now, but I'll try removing these plugins from my system and see what happens. But it should be a few simple test failures, not the kind of problem you are seeing. (As a side note, I use a Linux PC, and John and I originally got OSProcess working on Mac entirely through email with a bit of help from a couple other Mac users. Talk about a lengthy edit-compile-test-debug development cycle! But it was a good challenge and quite gratifying when we got it all working.). Thanks, Dave
Dave, It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation. Bill -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Wednesday, November 18, 2009 8:34 AM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - working on gnuplot On Wed, Nov 18, 2009 at 09:51:24AM +0100, Adrian Lienhard wrote:
Hi Dave,
Just to make sure I'm looking at the right code: The latest version (for Pharo) are in the projects OSProcess and CommandShell on SqueakSource.com?
Yes, these are the correct versions to use.
MC complains about missing classes when loading; I just ignored them (probably they are not relevant as they seem related to the GUI).
These are the MVC user interface classes for CommandShell, and it is safe to ignore them. The tests do not required these classes.
Running all tests crashes the Mac VM.
I have seen crashes on my Linux VM also (testing with an earlier Pharo image), although it was not happening yesterday when I was testing. Presumably there is inadequate parameter checking in one or more of the OSPP primitives, and whatever is going wrong in the tests has exposed this.
But even the simple test of chdir fails:
(UnixProcessAccessorTestCase selector: #testChDir) debug
It does not seem to invoke any Pharo-specific code. So I wonder if I miss something trivial. UnixOSProcessPlugin.bundle exists in my VM's Contents/Resources/ directory.
I do not have a Mac to test with. I would think that the fact that you have gotten the VM to crash is an indication that you have located the plugin bundle though ;) Just to double check, try doing this in a workspace: OSProcess accessor osppModuleVersionString ==> '4.3.3' The #osppModuleVersionString method will make a call to the plugin without doing much else, so that is a way to verify that you have loaded the bundle. Also, if the version is not fairly close to the latest (4.3.3) it might be an issue. On OS X, I don't think that you have the AioPlugin or XDisplayControlPlugin available, so some of the tests will fail as a result of this. I don't have time right now, but I'll try removing these plugins from my system and see what happens. But it should be a few simple test failures, not the kind of problem you are seeing. (As a side note, I use a Linux PC, and John and I originally got OSProcess working on Mac entirely through email with a bit of help from a couple other Mac users. Talk about a lengthy edit-compile-test-debug development cycle! But it was a good challenge and quite gratifying when we got it all working.). Thanks, Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Wed, Nov 18, 2009 at 08:55:50AM -0500, Schwab,Wilhelm K wrote:
Dave,
It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation.
Bill, Good idea, thanks. I'll take a look at that when I get home. Dave
On Wed, Nov 18, 2009 at 05:14:04PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 08:55:50AM -0500, Schwab,Wilhelm K wrote:
Dave,
It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation.
Bill,
Good idea, thanks. I'll take a look at that when I get home.
Well actually there were not any MVC references in #initialize. But CommandShell class>>initialize opens a shell window, and that was failing because text style 'Atlanta' no longer exists in Pharo. I fixed this, so if you update to CommandShell-dtl.40.mcz that failure will no longer occur. The shell window does not actually work, but at least the window opens properly now. One bug down, ??? to go ... Thanks, Dave
On Wed, Nov 18, 2009 at 09:44:08PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 05:14:04PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 08:55:50AM -0500, Schwab,Wilhelm K wrote:
Dave,
It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation.
Bill,
Good idea, thanks. I'll take a look at that when I get home.
Well actually there were not any MVC references in #initialize. But CommandShell class>>initialize opens a shell window, and that was failing because text style 'Atlanta' no longer exists in Pharo. I fixed this, so if you update to CommandShell-dtl.40.mcz that failure will no longer occur. The shell window does not actually work, but at least the window opens properly now. One bug down, ??? to go ...
Correction, the shell window *does* seem to be working, so this is actually looking much better now. However, the CommandShellTestCase does not pass (hangs up part way through, lots of zombie unix processes left unhandled). The other unit tests are passing, with the exception of the tests in AioEventHandlerTestCase that use sockets. This is due to changes in the socket support in Pharo, but I am not worried about these failures as they probably just require updates to the tests to match the Pharo socket support. I am testing on Linux with a VM that I built locally. For OS X, an updated OSProcessPlugin will need to be included with the VM, otherwise we will have VM crashes when handling external process exit. The main concerns now are: 1) Need an updated OSProcessPlugin for the Mac VM (for both Squeak and Pharo) 2) Why does CommandShellTestCase on Pharo get hung up and leave zombies? Note, CommandShellTestCase is a real stress test for OSProcess. It does a lot of concurrent process handling with pipelines of OS processes interacting with Squeak/Pharo. If this test does not pass, then I am not confident in the overall reliability of OSProcess. Dave
I took another try using a unix VM (3.11.3.2135 compiled from source) on OS X 10.5.8. It has the plugin version 4.3.3. Looks much better than with the Mac VM that has the outdated plugin! 409 run, 389 passes, 0 expected failures, 20 failures - The VM does not crash anymore - There are a couple of errors that go away if I change #useOldNetwork to return true - The following tests seem to fail due to assumptions about files and permissions that do not hold on OS X: ShellSyntaxTestCase>>#testExpandArgument03 ShellSyntaxTestCase>>#testExpandArgumentFrom01 UnixProcessAccessorTestCase>>#testIsExecutableForUserInGroup UnixProcessAccessorTestCase>>#testIsReadableForUserInGroup UnixProcessAccessorTestCase>>#testIsWritableForUserInGroup UnixProcessAccessorTestCase>>#testChDir - one or two tests fail sometimes, but not always. For example: UnixProcessWin32FileLockingTestCase>>#testCooperatingProcesses04 - The following tests, which fork the VM, do not work: PipeableOSProcessTestCase>>#testForkHeadlessSqueak PipeableOSProcessTestCase>>#testForkHeadlessSqueak2 PipeableOSProcessTestCase>>#testForkSqueak UnixProcessTestCase>>#testClassForkHeadlessSqueakAndDo UnixProcessTestCase>>#testClassForkHeadlessSqueakAndDoThenQuit UnixProcessTestCase>>#testClassForkSqueak UnixProcessTestCase>>#testClassForkSqueakAndDo UnixProcessTestCase>>#testClassForkSqueakAndDoThenQuit UnixProcessTestCase>>#testForkHeadlessSqueakAndDo UnixProcessTestCase>>#testForkHeadlessSqueakAndDoThenQuit UnixProcessTestCase>>#testForkSqueak UnixProcessTestCase>>#testForkSqueakAndDo UnixProcessTestCase>>#testForkSqueakAndDoThenQuit UnixProcessTestCase>>#testHeadlessChild For example running #testHeadlessChild I get to stdout: hello world from child process 1359 Segmentation fault 390493624 UnixProcess>forkHeadlessSqueakAndDoThenQuit: 390493524 >forkHeadlessSqueakAndDoThenQuit: 390493400 >headlessChild 390493264 UnixProcessTestCase>testHeadlessChild 390493172 TestCase>executeShould:inScopeOf: 390493080 BlockClosure>on:do: 390492988 TestCase>executeShould:inScopeOf: 390492876 TestCase>shouldnt:raise: 390491136 UnixProcessTestCase>testHeadlessChild 390491044 TestCase>performTest 390490952 TestCase>runCase If I run #testForSqueak I get The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. Segmentation fault 390953340 DisplayScreen>forceToScreen: 390953248 DisplayScreen>forceDamageToScreen: 390953124 OrderedCollection>do: 390952960 DisplayScreen>forceDamageToScreen: 390952840 WorldState>forceDamageToScreen: 390869724 WorldState>displayWorld:submorphs: 390869632 PasteUpMorph>privateOuterDisplayWorld 390869540 PasteUpMorph>displayWorld 390869448 WorldState>displayWorldSafely: 390869356 BlockClosure>on:do: 390869244 BlockClosure>ifError: 390869096 WorldState>displayWorldSafely: From these tests it seems that the critical parts of OSProcess are actually working with this configuration (modulo the forking). Adrian On Nov 19, 2009, at 06:03 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 09:44:08PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 05:14:04PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 08:55:50AM -0500, Schwab,Wilhelm K wrote:
Dave,
It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation.
Bill,
Good idea, thanks. I'll take a look at that when I get home.
Well actually there were not any MVC references in #initialize. But CommandShell class>>initialize opens a shell window, and that was failing because text style 'Atlanta' no longer exists in Pharo. I fixed this, so if you update to CommandShell-dtl.40.mcz that failure will no longer occur. The shell window does not actually work, but at least the window opens properly now. One bug down, ??? to go ...
Correction, the shell window *does* seem to be working, so this is actually looking much better now. However, the CommandShellTestCase does not pass (hangs up part way through, lots of zombie unix processes left unhandled).
The other unit tests are passing, with the exception of the tests in AioEventHandlerTestCase that use sockets. This is due to changes in the socket support in Pharo, but I am not worried about these failures as they probably just require updates to the tests to match the Pharo socket support.
I am testing on Linux with a VM that I built locally. For OS X, an updated OSProcessPlugin will need to be included with the VM, otherwise we will have VM crashes when handling external process exit.
The main concerns now are:
1) Need an updated OSProcessPlugin for the Mac VM (for both Squeak and Pharo)
2) Why does CommandShellTestCase on Pharo get hung up and leave zombies?
Note, CommandShellTestCase is a real stress test for OSProcess. It does a lot of concurrent process handling with pipelines of OS processes interacting with Squeak/Pharo. If this test does not pass, then I am not confident in the overall reliability of OSProcess.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Adrian, Thanks *very* much for the detailed feedback. I do want to ask a couple more questions if your patience holds up. First question: I am getting serious failures on CommandShellTestCase that still worry me. For example, if I evaluate this (on Linux VM with Pharo image): (CommandShellTestCase selector: #testEvaluateOrMakePipelinesFrom) debug I find that the test never completes. It is stuck waiting for the various external processes to be cleaned up, and the console output shows the zombie processes whose exit signals have not yet been handled by the image: lewis@linux-6xfc:~/squeak/Pharo/pharo1.0-10495-rc1dev09.11.3> ps PID TTY TIME CMD 3877 pts/51 00:00:00 bash 9695 pts/51 00:00:00 xterm 10060 pts/51 00:00:45 squeakvm 10268 pts/51 00:00:00 ls <defunct> 10269 pts/51 00:00:00 cat <defunct> 10271 pts/51 00:00:00 cat <defunct> 10272 pts/51 00:00:00 wc <defunct> 10273 pts/51 00:00:00 ps <defunct> 10274 pts/51 00:00:00 cat <defunct> 10275 pts/51 00:00:00 cat <defunct> 10279 pts/51 00:00:00 ps lewis@linux-6xfc:~/squeak/Pharo/pharo1.0-10495-rc1dev09.11.3> Do you see problems like this when you run CommandShellTestCase? Second question: Some of the failures you are getting look like things that I thought that I had fixed in the past (but I'm not sure). The latest versions from the SqueakSource repositories should be: OSProcess-dtl.53 Tests-OSProcess-dtl.20 CommandShell-dtl.40 Tests-CommandShell-dtl.14 Could you please double check your versions of OSProcess, CommandShell and the unit tests to see if they match these MC versions? Thanks a lot, I really appreciate the help. And obviously I am going to have to go buy a Mac one of these days so I will not have to ask all these dumb questions ;) Dave On Thu, Nov 19, 2009 at 10:05:26PM +0100, Adrian Lienhard wrote:
I took another try using a unix VM (3.11.3.2135 compiled from source) on OS X 10.5.8. It has the plugin version 4.3.3. Looks much better than with the Mac VM that has the outdated plugin!
409 run, 389 passes, 0 expected failures, 20 failures
- The VM does not crash anymore - There are a couple of errors that go away if I change #useOldNetwork to return true
- The following tests seem to fail due to assumptions about files and permissions that do not hold on OS X: ShellSyntaxTestCase>>#testExpandArgument03 ShellSyntaxTestCase>>#testExpandArgumentFrom01 UnixProcessAccessorTestCase>>#testIsExecutableForUserInGroup UnixProcessAccessorTestCase>>#testIsReadableForUserInGroup UnixProcessAccessorTestCase>>#testIsWritableForUserInGroup UnixProcessAccessorTestCase>>#testChDir
- one or two tests fail sometimes, but not always. For example: UnixProcessWin32FileLockingTestCase>>#testCooperatingProcesses04
- The following tests, which fork the VM, do not work: PipeableOSProcessTestCase>>#testForkHeadlessSqueak PipeableOSProcessTestCase>>#testForkHeadlessSqueak2 PipeableOSProcessTestCase>>#testForkSqueak UnixProcessTestCase>>#testClassForkHeadlessSqueakAndDo UnixProcessTestCase>>#testClassForkHeadlessSqueakAndDoThenQuit UnixProcessTestCase>>#testClassForkSqueak UnixProcessTestCase>>#testClassForkSqueakAndDo UnixProcessTestCase>>#testClassForkSqueakAndDoThenQuit UnixProcessTestCase>>#testForkHeadlessSqueakAndDo UnixProcessTestCase>>#testForkHeadlessSqueakAndDoThenQuit UnixProcessTestCase>>#testForkSqueak UnixProcessTestCase>>#testForkSqueakAndDo UnixProcessTestCase>>#testForkSqueakAndDoThenQuit UnixProcessTestCase>>#testHeadlessChild
For example running #testHeadlessChild I get to stdout:
hello world from child process 1359
Segmentation fault
390493624 UnixProcess>forkHeadlessSqueakAndDoThenQuit: 390493524 >forkHeadlessSqueakAndDoThenQuit: 390493400 >headlessChild 390493264 UnixProcessTestCase>testHeadlessChild 390493172 TestCase>executeShould:inScopeOf: 390493080 BlockClosure>on:do: 390492988 TestCase>executeShould:inScopeOf: 390492876 TestCase>shouldnt:raise: 390491136 UnixProcessTestCase>testHeadlessChild 390491044 TestCase>performTest 390490952 TestCase>runCase
If I run #testForSqueak I get
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug.
Segmentation fault
390953340 DisplayScreen>forceToScreen: 390953248 DisplayScreen>forceDamageToScreen: 390953124 OrderedCollection>do: 390952960 DisplayScreen>forceDamageToScreen: 390952840 WorldState>forceDamageToScreen: 390869724 WorldState>displayWorld:submorphs: 390869632 PasteUpMorph>privateOuterDisplayWorld 390869540 PasteUpMorph>displayWorld 390869448 WorldState>displayWorldSafely: 390869356 BlockClosure>on:do: 390869244 BlockClosure>ifError: 390869096 WorldState>displayWorldSafely:
From these tests it seems that the critical parts of OSProcess are actually working with this configuration (modulo the forking).
Adrian
On Nov 19, 2009, at 06:03 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 09:44:08PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 05:14:04PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 08:55:50AM -0500, Schwab,Wilhelm K wrote:
Dave,
It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation.
Bill,
Good idea, thanks. I'll take a look at that when I get home.
Well actually there were not any MVC references in #initialize. But CommandShell class>>initialize opens a shell window, and that was failing because text style 'Atlanta' no longer exists in Pharo. I fixed this, so if you update to CommandShell-dtl.40.mcz that failure will no longer occur. The shell window does not actually work, but at least the window opens properly now. One bug down, ??? to go ...
Correction, the shell window *does* seem to be working, so this is actually looking much better now. However, the CommandShellTestCase does not pass (hangs up part way through, lots of zombie unix processes left unhandled).
The other unit tests are passing, with the exception of the tests in AioEventHandlerTestCase that use sockets. This is due to changes in the socket support in Pharo, but I am not worried about these failures as they probably just require updates to the tests to match the Pharo socket support.
I am testing on Linux with a VM that I built locally. For OS X, an updated OSProcessPlugin will need to be included with the VM, otherwise we will have VM crashes when handling external process exit.
The main concerns now are:
1) Need an updated OSProcessPlugin for the Mac VM (for both Squeak and Pharo)
2) Why does CommandShellTestCase on Pharo get hung up and leave zombies?
Note, CommandShellTestCase is a real stress test for OSProcess. It does a lot of concurrent process handling with pipelines of OS processes interacting with Squeak/Pharo. If this test does not pass, then I am not confident in the overall reliability of OSProcess.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Hi Dave, On Nov 20, 2009, at 02:42 , David T. Lewis wrote: [...]
I am getting serious failures on CommandShellTestCase that still worry me. For example, if I evaluate this (on Linux VM with Pharo image):
(CommandShellTestCase selector: #testEvaluateOrMakePipelinesFrom) debug
Indeed, CommandShellTestCase isn't that stable as I first thought. I've run all tests of CommandShellTestCase in the Test runner multiple times: 1. run: all green 2. run: error in testPipeline75 (re-running this test succeeded, so I don't know what the error was) 3. run: test runner hangs in testIfThenElse02 because indefinitely waiting on completionSemaphore in #evaluatePipelines: restarted the VM 1. run: all green 2. run: same as 3 above 3. run: same as 3 above 4. run: error in testPipeline72 5. run: all green I haven't seen any defunct processes, though. [...]
Second question:
Some of the failures you are getting look like things that I thought that I had fixed in the past (but I'm not sure). The latest versions from the SqueakSource repositories should be:
OSProcess-dtl.53 Tests-OSProcess-dtl.20 CommandShell-dtl.40 Tests-CommandShell-dtl.14
I have the same versions except for CommandShell-dtl.39 (I updated to 40 now). Let me know if I can be of any more help. Adrian
Could you please double check your versions of OSProcess, CommandShell and the unit tests to see if they match these MC versions?
Thanks a lot, I really appreciate the help. And obviously I am going to have to go buy a Mac one of these days so I will not have to ask all these dumb questions ;)
Dave
On Thu, Nov 19, 2009 at 10:05:26PM +0100, Adrian Lienhard wrote:
I took another try using a unix VM (3.11.3.2135 compiled from source) on OS X 10.5.8. It has the plugin version 4.3.3. Looks much better than with the Mac VM that has the outdated plugin!
409 run, 389 passes, 0 expected failures, 20 failures
- The VM does not crash anymore - There are a couple of errors that go away if I change #useOldNetwork to return true
- The following tests seem to fail due to assumptions about files and permissions that do not hold on OS X: ShellSyntaxTestCase>>#testExpandArgument03 ShellSyntaxTestCase>>#testExpandArgumentFrom01 UnixProcessAccessorTestCase>>#testIsExecutableForUserInGroup UnixProcessAccessorTestCase>>#testIsReadableForUserInGroup UnixProcessAccessorTestCase>>#testIsWritableForUserInGroup UnixProcessAccessorTestCase>>#testChDir
- one or two tests fail sometimes, but not always. For example: UnixProcessWin32FileLockingTestCase>>#testCooperatingProcesses04
- The following tests, which fork the VM, do not work: PipeableOSProcessTestCase>>#testForkHeadlessSqueak PipeableOSProcessTestCase>>#testForkHeadlessSqueak2 PipeableOSProcessTestCase>>#testForkSqueak UnixProcessTestCase>>#testClassForkHeadlessSqueakAndDo UnixProcessTestCase>>#testClassForkHeadlessSqueakAndDoThenQuit UnixProcessTestCase>>#testClassForkSqueak UnixProcessTestCase>>#testClassForkSqueakAndDo UnixProcessTestCase>>#testClassForkSqueakAndDoThenQuit UnixProcessTestCase>>#testForkHeadlessSqueakAndDo UnixProcessTestCase>>#testForkHeadlessSqueakAndDoThenQuit UnixProcessTestCase>>#testForkSqueak UnixProcessTestCase>>#testForkSqueakAndDo UnixProcessTestCase>>#testForkSqueakAndDoThenQuit UnixProcessTestCase>>#testHeadlessChild
For example running #testHeadlessChild I get to stdout:
hello world from child process 1359
Segmentation fault
390493624 UnixProcess>forkHeadlessSqueakAndDoThenQuit: 390493524 >forkHeadlessSqueakAndDoThenQuit: 390493400 >headlessChild 390493264 UnixProcessTestCase>testHeadlessChild 390493172 TestCase>executeShould:inScopeOf: 390493080 BlockClosure>on:do: 390492988 TestCase>executeShould:inScopeOf: 390492876 TestCase>shouldnt:raise: 390491136 UnixProcessTestCase>testHeadlessChild 390491044 TestCase>performTest 390490952 TestCase>runCase
If I run #testForSqueak I get
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug. The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ () to debug.
Segmentation fault
390953340 DisplayScreen>forceToScreen: 390953248 DisplayScreen>forceDamageToScreen: 390953124 OrderedCollection>do: 390952960 DisplayScreen>forceDamageToScreen: 390952840 WorldState>forceDamageToScreen: 390869724 WorldState>displayWorld:submorphs: 390869632 PasteUpMorph>privateOuterDisplayWorld 390869540 PasteUpMorph>displayWorld 390869448 WorldState>displayWorldSafely: 390869356 BlockClosure>on:do: 390869244 BlockClosure>ifError: 390869096 WorldState>displayWorldSafely:
From these tests it seems that the critical parts of OSProcess are actually working with this configuration (modulo the forking).
Adrian
On Nov 19, 2009, at 06:03 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 09:44:08PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 05:14:04PM -0500, David T. Lewis wrote:
On Wed, Nov 18, 2009 at 08:55:50AM -0500, Schwab,Wilhelm K wrote:
Dave,
It would help to knock the MVC workspace out of the offending #initialize method and re-save the packages. In my experience, that simple change make the load hassle-free vs. the current situation.
Bill,
Good idea, thanks. I'll take a look at that when I get home.
Well actually there were not any MVC references in #initialize. But CommandShell class>>initialize opens a shell window, and that was failing because text style 'Atlanta' no longer exists in Pharo. I fixed this, so if you update to CommandShell-dtl.40.mcz that failure will no longer occur. The shell window does not actually work, but at least the window opens properly now. One bug down, ??? to go ...
Correction, the shell window *does* seem to be working, so this is actually looking much better now. However, the CommandShellTestCase does not pass (hangs up part way through, lots of zombie unix processes left unhandled).
The other unit tests are passing, with the exception of the tests in AioEventHandlerTestCase that use sockets. This is due to changes in the socket support in Pharo, but I am not worried about these failures as they probably just require updates to the tests to match the Pharo socket support.
I am testing on Linux with a VM that I built locally. For OS X, an updated OSProcessPlugin will need to be included with the VM, otherwise we will have VM crashes when handling external process exit.
The main concerns now are:
1) Need an updated OSProcessPlugin for the Mac VM (for both Squeak and Pharo)
2) Why does CommandShellTestCase on Pharo get hung up and leave zombies?
Note, CommandShellTestCase is a real stress test for OSProcess. It does a lot of concurrent process handling with pipelines of OS processes interacting with Squeak/Pharo. If this test does not pass, then I am not confident in the overall reliability of OSProcess.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Fri, Nov 20, 2009 at 09:45:48AM +0100, Adrian Lienhard wrote:
Hi Dave,
On Nov 20, 2009, at 02:42 , David T. Lewis wrote:
[...]
I am getting serious failures on CommandShellTestCase that still worry me. For example, if I evaluate this (on Linux VM with Pharo image):
(CommandShellTestCase selector: #testEvaluateOrMakePipelinesFrom) debug
Indeed, CommandShellTestCase isn't that stable as I first thought. I've run all tests of CommandShellTestCase in the Test runner multiple times:
1. run: all green 2. run: error in testPipeline75 (re-running this test succeeded, so I don't know what the error was) 3. run: test runner hangs in testIfThenElse02 because indefinitely waiting on completionSemaphore in #evaluatePipelines:
restarted the VM 1. run: all green 2. run: same as 3 above 3. run: same as 3 above 4. run: error in testPipeline72 5. run: all green
I haven't seen any defunct processes, though.
[...]
Second question:
Some of the failures you are getting look like things that I thought that I had fixed in the past (but I'm not sure). The latest versions from the SqueakSource repositories should be:
OSProcess-dtl.53 Tests-OSProcess-dtl.20 CommandShell-dtl.40 Tests-CommandShell-dtl.14
I have the same versions except for CommandShell-dtl.39 (I updated to 40 now).
Let me know if I can be of any more help.
Hi Adrian, Overall, it does look like the basic OSProcess functions that most people would use are working as long as an up to date plugin is available. The remaining issue that is relevant to the Pharo image (as opposed to OSProcess running on any other image) is this CommandShellTestCase issue. I think we are seeing different, but presumably related, symptoms of some underlying problem. The tests in CommandShellTestCase do some rather complex things with lots of processes, semaphore synchronization and so on. So I think I will need to somehow whittle the problem down to some simpler test that can provide some more reproduceable symptoms. My working assumption is that one of the following is true: 1) There is something different in Pharo versus Squeak that is exposing an existing bug in OSProcess/CommandShell. This is quite likely. 2) The CommandShellTestCase is exposing an issue in Pharo related to Semaphore handling, Delay scheduling, or Process scheduling. The seems less likely than #1, but still possible because these tests do drive a lot of interacting processes, and might expose an issue the "normal" users would never see. One thing that I know could produce the sort of symptoms that I am seeing would be anything that intermittently permitted a signal to a Semaphore to be missed. This type of problem causes OSProcess to "miss" the exit of one of its external processes, leading to zombie Unix processes and tests in Pharo/Squeak that fail to run to completion. For that reason, if anyone can think of any changes to process scheduling and semaphore handling, that might give me an idea of where to look. Thanks for all the help, Dave
david thanks for chasing this difference. I would love to be able to help but I'm concurrently bad :). Stef
Hi Adrian,
Overall, it does look like the basic OSProcess functions that most people would use are working as long as an up to date plugin is available.
The remaining issue that is relevant to the Pharo image (as opposed to OSProcess running on any other image) is this CommandShellTestCase issue. I think we are seeing different, but presumably related, symptoms of some underlying problem. The tests in CommandShellTestCase do some rather complex things with lots of processes, semaphore synchronization and so on. So I think I will need to somehow whittle the problem down to some simpler test that can provide some more reproduceable symptoms.
My working assumption is that one of the following is true:
1) There is something different in Pharo versus Squeak that is exposing an existing bug in OSProcess/CommandShell. This is quite likely.
2) The CommandShellTestCase is exposing an issue in Pharo related to Semaphore handling, Delay scheduling, or Process scheduling. The seems less likely than #1, but still possible because these tests do drive a lot of interacting processes, and might expose an issue the "normal" users would never see.
One thing that I know could produce the sort of symptoms that I am seeing would be anything that intermittently permitted a signal to a Semaphore to be missed. This type of problem causes OSProcess to "miss" the exit of one of its external processes, leading to zombie Unix processes and tests in Pharo/Squeak that fail to run to completion.
For that reason, if anyone can think of any changes to process scheduling and semaphore handling, that might give me an idea of where to look.
Thanks for all the help, Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Hi Dave, [...]
The remaining issue that is relevant to the Pharo image (as opposed to OSProcess running on any other image) is this CommandShellTestCase issue.
I'm not sure about this anymore. I just ran CommandShellTestCase a couple of times in a Squeak trunk image and the third time it also blocked on completionSemaphore. Do you see different problems on Linux?
I think we are seeing different, but presumably related, symptoms of some underlying problem. The tests in CommandShellTestCase do some rather complex things with lots of processes, semaphore synchronization and so on. So I think I will need to somehow whittle the problem down to some simpler test that can provide some more reproduceable symptoms.
That would certainly be a good next step! Cheers, Adrian [...]
On Sat, Nov 21, 2009 at 12:31:06PM +0100, Adrian Lienhard wrote:
Hi Dave,
[...]
The remaining issue that is relevant to the Pharo image (as opposed to OSProcess running on any other image) is this CommandShellTestCase issue.
I'm not sure about this anymore. I just ran CommandShellTestCase a couple of times in a Squeak trunk image and the third time it also blocked on completionSemaphore. Do you see different problems on Linux?
Yes I think that it must be behaving differently on Linux. On the combination of Linux + Pharo, I get bad failures on every test run.
I think we are seeing different, but presumably related, symptoms of some underlying problem. The tests in CommandShellTestCase do some rather complex things with lots of processes, semaphore synchronization and so on. So I think I will need to somehow whittle the problem down to some simpler test that can provide some more reproduceable symptoms.
That would certainly be a good next step!
Yes, I think so. I will see if I can narrow it down better. Thanks again, Dave
Hi Dave, OSProcess accessor osppModuleVersionString ==> '4.0.1' I get the same result with the Mac 4.2.2beta1U VM. Adrian On Nov 18, 2009, at 14:33 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 09:51:24AM +0100, Adrian Lienhard wrote:
Hi Dave,
Just to make sure I'm looking at the right code: The latest version (for Pharo) are in the projects OSProcess and CommandShell on SqueakSource.com?
Yes, these are the correct versions to use.
MC complains about missing classes when loading; I just ignored them (probably they are not relevant as they seem related to the GUI).
These are the MVC user interface classes for CommandShell, and it is safe to ignore them. The tests do not required these classes.
Running all tests crashes the Mac VM.
I have seen crashes on my Linux VM also (testing with an earlier Pharo image), although it was not happening yesterday when I was testing. Presumably there is inadequate parameter checking in one or more of the OSPP primitives, and whatever is going wrong in the tests has exposed this.
But even the simple test of chdir fails:
(UnixProcessAccessorTestCase selector: #testChDir) debug
It does not seem to invoke any Pharo-specific code. So I wonder if I miss something trivial. UnixOSProcessPlugin.bundle exists in my VM's Contents/Resources/ directory.
I do not have a Mac to test with. I would think that the fact that you have gotten the VM to crash is an indication that you have located the plugin bundle though ;) Just to double check, try doing this in a workspace:
OSProcess accessor osppModuleVersionString ==> '4.3.3'
The #osppModuleVersionString method will make a call to the plugin without doing much else, so that is a way to verify that you have loaded the bundle. Also, if the version is not fairly close to the latest (4.3.3) it might be an issue.
On OS X, I don't think that you have the AioPlugin or XDisplayControlPlugin available, so some of the tests will fail as a result of this. I don't have time right now, but I'll try removing these plugins from my system and see what happens. But it should be a few simple test failures, not the kind of problem you are seeing.
(As a side note, I use a Linux PC, and John and I originally got OSProcess working on Mac entirely through email with a bit of help from a couple other Mac users. Talk about a lengthy edit-compile-test-debug development cycle! But it was a good challenge and quite gratifying when we got it all working.).
Thanks, Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Wed, Nov 18, 2009 at 02:56:56PM +0100, Adrian Lienhard wrote:
Hi Dave,
OSProcess accessor osppModuleVersionString ==> '4.0.1'
I get the same result with the Mac 4.2.2beta1U VM.
I think that's from about November 2005, so a bit out of date. The source is on SqueakSource OSProcessPlugin project if you want to see what has changed since then. Nevertheless it should still be good enough to get most of the tests to pass, so I would not worry about the version difference for now. Dave
OK, I'll stick to this VM version. I compared to the Squeak trunk image and the very same test cases crash the VM. So at least the crashing problem (on Mac) is not likely caused by a Pharo-specific change. Adrian On Nov 18, 2009, at 16:18 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 02:56:56PM +0100, Adrian Lienhard wrote:
Hi Dave,
OSProcess accessor osppModuleVersionString ==> '4.0.1'
I get the same result with the Mac 4.2.2beta1U VM.
I think that's from about November 2005, so a bit out of date. The source is on SqueakSource OSProcessPlugin project if you want to see what has changed since then. Nevertheless it should still be good enough to get most of the tests to pass, so I would not worry about the version difference for now.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
I attached gdb to the VM. When running UnixProcessTestCase the VM crashes with: Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000083 0x000e7e9d in resume () (gdb) bt #0 0x000e7e9d in resume () #1 0x4b04264b in ?? () Previous frame inner to this frame (gdb could not unwind past this frame) Adrian On Nov 18, 2009, at 17:19 , Adrian Lienhard wrote:
OK, I'll stick to this VM version.
I compared to the Squeak trunk image and the very same test cases crash the VM. So at least the crashing problem (on Mac) is not likely caused by a Pharo-specific change.
Adrian
On Nov 18, 2009, at 16:18 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 02:56:56PM +0100, Adrian Lienhard wrote:
Hi Dave,
OSProcess accessor osppModuleVersionString ==> '4.0.1'
I get the same result with the Mac 4.2.2beta1U VM.
I think that's from about November 2005, so a bit out of date. The source is on SqueakSource OSProcessPlugin project if you want to see what has changed since then. Nevertheless it should still be good enough to get most of the tests to pass, so I would not worry about the version difference for now.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Hmmm ... I smell a thread problem. I looked a little more carefully at the change log for OSPP on SqueakSource, and I see that I added this for OSPP 4.0.2 in March 2006: VMConstruction-Plugins-OSProcessPlugin-dtl.3.mcz Dave Lewis, 5 March 2006 6:08:38 pm Version 4.0.2 Add pthread access primitive. Add pthread signal masking to ensure that forwarded signals are delivered to the interpreter thread. This is required for OS X. Strategy: Signal handler checks to see if it is executing in the context of the interpreter pthread. If yes, signal the Squeak semaphore, otherwise mask this pthread to prevent future delivery if this signum and resend the signal to the interpreter pthread. IIRC, this update was needed to ensure that SIGCHLD signals would be delivered to the pthread in which the interpreter is running. Prior to that change, Mac users would get VM crashes when a child process exited and OSPP handled the resulting SIGCHLD signal (which involves signaling a Semaphore in the image with #signalSemaphoreWithIndex:). So I think that you will need a more up to date OSPP plugin in order to run the tests on OS X. Note: A quick google check shows that Igor Stasenko solved this same problem differently in Hydra, see: http://lists.squeakfoundation.org/pipermail/vm-dev/2009-September/003223.htm... Dave On Wed, Nov 18, 2009 at 05:59:30PM +0100, Adrian Lienhard wrote:
I attached gdb to the VM. When running UnixProcessTestCase the VM crashes with:
Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000083 0x000e7e9d in resume ()
(gdb) bt #0 0x000e7e9d in resume () #1 0x4b04264b in ?? () Previous frame inner to this frame (gdb could not unwind past this frame)
Adrian
On Nov 18, 2009, at 17:19 , Adrian Lienhard wrote:
OK, I'll stick to this VM version.
I compared to the Squeak trunk image and the very same test cases crash the VM. So at least the crashing problem (on Mac) is not likely caused by a Pharo-specific change.
Adrian
On Nov 18, 2009, at 16:18 , David T. Lewis wrote:
On Wed, Nov 18, 2009 at 02:56:56PM +0100, Adrian Lienhard wrote:
Hi Dave,
OSProcess accessor osppModuleVersionString ==> '4.0.1'
I get the same result with the Mac 4.2.2beta1U VM.
I think that's from about November 2005, so a bit out of date. The source is on SqueakSource OSProcessPlugin project if you want to see what has changed since then. Nevertheless it should still be good enough to get most of the tests to pass, so I would not worry about the version difference for now.
Dave
participants (7)
-
Adrian Lienhard -
David T. Lewis -
Henrik Johansen -
Igor Stasenko -
Levente Uzonyi -
Schwab,Wilhelm K -
Stéphane Ducasse