Re: [Pharo-project] OSProcess - is process done?
Begin forwarded message:
From: "David T. Lewis" <lewis@mail.msen.com> Subject: Re: OSProcess - is process done? Date: April 26, 2012 8:03:50 PM EDT
Sean,
Excellent, thanks for the feedback. As for #upToEnd, that's a tricky question when you are reading from a pipe (or a socket). It is not generally possible to ask the pipe whether more data is going to be available on it. About the best you can do is ask it whether an end of file condition was detected on the last read operation (there are fancy ways to set up notification that a pipe was closed, but nothing that I would want to try to implement in the OSPP plugin).
Based on that, I ended up doing #upToEndOfFile to mean "read all available data, and keep trying until end of file is detected, then answer the result" and #upToEnd to mean "read all currently available data". In the case of a blocking stream (i.e. the end of the pipe from which you are reading data), you want the underlying reads to block (because that is the whole point of a blocking read), and you want the reads to give you available data if the stream is set to non-blocking. But there is no practical way to ask a blocking pipe to give you all available data, because you cannot ask it how much data it has available in advance of doing the read.
The combination of a pipe set to non-blocking mode, along with an #upToEnd that reads whatever data is currently available, gives you the behavior that you probably expect. But that implies that you will poll for data (loop and keep reading until you get all the data) or arrange for event notification to let you know when data is available (this the AioPlugin, which notifies the image whenever data becomes available for reading on a pipe).
There is a mechanism in OSProcess/CommandShell for converting a blocking pipe into an event-driven buffered reader that handles these issues. It is not well documented, but try this to see how it works:
p := PipeableOSProcess command: 'ls -l /usr/bin'. p pipeFromOutput setBufferedReader. p upToEndOfFile
This approach is used in CommandShell to provide the expected behavior when the image is reading from a pipe connected to some external process. And when pipes are connected end-to-end in a unix style pipeline, they can be left in the normal blocking mode so that external programs will block on reads from their stdin, which is the expected behavior for normal programs in a unix command pipeline.
Dave
Sean P. DeNigris wrote
As for #upToEnd
Dave, thanks for being so patient and generous with the explanations! I'm still a little confused about blocking and output. I'm exploring some of your suggestions from the last message, but one thing is bothering me... PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered read by "self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000). I'm pretty sure this sets up a race condition. In http://forum.world.st/Standard-input-in-Pharo-td2173080.html#a4607687 , I found out that fread blocks forever under the following conditions: 1. there is less data available than was asked for 2. no EOF is encountered I think this is why it seems that my (and others') reports of hanging OSP images are inconsistent. If the process has generated at least 1000 characters output before the call to #upToEnd, or if (e.g. "p pipeFromOutput next: 10") less output is asked for than is available, there is no problem. But as soon as one more character is asked for than is available, the vm hangs in fread forever. I don't know what the solution is, but the current behavior doesn't seem workable. I'd be happy to work with you as much as I can to fix it... Thanks again. Sean -- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4607761.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
Sean P. DeNigris wrote
PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered read by "self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000).
After further investigation, it seems to me that blocking #upToEnd is functionally the same as #upToEndOfFile, because its test to stop reading data is StandardFileStream>>atEnd, which calls feof(). Therefore, if there is no EOF, it will keep reading until the pipe is out of data, and then hang in fread on the following iteration. -- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4609991.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On 5 May 2012 00:21, Sean P. DeNigris <sean@clipperadams.com> wrote:
Sean P. DeNigris wrote
PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered read by "self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000).
After further investigation, it seems to me that blocking #upToEnd is functionally the same as #upToEndOfFile, because its test to stop reading data is StandardFileStream>>atEnd, which calls feof(). Therefore, if there is no EOF, it will keep reading until the pipe is out of data, and then hang in fread on the following iteration.
imo stdin upToEnd makes no sense.. one should not expect that data which comes to stdin has any notion of "end", therefore he should never use this method on such stream. stdin/out are unbound (endless) streams , and use things like eof(), and other of such sort should be discouraged.. since it is same as asking infinity atEnd.
-- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4609991.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
-- Best regards, Igor Stasenko.
Igor Stasenko wrote
stdin/out are unbound (endless) streams , and use things like eof(), should be discouraged..
Yes! It is nonsensical!! Igor Stasenko wrote
stdin upToEnd makes no sense..
Well #upToEndOfFile makes no sense, but as I understood Dave's description, #upToEnd means "return all available data". However, eof is still getting checked, which I think is a bug. Right now, OSP delegates to the same stream primitives as all other files, which relies on eof checks. Maybe that is the problem... -- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4610040.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On Fri, May 4, 2012 at 3:28 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 5 May 2012 00:21, Sean P. DeNigris <sean@clipperadams.com> wrote:
Sean P. DeNigris wrote
PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered read
by
"self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000).
After further investigation, it seems to me that blocking #upToEnd is functionally the same as #upToEndOfFile, because its test to stop reading data is StandardFileStream>>atEnd, which calls feof(). Therefore, if there is no EOF, it will keep reading until the pipe is out of data, and then hang in fread on the following iteration.
imo stdin upToEnd makes no sense.. one should not expect that data which comes to stdin has any notion of "end", therefore he should never use this method on such stream.
Um, no. See below.
stdin/out are unbound (endless) streams , and use things like eof(), and other of such sort should be discouraged.. since it is same as asking infinity atEnd.
Um, no. One can redirect a file to stdin. One can type EOF to stdin. EOF definitely *does* make sense for stdin. stdin. stdout and stderr are merely well-defined stream names. they can be bound to arbitrary streams, infinite or otherewise. In unix shells piping is built using dup with fork & exec to arrange that some program reads and writes to specific pipe files in a full pipe.
-- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4609991.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
-- Best regards, Igor Stasenko.
-- best, Eliot
Eliot Miranda-2 wrote
One can redirect a file to stdin. One can type EOF to stdin. EOF definitely *does* make sense for stdin.
Ah, good points. Let me rephrase... in the most common cases, I think waiting for EOF in stdin is not what one wants/expects. -- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4610052.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On 5 May 2012 00:56, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Fri, May 4, 2012 at 3:28 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 5 May 2012 00:21, Sean P. DeNigris <sean@clipperadams.com> wrote:
Sean P. DeNigris wrote
PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered read by "self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000).
After further investigation, it seems to me that blocking #upToEnd is functionally the same as #upToEndOfFile, because its test to stop reading data is StandardFileStream>>atEnd, which calls feof(). Therefore, if there is no EOF, it will keep reading until the pipe is out of data, and then hang in fread on the following iteration.
imo stdin upToEnd makes no sense.. one should not expect that data which comes to stdin has any notion of "end", therefore he should never use this method on such stream.
Um, no. Â See below.
stdin/out are unbound (endless) streams , and use things like eof(), and other of such sort should be discouraged.. since it is same as asking infinity atEnd.
Um, no. Â One can redirect a file to stdin. Â One can type EOF to stdin. EOF definitely *does* make sense for stdin.
what? like putc(EOF)? but that is a convention between two ends. if another end does not recognizing EOF character as an "end of input" signal, it will keep waiting for more data. since these streams are natually binary, i would be really surprised if some characters is reserved for special purposes.
 stdin. stdout and stderr are merely well-defined stream names.  they can be bound to arbitrary streams, infinite or otherewise.  In unix shells piping is built using dup with fork & exec to arrange that some program reads and writes to specific pipe files in a full pipe.
right. but as you gave an example, imagine that i used dup() and one fork keeps writing to stream, while other closes own copy, do receiving side receives any "EOF" signal? i doubt. Here the excert from feof man page: The function feof() tests the end-of-file indicator for the stream pointed to by stream, returning non-zero if it is set. The end-of-file indicator may be cleared by explic- itly calling clearerr(), or as a side-effect of other operations, e.g. fseek(). which means that actually, eof is nothing else than error which captured while attempting to read moar from stream, nicely converted to eof() by higher abstraction level functions (fxxxx C functions). but if you look at basic infrastructure, supported by kernel (read(), write()), there is no notion of "eof" for descriptors. all you can have is an error while attempting to read from or write to descriptor, and then you can decide how to handle that error by either treating it as end-of-file or signaling exception etc.
-- View this message in context: http://forum.world.st/OSProcess-is-process-done-tp4569088p4609991.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
-- Best regards, Igor Stasenko.
-- best, Eliot
-- Best regards, Igor Stasenko.
On Fri, May 4, 2012 at 5:24 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 5 May 2012 00:56, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Fri, May 4, 2012 at 3:28 PM, Igor Stasenko <siguctua@gmail.com>
wrote:
On 5 May 2012 00:21, Sean P. DeNigris <sean@clipperadams.com> wrote:
Sean P. DeNigris wrote
PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered
read
by "self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000).
After further investigation, it seems to me that blocking #upToEnd is functionally the same as #upToEndOfFile, because its test to stop reading data is StandardFileStream>>atEnd, which calls feof(). Therefore, if there is no EOF, it will keep reading until the pipe is out of data, and then hang in fread on the following iteration.
imo stdin upToEnd makes no sense.. one should not expect that data which comes to stdin has any notion of "end", therefore he should never use this method on such stream.
Um, no. See below.
stdin/out are unbound (endless) streams , and use things like eof(), and other of such sort should be discouraged.. since it is same as asking infinity atEnd.
Um, no. One can redirect a file to stdin. One can type EOF to stdin. EOF definitely *does* make sense for stdin.
what? like putc(EOF)?
No. One types an EOF character to the shell (see stty) and the shell responds by closing the pipe to the process. The process then detects an eof condition once it has read all the data from the pipe. There are no EOF characters in a stream on unix. but that is a convention between two ends. if another end does not
recognizing EOF character as an "end of input" signal, it will keep waiting for more data. since these streams are natually binary, i would be really surprised if some characters is reserved for special purposes.
stdin. stdout and stderr are merely well-defined stream names. they can be bound to arbitrary streams, infinite or otherewise. In unix shells piping is built using dup with fork & exec to arrange that some program reads and writes to specific pipe files in a full pipe.
right. but as you gave an example, imagine that i used dup() and one fork keeps writing to stream, while other closes own copy, do receiving side receives any "EOF" signal? i doubt.
Um, that's not how it is used. One process (e.g. parent) holds the write end of the pipe and it can close that end. The other process (e.g. child) holds teh read end and it can detect eof when it consumes all available input.
Here the excert from feof man page: The function feof() tests the end-of-file indicator for the stream pointed to by stream, returning non-zero if it is set. The end-of-file indicator may be cleared by explic- itly calling clearerr(), or as a side-effect of other operations, e.g. fseek().
which means that actually, eof is nothing else than error which captured while attempting to read moar from stream, nicely converted to eof() by higher abstraction level functions (fxxxx C functions).
Right.
but if you look at basic infrastructure, supported by kernel (read(), write()), there is no notion of "eof" for descriptors. all you can have is an error while attempting to read from or write to descriptor, and then you can decide how to handle that error by either treating it as end-of-file or signaling exception etc.
So for files, where eof is an attempt to read beyond end-of-file but not so for pipes and socket streams there is a notion of eof, which is when all data has been read and the write side of the pipe/socketstream is closed. See pipe(2), & socketpair(2)
-- View this message in context:
http://forum.world.st/OSProcess-is-process-done-tp4569088p4609991.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
-- Best regards, Igor Stasenko.
-- best, Eliot
-- Best regards, Igor Stasenko.
-- best, Eliot
On 5 May 2012 02:36, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Fri, May 4, 2012 at 5:24 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 5 May 2012 00:56, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Fri, May 4, 2012 at 3:28 PM, Igor Stasenko <siguctua@gmail.com> wrote:
On 5 May 2012 00:21, Sean P. DeNigris <sean@clipperadams.com> wrote:
Sean P. DeNigris wrote
PipeableOSProcess>>#upToEnd eventually calls AttachableFileStream>>#upToEnd, which tries to perform a buffered read by "self nextInto: 1000" (which eventually calls primitiveFileRead which calls sqFileReadIntoAt which calls fread with count arg of 1000).
After further investigation, it seems to me that blocking #upToEnd is functionally the same as #upToEndOfFile, because its test to stop reading data is StandardFileStream>>atEnd, which calls feof(). Therefore, if there is no EOF, it will keep reading until the pipe is out of data, and then hang in fread on the following iteration.
imo stdin upToEnd makes no sense.. one should not expect that data which comes to stdin has any notion of "end", therefore he should never use this method on such stream.
Um, no. Â See below.
stdin/out are unbound (endless) streams , and use things like eof(), and other of such sort should be discouraged.. since it is same as asking infinity atEnd.
Um, no. Â One can redirect a file to stdin. Â One can type EOF to stdin. EOF definitely *does* make sense for stdin.
what? like putc(EOF)?
No. Â One types an EOF character to the shell (see stty) and the shell responds by closing the pipe to the process. Â The process then detects an eof condition once it has read all the data from the pipe. Â There are no EOF characters in a stream on unix.
whatever. its only a shell convention. it just closes the stream by handling input from user. if you look at general case (between two unfamiliar processes), you cannot assume anything.
but that is a convention between two ends. if another end does not recognizing EOF character as an "end of input" signal, it will keep waiting for more data. since these streams are natually binary, i would be really surprised if some characters is reserved for special purposes.
 stdin. stdout and stderr are merely well-defined stream names.  they can be bound to arbitrary streams, infinite or otherewise.  In unix shells piping is built using dup with fork & exec to arrange that some program reads and writes to specific pipe files in a full pipe.
right. but as you gave an example, imagine that i used dup() and one fork keeps writing to stream, while other closes own copy, do receiving side receives any "EOF" signal? i doubt.
Um, that's not how it is used. Â One process (e.g. parent) holds the write end of the pipe and it can close that end. Â The other process (e.g. child) holds teh read end and it can detect eof when it consumes all available input.
but how you can detect end of available input if other side never closing own end? and it is free to do so. that's why i say that using upToEnd for stdin is bad practice... because i can always redirect endless stream as input for stdin .. and so your program will run in endless loop.
Here the excert from feof man page: Â Â The function feof() tests the end-of-file indicator for the stream pointed to by stream, returning non-zero if it is set. Â The end-of-file indicator may be cleared by explic- Â Â itly calling clearerr(), or as a side-effect of other operations, e.g. fseek().
which means that actually, eof is nothing else than error which captured while attempting to read moar from stream, nicely converted to eof() by higher abstraction level functions (fxxxx C functions).
Right.
but if you look at basic infrastructure, supported by kernel (read(), write()), there is no notion of "eof" for descriptors. all you can have is an error while attempting to read from or write to descriptor, and then you can decide how to handle that error by either treating it as end-of-file or signaling exception etc.
So for files, where eof is an attempt to read beyond end-of-file but not so for pipes and socket streams there is a notion of eof, which is when all data has been read and the write side of the pipe/socketstream is closed. Â See pipe(2), &Â socketpair(2)
but its again about handling error status. it even says so in man page: --- A pipe whose read or write end has been closed is considered widowed. Writing on such a pipe causes the writing process to receive a SIGPIPE signal. Widowing a pipe is the only way to deliver end-of-file to a reader: after the reader consumes any buffered data, reading a widowed pipe returns a zero count. --- but as you can see, for system there is still no notion of end-of-file. it is again, convention where you can assume that upon receiving SIGPIPE you meet the end of input, except from cases.. when such condition is unexpectable.. like reading from /dev/random :) one application can treat SIGPIPE as end of input, while another can treat it as "there something wrong with process, which delivering data to me", and so he will attempt to reconnect. -- Best regards, Igor Stasenko.
participants (4)
-
DeNigris Sean -
Eliot Miranda -
Igor Stasenko -
Sean P. DeNigris