[Pharo-project] OSProcess - can it pipe 1 MB?
I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data. I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself. But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string. Any ideas? Bill
On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data. I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself. But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string. Any ideas?
Bill
Try googling "maximum size of write to linux pipe". Unix pipes have a capacity limit, and the write will fail if you try to write too large a chunk all at once. Dave
Dave, Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable. The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece? One thought would be AttachableFileStream>>nextPutAll:blob | in | in := ReadStream on:blob. [ in atEnd ] whileFalse:[ self nextPutAll:( in next:32000). ]. but it should run into the same problem after a few times through the loop. It needs some "flow control" and could easily belong elsewhere in the hierarch?? The above assumes the (should be changed) silent truncation of #next:. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis [lewis@mail.msen.com] Sent: Sunday, August 29, 2010 12:37 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB? On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data. I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself. But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string. Any ideas?
Bill
Try googling "maximum size of write to linux pipe". Unix pipes have a capacity limit, and the write will fail if you try to write too large a chunk all at once. Dave _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
It does not belong in OSProcess unless it's something that the underlying operating system actually supports, which is not the case here. As far as I know, there is no Unix system call for determining how much data a pipe is ready to accept. You'll need to manage your flow control yourself. You may also need to call #setNonBlocking on the pipe writer, otherwise you can block on write and lock up the virtual machine(I think you know that already but just in case). If the process at the other end of your pipe is keeping up with the data, it won't be a problem, but if you try to write to a full pipe without pulling something out of the other end, you will lock up your VM completely unless the stream has been set to non-blocking mode. Dave On Sun, Aug 29, 2010 at 02:28:50PM -0400, Schwab,Wilhelm K wrote:
Dave,
Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable. The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece?
One thought would be
AttachableFileStream>>nextPutAll:blob | in | in := ReadStream on:blob. [ in atEnd ] whileFalse:[ self nextPutAll:( in next:32000). ].
but it should run into the same problem after a few times through the loop. It needs some "flow control" and could easily belong elsewhere in the hierarch?? The above assumes the (should be changed) silent truncation of #next:.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis [lewis@mail.msen.com] Sent: Sunday, August 29, 2010 12:37 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data. I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself. But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string. Any ideas?
Bill
Try googling "maximum size of write to linux pipe". Unix pipes have a capacity limit, and the write will fail if you try to write too large a chunk all at once.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Dave, Why does primWrite: id from: stringOrByteArray startingAt: startIndex count: count "Write count bytes onto this file from the given string or byte array starting at the given index. Answer the number of bytes written." raise an error? Perhaps it or a related method should answer the number of bytes written. There should be a compromise between locking up the VM and having no idea how quickly the other side can accept data. Given a count of what was written (as suggested in the comment, and perhaps zero bytes if the other side is struggling), then one could design a reasonable loop with a delay to provide the needed flow control. Otherwise, it's just guesswork. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis [lewis@mail.msen.com] Sent: Sunday, August 29, 2010 3:05 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB? It does not belong in OSProcess unless it's something that the underlying operating system actually supports, which is not the case here. As far as I know, there is no Unix system call for determining how much data a pipe is ready to accept. You'll need to manage your flow control yourself. You may also need to call #setNonBlocking on the pipe writer, otherwise you can block on write and lock up the virtual machine(I think you know that already but just in case). If the process at the other end of your pipe is keeping up with the data, it won't be a problem, but if you try to write to a full pipe without pulling something out of the other end, you will lock up your VM completely unless the stream has been set to non-blocking mode. Dave On Sun, Aug 29, 2010 at 02:28:50PM -0400, Schwab,Wilhelm K wrote:
Dave,
Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable. The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece?
One thought would be
AttachableFileStream>>nextPutAll:blob | in | in := ReadStream on:blob. [ in atEnd ] whileFalse:[ self nextPutAll:( in next:32000). ].
but it should run into the same problem after a few times through the loop. It needs some "flow control" and could easily belong elsewhere in the hierarch?? The above assumes the (should be changed) silent truncation of #next:.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis [lewis@mail.msen.com] Sent: Sunday, August 29, 2010 12:37 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data. I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself. But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string. Any ideas?
Bill
Try googling "maximum size of write to linux pipe". Unix pipes have a capacity limit, and the write will fail if you try to write too large a chunk all at once.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Bill, I think that you should be able to design a reasonable loop with a delay to provide the needed flow control without modifying the underlying #primWrite:from:startingAt:count: semantics. That is exactly what I think you should do in this case. Be pragmatic about it; break your 1MB data into digestible pieces, set the pipe writer to be nonblocking, and loop with a delay until the whole thing gets digested by whatever is reading it on the other end of the pipe. HTH, Dave On Sun, Aug 29, 2010 at 03:47:18PM -0400, Schwab,Wilhelm K wrote:
Dave,
Why does
primWrite: id from: stringOrByteArray startingAt: startIndex count: count "Write count bytes onto this file from the given string or byte array starting at the given index. Answer the number of bytes written."
raise an error? Perhaps it or a related method should answer the number of bytes written. There should be a compromise between locking up the VM and having no idea how quickly the other side can accept data. Given a count of what was written (as suggested in the comment, and perhaps zero bytes if the other side is struggling), then one could design a reasonable loop with a delay to provide the needed flow control. Otherwise, it's just guesswork.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis [lewis@mail.msen.com] Sent: Sunday, August 29, 2010 3:05 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
It does not belong in OSProcess unless it's something that the underlying operating system actually supports, which is not the case here. As far as I know, there is no Unix system call for determining how much data a pipe is ready to accept.
You'll need to manage your flow control yourself. You may also need to call #setNonBlocking on the pipe writer, otherwise you can block on write and lock up the virtual machine(I think you know that already but just in case). If the process at the other end of your pipe is keeping up with the data, it won't be a problem, but if you try to write to a full pipe without pulling something out of the other end, you will lock up your VM completely unless the stream has been set to non-blocking mode.
Dave
On Sun, Aug 29, 2010 at 02:28:50PM -0400, Schwab,Wilhelm K wrote:
Dave,
Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable. The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece?
One thought would be
AttachableFileStream>>nextPutAll:blob | in | in := ReadStream on:blob. [ in atEnd ] whileFalse:[ self nextPutAll:( in next:32000). ].
but it should run into the same problem after a few times through the loop. It needs some "flow control" and could easily belong elsewhere in the hierarch?? The above assumes the (should be changed) silent truncation of #next:.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis [lewis@mail.msen.com] Sent: Sunday, August 29, 2010 12:37 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data. I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself. But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string. Any ideas?
Bill
Try googling "maximum size of write to linux pipe". Unix pipes have a capacity limit, and the write will fail if you try to write too large a chunk all at once.
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
participants (2)
-
David T. Lewis -
Schwab,Wilhelm K