[Pharo-project] reading *exactly* n bytes from socket
Hey guys I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something? Cheers, Max
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/ Cheers, Max On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
Max, I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows: http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/ Cheers, Max On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
On Jan 15, 2012, at 8:10 PM, Schwab,Wilhelm K wrote:
Max,
I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows:
http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware
Yes :) I know.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/
Cheers, Max
On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
Good to know, that you're working on it. Took me a while to figure out that #next: would not fill the entiry buffer... Unfortunately, I wasn't able to resolve my afore mentioned problem completely. To make things easier, I sent #upToEnd to my SocketStream, expecting to get all the data (and then read the lines later). However, towards the end of the transmission the connection is suddenly closed (ConnectionClosed is signaled by Socket>>waitForDataFor:) and I lose a variable amount of data (up to about 300KB out of 4.7MB). The primitive says that the server closed the connection (which might of course be true) but I can't see where my data went missing. In one case I even had a singel byte missing inside a line (the line was one byte shorter than advertised). Now this would probably be a totally different proplem and, to be fair, I couldn' reproduce it, so ignore this for now. Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help. Cheers, Max On 15.01.2012, at 20:56, Stéphane Ducasse wrote:
On Jan 15, 2012, at 8:10 PM, Schwab,Wilhelm K wrote:
Max,
I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows:
http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware
Yes :)
I know.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/
Cheers, Max
On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
On 15 Jan 2012, at 21:19, Max Leske wrote:
Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help.
Have a look at Zodiac too, although it is an implementation of a secure socket stream, it also contains different re-implementations of SocketStream, because I found the existing one to be too hard to subclass. In terms of API and behavior, it does try to mimic the original of course. Sven
Thanks Sven, I'll give it a shot. Max On 15.01.2012, at 21:26, Sven Van Caekenberghe wrote:
On 15 Jan 2012, at 21:19, Max Leske wrote:
Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help.
Have a look at Zodiac too, although it is an implementation of a secure socket stream, it also contains different re-implementations of SocketStream, because I found the existing one to be too hard to subclass. In terms of API and behavior, it does try to mimic the original of course.
Sven
On Jan 15, 2012, at 9:19 PM, Max Leske wrote:
Good to know, that you're working on it. Took me a while to figure out that #next: would not fill the entiry bufferâ¦
I'm not. I just know that bill tried to explain to us what was the problem :) and since the emails/sentences were too long or too english I was always lost but I know that there is this point that we should address.
Unfortunately, I wasn't able to resolve my afore mentioned problem completely. To make things easier, I sent #upToEnd to my SocketStream, expecting to get all the data (and then read the lines later). However, towards the end of the transmission the connection is suddenly closed (ConnectionClosed is signaled by Socket>>waitForDataFor:) and I lose a variable amount of data (up to about 300KB out of 4.7MB). The primitive says that the server closed the connection (which might of course be true) but I can't see where my data went missing.
In one case I even had a singel byte missing inside a line (the line was one byte shorter than advertised). Now this would probably be a totally different proplem and, to be fair, I couldn' reproduce it, so ignore this for now.
Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help.
Cheers, Max
On 15.01.2012, at 20:56, Stéphane Ducasse wrote:
On Jan 15, 2012, at 8:10 PM, Schwab,Wilhelm K wrote:
Max,
I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows:
http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware
Yes :)
I know.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/
Cheers, Max
On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
I don't just write English, I also write Smalltalk :) I had put some related code on SqueakMap, but I suspect it disappeared in one of the meltdowns. It also picked up some code that I had not intended to release, which led me to be unhappy with http repositories. The code varies in quality, covering things missing in streams, FFI, value adapters, among other things. I recently noted that the FFI code will require tweaking to be of use with 1.3 and up. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Sunday, January 15, 2012 3:47 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket On Jan 15, 2012, at 9:19 PM, Max Leske wrote:
Good to know, that you're working on it. Took me a while to figure out that #next: would not fill the entiry bufferâ¦
I'm not. I just know that bill tried to explain to us what was the problem :) and since the emails/sentences were too long or too english I was always lost but I know that there is this point that we should address.
Unfortunately, I wasn't able to resolve my afore mentioned problem completely. To make things easier, I sent #upToEnd to my SocketStream, expecting to get all the data (and then read the lines later). However, towards the end of the transmission the connection is suddenly closed (ConnectionClosed is signaled by Socket>>waitForDataFor:) and I lose a variable amount of data (up to about 300KB out of 4.7MB). The primitive says that the server closed the connection (which might of course be true) but I can't see where my data went missing.
In one case I even had a singel byte missing inside a line (the line was one byte shorter than advertised). Now this would probably be a totally different proplem and, to be fair, I couldn' reproduce it, so ignore this for now.
Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help.
Cheers, Max
On 15.01.2012, at 20:56, Stéphane Ducasse wrote:
On Jan 15, 2012, at 8:10 PM, Schwab,Wilhelm K wrote:
Max,
I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows:
http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware
Yes :)
I know.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/
Cheers, Max
On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
On Jan 16, 2012, at 5:19 AM, Schwab,Wilhelm K wrote:
I don't just write English, I also write Smalltalk :)
:)
I had put some related code on SqueakMap, but I suspect it disappeared in one of the meltdowns. It also picked up some code that I had not intended to release, which led me to be unhappy with http repositories.
The code varies in quality, covering things missing in streams, FFI, value adapters, among other things. I recently noted that the FFI code will require tweaking to be of use with 1.3 and up.
if you have fixes for FFi please publish it because we need a good FFI.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Sunday, January 15, 2012 3:47 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
On Jan 15, 2012, at 9:19 PM, Max Leske wrote:
Good to know, that you're working on it. Took me a while to figure out that #next: would not fill the entiry bufferâ¦
I'm not. I just know that bill tried to explain to us what was the problem :) and since the emails/sentences were too long or too english I was always lost but I know that there is this point that we should address.
Unfortunately, I wasn't able to resolve my afore mentioned problem completely. To make things easier, I sent #upToEnd to my SocketStream, expecting to get all the data (and then read the lines later). However, towards the end of the transmission the connection is suddenly closed (ConnectionClosed is signaled by Socket>>waitForDataFor:) and I lose a variable amount of data (up to about 300KB out of 4.7MB). The primitive says that the server closed the connection (which might of course be true) but I can't see where my data went missing.
In one case I even had a singel byte missing inside a line (the line was one byte shorter than advertised). Now this would probably be a totally different proplem and, to be fair, I couldn' reproduce it, so ignore this for now.
Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help.
Cheers, Max
On 15.01.2012, at 20:56, Stéphane Ducasse wrote:
On Jan 15, 2012, at 8:10 PM, Schwab,Wilhelm K wrote:
Max,
I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows:
http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware
Yes :)
I know.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/
Cheers, Max
On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
Stef, It's not so much fixes for FFI as it is filling gaps. We need a full set of signed and unsigned word, dword, float and double arrays. Toward that end, I have a sloppy set that appears to work. I recently discovered that I will have to explicitly assign endianness due to changes in Pharo since 1.1. One thing that really disappoints me about my fixed type arrays is that they all ultimately use byte array to hold the data; I can't pass my DWORDArray to a ulong*; I have to specify void* to get it to work. Likewise with DOUBLEArray and double*. There are also alignment questions that need attention. Bill ________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Monday, January 16, 2012 1:21 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket On Jan 16, 2012, at 5:19 AM, Schwab,Wilhelm K wrote:
I don't just write English, I also write Smalltalk :)
:)
I had put some related code on SqueakMap, but I suspect it disappeared in one of the meltdowns. It also picked up some code that I had not intended to release, which led me to be unhappy with http repositories.
The code varies in quality, covering things missing in streams, FFI, value adapters, among other things. I recently noted that the FFI code will require tweaking to be of use with 1.3 and up.
if you have fixes for FFi please publish it because we need a good FFI.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Stéphane Ducasse [stephane.ducasse@inria.fr] Sent: Sunday, January 15, 2012 3:47 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
On Jan 15, 2012, at 9:19 PM, Max Leske wrote:
Good to know, that you're working on it. Took me a while to figure out that #next: would not fill the entiry bufferâ¦
I'm not. I just know that bill tried to explain to us what was the problem :) and since the emails/sentences were too long or too english I was always lost but I know that there is this point that we should address.
Unfortunately, I wasn't able to resolve my afore mentioned problem completely. To make things easier, I sent #upToEnd to my SocketStream, expecting to get all the data (and then read the lines later). However, towards the end of the transmission the connection is suddenly closed (ConnectionClosed is signaled by Socket>>waitForDataFor:) and I lose a variable amount of data (up to about 300KB out of 4.7MB). The primitive says that the server closed the connection (which might of course be true) but I can't see where my data went missing.
In one case I even had a singel byte missing inside a line (the line was one byte shorter than advertised). Now this would probably be a totally different proplem and, to be fair, I couldn' reproduce it, so ignore this for now.
Camillo is now looking into ithe SocketStream stuff but if any of you have a clue what could be going on, I'd appreciate your help.
Cheers, Max
On 15.01.2012, at 20:56, Stéphane Ducasse wrote:
On Jan 15, 2012, at 8:10 PM, Schwab,Wilhelm K wrote:
Max,
I wouldn't forget it too soon. Streams should work as advertised or raise an error. My (compromise) proposal remains as follows:
http://code.google.com/p/pharo/wiki/StreamsForRobustSoftware
Yes :)
I know.
Bill
________________________________________ From: pharo-project-bounces@lists.gforge.inria.fr [pharo-project-bounces@lists.gforge.inria.fr] on behalf of Max Leske [maxleske@gmail.com] Sent: Sunday, January 15, 2012 6:18 AM To: pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] reading *exactly* n bytes from socket
Sorry, forget what I just wrote⦠I found the bug in my code. Should have checked if the connection is still open :-/
Cheers, Max
On 15.01.2012, at 12:09, Max Leske wrote:
Hey guys
I'm having a problem with Socket / SocketStream. When I know that the next packet of data from the server is going to be 10'000 bytes I want to ask the socket for exactly 10'000 bytes of data (I don't care how long it takes). However, the comments in the Socket class suggest that the buffer might not be filled entirely when the message answers. As a consequence, my code fails because the ByteArray sometimes has a number of zero bytes at the end which obviously wasn't expected. I also tried to use SocketStream to get around this problem but wasn't successful. Am I supposed to handle this case myself or did I overlook something?
Cheers, Max
participants (4)
-
Max Leske -
Schwab,Wilhelm K -
Stéphane Ducasse -
Sven Van Caekenberghe