Thread overview | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
November 03, 2009 (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
While trying to learn sockets in Phobos, I ran into a problem. I was trying to download the google logo via an HTTP GET. when I do a socketStream.read, the returned "bytes read" value indicates that less bytes were read in than were available (4097 bytes to be exact). The google logo is 8558 bytes in size as indicated by the HTTP headers received. Here is my code: module HttpManager; import std.stream; import std.stdio; import std.socket; import std.socketstream; import std.string; int main() { char[] line; ubyte[] data = new ubyte[8558]; uint num = 0; TcpSocket socket = new TcpSocket(new InternetAddress("www.google.com", 80)); socket.send("GET /intl/en_ALL/images/logo.gif HTTP/1.0\r\n\r\n"); SocketStream socketStream = new SocketStream(socket); while(!socketStream.eof) { line = socketStream.readLine(); if (line=="") break; writef("%s\n", line); } num = socketStream.read(data); writef("\n\nNum: %d\n\n", num); socketStream.close; socket.close; File file = new File("logo.gif", FileMode.Out); file.write(data); file.close; return 0; } The console output is: HTTP/1.1 200 OK Content-Type: image/gif Last-Modified: Wed, 07 Jun 2006 19:38:24 GMT Date: Tue, 03 Nov 2009 13:04:57 GMT Expires: Wed, 03 Nov 2010 13:04:57 GMT X-Content-Type-Options: nosniff Server: gws Content-Length: 8558 Cache-Control: public, max-age=31536000 Age: 28322 X-XSS-Protection: 0 Num: 4097 I am using dmd v1.050 with dsss v0.78 Sorry if this is a stupid question. Any help would be very appreciated :) Thanks, Zane |
November 03, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Zane | Hello Zane,
> While trying to learn sockets in Phobos, I ran into a problem. I was
> trying to download the google logo via an HTTP GET. when I do a
> socketStream.read, the returned "bytes read" value indicates that less
> bytes were read in than were available (4097 bytes to be exact). The
> google logo is 8558 bytes in size as indicated by the HTTP headers
> received. Here is my code:
>
The network connection might not have all the data yet. Try calling it in a loop kind of like your header reading code.
|
November 03, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to BCS | BCS Wrote:
> Hello Zane,
>
> > While trying to learn sockets in Phobos, I ran into a problem. I was trying to download the google logo via an HTTP GET. when I do a socketStream.read, the returned "bytes read" value indicates that less bytes were read in than were available (4097 bytes to be exact). The google logo is 8558 bytes in size as indicated by the HTTP headers received. Here is my code:
> >
>
> The network connection might not have all the data yet. Try calling it in a loop kind of like your header reading code.
>
>
I do not see how this can be the case. Per the Phobos spec for InputStream, "Read a block of data big enough to fill the given array buffer. " This sounds to me like it should do just that, fill my array....and not return until done. readLine needs to be looped because it only reads up to a carriage return / newline combination (and there are multiple lines). Can anyone confirm that my speculations are correct? Otherwise, the documentation seems misleading and I will need help with a correct implementation.
Zane
|
November 04, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Zane | Hello Zane,
> BCS Wrote:
>
>> Hello Zane,
>>
>>> While trying to learn sockets in Phobos, I ran into a problem. I
>>> was trying to download the google logo via an HTTP GET. when I do a
>>> socketStream.read, the returned "bytes read" value indicates that
>>> less bytes were read in than were available (4097 bytes to be
>>> exact). The google logo is 8558 bytes in size as indicated by the
>>> HTTP headers received. Here is my code:
>>>
>> The network connection might not have all the data yet. Try calling
>> it in a loop kind of like your header reading code.
>>
> I do not see how this can be the case. Per the Phobos spec for
> InputStream, "Read a block of data big enough to fill the given array
> buffer. " This sounds to me like it should do just that, fill my
> array....and not return until done. readLine needs to be looped
> because it only reads up to a carriage return / newline combination
> (and there are multiple lines). Can anyone confirm that my
> speculations are correct? Otherwise, the documentation seems
> misleading and I will need help with a correct implementation.
If I were you, I'd just try the looped version because there might be a bug. I seem to recall that the stream stuff in some places gets defined to return on out of data conditions. (BTW this would be consistent with the liked named "read" function from posix.)
Looking into the code (Stream.read() and SocketStream.readBlock()) it doesn't block if there isn't enough data.
If this is a code bug or a documentation deficiency is another question.
|
November 04, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to BCS | BCS Wrote:
> Hello Zane,
>
> > BCS Wrote:
> >
> >> Hello Zane,
> >>
> >>> While trying to learn sockets in Phobos, I ran into a problem. I was trying to download the google logo via an HTTP GET. when I do a socketStream.read, the returned "bytes read" value indicates that less bytes were read in than were available (4097 bytes to be exact). The google logo is 8558 bytes in size as indicated by the HTTP headers received. Here is my code:
> >>>
> >> The network connection might not have all the data yet. Try calling it in a loop kind of like your header reading code.
> >>
> > I do not see how this can be the case. Per the Phobos spec for InputStream, "Read a block of data big enough to fill the given array buffer. " This sounds to me like it should do just that, fill my array....and not return until done. readLine needs to be looped because it only reads up to a carriage return / newline combination (and there are multiple lines). Can anyone confirm that my speculations are correct? Otherwise, the documentation seems misleading and I will need help with a correct implementation.
>
> If I were you, I'd just try the looped version because there might be a bug. I seem to recall that the stream stuff in some places gets defined to return on out of data conditions. (BTW this would be consistent with the liked named "read" function from posix.)
>
> Looking into the code (Stream.read() and SocketStream.readBlock()) it doesn't block if there isn't enough data.
>
> If this is a code bug or a documentation deficiency is another question.
>
>
I assumed since the underlying TcpSocket was blocking, then calling read through the stream would also block....dunno. Anyway, so if I were to use a loop, how could I do this with read? The size of the read buffer must be initialized before the reading takes place, however, I do not know how much will be read for the next "chunk". If I am to receive these in arbitrarily sized chunks for concatenation, I don't see a sensible way of constructing a loop. Example?
Zane
|
November 04, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Zane | On Tue, 03 Nov 2009 20:05:17 -0500, Zane wrote:
> If I am to receive
> these in arbitrarily sized chunks for concatenation, I don't see a
> sensible way of constructing a loop. Example?
>
> Zane
You can use the number of bytes read to determine and use string slicing to concatenation into the final array.
|
November 04, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Zane | On Tue, 03 Nov 2009 20:05:17 -0500, Zane <zane.sims@gmail.com> wrote:
> I assumed since the underlying TcpSocket was blocking, then calling read through the stream would also block....dunno. Anyway, so if I were to use a loop, how could I do this with read? The size of the read buffer must be initialized before the reading takes place, however, I do not know how much will be read for the next "chunk". If I am to receive these in arbitrarily sized chunks for concatenation, I don't see a sensible way of constructing a loop. Example?
Your interpretation of blocking sockets is incorrect. A socket blocks on a read only if *no* data is available. If any data is available, it reads as much as it can and returns.
A non blocking socket returns immediately, even if no data is available (usually returning an error like EAGAIN).
A simple loop (keep in mind, I don't know phobos, and I didn't look up the exact usage):
// ubyte[] buf is the array we want to fill, predetermined length
ubyte[] tmp = buf[];
while(tmp.length > 0)
{
auto bytesread = socket.read(tmp);
if(bytesread == 0) break; // EOF (also should check for error here if necessary)
tmp = tmp[bytesread..$];
}
This kind of thing can *easily* be put into a wrapper function.
-Steve
|
November 04, 2009 Re: (Phobos - SocketStream) Am I doing something wrong or is this a bug? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Zane | Hello Zane,
> BCS Wrote:
>
>> Hello Zane,
>>
>>> BCS Wrote:
>>>
>>>> Hello Zane,
>>>>
>>>>> While trying to learn sockets in Phobos, I ran into a problem. I
>>>>> was trying to download the google logo via an HTTP GET. when I do
>>>>> a socketStream.read, the returned "bytes read" value indicates
>>>>> that less bytes were read in than were available (4097 bytes to be
>>>>> exact). The google logo is 8558 bytes in size as indicated by the
>>>>> HTTP headers received. Here is my code:
>>>>>
>>>> The network connection might not have all the data yet. Try calling
>>>> it in a loop kind of like your header reading code.
>>>>
>>> I do not see how this can be the case. Per the Phobos spec for
>>> InputStream, "Read a block of data big enough to fill the given
>>> array buffer. " This sounds to me like it should do just that, fill
>>> my array....and not return until done. readLine needs to be looped
>>> because it only reads up to a carriage return / newline combination
>>> (and there are multiple lines). Can anyone confirm that my
>>> speculations are correct? Otherwise, the documentation seems
>>> misleading and I will need help with a correct implementation.
>>>
>> If I were you, I'd just try the looped version because there might be
>> a bug. I seem to recall that the stream stuff in some places gets
>> defined to return on out of data conditions. (BTW this would be
>> consistent with the liked named "read" function from posix.)
>>
>> Looking into the code (Stream.read() and SocketStream.readBlock()) it
>> doesn't block if there isn't enough data.
>>
>> If this is a code bug or a documentation deficiency is another
>> question.
>>
> I assumed since the underlying TcpSocket was blocking, then calling
> read through the stream would also block....dunno. Anyway, so if I
> were to use a loop, how could I do this with read? The size of the
> read buffer must be initialized before the reading takes place,
> however, I do not know how much will be read for the next "chunk". If
> I am to receive these in arbitrarily sized chunks for concatenation, I
> don't see a sensible way of constructing a loop. Example?
ubyte[] buffer = new ubyte[size];
auto window = buffer;
while(!ss.eof() && window.length > 0)
{
int i = ss.read(windows);
window = window[i..$];
}
that, or take a look at readExect.
|
Copyright © 1999-2021 by the D Language Foundation