While trying to learn sockets in Phobos, I ran into a problem.  I was trying to download the google logo via an HTTP GET.  when I do a socketStream.read, the returned "bytes read" value indicates that less bytes were read in than were available (4097 bytes to be exact).  The google logo is 8558 bytes in size as indicated by the HTTP headers received.  Here is my code:

module HttpManager;

import std.stream;
import std.stdio;
import std.socket;
import std.socketstream;

import std.string;

int main()
{
	char[] line;
	ubyte[] data = new ubyte[8558];
	uint num = 0;

	TcpSocket socket = new TcpSocket(new InternetAddress("www.google.com", 80));

	socket.send("GET /intl/en_ALL/images/logo.gif HTTP/1.0\r\n\r\n");

	SocketStream socketStream = new SocketStream(socket);

	while(!socketStream.eof)
	{
		line = socketStream.readLine();

		if (line=="")
			break;

		writef("%s\n", line);
	}

	num = socketStream.read(data);

	writef("\n\nNum: %d\n\n", num);

	socketStream.close;
	socket.close;

	File file = new File("logo.gif", FileMode.Out);
	file.write(data);
	file.close;

	return 0;
}

The console output is:

	HTTP/1.1 200 OK
	Content-Type: image/gif
	Last-Modified: Wed, 07 Jun 2006 19:38:24 GMT
	Date: Tue, 03 Nov 2009 13:04:57 GMT
	Expires: Wed, 03 Nov 2010 13:04:57 GMT
	X-Content-Type-Options: nosniff
	Server: gws
	Content-Length: 8558
	Cache-Control: public, max-age=31536000
	Age: 28322
	X-XSS-Protection: 0


	Num: 4097

I am using dmd v1.050 with dsss v0.78

Sorry if this is a stupid question.  Any help would be very appreciated :)

Thanks,
Zane

Hello Zane,

> While trying to learn sockets in Phobos, I ran into a problem.  I was
> trying to download the google logo via an HTTP GET.  when I do a
> socketStream.read, the returned "bytes read" value indicates that less
> bytes were read in than were available (4097 bytes to be exact).  The
> google logo is 8558 bytes in size as indicated by the HTTP headers
> received.  Here is my code:
> 

The network connection might not have all the data yet. Try calling it in a loop kind of like your header reading code.

BCS Wrote:

> Hello Zane,
> 
> > While trying to learn sockets in Phobos, I ran into a problem.  I was trying to download the google logo via an HTTP GET.  when I do a socketStream.read, the returned "bytes read" value indicates that less bytes were read in than were available (4097 bytes to be exact).  The google logo is 8558 bytes in size as indicated by the HTTP headers received.  Here is my code:
> > 
> 
> The network connection might not have all the data yet. Try calling it in a loop kind of like your header reading code.
> 
> 

I do not see how this can be the case.  Per the Phobos spec for InputStream, "Read a block of data big enough to fill the given array buffer. "  This sounds to me like it should do just that, fill my array....and not return until done.  readLine needs to be looped because it only reads up to a carriage return / newline combination (and there are multiple lines).  Can anyone confirm that my speculations are correct?  Otherwise, the documentation seems misleading and I will need help with a correct implementation.

Zane

Hello Zane,

> BCS Wrote:
> 
>> Hello Zane,
>> 
>>> While trying to learn sockets in Phobos, I ran into a problem.  I
>>> was trying to download the google logo via an HTTP GET.  when I do a
>>> socketStream.read, the returned "bytes read" value indicates that
>>> less bytes were read in than were available (4097 bytes to be
>>> exact).  The google logo is 8558 bytes in size as indicated by the
>>> HTTP headers received.  Here is my code:
>>> 
>> The network connection might not have all the data yet. Try calling
>> it in a loop kind of like your header reading code.
>> 
> I do not see how this can be the case.  Per the Phobos spec for
> InputStream, "Read a block of data big enough to fill the given array
> buffer. "  This sounds to me like it should do just that, fill my
> array....and not return until done.  readLine needs to be looped
> because it only reads up to a carriage return / newline combination
> (and there are multiple lines).  Can anyone confirm that my
> speculations are correct?  Otherwise, the documentation seems
> misleading and I will need help with a correct implementation.

If I were you, I'd just try the looped version because there might be a bug. I seem to recall that the stream stuff in some places gets defined to return on out of data conditions. (BTW this would be consistent with the liked named "read" function from posix.)

Looking into the code (Stream.read() and SocketStream.readBlock()) it doesn't block if there isn't enough data. 

If this is a code bug or a documentation deficiency is another question.

BCS Wrote:

> Hello Zane,
> 
> > BCS Wrote:
> > 
> >> Hello Zane,
> >> 
> >>> While trying to learn sockets in Phobos, I ran into a problem.  I was trying to download the google logo via an HTTP GET.  when I do a socketStream.read, the returned "bytes read" value indicates that less bytes were read in than were available (4097 bytes to be exact).  The google logo is 8558 bytes in size as indicated by the HTTP headers received.  Here is my code:
> >>> 
> >> The network connection might not have all the data yet. Try calling it in a loop kind of like your header reading code.
> >> 
> > I do not see how this can be the case.  Per the Phobos spec for InputStream, "Read a block of data big enough to fill the given array buffer. "  This sounds to me like it should do just that, fill my array....and not return until done.  readLine needs to be looped because it only reads up to a carriage return / newline combination (and there are multiple lines).  Can anyone confirm that my speculations are correct?  Otherwise, the documentation seems misleading and I will need help with a correct implementation.
> 
> If I were you, I'd just try the looped version because there might be a bug. I seem to recall that the stream stuff in some places gets defined to return on out of data conditions. (BTW this would be consistent with the liked named "read" function from posix.)
> 
> Looking into the code (Stream.read() and SocketStream.readBlock()) it doesn't block if there isn't enough data.
> 
> If this is a code bug or a documentation deficiency is another question.
> 
> 

I assumed since the underlying TcpSocket was blocking, then calling read through the stream would also block....dunno.  Anyway, so if I were to use a loop, how could I do this with read?  The size of the read buffer must be initialized before the reading takes place, however, I do not know how much will be read for the next "chunk".  If I am to receive these in arbitrarily sized chunks for concatenation, I don't see a sensible way of constructing a loop.  Example?

Zane

On Tue, 03 Nov 2009 20:05:17 -0500, Zane wrote:

> If I am to receive
> these in arbitrarily sized chunks for concatenation, I don't see a
> sensible way of constructing a loop.  Example?
> 
> Zane

You can use the number of bytes read to determine and use string slicing to concatenation into the final array.

On Tue, 03 Nov 2009 20:05:17 -0500, Zane <zane.sims@gmail.com> wrote:

> I assumed since the underlying TcpSocket was blocking, then calling read through the stream would also block....dunno.  Anyway, so if I were to use a loop, how could I do this with read?  The size of the read buffer must be initialized before the reading takes place, however, I do not know how much will be read for the next "chunk".  If I am to receive these in arbitrarily sized chunks for concatenation, I don't see a sensible way of constructing a loop.  Example?

Your interpretation of blocking sockets is incorrect. A socket blocks on a read only if *no* data is available.  If any data is available, it reads as much as it can and returns.

A non blocking socket returns immediately, even if no data is available (usually returning an error like EAGAIN).

A simple loop (keep in mind, I don't know phobos, and I didn't look up the exact usage):

// ubyte[] buf is the array we want to fill, predetermined length

ubyte[] tmp = buf[];

while(tmp.length > 0)
{
  auto bytesread = socket.read(tmp);
  if(bytesread == 0) break; // EOF (also should check for error here if necessary)
  tmp = tmp[bytesread..$];
}

This kind of thing can *easily* be put into a wrapper function.

-Steve

Hello Zane,

> BCS Wrote:
> 
>> Hello Zane,
>> 
>>> BCS Wrote:
>>> 
>>>> Hello Zane,
>>>> 
>>>>> While trying to learn sockets in Phobos, I ran into a problem.  I
>>>>> was trying to download the google logo via an HTTP GET.  when I do
>>>>> a socketStream.read, the returned "bytes read" value indicates
>>>>> that less bytes were read in than were available (4097 bytes to be
>>>>> exact).  The google logo is 8558 bytes in size as indicated by the
>>>>> HTTP headers received.  Here is my code:
>>>>> 
>>>> The network connection might not have all the data yet. Try calling
>>>> it in a loop kind of like your header reading code.
>>>> 
>>> I do not see how this can be the case.  Per the Phobos spec for
>>> InputStream, "Read a block of data big enough to fill the given
>>> array buffer. "  This sounds to me like it should do just that, fill
>>> my array....and not return until done.  readLine needs to be looped
>>> because it only reads up to a carriage return / newline combination
>>> (and there are multiple lines).  Can anyone confirm that my
>>> speculations are correct?  Otherwise, the documentation seems
>>> misleading and I will need help with a correct implementation.
>>> 
>> If I were you, I'd just try the looped version because there might be
>> a bug. I seem to recall that the stream stuff in some places gets
>> defined to return on out of data conditions. (BTW this would be
>> consistent with the liked named "read" function from posix.)
>> 
>> Looking into the code (Stream.read() and SocketStream.readBlock()) it
>> doesn't block if there isn't enough data.
>> 
>> If this is a code bug or a documentation deficiency is another
>> question.
>> 
> I assumed since the underlying TcpSocket was blocking, then calling
> read through the stream would also block....dunno.  Anyway, so if I
> were to use a loop, how could I do this with read?  The size of the
> read buffer must be initialized before the reading takes place,
> however, I do not know how much will be read for the next "chunk".  If
> I am to receive these in arbitrarily sized chunks for concatenation, I
> don't see a sensible way of constructing a loop.  Example?

ubyte[] buffer = new ubyte[size];
auto window = buffer;

while(!ss.eof() && window.length > 0)
{
  int i = ss.read(windows);
  window = window[i..$];
}


that, or take a look at readExect.