Greetings!

Seeing that everyone just want to argue, ;-), I hope some of you could help on this one:  I am downloading a bunch of xml file using socket and datastream, but from time to time, the program that I wrote would finish with

Error: not enough data in stream

I know that it's downloading lots of xml data every time this happens.  I don't know how much, but my question is, is there a definite amount of data that a stream can handle?  Don't tell me that this is because of memory, cuz this server has over 1.5GB of memory and I know I am not downloading that much. ;-)

any help would be greatly appreciated.

thanks,

jic

On Tue, 8 Mar 2005 09:26:31 +0000 (UTC), jicman <jicman_member@pathlink.com> wrote:
> Greetings!
>
> Seeing that everyone just want to argue, ;-), I hope some of you could help on
> this one:  I am downloading a bunch of xml file using socket and datastream, but
> from time to time, the program that I wrote would finish with
>
> Error: not enough data in stream

I cannot find the string "not enough data in stream" in the phobos source.. is that your own error message?

> I know that it's downloading lots of xml data every time this happens.  I don't
> know how much, but my question is, is there a definite amount of data that a
> stream can handle?  Don't tell me that this is because of memory, cuz this
> server has over 1.5GB of memory and I know I am not downloading that much. ;-)
>
> any help would be greatly appreciated.

Perhaps posting some code would help? esp if you can cut the code down to a small sample which reproduces the problem. I find that attempting to do this, often helps find the problem.

Regan

Regan Heath wrote:

>> Error: not enough data in stream
> 
> I cannot find the string "not enough data in stream" in the phobos  source.. is that your own error message?

It's in std.stream:
>   // reads block of data of specified size,
>   // throws ReadException on error
>   void readExact(void* buffer, uint size)
>   {
>     uint readsize = readBlock(buffer, size);
>     if (readsize != size)
>       throw new ReadException("not enough data in stream");
>   }

I think it's a flawed design, that EOF is an Exception ?

I'd rather have "read" return 'false' on EOF, myself.
And save the exceptions for the I/O errors and such...

--anders

Regan Heath says...
>
>On Tue, 8 Mar 2005 09:26:31 +0000 (UTC), jicman <jicman_member@pathlink.com> wrote:
>> Greetings!
>>
>> Seeing that everyone just want to argue, ;-), I hope some of you could
>> help on
>> this one:  I am downloading a bunch of xml file using socket and
>> datastream, but
>> from time to time, the program that I wrote would finish with
>>
>> Error: not enough data in stream
>
>I cannot find the string "not enough data in stream" in the phobos source.. is that your own error message?

Nope.   Not my own message.  It may be what the server is giving my program which d.DataStream is probably getting and printing.

But, I think I figured it out.  The server from where the XML data is coming from may have had a cap on its maximum data output.  It was set to 128MB, so I raised it to 1GB and now the error has disappeared, but the user which had this problem previously is still downloading xml data and it's up to 500+MB!  YIKES! This is bad design at its best. :-)

>> I know that it's downloading lots of xml data every time this happens.
>> I don't
>> know how much, but my question is, is there a definite amount of data
>> that a
>> stream can handle?  Don't tell me that this is because of memory, cuz
>> this
>> server has over 1.5GB of memory and I know I am not downloading that
>> much. ;-)
>>
>> any help would be greatly appreciated.
>
>Perhaps posting some code would help? esp if you can cut the code down to a small sample which reproduces the problem. I find that attempting to do this, often helps find the problem.

I would, but I don't have the time, right now.  I am at a customer site and I am trying to clean up a DB and I'm changing code on the fly.

Did I say that I love d?

thanks.

jic

On Tue, 08 Mar 2005 10:51:26 +0100, Anders F Björklund <afb@algonet.se> wrote:
> Regan Heath wrote:
>
>>> Error: not enough data in stream
>>  I cannot find the string "not enough data in stream" in the phobos  source.. is that your own error message?
>
> It's in std.stream:

Aha! That'll teach me to forget the "search subfolders" checkbox. :)

>>   // reads block of data of specified size,
>>   // throws ReadException on error
>>   void readExact(void* buffer, uint size)
>>   {
>>     uint readsize = readBlock(buffer, size);
>>     if (readsize != size)
>>       throw new ReadException("not enough data in stream");
>>   }
>
> I think it's a flawed design, that EOF is an Exception ?
>
> I'd rather have "read" return 'false' on EOF, myself.
> And save the exceptions for the I/O errors and such...

What if you're trying to read an 'int' i.e. 4 bytes and there is only 1, 2 or 3 bytes available?
(as it might be doing above)

Regan

Regan Heath wrote:

>> I think it's a flawed design, that EOF is an Exception ?
>>
>> I'd rather have "read" return 'false' on EOF, myself.
>> And save the exceptions for the I/O errors and such...
> 
> What if you're trying to read an 'int' i.e. 4 bytes and there is only 1, 2  or 3 bytes available?
> (as it might be doing above)

Actually, I was only thinking about bytes and lines...

You might be right. Either way, that's how it works now.

--anders

"Anders F Björklund" <afb@algonet.se> wrote in message news:d0jsiv$1vrf$1@digitaldaemon.com...
> Regan Heath wrote:
>
>>> Error: not enough data in stream
>>
>> I cannot find the string "not enough data in stream" in the phobos source.. is that your own error message?
>
> It's in std.stream:
>>   // reads block of data of specified size,
>>   // throws ReadException on error
>>   void readExact(void* buffer, uint size)
>>   {
>>     uint readsize = readBlock(buffer, size);
>>     if (readsize != size)
>>       throw new ReadException("not enough data in stream");
>>   }
>
> I think it's a flawed design, that EOF is an Exception ?
>
> I'd rather have "read" return 'false' on EOF, myself.
> And save the exceptions for the I/O errors and such...
>
> --anders

Hence the name "readExact". The function "readBlock", as you could guess from the pasted snippet, returns the number of bytes read. The reason this is exceptional is that by calling readExact the coder is asserting there are enough bytes in the stream to complete the read. If the stream closes prematurely there won't be enough bytes so it throws.

> But, I think I figured it out.  The server from where the XML data is
> coming
> from may have had a cap on its maximum data output.  It was set to 128MB,
> so I
> raised it to 1GB and now the error has disappeared, but the user which had
> this
> problem previously is still downloading xml data and it's up to 500+MB!
> YIKES!
> This is bad design at its best. :-)

I don't understand - is there a bug in std.stream? I can't tell from that paragraph.

Ben Hinkle says...
>
>
>> But, I think I figured it out.  The server from where the XML data is
>> coming
>> from may have had a cap on its maximum data output.  It was set to 128MB,
>> so I
>> raised it to 1GB and now the error has disappeared, but the user which had
>> this
>> problem previously is still downloading xml data and it's up to 500+MB!
>> YIKES!
>> This is bad design at its best. :-)
>
>I don't understand - is there a bug in std.stream? I can't tell from that paragraph.

Sorry.  typing and thinking at the same time has not been one of my strongest showings ever! :-)

No, there is no bug on this post.  The problem was caused by the server's software where I was getting the data from, which, by the way, is written in java.  I was trying to acquire data from that server using d.   D acted accordingly, so the problem was java memory cap setting.  The cap was set for 128MB of memory.  The amount of XML that I was downloading exceeded that amount and I kept getting that "Error: not enough data in stream."  After raising the memory cap to 1024MB, the problem went away.

Apparently, d was reading socket.datastream from the server ok, but when the amount of data from the server exceeded 128MB, the server stopped sending data and d aborted with "Error: not enough data in stream."  This confused me a bit and I thought that it was something on my side.  (Perhaps we should change that error.)  But, this only meant that no data at all and the server closed the socket connection.  It didn't even send any http headers or anything.  Weird. It just returned a null string.  Go figured.  Of course, the server also went down hard. :-)  So, that's how the West was won.

Or something like that.

So, I am ok and you should ignore this post.  But, perhaps the error is a little confusing, since I thought it was my problem, when it really was the other side.

Thanks.

jic