Hi,

I need to determine in advance the number of bytes I can read from a stream without that stream blocking. Or, equivalently, receive a yes/no answer to the question "if I attempt to read n bytes from this stream, will it block?"

Is there any way of doing that with a std.stream.Stream ? (I don't mind if the answer is "Not yet, but I'll add that one at the next release").

Failing that, can anyone think of any other way of achieving this?

All offers of advice appreciated.
Jill

In article <c9tqup$9cg$1@digitaldaemon.com>, Arcane Jill says...
>
>Hi,
>
>I need to determine in advance the number of bytes I can read from a stream without that stream blocking. Or, equivalently, receive a yes/no answer to the question "if I attempt to read n bytes from this stream, will it block?"
>
>Is there any way of doing that with a std.stream.Stream ? (I don't mind if the answer is "Not yet, but I'll add that one at the next release").
>
>Failing that, can anyone think of any other way of achieving this?
>
>All offers of advice appreciated.
>Jill

I haven't tried this, but could you create a seperate thread that did a read, and once it got data, it could (under synchronization) put the results in a buffer?  Then your "main" thread could (under synch) check the buffer object.

You would still be doing a blocking read, but the main thread could stay out of the way.  Assuming that the blocking doesn't block the whole D process.

I would think you could also use the "select" or "poll" system calls if you are in a unix-like environment.

Kevin

In article <c9u86n$r8r$1@digitaldaemon.com>, Kevin Bealer says...
>
>I haven't tried this, but could you create a seperate thread that did a read, and once it got data, it could (under synchronization) put the results in a buffer?  Then your "main" thread could (under synch) check the buffer object.

A feasable workaround, though I'd hope it would only be temporary. Thanks for the idea.


>I would think you could also use the "select" or "poll" system calls if you are in a unix-like environment.

Ah, well you see, that's a problem. I like to write code that is as platform independent as possible. As it happens, I'm using Windows, but even if I weren't, it would irritate the hell out of me to have to write Linux-only code.

And in actual fact, that wouldn't work anyway - even on Linux. The reason is that a Stream doesn't have a filehandle. This is because not all Streams are built around files or sockets. A Stream is a generic concept. It might simply iterate through an array. Such a stream would have to block until more stuff got appended to the array. I need a solution which will work for ALL Streams, and ideally on all platforms.


Let me restate the question, because my need is actually less specific than I originally asked. All I *ACTUALLY* want is a yes/no answer to the following question: WILL THIS STREAM BLOCK ON THE NEXT CALL TO readBlock(...)?

There appears to be no way to find out under the existing API, but it strikes me it ought to be quite easy for the Phobos folk to add an API function to Stream which answered that very question. (Of course, it would have to be implemented by all subclasses of Stream, which might be a bit of a headache).

In any case, I suspect that, in time, I won't be the only person who needs to do this.

Arcane Jill

>>I would think you could also use the "select" or "poll" system calls if you are in a unix-like environment.

These work in win32 also.

>In any case, I suspect that, in time, I won't be the only person who needs to do this.

This sounds like a great idea, good call!  .I think trying to convince Ben (H)
is your best bet ;).

C

In article <c9usbu$1mcd$1@digitaldaemon.com>, Arcane Jill says...
>
>In article <c9u86n$r8r$1@digitaldaemon.com>, Kevin Bealer says...
>>
>>I haven't tried this, but could you create a seperate thread that did a read, and once it got data, it could (under synchronization) put the results in a buffer?  Then your "main" thread could (under synch) check the buffer object.
>
>A feasable workaround, though I'd hope it would only be temporary. Thanks for the idea.
>
>
>>I would think you could also use the "select" or "poll" system calls if you are in a unix-like environment.
>
>Ah, well you see, that's a problem. I like to write code that is as platform independent as possible. As it happens, I'm using Windows, but even if I weren't, it would irritate the hell out of me to have to write Linux-only code.
>
>And in actual fact, that wouldn't work anyway - even on Linux. The reason is that a Stream doesn't have a filehandle. This is because not all Streams are built around files or sockets. A Stream is a generic concept. It might simply iterate through an array. Such a stream would have to block until more stuff got appended to the array. I need a solution which will work for ALL Streams, and ideally on all platforms.
>
>
>Let me restate the question, because my need is actually less specific than I originally asked. All I *ACTUALLY* want is a yes/no answer to the following question: WILL THIS STREAM BLOCK ON THE NEXT CALL TO readBlock(...)?
>
>There appears to be no way to find out under the existing API, but it strikes me it ought to be quite easy for the Phobos folk to add an API function to Stream which answered that very question. (Of course, it would have to be implemented by all subclasses of Stream, which might be a bit of a headache).
>
>In any case, I suspect that, in time, I won't be the only person who needs to do this.
>
>Arcane Jill
>
>

[snip]
> Let me restate the question, because my need is actually less specific
than I
> originally asked. All I *ACTUALLY* want is a yes/no answer to the
following
> question: WILL THIS STREAM BLOCK ON THE NEXT CALL TO readBlock(...)?

In some sense readBlock by definition blocks until the data is read, so the
question should be "will the next readBlock block for more than xxx seconds"
which is a hard question to answer in any case.
Another possible solution would be to wrap the stream in a BufferedStream
and add a small function to BufferedStream that returns true if the buffer
contains xxx bytes from the current position.
Adding a general "will this block" function would be nice but I honestly
don't know what it would do to answer yes or no.

> There appears to be no way to find out under the existing API, but it
strikes me
> it ought to be quite easy for the Phobos folk to add an API function to
Stream
> which answered that very question. (Of course, it would have to be
implemented
> by all subclasses of Stream, which might be a bit of a headache).
>
> In any case, I suspect that, in time, I won't be the only person who needs
to do
> this.
>
> Arcane Jill
>
>

In article <ca2don$r85$1@digitaldaemon.com>, Ben Hinkle says...
>
>In some sense readBlock by definition blocks until the data is read, so the question should be "will the next readBlock block for more than xxx seconds"

Not so. I want to know "Will the next readBlock block AT ALL?". As in, in zero seconds.

>which is a hard question to answer in any case.

I think it's not only easy, but /trivially/ easy. I hope to explain in the course of this post.

>Another possible solution would be to wrap the stream in a BufferedStream

I think I'm in the wrong language. Java has BufferedInputStream. D does not. Or at least, Phobos does not. Of course it would be easy for me to write my own. I could even add it to Deimos.

>and add a small function to BufferedStream that returns true if the buffer contains xxx bytes from the current position.

To be honest, I had rather assumed that the class std.stream.File *WAS* buffered. Are you telling me it isn't? It's important that we know these things, because lots of read()s from an unbuffered stream is horribly inefficient, but adding buffering to an already buffered stream is ALSO horribly inefficient.

>Adding a general "will this block" function would be nice but I honestly don't know what it would do to answer yes or no.

I do. Java, as you probably know, has the abstract class InputStream. Now, InputStream has exactly this funtion. It's called available(). It is documented as, and I quote, "Returns the number of bytes that can be read (or skipped over) from this input stream without blocking by the next caller of a method for this input stream." Knowing HOW MANY bytes you can read is, of course, even better than knowing whether or not you will be able to read just one.

Implementation is child's play. In std.stream.Stream, you do this:

>       abstract ulong available();

In general, a non-buffered subclass would do this:

>       override ulong available() { return 0; }

In any buffered or filtering subclass, you do this:

>       override ulong available()
>       {
>           return numBytesRemainingInBufer + s.available();
>       }

(where s is the underlying Stream - the one being buffered). In the special case of std.stream.MemoryStream, you return the number of bytes to the end of the array (although arguably this is a special case of buffering).

If Java can do this, D should be able to do it to. We're supposed to be trying to make D /better/ than Java, not inferior to it in basic functionality. The above suggested implementation, plus the addition to Phobos (or Deimos) of the class BufferedStream, would do the job nicely.

Arcane Jill

In article <ca3h7h$2lll$1@digitaldaemon.com>, Arcane Jill says...
>
>In article <ca2don$r85$1@digitaldaemon.com>, Ben Hinkle says...
>>
>>Another possible solution would be to wrap the stream in a BufferedStream
>
>I think I'm in the wrong language. Java has BufferedInputStream. D does not. Or at least, Phobos does not. Of course it would be easy for me to write my own. I could even add it to Deimos.

Just user BufferedStream like Ben says.  I don't think it's mentioned in the docs, but the class does exist.

>To be honest, I had rather assumed that the class std.stream.File *WAS* buffered. Are you telling me it isn't? It's important that we know these things, because lots of read()s from an unbuffered stream is horribly inefficient, but adding buffering to an already buffered stream is ALSO horribly inefficient.

None of the base classes are buffered.  There is a BufferedStream class that adds buffering if you want it.


Sean

Might I suggest Mango.io instead of Phobos modules?  I've been pretty happy with this part of the Mango tree, even though there's a nice http client, server, and servlet container there as well.

I'm not sure about a count of how many characters you have left to read w/o blocking, but Kris is pretty good with avoiding frivolous memory allocations and inefficient code in general.  (Live up to that, Mr. Bell)

BA

http://mango.dsource.org or http://www.dsource.org/projects/mango




Sean Kelly wrote:
> In article <ca3h7h$2lll$1@digitaldaemon.com>, Arcane Jill says...
> 
>>In article <ca2don$r85$1@digitaldaemon.com>, Ben Hinkle says...
>>
>>>Another possible solution would be to wrap the stream in a BufferedStream
>>
>>I think I'm in the wrong language. Java has BufferedInputStream. D does not. Or
>>at least, Phobos does not. Of course it would be easy for me to write my own. I
>>could even add it to Deimos.
> 
> 
> Just user BufferedStream like Ben says.  I don't think it's mentioned in the
> docs, but the class does exist.
> 
> 
>>To be honest, I had rather assumed that the class std.stream.File *WAS*
>>buffered. Are you telling me it isn't? It's important that we know these things,
>>because lots of read()s from an unbuffered stream is horribly inefficient, but
>>adding buffering to an already buffered stream is ALSO horribly inefficient.
> 
> 
> None of the base classes are buffered.  There is a BufferedStream class that
> adds buffering if you want it.
> 
> 
> Sean
> 
> 

In article <ca3hvs$2mse$1@digitaldaemon.com>, Sean Kelly says...

>Just user BufferedStream like Ben says.  I don't think it's mentioned in the docs, but the class does exist.

Thanks for the info that D has BufferedStream. I didn't know that, and it really should be in the docs.

But as to the advice that I use it, I believe you have completely misunderstood my needs. I guess that doesn't matter though - thanks for trying to help anyway.

Jill

In article <ca3ia2$2nf5$1@digitaldaemon.com>, Brad Anderson says...
>
>Might I suggest Mango.io instead of Phobos modules?  I've been pretty happy with this part of the Mango tree, even though there's a nice http client, server, and servlet container there as well.

Everyone on this newsgroup is really nice, and I really do appreciate all these offers of help from people. (And I will look at mango as well). But, alas, you have also completely misunderstood my needs.

I want a function equivalent to Java's java.io.InputSream.available() added to
D's std.stream.Stream() and its subclasses. [Or alternatively, a willBlock()
function]. Nothing less will suffice.

If it can't be done, then it can't be done. End of story. Then I just move on to something else.

Arcane Jill