I think the D streams provide a good base but are missing some of the more complex features that I miss from C++, mostly to do with formatted i/o.  This raises the rather large issue of localization which I'm going to ignore for the moment.

As things stand, the basic D stream framework has an InputStream and OutputStream interface and then a single Stream class that implements both of these.  File, MemoryStream, SliceStream, and BufferedStream all inherit from Stream, while BufferedFile inherits from BufferedStream.  Stream and File do not buffer by default, BufferedStream acts as an adaptor class, and BufferedFile is the buffered version of the File stream class.  Currently, all formatted i/o goes through the printf method, with the read/write methods offering unformatted i/o.  There is also an OutBuffer class (though no corresponding InBuffer) which seems functionally similar to MemoryStream.

I'd like separate InputStream, OutputStream, and Stream base classes to allow support for streams where only input or output makes sense.  To make it clear these are base classes and prevent naming problems with the interfaces I propose these names: InputStreamBase, OutputStreamBase, and StreamBase (or InputOutputStreamBase, though that's pretty darn verbose).  Because of the single inheritance rule, StreamBase won't be able to inherit from InputStreamBase and OutputStreamBase which will liekly result in some toying with the design to avoid code duplication.  These classes will be obviously be abstract base classes.  If I wanted to nit-pick I might suggest renaming File to FileStream for consistency.

In addition to the existing functionality I propose new formatted i/o methods for all types that already have unformatted methods.  Apart from a few instances these can forward most of the work to the existing printf method.  The default methods would be "get" and "put" with the possibility of adding overloaded operator support in the future.  I'd also like to see something similar to the callback mechanism in C++ streams so arbitrary classes can hook in formatting information.  And Jill had mentioned wanting something similar to readsome() from C++ streams so I suspect this list will grow a bit.

As for open issues, someone with more experience than myself should probably address localization.  At the very least, there should be some support for proper numeric formatting, but that raises the issue of how formatting rules will be supplied to the stream classes and what else the localization rules should cover.

I'm willing to go and do this myself, but as it's a modification of existing Phobos stuff I wanted to get some feedback.  Does anyone have any objections or suggestions?  Assuming the design doesn't suck will it have a chance for inclusion?


Sean

Sean Kelly wrote:

> I think the D streams provide a good base but are missing some of the more complex features that I miss from C++, mostly to do with formatted i/o. This raises the rather large issue of localization which I'm going to ignore for the moment.
> 
> As things stand, the basic D stream framework has an InputStream and
> OutputStream interface and then a single Stream class that implements both
> of
> these.  File, MemoryStream, SliceStream, and BufferedStream all inherit
> from
> Stream, while BufferedFile inherits from BufferedStream.  Stream and File
> do not buffer by default, BufferedStream acts as an adaptor class, and
> BufferedFile is
> the buffered version of the File stream class.  Currently, all formatted
> i/o goes through the printf method, with the read/write methods offering
> unformatted
> i/o.  There is also an OutBuffer class (though no corresponding InBuffer)
> which seems functionally similar to MemoryStream.
> 
> I'd like separate InputStream, OutputStream, and Stream base classes to
> allow
> support for streams where only input or output makes sense.  To make it
> clear these are base classes and prevent naming problems with the
> interfaces I propose these names: InputStreamBase, OutputStreamBase, and
> StreamBase (or
> InputOutputStreamBase, though that's pretty darn verbose).  Because of the
> single inheritance rule, StreamBase won't be able to inherit from
> InputStreamBase and OutputStreamBase which will liekly result in some
> toying
> with the design to avoid code duplication.  These classes will be
> obviously be
> abstract base classes.  If I wanted to nit-pick I might suggest renaming
> File to FileStream for consistency.
> 
> In addition to the existing functionality I propose new formatted i/o
> methods
> for all types that already have unformatted methods.  Apart from a few
> instances
> these can forward most of the work to the existing printf method.  The
> default methods would be "get" and "put" with the possibility of adding
> overloaded
> operator support in the future.  I'd also like to see something similar to
> the callback mechanism in C++ streams so arbitrary classes can hook in
> formatting
> information.  And Jill had mentioned wanting something similar to
> readsome() from C++ streams so I suspect this list will grow a bit.
> 
> As for open issues, someone with more experience than myself should
> probably
> address localization.  At the very least, there should be some support for
> proper numeric formatting, but that raises the issue of how formatting
> rules will be supplied to the stream classes and what else the
> localization rules should cover.
> 
> I'm willing to go and do this myself, but as it's a modification of
> existing
> Phobos stuff I wanted to get some feedback.  Does anyone have any
> objections or
> suggestions?  Assuming the design doesn't suck will it have a chance for
> inclusion?
> 
> 
> Sean

It would be nice to have mixins DefaultInputStream and DefaultOutputStream
that would contain the default implementations from Stream and then Stream
would be a mixin of those and the seeking and other odds and ends. That
would make it easier to implement InputStream and OutputStream since users
would just have to supply readBlock and writeBlock, respectively. So for
example to define a FooInputStream
class FooInputStream : InputStream {
  mixin DefaultInputStream; // ultimately call readBlock
  uint readBlock(...) {...}
  ... whatever else ...
}

In article <ca82sq$u9n$1@digitaldaemon.com>, Sean Kelly says...

>As things stand, the basic D stream framework has an InputStream and OutputStream interface and then a single Stream class that implements both of these.  File, MemoryStream, SliceStream, and BufferedStream all inherit from Stream, while BufferedFile inherits from BufferedStream.  Stream and File do not buffer by default, BufferedStream acts as an adaptor class, and BufferedFile is the buffered version of the File stream class.  Currently, all formatted i/o goes through the printf method, with the read/write methods offering unformatted i/o.  There is also an OutBuffer class (though no corresponding InBuffer) which seems functionally similar to MemoryStream.

As things stand, the Phobos documentation mentions almost none of this. I didn't even know that file streams were unbuffered, or that the class BufferedStream even existed, until a few days ago when someone told me.


Anyway, I'd want a function available() to be available in input streams. (Same
as Java's java.io.InputStream). The implementation of this is /childs play/. It
goes like this:

In InputStream and Stream
>    abstract uint available();

In unbuffered streams:
>   override uint available() { return 0; }

In buffered streams:
>   override uint available()
>   {
>       return numBytesRemainingInBuffer + s.available();
>   }

where s is the underlying stream. (This is likely to be itself unbuffered, and so it's available() function is likely to return zero - but there will always be special cases, so it's best to assume not).

As one special case, for example a string input stream could have available() return the remaining length of the string.

In any case, the most important thing about available() is that it MUST NOT BLOCK. It doesn't matter if it underestimates, but it must never overestimate.


Arcane Jill

In article <ca90hu$2bgd$1@digitaldaemon.com>, Arcane Jill says...
>
>As things stand, the Phobos documentation mentions almost none of this. I didn't even know that file streams were unbuffered, or that the class BufferedStream even existed, until a few days ago when someone told me.

At this point, I think it's still better to just look at the Phobos source code. There's a lot of stuff in there that's largely undocumented.

>Anyway, I'd want a function available() to be available in input streams. (Same
>as Java's java.io.InputStream). The implementation of this is /childs play/. It
>goes like this:
>
>In InputStream and Stream
>>    abstract uint available();
>
>In unbuffered streams:
>>   override uint available() { return 0; }
>
>In buffered streams:
>>   override uint available()
>>   {
>>       return numBytesRemainingInBuffer + s.available();
>>   }
>
>where s is the underlying stream. (This is likely to be itself unbuffered, and so it's available() function is likely to return zero - but there will always be special cases, so it's best to assume not).
>
>As one special case, for example a string input stream could have available() return the remaining length of the string.
>
>In any case, the most important thing about available() is that it MUST NOT BLOCK. It doesn't matter if it underestimates, but it must never overestimate.

I was planning on adding available() to BufferedStream only, as it doesn't seem to make sense for an unbuffered stream class.  I guess the alternative would be to document rules that unbuffered classes could follow, like return 0 if they don't know and -1 if at the end of file.

Sean

In article <ca8kcl$1nvb$1@digitaldaemon.com>, Ben Hinkle says...
>
>It would be nice to have mixins DefaultInputStream and DefaultOutputStream that would contain the default implementations from Stream and then Stream would be a mixin of those and the seeking and other odds and ends.

What a wonderful idea :)  I knew those mixins would come in handy for something!

I tend to develop ideas by coding so I've gone ahead and started the modifications.  I decided to refactor the design a bit so the streams won't throw exceptions on eof() or on read errors.  At the moment I'm going with having the streams castable to bit as an easy way to check state, much the same way C++ streams work.  Otherwise the existing methods are staying pretty much as-is.  I'll post more in a few days once I get more of the grunt work done.

One question about character output.  The docs say dchar is a 32-bit unicode character but I'm pretty sure that it's currently implemented as wchar_t in Windows right now (which would make it compatible with wchar).  Is there any reason there shouldn't be methods to operate on all three char types: char, wchar, and dchar?


Sean

On Thu, 10 Jun 2004 23:35:52 +0000 (UTC), Sean Kelly <sean@f4.ca>
wrote:

>In article <ca8kcl$1nvb$1@digitaldaemon.com>, Ben Hinkle says...
>>
>>It would be nice to have mixins DefaultInputStream and DefaultOutputStream that would contain the default implementations from Stream and then Stream would be a mixin of those and the seeking and other odds and ends.
>
>What a wonderful idea :)  I knew those mixins would come in handy for something!
>
>I tend to develop ideas by coding so I've gone ahead and started the modifications.

cool. That's usually the best way to gives ideas focus.

>I decided to refactor the design a bit so the streams won't throw exceptions on eof() or on read errors.

Are you talking about the ReadError in readExact? I think that error should stay since readBlock returns the number of bytes read if there isn't enough data. If one asks to read exactly 10 bytes and that is impossible it should throw an error.

>At the moment I'm going with
>having the streams castable to bit as an easy way to check state, much the same
>way C++ streams work.

Two things come to mind:
1) Can opCast be implicit? The DMD spec says "Overloading the cast
operator does not affect implicit casts, it only applies to explicit
casts." Since writing cast(bit)foo isn't more readable than foo.eof()
I'd stay away from opCast for bit.
2) if (foo) ... for any other object checks that foo is non-null.
Making it anything else would be wierd.

>Otherwise the existing methods are staying pretty much
>as-is.  I'll post more in a few days once I get more of the grunt work done.

If you have time and/or desire check out the MmFile wrapper for
streams that I mentioned on digitalmars.D.bugs available at
http://home.comcast.net/~benhinkle/stream.d
Down near the bottom there is ArrayStream(Buffer) that wraps
"array-like" types with a stream API.

>One question about character output.  The docs say dchar is a 32-bit unicode character but I'm pretty sure that it's currently implemented as wchar_t in Windows right now (which would make it compatible with wchar).  Is there any reason there shouldn't be methods to operate on all three char types: char, wchar, and dchar?

I wondered that, too, when I was poking around. It could be that stream.d came before dchar? I don't know. I haven't thought very much about it but adding dchar support seems reasonable to me. I can't think of any big reason not to.

>
>Sean
>

In article <3i7ic0dlu2mdmb8q47bmvro8n4og30jvad@4ax.com>, Ben Hinkle says...
>
>On Thu, 10 Jun 2004 23:35:52 +0000 (UTC), Sean Kelly <sean@f4.ca>
>wrote:
>
>>I decided to refactor the design a bit so the streams won't throw exceptions on eof() or on read errors.
>
>Are you talking about the ReadError in readExact? I think that error should stay since readBlock returns the number of bytes read if there isn't enough data. If one asks to read exactly 10 bytes and that is impossible it should throw an error.

What bothered me about the exception throwing in Stream was that since all methods are implemented in terms of readExact, an exception will always be throw when eof() is reached.  In my opinion, end of file is an expected rather than an exceptional condition.  If I had to wrap all my stream operations in try blocks I'd go nuts :)

>>At the moment I'm going with
>>having the streams castable to bit as an easy way to check state, much the same
>>way C++ streams work.
>
>Two things come to mind:
>1) Can opCast be implicit? The DMD spec says "Overloading the cast
>operator does not affect implicit casts, it only applies to explicit
>casts." Since writing cast(bit)foo isn't more readable than foo.eof()
>I'd stay away from opCast for bit.

Darn.  I'd forgotten about this.  And I so liked the C++ semantics:

while( istr.get(x) );

This would work if all operations returned a fail state, but it's much nicer to have them return the stream itself so operations can be chained.  I suppose this works however:

while( !istr.get(x).fail() );

and it is perhaps more clear.

>If you have time and/or desire check out the MmFile wrapper for
>streams that I mentioned on digitalmars.D.bugs available at
>http://home.comcast.net/~benhinkle/stream.d
>Down near the bottom there is ArrayStream(Buffer) that wraps
>"array-like" types with a stream API.

Will do.

I have to say, this lack of multiple inheritance has been interesting to deal with.  As both InputStream and OutputStream share some common features (state tracking) it made sense to have them both inherit from a common interface.  But as the IOStream class was to incorporate stuff from both of them this meant a diamond-shaped inheritance tree, which I think tosses the idea of having only two mixins.  I haven't tested it, but assuming this code:

class CommonBase
{
int x;
}


class DerivedA
{
mixin CommonBase;
}

template DerivedB
{
mixin CommonBase;
}

template Derived
{
mixin DerivedA;
mixin DerivedB;
}

Derived will end up with two instances of x defined, won't it?  Or perhaps a compiler error?  Or is the code intelligent enough to gurantee that the stuff in CommonBase is only imported once?


Sean

err... replace "template" with "class" in my example.

Sean

"Sean Kelly" <sean@f4.ca> wrote in message news:cacidg$1jn3$1@digitaldaemon.com...
> In article <3i7ic0dlu2mdmb8q47bmvro8n4og30jvad@4ax.com>, Ben Hinkle
says...
> >
> >On Thu, 10 Jun 2004 23:35:52 +0000 (UTC), Sean Kelly <sean@f4.ca>
> >wrote:
> >
> >>I decided to refactor the design a bit so the streams won't throw exceptions on eof() or on read errors.
> >
> >Are you talking about the ReadError in readExact? I think that error should stay since readBlock returns the number of bytes read if there isn't enough data. If one asks to read exactly 10 bytes and that is impossible it should throw an error.
>
> What bothered me about the exception throwing in Stream was that since all methods are implemented in terms of readExact, an exception will always be
throw
> when eof() is reached.  In my opinion, end of file is an expected rather
than an
> exceptional condition.  If I had to wrap all my stream operations in try
blocks
> I'd go nuts :)

Eof in the middle of reading an int (for example) shouldn't be expected,
though. If one is trying to read an int and you only get half an int and a
return value of false then every call to read(out int) will have to be
wrapped in "if" tests instead of try-catches. Reading multiple values would
look like
 if (!s.read(x1) || !s.read(x2) || !s.read(x3)) {evasive action}
instead of
 try{
  s.read(x1);
  s.read(x2);
  s.read(x3);
 } catch (ReadError re) { evasive action }

Either way I think eof should be the "usual" way of checking if any data remains.

[snip]

> I have to say, this lack of multiple inheritance has been interesting to
deal
> with.  As both InputStream and OutputStream share some common features
(state
> tracking) it made sense to have them both inherit from a common interface.
But
> as the IOStream class was to incorporate stuff from both of them this
meant a
> diamond-shaped inheritance tree, which I think tosses the idea of having
only
> two mixins.  I haven't tested it, but assuming this code:
>
> class CommonBase
> {
> int x;
> }
>
>
> class DerivedA
> {
> mixin CommonBase;
> }
>
> template DerivedB
> {
> mixin CommonBase;
> }
>
> template Derived
> {
> mixin DerivedA;
> mixin DerivedB;
> }
>
> Derived will end up with two instances of x defined, won't it?  Or perhaps
a
> compiler error?  Or is the code intelligent enough to gurantee that the
stuff in
> CommonBase is only imported once?

Is it possible to just have the mixins be the non-overlapping parts? I'm
looking at Stream class and the routines that implement InputStream only
call readBlock and eof to do their thing. I can't see any overlap with the
implementations of OutputStream. In other words, lines 219-796 (plus line
209 for "readable") form DefaultInputStream and lines 802-932 (plus 210)
form DefaultOutputStream. So I'm picturing
 interface InputStream{...}
 interface OutputStream {...}
 template DefaultInputStream() {...}
 template DefaultOutputStream() {...}
 class Stream {
    mixin DefaultInputStream;
    mixin DefaultOutputStream;
    ...
 }
That way everything is backwards compatible with the existing stream.d.
Users can't instantiate a DefaultInputStream by itself but that's ok.


> Sean

>  interface InputStream{...}
>  interface OutputStream {...}
>  template DefaultInputStream() {...}
>  template DefaultOutputStream() {...}
>  class Stream {
>     mixin DefaultInputStream;
>     mixin DefaultOutputStream;
>     ...
>  }

oops - I forgot that Stream wants to implement InputStream and OutputStream
  class Stream : InputStream, OutputStream {