Typical stream implementations I have seen use the following heirarchy:

class Stream {
}
class FileStream : Stream {
}
class SocketStream : Stream {
}

in other words you have to design/derive a class from Stream for each type of Stream you want to implement.

I always thought this was kinda backwards, and limited, in fact after Arcane Jill said something along the lines of "anything that reads and writes bytes _is_ a stream" I got to thinking surely I can use templates to make that statement true.


The way I have done it is to define some interfaces:

interface InputStream
{
	ulong read(void* buffer, ulong length);
	ulong available();
	bool eof();
}

interface OutputStream
{
	ulong write(void* buffer, ulong length);
}

and a Stream and BufferedStream template class. Any object that supports those interfaces (as does my RawFile class) can then be extended by the two template classes to give full stream functionality by some simple aliases, example:

alias Stream!(RawFile) FileStream;
alias BufferedStream!(FileStream,2048) File;

I have attached the code, it is far from complete and probably buggy as heck as I have not done any testing (or unittests - I know I'm bad) this was simply proof of concept.

Any and all comments are welcome/desired.

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

"Regan Heath" <regan@netwin.co.nz> wrote in message news:opsb4onie15a2sq9@digitalmars.com...
> Typical stream implementations I have seen use the following heirarchy:
>
> class Stream {
> }
> class FileStream : Stream {
> }
> class SocketStream : Stream {
> }
>
> in other words you have to design/derive a class from Stream for each type of Stream you want to implement.
>
> I always thought this was kinda backwards, and limited, in fact after Arcane Jill said something along the lines of "anything that reads and writes bytes _is_ a stream" I got to thinking surely I can use templates to make that statement true.
>
>
> The way I have done it is to define some interfaces:
>
> interface InputStream
> {
> ulong read(void* buffer, ulong length);
> ulong available();
> bool eof();
> }
>
> interface OutputStream
> {
> ulong write(void* buffer, ulong length);
> }
>
> and a Stream and BufferedStream template class. Any object that supports those interfaces (as does my RawFile class) can then be extended by the two template classes to give full stream functionality by some simple aliases, example:
>
> alias Stream!(RawFile) FileStream;
> alias BufferedStream!(FileStream,2048) File;
>
> I have attached the code, it is far from complete and probably buggy as heck as I have not done any testing (or unittests - I know I'm bad) this was simply proof of concept.
>
> Any and all comments are welcome/desired.

They are remarkably similar to two interfaces - IInputStream & IOutputStream - that I've recently written for a client's networking project.

I can't include the code, of course. ;)

I agree with the general principle you outline. IStream is one of the elegant parts of COM. (IStorage one of the hideous ones.<g>).

Regan Heath wrote:

> Typical stream implementations I have seen use the following heirarchy:
> 
> class Stream {
> }
> class FileStream : Stream {
> }
> class SocketStream : Stream {
> }
> 
> in other words you have to design/derive a class from Stream for each type of Stream you want to implement.
> 
> I always thought this was kinda backwards, and limited, in fact after Arcane Jill said something along the lines of "anything that reads and writes bytes _is_ a stream" I got to thinking surely I can use templates to make that statement true.
> 
> 
> The way I have done it is to define some interfaces:
> 
> interface InputStream
> {
>     ulong read(void* buffer, ulong length);
>     ulong available();
>     bool eof();
> }
> 
> interface OutputStream
> {
>     ulong write(void* buffer, ulong length);
> }
> 
> and a Stream and BufferedStream template class. Any object that supports those interfaces (as does my RawFile class) can then be extended by the two template classes to give full stream functionality by some simple aliases, example:
> 
> alias Stream!(RawFile) FileStream;
> alias BufferedStream!(FileStream,2048) File;
> 
> I have attached the code, it is far from complete and probably buggy as heck as I have not done any testing (or unittests - I know I'm bad) this was simply proof of concept.
> 
> Any and all comments are welcome/desired.
> 
> Regan

I am not entirely clear on how what you are doing is different from the typical stream implementations you have seen.

Look through the thread "OT scanf in Java" for a long post of mine towards the top that explains why the Stream implementations that you think of as being backwards actually do exactly what you want.

To give a hint you should ask yourself how you can combine these streams:

(--Stream means InputStream or OutputStream)

File--Stream
Network--Stream
Memory--Stream

Buffered--Stream
Compress--Stream
CRC32 Digest--Stream (eg CRC32 or MD5)

Image--Stream
Video--Stream
Audio--Stream


Sometimes I will want my data to come from a File, other times I want it to come from the Network. Sometimes I want a digest of the compressed data other times I want a digest of the uncompressed data.

These actions are performed in the order I want based on the order in which I construct them:

new ImageStream( DigestStream( CompressStream ( FileStream ) ) )

This would read file data, decompresses the data and then compute the CRC32 of the uncompressed data and read image data from it. However:

new ImageStream( CompressStream ( DigestStream( FileStream ) ) )

This would read file data, compute the CRC32 of the uncompressed data and then decompress the data and read image data from it.


I suspect the class you want to implement will either duplicate the way Stream libraries work or you will find yourself writing seperate classes to handle the two cases above which is bad. Just try to write out all the possible combinations that these 9 classes have already accomplished and you will see what I mean.

On Mon, 02 Aug 2004 20:21:28 -0400, parabolis <parabolis@softhome.net> wrote:
> Regan Heath wrote:
>
>> Typical stream implementations I have seen use the following heirarchy:
>>
>> class Stream {
>> }
>> class FileStream : Stream {
>> }
>> class SocketStream : Stream {
>> }
>>
>> in other words you have to design/derive a class from Stream for each type of Stream you want to implement.
>>
>> I always thought this was kinda backwards, and limited, in fact after Arcane Jill said something along the lines of "anything that reads and writes bytes _is_ a stream" I got to thinking surely I can use templates to make that statement true.
>>
>>
>> The way I have done it is to define some interfaces:
>>
>> interface InputStream
>> {
>>     ulong read(void* buffer, ulong length);
>>     ulong available();
>>     bool eof();
>> }
>>
>> interface OutputStream
>> {
>>     ulong write(void* buffer, ulong length);
>> }
>>
>> and a Stream and BufferedStream template class. Any object that supports those interfaces (as does my RawFile class) can then be extended by the two template classes to give full stream functionality by some simple aliases, example:
>>
>> alias Stream!(RawFile) FileStream;
>> alias BufferedStream!(FileStream,2048) File;
>>
>> I have attached the code, it is far from complete and probably buggy as heck as I have not done any testing (or unittests - I know I'm bad) this was simply proof of concept.
>>
>> Any and all comments are welcome/desired.
>>
>> Regan
>
> I am not entirely clear on how what you are doing is different from the typical stream implementations you have seen.

I have written the code to read different types etc once in my Stream template, and can apply it to _any_ class at all that implements one read method of the form:

  ulong read(void* buffer, ulong length);

the same applies to writing.


So, if you added methods in the forms given in my interfaces to std.socket you could say:

alias Stream!(Socket) SocketStream;
alias BufferedStream!(SocketStream,2048) BufferedSocket;

and you have a socket you can instantiate like so:

  SocketStream s = new SocketStream();

which is a buffered stream.

> Look through the thread "OT scanf in Java" for a long post of mine towards the top that explains why the Stream implementations that you think of as being backwards actually do exactly what you want.
>
> To give a hint you should ask yourself how you can combine these streams:

Combination is the next step, what I have so far is not intended to handle combinations.

> (--Stream means InputStream or OutputStream)
>
> File--Stream
> Network--Stream
> Memory--Stream
>
> Buffered--Stream
> Compress--Stream
> CRC32 Digest--Stream (eg CRC32 or MD5)
>
> Image--Stream
> Video--Stream
> Audio--Stream
>
>
> Sometimes I will want my data to come from a File, other times I want it to come from the Network. Sometimes I want a digest of the compressed data other times I want a digest of the uncompressed data.
>
> These actions are performed in the order I want based on the order in which I construct them:
>
> new ImageStream( DigestStream( CompressStream ( FileStream ) ) )
>
> This would read file data, decompresses the data and then compute the CRC32 of the uncompressed data and read image data from it. However:
>
> new ImageStream( CompressStream ( DigestStream( FileStream ) ) )
>
> This would read file data, compute the CRC32 of the uncompressed data and then decompress the data and read image data from it.

My next step is to implement what I call 'filters' they are like your CompressStream etc above, they will link one stream to another and filter the data passing through them.

> I suspect the class you want to implement will either duplicate the way Stream libraries work or you will find yourself writing seperate classes to handle the two cases above which is bad. Just try to write out all the possible combinations that these 9 classes have already accomplished and you will see what I mean.

As I said above, combinations are my next step. I may end up handling them in the same way as you have shown above but I'd rather not, I dislike instantiation in the form:

  new ImageStream( CompressStream ( DigestStream( FileStream ) ) )

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

Regan Heath wrote:

> As I said above, combinations are my next step. I may end up handling them in the same way as you have shown above but I'd rather not, I dislike instantiation in the form:
> 
>   new ImageStream( CompressStream ( DigestStream( FileStream ) ) )

What will your eq. version of the above be?

On Mon, 02 Aug 2004 21:00:57 -0400, parabolis <parabolis@softhome.net> wrote:

> Regan Heath wrote:
>
>> As I said above, combinations are my next step. I may end up handling them in the same way as you have shown above but I'd rather not, I dislike instantiation in the form:
>>
>>   new ImageStream( CompressStream ( DigestStream( FileStream ) ) )
>
> What will your eq. version of the above be?

I'm not at all sure yet.

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

Regan Heath wrote:

> Typical stream implementations I have seen use the following heirarchy:
> 
> class Stream {
> }
> class FileStream : Stream {
> }
> class SocketStream : Stream {
> }
> 
> in other words you have to design/derive a class from Stream for each type of Stream you want to implement.
> 
> I always thought this was kinda backwards, and limited, in fact after Arcane Jill said something along the lines of "anything that reads and writes bytes _is_ a stream" I got to thinking surely I can use templates to make that statement true.
> 
> 
> The way I have done it is to define some interfaces:
> 
> interface InputStream
> {
> ulong read(void* buffer, ulong length);
> ulong available();
> bool eof();
> }
> 
> interface OutputStream
> {
> ulong write(void* buffer, ulong length);
> }
> 
> and a Stream and BufferedStream template class. Any object that supports those interfaces (as does my RawFile class) can then be extended by the two template classes to give full stream functionality by some simple aliases, example:
> 
> alias Stream!(RawFile) FileStream;
> alias BufferedStream!(FileStream,2048) File;
> 
> I have attached the code, it is far from complete and probably buggy as heck as I have not done any testing (or unittests - I know I'm bad) this was simply proof of concept.
> 
> Any and all comments are welcome/desired.
> 
> Regan
> 

cool idea(s). I like the "bolt-in" (or it bolt-on?) stuff. One possible
problem, though, is that if read(out int x), for example, is in
Stream!(File) and it call super.read(...) then the
BufferedStream!(FileStream) subclass will not read ints from the buffer
(the super.read goes directly to RawFile.read - not BufferedStream.read). I
guess I'm a little lost about how the buffering works.

At one point there was the suggestion of moving the default implementations
of all the read(out int x) etc from Stream (the original one in
std/stream.d) to a mixin so that instead of subclassing to get the default
implementation you mix it in. I don't know how that approach would change
around your code but it's worth thinking about.
To be more concrete, here's how that would work:
template DefaultInputStream() {
  void read(out int x) {readExact(...);}
  char[] readLine() {...}
  ...
}
template DefaultOutputStream() {
  void write(int x) {...}
  void writeLine(char[] x) {...}
  ...
}
class Stream : InputStream, OutputStream {
 mixin DefaultInputStream;
 mixin DefaultOutputStream;
 ...
}

Anyhow, keep pluggin'.

-Ben

On Mon, 02 Aug 2004 22:19:19 -0400, Ben Hinkle <bhinkle4@juno.com> wrote:
> Regan Heath wrote:
>
>> Typical stream implementations I have seen use the following heirarchy:
>>
>> class Stream {
>> }
>> class FileStream : Stream {
>> }
>> class SocketStream : Stream {
>> }
>>
>> in other words you have to design/derive a class from Stream for each type
>> of Stream you want to implement.
>>
>> I always thought this was kinda backwards, and limited, in fact after
>> Arcane Jill said something along the lines of "anything that reads and
>> writes bytes _is_ a stream" I got to thinking surely I can use templates
>> to make that statement true.
>>
>>
>> The way I have done it is to define some interfaces:
>>
>> interface InputStream
>> {
>> ulong read(void* buffer, ulong length);
>> ulong available();
>> bool eof();
>> }
>>
>> interface OutputStream
>> {
>> ulong write(void* buffer, ulong length);
>> }
>>
>> and a Stream and BufferedStream template class. Any object that supports
>> those interfaces (as does my RawFile class) can then be extended by the
>> two template classes to give full stream functionality by some simple
>> aliases, example:
>>
>> alias Stream!(RawFile) FileStream;
>> alias BufferedStream!(FileStream,2048) File;
>>
>> I have attached the code, it is far from complete and probably buggy as
>> heck as I have not done any testing (or unittests - I know I'm bad) this
>> was simply proof of concept.
>>
>> Any and all comments are welcome/desired.
>>
>> Regan
>>
>
> cool idea(s). I like the "bolt-in" (or it bolt-on?) stuff.

Yeah, I think they're a great idea.

> One possible
> problem, though, is that if read(out int x), for example, is in
> Stream!(File) and it call super.read(...) then the
> BufferedStream!(FileStream) subclass will not read ints from the buffer
> (the super.read goes directly to RawFile.read - not BufferedStream.read). I
> guess I'm a little lost about how the buffering works.

You're right. :(

I think to fix it I change:

  alias Stream!(RawFile) FileStream;
  alias BufferedStream!(FileStream,2048) File;

to

  alias BufferedStream!(RawFile,2048) File;
  alias Stream!(File) FileStream;

and then change the signatures of the methods in BufferedStream from

  ulong read(ubyte[] data)

to

  ulong read(void* buffer, ulong length)

etc.

This highlights an 'inclusion order' weakness to this design which needs to be fixed.

> At one point there was the suggestion of moving the default implementations
> of all the read(out int x) etc from Stream (the original one in
> std/stream.d) to a mixin so that instead of subclassing to get the default
> implementation you mix it in. I don't know how that approach would change
> around your code but it's worth thinking about.

I think this might just be the solution to the inclusion order weakness in my current design.
I'm going to turn my Stream template class into a pair of mixins.

> To be more concrete, here's how that would work:
> template DefaultInputStream() {
>   void read(out int x) {readExact(...);}
>   char[] readLine() {...}
>   ...
> }
> template DefaultOutputStream() {
>   void write(int x) {...}
>   void writeLine(char[] x) {...}
>   ...
> }
> class Stream : InputStream, OutputStream {
>  mixin DefaultInputStream;
>  mixin DefaultOutputStream;
>  ...
> }
>
> Anyhow, keep pluggin'.

Thanks. I'll post version 2 when I have one.

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

> and then change the signatures of the methods in BufferedStream from
>
>    ulong read(ubyte[] data)
>
> to
>
>    ulong read(void* buffer, ulong length)
>
> etc.
>

Why not use void[] for general read and write?

On Tue, 3 Aug 2004 00:19:15 -0400, Vathix <vathixSpamFix@dprogramming.com> wrote:
>> and then change the signatures of the methods in BufferedStream from
>>
>>    ulong read(ubyte[] data)
>>
>> to
>>
>>    ulong read(void* buffer, ulong length)
>>
>> etc.
>>
>
> Why not use void[] for general read and write?

So you can go:

bool read(out byte x)	{ if (this.read(&x,x.sizeof) ..etc..
bool read(out ubyte x)	{ if (this.read(&x,x.sizeof) ..etc..
bool read(out short x)	{ if (this.read(&x,x.sizeof) ..etc..

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/