So I've been working on a small side project for the last few days, and I think that it's gotten to the point where I think that it's ready to be reviewed/critiqued.

The project is available on GitHub here: https://github.com/andrewlalis/streams

It introduces the concept of Streams, which is anything with either of the following function signatures:

• int read(T[] buffer) - this is an input stream.
• int write(T[] buffer) - this is an output stream.

The README.md on the project's homepage describes the motivation in more detail, but in short, I'm not 100% satisfied with Phobos' ranges, and I think that streams could be introduced as a lower-level primitive that's also more familiar to programmers coming from a variety of other languages, while still trying to be as idiomatically D as possible.

Just for the sake of demonstration, here's an example of using streams to transfer the contents of a file to some arbitrary output stream. Of course pretty much anything done with streams can also be done with ranges, but I think that the simpler interface will make some things more ergonomic.

import streams;

void readFileTo(S)(string filename, S stream) if (isOutputStream!(S, ubyte)) {
  import std.stdio;
  auto fIn = FileInputStream(filename);
  transferTo(fIn, stream);
}

So, I'd appreciate if anyone could take a look at my project, tell me if you think this is a good idea or not, if I should introduce a DIP for this change if added to Phobos (I know the DIP process is closed at the moment), or if you have any other feedback for this.

On Monday, 15 May 2023 at 09:53:22 UTC, Andrew wrote:

Thanks for sharing!

Probably will have a look later, just a couple of questions:

1. Does it support auto buffer for performance purposes (something like BufReader/BufWriter in other langs , for example https://zig.news/kristoff/how-to-add-buffering-to-a-writer-reader-in-zig-7jd)

2. while implementing have you consider to look into undead repo? https://github.com/dlang/undeaD/blob/master/src/undead/stream.d

3. will it be possible to connect it with things like Kafka?

On Monday, 15 May 2023 at 10:26:24 UTC, Sergey wrote:

I haven't added such a buffered wrapper type to this library yet, but now that I read that article, it seems entirely doable to add that to this implementation, so I'll do that shortly!

undead/stream.d is, as far as I can see, a purely OOP-style approach to IO streams, which looks like it's loosely inspired by Java/C#, while my streams implementation does offer an OOP interface (like Phobos does for ranges), the main goal is to use compile-time checks to let anything be a stream if it behaves like one.

Well, yes, that is possible, but I don't personally have much experience with Kafka's binary protocol. But generally, it should be rather trivial to translate existing implementations that use a similar IO approach to my proposed streams implementation.

On Monday, 15 May 2023 at 09:53:22 UTC, Andrew wrote:

import streams;

void readFileTo(S)(string filename, S stream) if (isOutputStream!(S, ubyte)) {
  import std.stdio;
  auto fIn = FileInputStream(filename);
  transferTo(fIn, stream);
}

How would this help me with say rendering video and enforcing frames are syncef? If T[] doesnt have any flexable logic?

On Wednesday, 17 May 2023 at 00:50:19 UTC, Monkyyy wrote:

I'm not really sure how you think that an IO stream library would help you particularly more than any other one would... that said, it would certainly be easier to enforce that frames are synced using this library than, say, Phobos ranges, because you can more gracefully handle stream errors without having to use exceptions/GC stuff.

Additionally, like Sergey suggested, I've added "buffered" streams, as decorators for any base stream, so that you could, for example, use a buffered input stream to read exactly as many bytes from a video stream as needed to fill a framebuffer (or something like that, I'm not familiar with video stuff).

On Wednesday, 17 May 2023 at 08:29:38 UTC, Andrew wrote:

On Wednesday, 17 May 2023 at 13:47:15 UTC, Monkyyy wrote:

Yes, I am reducing the expressiveness, but I think it's a good idea. Ranges aren't @nogc/betterC compatible, and they don't support giving extra context about how many items were written or read in a consistent way. Streams are also defining their primitive as anything implementing int readFromStream(T[] items) or int writeToStream(T[] items), or both.

I know it's mostly a matter of personal preference, but I think there is value in having the standard library use a restrictive interface, instead of duck-typing, since it'll (hopefully) be used all over the place.

On Wednesday, 17 May 2023 at 14:40:48 UTC, Andrew wrote:

What does duck typing have to do with nogc?

If you said writeToStream(T)(T items) and assumed the user would provide a T that defined opSlice, opIndex and a length why couldnt whatever systems have nogc somewhere in the pipeline that makes it werk

On Wednesday, 17 May 2023 at 14:53:04 UTC, monkyyy wrote:

Nothing; duck typing just results in code that's harder to read, and harder to reason about than restrictive code, usually.

I don't understand what you're trying to say here. Yes, the intention is that my library is @nogc compatible by default, and anyone can choose to make a stream that is or isn't @nogc compatible, and it'll work with the library.

On Monday, 15 May 2023 at 09:53:22 UTC, Andrew wrote:

First of all, thanks for investing your time into this!

Have got some questions:

1. Have you looked at the IOPipe library?
2. What are the main benefits over the existing Ranges concept? Say, given your example:

import streams;

void readFileTo(S)(string filename, S stream) if (isOutputStream!(S, ubyte)) {
    import std.stdio;
    auto fIn = FileInputStream(filename);
    transferTo(fIn, stream);
}

With ranges, this would look something like:

import std;

void readFileTo(S)(string filename, ref S stream) if (isOutputRange!(S, ubyte[])) {
    File(filename).byChunk().copy(stream);
}

Which is more or less the same.

3. In the README you write:

Phobos' concept of an Input Range relies on implicit buffering of results... This doesn't map as easily to many low-level resources

AFAIK, the read/write buffers are anywhere, except, probably, sendfile() and some combination of mmap and write. But I'm struggling to get how this streams concept maps onto sendfile as well.