I'd like to start utilizing IOPipe[1] more. Right now I have an interest in utilizing it for buffering output (actually I don't have a need for buffering, I just want to utilize iopipe)

Looking through some different examples[2][3] I thought I would have something with this:

```dlang
/+ dub.sdl:
    name "iobuftofile"
    dependency "iopipe" version="~>0.1.7"
    dependency "io" version="~>0.2.4"
+/

void main() {
    import iopipe.valve;
    import iopipe.textpipe;
    import iopipe.bufpipe;
    import iopipe.buffer;
    import std.io;
    import std.typecons;
    auto output() {
        return std.io.File("somefile.txt", mode!"w").refCounted;
    }
    auto outputBuffered = bufd!char
        .outputPipe(output());
}
```

1. https://code.dlang.org/packages/iopipe
2. https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d#L13
3. https://youtu.be/9fzttyj4JCs?t=1210

Just as I'm hitting send the part I'm missing clicked:

I needed to add the text encoding because my buffer is `char` but File writes `ubyte`

```dlang
/+ dub.sdl:
    name "iobuftofile"
    dependency "iopipe" version="~>0.1.7"
    dependency "io" version="~>0.2.4"
+/

void main() {
    import iopipe.valve;
    import iopipe.textpipe;
    import iopipe.bufpipe;
    import iopipe.buffer;
    import std.io;
    import std.typecons;
    auto output() {
        return std.io.File("somefile.txt", mode!"w").refCounted;
    }
    auto outputBuffered = bufd!char
        .encodeText // Missing This part
        .outputPipe(output());
}
```

On Monday, 27 January 2020 at 01:50:00 UTC, Jesse Phillips wrote:
> Just as I'm hitting send the part I'm missing clicked:
>
> I needed to add the text encoding because my buffer is `char` but File writes `ubyte`
>
> ```dlang
>     auto output() {
>         return std.io.File("somefile.txt", mode!"w").refCounted;
>     }
>     auto outputBuffered = bufd!char
>         .encodeText // Missing This part
>         .outputPipe(output());
> }
> ```

Unfortunately this did not write a text file as I expected.

On 1/26/20 11:59 PM, Jesse Phillips wrote:
> On Monday, 27 January 2020 at 01:50:00 UTC, Jesse Phillips wrote:
>> Just as I'm hitting send the part I'm missing clicked:
>>
>> I needed to add the text encoding because my buffer is `char` but File writes `ubyte`
>>
>> ```dlang
>>     auto output() {
>>         return std.io.File("somefile.txt", mode!"w").refCounted;
>>     }
>>     auto outputBuffered = bufd!char
>>         .encodeText // Missing This part
>>         .outputPipe(output());
>> }
>> ```
> 
> Unfortunately this did not write a text file as I expected.

This is because output is buffered differently from input. With output you are writing to the buffer, then flushing the result as needed. But I constructed iopipe to be able to utilize all mechanisms as both input and output.

Before I show you what to do, let me explain what the above actually does.

1. You constructed a buffer of characters. Good, this is the first step.
2. You used encodeText to convert the data to ubyte. Note that for char buffer, this basically is just a cast. Other encodings might do byteswapping. But you have done this step a bit early. At this point now, the window type is ubyte[]. You want to put data into the buffer as char[].
3. You appended an outputPipe, which is a pass through while writing the data to a file (which is fed a stream of uninitialized data I think, so you probably got a file with garbage in it). Writing to this pipe's buffer is kind of pointless because the writing has already happened (pretty much all effects and actions performed on the buffer happen in the .extend function).

What you need is to access the buffer for writing BEFORE encoding and streaming to the file. For this, you need the push [1] mechanism, which wraps up everything you want to happen to data you have written to the buffer into a lambda template:

auto outputBuffered = bufd!char
   .push!(p => p
      .encodeText
      .outputPipe(output()));

now, you write to the outputBuffered "buffer" as needed. releasing any data that is ready to go. As soon as it needs more buffer space and there is pending data, it will write the data, giving you back the buffer space. It also auto-flushes on the last reference destructor (this behavior is configurable).

A simple writeln on such a pipe would look like:

void writeln(Pipe)(ref Pipe sink, string text)
{
   enforce(sink.ensureElems(text.size) >= text.length); // make sure there's enough buffer space to hold the text
   sink[0 .. text.length] = text; // write to the buffer
   sink.release(text.length); // release to be written
}

I really need to do an article on iopipe. I've been neglecting that...

-Steve

[1] http://schveiguy.github.io/iopipe/iopipe/valve/push.html

On 1/27/20 1:12 PM, Steven Schveighoffer wrote:

> void writeln(Pipe)(ref Pipe sink, string text)
> {
>     enforce(sink.ensureElems(text.size) >= text.length); // make sure there's enough buffer space to hold the text
>     sink[0 .. text.length] = text; // write to the buffer
>     sink.release(text.length); // release to be written
> }
> 

Ugh, should have tested.

sink.window[0 .. text.length] = text;

-Steve

On Monday, 27 January 2020 at 18:12:40 UTC, Steven Schveighoffer wrote:
> Before I show you what to do, let me explain what the above actually does.
>
> 1. You constructed a buffer of characters. Good, this is the first step.
> 2. You used encodeText to convert the data to ubyte. Note that for char buffer, this basically is just a cast. Other encodings might do byteswapping. But you have done this step a bit early. At this point now, the window type is ubyte[]. You want to put data into the buffer as char[].
> 3. You appended an outputPipe, which is a pass through while writing the data to a file (which is fed a stream of uninitialized data I think, so you probably got a file with garbage in it). Writing to this pipe's buffer is kind of pointless because the writing has already happened (pretty much all effects and actions performed on the buffer happen in the .extend function).
>
> What you need is to access the buffer for writing BEFORE encoding and streaming to the file. For this, you need the push [1] mechanism, which wraps up everything you want to happen to data you have written to the buffer into a lambda template:
>
> auto outputBuffered = bufd!char
>    .push!(p => p
>       .encodeText
>       .outputPipe(output()));
>
> [1] http://schveiguy.github.io/iopipe/iopipe/valve/push.html

I really feel like this is all very well thought out and clean, I don't appear to have a previous model to help visualize this output approach. Right now something like tee is coming to mind. Thank you for explaining with the answer.

On 1/28/20 1:25 AM, Jesse Phillips wrote:

> I really feel like this is all very well thought out and clean, I don't appear to have a previous model to help visualize this output approach. Right now something like tee is coming to mind. Thank you for explaining with the answer.

Thanks! Tee is actually a pretty close approximation. For example, if you wanted to save to 2 files, you could do:

push!(p => p
   .encodeText
   .outputPipe(file1)
   .outputPipe(file2)
)

The whole advantage is that you don't need to use the output pipes as buffered output. For example, the convert function doesn't need them, it just pulls the whole thing (using the process function).

Everything is pulled with iopipe, even output, so it's just a matter of who is pulling and when. Pushing is a matter of telling the other end to pull.

-Steve

On Tuesday, 28 January 2020 at 16:09:55 UTC, Steven Schveighoffer wrote:
>
> Everything is pulled with iopipe, even output, so it's just a matter of who is pulling and when. Pushing is a matter of telling the other end to pull.
>
> -Steve

That statement I think will be very helpful to me.

The push would control the buffer, creating that value concept, where the buffer is flush which creates a pull, specified in the delegate.

Pusher(buffer) <- put(content)

An output range wrapper would provide a push interface which would fill in the buffer of the Pusher. When the buffer fills the range wrapper would ask to release which Pusher does by calling the delegate.

Hopefully I'm following this correctly.

On 1/30/20 12:59 AM, Jesse Phillips wrote:
> On Tuesday, 28 January 2020 at 16:09:55 UTC, Steven Schveighoffer wrote:
>>
>> Everything is pulled with iopipe, even output, so it's just a matter of who is pulling and when. Pushing is a matter of telling the other end to pull.
>>
> 
> That statement I think will be very helpful to me.
> 
> The push would control the buffer, creating that value concept, where the buffer is flush which creates a pull, specified in the delegate.

The push function divides the buffer into 2 parts, the "writeable" items (i.e. items ready to push), and the buffer that can be accessed for manipulation (this is where you put your data). Eventually, the latter fills up and you need more buffer space. At this point, it performs the push, which is done by running the push chain as if it were an input stream, where the data source is the writeable data.

> 
> Pusher(buffer) <- put(content)
> 
> An output range wrapper would provide a push interface which would fill in the buffer of the Pusher. When the buffer fills the range wrapper would ask to release which Pusher does by calling the delegate.

An output range wrapper should fill in the accessible buffer and then immediately release it (same as that writeln example I showed before).

> Hopefully I'm following this correctly.

I was going to write an ascii art concept to show how the pushing works, but I think I'll maybe draw an actual picture. I need some time to accomplish this, though.

It's very clear in my head how this works, but very hard to describe :)

-Steve