iopipe version 0.1.0 has been released.

iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible.

Nothing has really been changed, but it now has Windows i/o support. I will note at this time, however, that ring buffers are not yet supported on Windows.

This version deprecates the IODev type that I had included, in favor of a new io library that shows extreme promise.

This version ONLY builds on 2.080.1 or later (the bug fix that I submitted at dconf has been merged in that version, and so iopipe will now build against Martin Nowak's io library). In fact, iopipe development was kind of stalled due to this bug, so I'm super-happy to see it fixed and released!

Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries.

Near future plans:

1. Utilize a CI to make sure it continues to work on all platforms.
2. Add RingBuffer support on Windows
3. Add more driver support for std.io.
4. Continue development of JSON library that depends on iopipe (not yet on code.dlang.org).

git - https://github.com/schveiguy/iopipe
dub - https://code.dlang.org/packages/iopipe
docs - http://schveiguy.github.io/iopipe/

-Steve

On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
> iopipe version 0.1.0 has been released.
>
> iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible.

All I can say (again, like I repeated on IRC many times) is that iopipe should be in the current form, or another, in Phobos. I just love it!

On 6/11/18 6:14 AM, Dejan Lekic wrote:
> On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
>> iopipe version 0.1.0 has been released.
>>
>> iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible.
> 
> All I can say (again, like I repeated on IRC many times) is that iopipe should be in the current form, or another, in Phobos. I just love it!

Thanks for the kind words! If Martin's std.io makes it in to Phobos, there is a realistic chance iopipe could go as well.

Just curious, do you have any projects based on iopipe?

-Steve

On 6/10/18 4:10 PM, Steven Schveighoffer wrote:

> Nothing has really been changed, but it now has Windows i/o support. I will note at this time, however, that ring buffers are not yet supported on Windows.
> 

I just pushed v0.1.1 -- I realized that I never *actually* compiled on windows, and there were a couple things that didn't work.

Note: the examples still don't work as they rely on openDev, which is only available on Posix systems now.

I need to figure out a good way to open stdin/stdout in a cross platform way with std.io.

-Steve

On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
> iopipe version 0.1.0 has been released.
>
> iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible.
>
> Nothing has really been changed, but it now has Windows i/o support. I will note at this time, however, that ring buffers are not yet supported on Windows.
>
> This version deprecates the IODev type that I had included, in favor of a new io library that shows extreme promise.
>
> This version ONLY builds on 2.080.1 or later (the bug fix that I submitted at dconf has been merged in that version, and so iopipe will now build against Martin Nowak's io library). In fact, iopipe development was kind of stalled due to this bug, so I'm super-happy to see it fixed and released!
>
> Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries.
>
> Near future plans:
>
> 1. Utilize a CI to make sure it continues to work on all platforms.
> 2. Add RingBuffer support on Windows
> 3. Add more driver support for std.io.
> 4. Continue development of JSON library that depends on iopipe (not yet on code.dlang.org).
>
> git - https://github.com/schveiguy/iopipe
> dub - https://code.dlang.org/packages/iopipe
> docs - http://schveiguy.github.io/iopipe/
>
> -Steve

Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others?

On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
> iopipe version 0.1.0 has been released.
>
> iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible.
>

I saw iopipe a while back, but never looked at it closely. Now I did and... it implements its own kind of pattern and not ranges? I guess it is done for the sake of performance, but how easy it is to use iopipe with standard range-based methods from std.algorithm for example?

As long as it's easy to use with the rest of the phobos - I'd like to see it in the standard library.

>
> This version ONLY builds on 2.080.1 or later (the bug fix that I submitted at dconf has been merged in that version, and so iopipe will now build against Martin Nowak's io library). In fact, iopipe development was kind of stalled due to this bug, so I'm super-happy to see it fixed and released!
>

These are great news, keep it up!

On 6/12/18 1:51 AM, DigitalDesigns wrote:
> 
> Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others?

The things that I think make this approach better are:

1. Direct buffer access

Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time.

This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'.

2. Using templates to their fullest

Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o.

This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8.

For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation:

https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d

3. Compiler optimization for everything

All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well.

An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d

I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse.

I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things.

-Steve

On 6/12/18 3:08 AM, Anton Fediushin wrote:
> On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
>> iopipe version 0.1.0 has been released.
>>
>> iopipe is a high-performance pipe processing system that makes it easy to string together pipelines to process data with as little buffer copying as possible.
>>
> 
> I saw iopipe a while back, but never looked at it closely. Now I did and... it implements its own kind of pattern and not ranges?

Correct, although it's very similar to ranges.

> I guess it is done for the sake of performance, but how easy it is to use iopipe with standard range-based methods from std.algorithm for example?

Very simple. You just have to define what is an "element" of a range that is a sliding window of data.

For example: https://github.com/schveiguy/iopipe/blob/master/source/iopipe/textpipe.d#L542

Note that asInputRange simply treats the current window as "front", and "popFront" discards that window and loads the next. It's a crude but effective tool to convert any iopipe into a range.

The reason iopipe is not based on ranges exactly (the window must be a random-access range), is because ranges don't handle i/o very performantly. E.g. I would never use lockingTextReader to process text data, it would be slow as hell, and too limiting.

File.byLine is fast, but only because of the underlying non-range tricks it uses to achieve performance. And iopipe still is 2x faster.

> As long as it's easy to use with the rest of the phobos - I'd like to see it in the standard library.

It should be easy to use on its own, and with algorithms from phobos. I've done so in some of the toy parsers I've written.

I think the one sticking point (which I'm not sure how to reconcile, but we can probably hash it out) is that for iopipes, char and wchar arrays are arrays, not auto-decoding ranges.

-Steve

> I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things.
>
> -Steve

You should definitely tackle xml by branching dxml. I'm really liking the api.

On 6/12/18 10:19 AM, Jesse Phillips wrote:
>> I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things.
>>
> 
> You should definitely tackle xml by branching dxml. I'm really liking the api.

Of course that is my plan! I would never want to have to build an xml parser from scratch :)

-Steve