On Tuesday, 12 June 2018 at 13:51:56 UTC, Steven Schveighoffer wrote:
> On 6/12/18 1:51 AM, DigitalDesigns wrote:
>> 
>> Could you explain some benefits specific to this implementation and a bit of the functional aspects for a proper overview of it's capabilities and why I should chose this method over others?
>
> The things that I think make this approach better are:
>
> 1. Direct buffer access
>
> Direct buffer access I have found is one of those ideas that doesn't seem like it's that impressive until you start using it. Many times, buffering is used in a generic library for optimization (basically amortizing reads) and is an implementation detail hidden from your view. Think of how FILE * keeps a large buffer of data inside itself, but only gives you access to one character at a time.
>
> This forces you to create your *own* buffering scheme on top of that. What a waste! Iopipe allows you to use buffering for your purposes on top of the benefits of amortization. It's my belief that this is why iopipe's byline feature is 2x faster than Phobos'.
>
> 2. Using templates to their fullest
>
> Iopipes are all templated on the buffer or iopipe underneath it. This makes tings easily swappable. It's really cool to be able to take your JSON or XML parser, and hook it onto an in-memory string in one line, and then hook it onto a socket, and everything is optimized for that situation. It takes the fun and flexibility of range programming and brings it to i/o.
>
> This is why iopipe's byline handles all forms of UTF, compared to Phobos which only handles UTF8.
>
> For example, I handle all forms of UTF with iopipe, with a decent set of utilities. Here is a complete program using iopipe that converts any type of UTF into another type, optimized for the specific situation:
>
> https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d
>
> 3. Compiler optimization for everything
>
> All parts of iopipe, except for the low-level reads and writes (which ironically are not really part of iopipe) are visible to the compiler for inlining and optimization. I'm leveraging the power of the decades of optimization experience that the compiler can provide. This makes it easy to write code that performs well.
>
> An anecdote: For my talk on iopipe in 2017 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live demo showing the performance power. I literally was still working on it 2 or 3 days before, while at dconf. I had already written a JSON parser, which was part of my presentation, but when I was showing it to another D user (Daniel Murphy), I couldn't really answer the question "how does it perform?". So he gave me a challenge -- do pretty printing on a JSON file. Simple enough, with the parser I had already written, took me about 1 hour to write it. It performed poorly compared to what we would have expected, but tweaking a few things (almost all were due to using some algorithms incorrectly), I got it to go faster than RapidJson in certain use cases, and reasonably close in others. And I did nothing in terms of lookup tables or using any special instructions. All in all, it was probably 2 hours of work, and the code is beautiful IMO! https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d
>
> I think anyone who is doing parsing should have a look at iopipe, it not only may make your code much simpler and easier to read and write, but it would help me tune iopipe to cater to parsers, which I think is its wheelhouse.
>
> I plan to eventually finish the JSON parser for a releasable state, and eventually tackle XML and a few other things.
>
> -Steve

Thanks!

On Tuesday, 12 June 2018 at 14:00:32 UTC, Steven Schveighoffer wrote:
> File.byLine is fast, but only because of the underlying non-range tricks it uses to achieve performance. And iopipe still is 2x faster.

I wish iopipe was around a little bit earlier so I could use it in my small project. It doesn't use IO much, just reads large file (60GB+) and computes some hashes. Now I'd like to rewrite it using iopipe just to compare the performance.

On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:
> -Steve

Does iopipe work with CTFE?

On 6/13/18 8:35 AM, bauss wrote:
> Does iopipe work with CTFE?

It may work in some cases. Some of the things it does are not conducive to CTFE working well -- like using a buffer. But generally at compile time, you don't want to use a buffer.

But I would expect, for instance, using algorithms and iopipes on a string would probably work.

hm... just thought I'd try it:

    import std.range : walkLength;
    static assert("hello\nworld\nthis\nis\na\ntest".byLineRange.walkLength == 6);

It didn't work at first, as I'm using memchr for searching for the newlines, but with a nice little if(__ctfe) override, it works just great!

I'm going to push this (I'll do some tests for the other widths to make sure it works for all UTF), but if you have any more things you want to work at CTFE, submit some issues on the github project.

I don't expect a lot of the array casting stuff to work, but maybe there are ways. I actually never thought much about making it work at compile-time. I suppose though, it would be cool to import a file at compile time, and generate the JSON or XML DOM at compile time too :)

I think zip and unzip are never going to work since the underlying C calls are not CTFE-able.

-Steve

On 6/13/18 12:03 PM, Steven Schveighoffer wrote:

> I'm going to push this (I'll do some tests for the other widths to make sure it works for all UTF), but if you have any more things you want to work at CTFE, submit some issues on the github project.

v0.1.2 released

-Steve

On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:
> Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries.

Note that the WIP std.io library is fully @nogc @safe, so it's a bit
edgy on using latest features. Soon want to move to use DIP10008 instead
of preallocated exceptions.
With that and @nogc in the Driver interface¹ it's still to be seen
whether we can adapt this well with vibe.d or need to adjust the
low-level design.

-Martin

¹: https://martinnowak.github.io/io/std/io/driver/Driver.html

On Sunday, 17 June 2018 at 04:52:07 UTC, Martin Nowak wrote:
> On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:
>> Note that the new io library also supports sockets, which IODev did not have support for, AND has a pluggable driver system, so you could potentially use fiber-based async io without rebuilding. It just makes a lot of sense for D to have a standard low-level io library that everything can use without having to kludge together multiple types of io libraries.
>
> Note that the WIP std.io library is fully @nogc @safe, so it's a bit
> edgy on using latest features. Soon want to move to use DIP10008 instead
> of preallocated exceptions.

This is very encouraging. I’d like to see it working well with Photon (though my time is very limited atm). Any thoughts on what set of syscalls I need to support?

Maybe I could just provide my own “native” driver that fits your concept of I/O driver in io library.

> With that and @nogc in the Driver interface¹ it's still to be seen
> whether we can adapt this well with vibe.d or need to adjust the
> low-level design.
>
> -Martin
>
> ¹: https://martinnowak.github.io/io/std/io/driver/Driver.html

On 2018-06-11 16:45, Steven Schveighoffer wrote:

> I just pushed v0.1.1 -- I realized that I never *actually* compiled on
> windows, and there were a couple things that didn't work.
>
> Note: the examples still don't work as they rely on openDev, which is
> only available on Posix systems now.
>
> I need to figure out a good way to open stdin/stdout in a cross platform
> way with std.io.

You should setup AppVeyor [1] to make it works on Windows (when it works).

[1] https://www.appveyor.com

-- 
/Jacob Carlborg

On 6/19/18 7:18 AM, Jacob Carlborg wrote:
> On 2018-06-11 16:45, Steven Schveighoffer wrote:
> 
>> I just pushed v0.1.1 -- I realized that I never *actually* compiled on
>> windows, and there were a couple things that didn't work.
>>
>> Note: the examples still don't work as they rely on openDev, which is
>> only available on Posix systems now.
>>
>> I need to figure out a good way to open stdin/stdout in a cross platform
>> way with std.io.
> 
> You should setup AppVeyor [1] to make it works on Windows (when it works).
> 
> [1] https://www.appveyor.com
> 

I just set up travis to do the Linux/mac testing. I need to add appveyor as well, but haven't gotten to it. I'm a complete CI noob, so I'm learning slowly :)

-Steve

On 2018-06-19 15:04, Steven Schveighoffer wrote:

> I just set up travis to do the Linux/mac testing. I need to add appveyor
> as well, but haven't gotten to it. I'm a complete CI noob, so I'm
> learning slowly :)

To save you some trouble, AppVeyor supports both a YAML, like Travis, and a web UI to configure the CI system. If you're not including some parts of the YAML file, like "build_script", it will use the default, which is preforming some Visual Studio specific task. You also need to download the D compiler manually since AppVeyor doesn't have built-in support for D the same way as Travis.

-- 
/Jacob Carlborg