On Monday, 8 February 2016 at 12:02:08 UTC, Kagamin wrote:
> In the output stream you compare output data length to the input data length. In case of a transcoding stream they can be different. Are you trying to account for partial writes?

Reads and writes are not guaranteed to fill/write the entire buffer you throw at it. This is what readExactly/writeExactly are for. Those will throw an exception if the entire read/write cannot be done.

On Mon, 08 Feb 2016 18:52:52 +0000, Atila Neves wrote:

> On Monday, 8 February 2016 at 17:11:56 UTC, Chris Wright wrote:
>> On Mon, 08 Feb 2016 12:19:59 +0000, Atila Neves wrote:
>>
>>> On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:
>>>> [...]
>>> 
>>> I like boost. Well, sometimes. I _severely_ dislike boost::asio. The new couroutines may make it easier to use, I'd have to try 1st, but what's the point when vibe.d exists?
>>> 
>>> Atila
>>
>> I can't use vibe.d for one of my projects because I need to keep tight control of coroutine scheduling. Better async IO routines in Phobos would help me -- though I'm mainly looking for higher level stuff, like a telnet handler.
> 
> You can always use fibers yourself and just do the IO there. It's cooperative multithreading so you control when one of them yields.

Which is exactly what I'm doing now.

The point is, it's useful to have multiple types of concurrency handling with IO, and async IO that's likely to end up in Phobos doesn't necessarily obviate vibe.d or duplicate work.

On Monday, 8 February 2016 at 20:21:31 UTC, Jason White wrote:
> Reads and writes are not guaranteed to fill/write the entire buffer you throw at it. This is what readExactly/writeExactly are for. Those will throw an exception if the entire read/write cannot be done.

You mean posix non-blocking IO?

On Tue, 09 Feb 2016 09:45:03 +0000, Kagamin wrote:

> On Monday, 8 February 2016 at 20:21:31 UTC, Jason White wrote:
>> Reads and writes are not guaranteed to fill/write the entire buffer you throw at it. This is what readExactly/writeExactly are for. Those will throw an exception if the entire read/write cannot be done.
> 
> You mean posix non-blocking IO?

No, Posix non-blocking IO will fill a buffer with all data that is currently available and tell you the number of bytes it managed to give you. And then it will set errno to EAGAIN.

readExactly will either fill the entire buffer or throw an exception saying that it was unable to do so.

It's using Posix functions underneath, so if it repeatedly calls them until it's sent the whole buffer (assuming EAGAIN is set), that saves me work sometimes (but I think phobos does that already).

But if it throws an exception even when errno is set to EAGAIN, that would be useful for writing to a file atomically.

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> I'm also interested in a discussion of what IO-related functionality people are missing in Phobos.

This is still a very interesting approach that could even become a candidate for std.io at some point. Would be great if we could get a few more people behind this, for sure there are already plenty of D socket implementations out there.

https://github.com/jasonwhite/io/pull/3#issuecomment-234775601

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> I'm interested in feedback on this library. What is it missing? How can be better?

I think making the actual read/write/open/accept/et.al. functions used to talk to the kernel pluggable would be a good extension point to hook in evented/threaded IO frameworks.
This restricts the async integration to synchronous APIs, e.g. suspendable Fibers or async-await, but that might be a reasonable choice over nodejs-style inversion of control.

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> I see the subject of IO streams brought up here occasionally. The general consensus seems to be that we need something better than what Phobos provides.
>
> I wrote a library "io" that can work as a replacement for std.stdio, std.mmfile, std.cstream, and parts of std.stream:
>
>     GitHub:  https://github.com/jasonwhite/io
>     Package: https://code.dlang.org/packages/io
>
> This library provides an input and output range interface for streams (which is more efficient if the stream is buffered). Thus, many of the wonderful range operations from std.range and std.algorithm can be used with this.
>
> I'm interested in feedback on this library. What is it missing? How can be better?
>
> I'm also interested in a discussion of what IO-related functionality people are missing in Phobos.
>
> Please destroy!

Hello,

I don't know if it is good practice or not, but sometimes it make life easier if you can put part of the data back into the input stream.

Am Mon, 25 Jul 2016 13:10:42 +0000
schrieb ikod <geller.garry@gmail.com>:

> On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> > I see the subject of IO streams brought up here occasionally. The general consensus seems to be that we need something better than what Phobos provides.
> >
> > I wrote a library "io" that can work as a replacement for std.stdio, std.mmfile, std.cstream, and parts of std.stream:
> >
> >     GitHub:  https://github.com/jasonwhite/io
> >     Package: https://code.dlang.org/packages/io
> >
> > This library provides an input and output range interface for streams (which is more efficient if the stream is buffered). Thus, many of the wonderful range operations from std.range and std.algorithm can be used with this.
> >
> > I'm interested in feedback on this library. What is it missing? How can be better?
> >
> > I'm also interested in a discussion of what IO-related functionality people are missing in Phobos.
> >
> > Please destroy!
> 
> Hello,
> 
> I don't know if it is good practice or not, but sometimes it make life easier if you can put part of the data back into the input stream.
> 

Writing data back to a stream is quite uncommon. The standard way to solve such problems is a peek method for buffered streams:

auto buf = stream.peek(length)
// You can now look at the data in buf
stream.read() will still return the data read by peek, no need to write
data back into the stream.

Am Sun, 24 Jul 2016 13:03:01 +0000
schrieb Martin Nowak <code@dawg.eu>:

> On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> > I'm interested in feedback on this library. What is it missing? How can be better?
> 
> I think making the actual read/write/open/accept/et.al. functions
> used to talk to the kernel pluggable would be a good extension
> point to hook in evented/threaded IO frameworks.
> This restricts the async integration to synchronous APIs, e.g.
> suspendable Fibers or async-await, but that might be a reasonable
> choice over nodejs-style inversion of control.
> 

Can you please elaborate why this is useful? As streams already provide a source-independent way to process data it should always possible to just implement another stream to wrap low level calls. So why add another API layer? And if the API slightly varies (e.g. different open flags) implementing another wrapper is much simpler than having one wrapper dealing with different low level functions.

Am 26.07.2016 um 16:50 schrieb Johannes Pfau:
> Am Mon, 25 Jul 2016 13:10:42 +0000
>>
>> Hello,
>>
>> I don't know if it is good practice or not, but sometimes it make
>> life easier if you can put part of the data back into the input
>> stream.
>>
>
> Writing data back to a stream is quite uncommon. The standard way to
> solve such problems is a peek method for buffered streams:
>
> auto buf = stream.peek(length)
> // You can now look at the data in buf
> stream.read() will still return the data read by peek, no need to write
> data back into the stream.
>

With the notable exception of ungetc() for C's file streams. But working on the byte level is something that ranges should be used for in D's case and not streams, because the latter tend to have a high call overhead. So such a feature could make sense for a StreamInputRange/InputStreamRange wrapper.