May 18, 2012
On Friday, 18 May 2012 at 15:49:23 UTC, Steven Schveighoffer wrote:
>>
>> I beg to differ..
>>
>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365469.aspx
>
> It still reads into an array of buffers, which are slices.  And the resulting "range" looks *exactly* like a range of T[].
>
> -Steve

Uh, the resulting range can be totally discontiguous...
May 18, 2012
On Fri, 18 May 2012 11:52:43 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On Friday, 18 May 2012 at 15:49:23 UTC, Steven Schveighoffer wrote:
>>>
>>> I beg to differ..
>>>
>>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365469.aspx
>>
>> It still reads into an array of buffers, which are slices.  And the resulting "range" looks *exactly* like a range of T[].
>>
>> -Steve
>
> Uh, the resulting range can be totally discontiguous...

So?  So can a range of T[].

I'm not getting your point yet...

-Steve
May 18, 2012
On Friday, 18 May 2012 at 15:57:20 UTC, Steven Schveighoffer wrote:
> So?  So can a range of T[].
>
> I'm not getting your point yet...
>
> -Steve

Well you mentioned "There isn't an OS primitive that reads a file descriptor into an e.g. linked list, anything other than a slice would go through a translation.", but I was just pointing out that there is, and that it doesn't go through any translation. I guess you /can/ call it a "range of T[]", but then, you can *also* call a linked list "a range of T[]"... where each T[] has one element. That's not very useful tho, since their nature is kinda different..
May 18, 2012
On Fri, 18 May 2012 12:02:16 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On Friday, 18 May 2012 at 15:57:20 UTC, Steven Schveighoffer wrote:
>> So?  So can a range of T[].
>>
>> I'm not getting your point yet...
>>
>> -Steve
>
> Well you mentioned "There isn't an OS primitive that reads a file descriptor into an e.g. linked list, anything other than a slice would go through a translation.", but I was just pointing out that there is, and that it doesn't go through any translation. I guess you /can/ call it a "range of T[]", but then, you can *also* call a linked list "a range of T[]"... where each T[] has one element. That's not very useful tho, since their nature is kinda different..

No, by range of T[] I mean this:

static assert(isInputRange!Range && is(ElementType!Range == T[]));

-Steve
May 18, 2012
On Friday, 18 May 2012 at 16:16:21 UTC, Steven Schveighoffer wrote:
> No, by range of T[] I mean this:
>
> static assert(isInputRange!Range && is(ElementType!Range == T[]));
>
> -Steve

Yes, I believe I understood it correctly...

In the case of ReadFileScatter, each T[] has the size of (at most) 1 page.

In the case of a random linked list, each T[] has the size of 1 element.

Hence you can represent both of them as "a range of T[]", but really, that's just trying to fit it into a mold instead of creating the mold based on the actual thing.

What it *really* is is just a discontiguous buffer...
May 18, 2012
On 5/18/12 8:27 AM, Steven Schveighoffer wrote:
> But be clear, I am *not* going to remove the existing stream I/O
> primitives I had for buffered i/o, I'm rather *adding* range primitives
> as well.

That sounds very promising.

Andrei
May 18, 2012
On 5/18/12 8:51 AM, kenji hara wrote:
> OK. If reading bytes from underlying device failed, your 'fronts' can
> return empty slice. I understood.
> But, It is still *not efficient*. The returned slice will specifies a
> buffer controlled by underlying device. If you want to gather bytes
> into one chunk, you must copy bytes from returned slice to your chunk.
> We should reduce copying memories as much as possible.

Yah, this goes back to the fact that ranges are by definition buffered; there's no way to escape that.

So as I said, we need to add unbuffered primitives (e.g. "Here's a buffer, fill it with data). They would work with both inputs that have no buffering at all, and with ranges.

> And, 'put' primitive in output range concept doesn't support non-blocikng write.
> 'put' should consume *all* of given data and write it  to underlying
> device, then it would block.

Right.

Zero-copy I/O is a possibility, we need to define primitives for destructively transferring buffers to and from streams/ranges.

> Therefore, whole of range concept doesn't cover non-blocking I/O.

Correct. Doesn't cover zero-copy I/O either.

It's interesting to think what primitives we should define, and what algorithms can take advantage of them beyond just copy().

>>> I'm designing experimental IO primitives:
>>> https://github.com/9rnsr/dio

Had only time to skim it, looks very promising.

> Range concept is good abstraction if underlying container controlls
> ownership. But, in I/O we want to *move* ownership of bytes. Range is
> not designed efficiently for the purpose, IMO.

Yes, yes, yes. Perfect thinking. (What ranges are good at though is for algorithms to mess with.)



Andrei
May 18, 2012
On Friday, 18 May 2012 at 16:38:22 UTC, Andrei Alexandrescu wrote:
>> Range concept is good abstraction if underlying container controlls
>> ownership. But, in I/O we want to *move* ownership of bytes. Range is
>> not designed efficiently for the purpose, IMO.
>
> Yes, yes, yes. Perfect thinking.

And I think the issues you brought up some time ago regarding to orphan ranges and non-GC allocators are also rooted in this fact, i.e. that the design of ranges is completely oblivious to data ownership concerns.

But as you said, it's a very convenient interface for algorithms, so…

David
May 18, 2012
2012/5/19 Steven Schveighoffer <schveiguy@yahoo.com>:
> On Fri, 18 May 2012 10:39:55 -0400, kenji hara <k.hara.pg@gmail.com> wrote:
>>>> I'm designing experimental IO primitives: https://github.com/9rnsr/dio
>
> I'm having trouble following the code, is there a place with the generated docs?   I'm looking for an overview to understand where to look.

I have created gh-pages: http://9rnsr.github.com/dio/d/io_core.html

Kenji Hara
May 18, 2012
On 05/18/12 17:43, kenji hara wrote:
> 2012/5/19 Artur Skawina <art.08.09@gmail.com>:
>> On 05/18/12 15:51, kenji hara wrote:
>>> OK. If reading bytes from underlying device failed, your 'fronts' can
>>> return empty slice. I understood.
>>> But, It is still *not efficient*. The returned slice will specifies a
>>> buffer controlled by underlying device. If you want to gather bytes
>>> into one chunk, you must copy bytes from returned slice to your chunk.
>>> We should reduce copying memories as much as possible.
>>
>> Depends if your input range supports zero-copy or not. IOW you avoid
>> the copy iff the range can somehow write the data directly to the caller
>> provided buffer. This can be true eg for file reads, where you can tell
>> the read(2) syscall to write into the user buffer. But what if you need to
>> buffer the stream? An intermediate buffer can become necessary anyway.
>> But, as i said before, i agree that a caller-provided-buffer-interface
>> is useful.
>>
>>   E[] fronts();
>>   void fronts(ref E[]);
>>
>> And one can be implemented in terms of the other, ie:
>>
>>  E[] fronts[] { E[] els; fronts(els); return els; }
>>  void fronts(ref E[] e) { e[] = fronts()[]; }
> 
> The flaw of your design is, the memory to store read bytes/elements is allocated by the lower layer.

It's a feature. :)

> E.g. If you want to construct linked list of some some elements, you must copy elements from returned slice to new allocated node. I think it is still inefficient.
> 
>> depending on which is more efficient. A range can provide
>>
>>  enum bool HasBuffer = 0 || 1;
>>
>> so that the user can pick the more suited alternative.
> 
> I think fewer primitives as possible is better design than adding extra/optional primitives.

If you pick just one scheme, then you will end up with an unnecessary copy sometimes. Or using non-std APIs. Again, I'm saying *both* caller- owned-buffer *and* range-owned-buffer interfaces should be defined. Otherwise, code that needs decent performance will not be able to use the pure range API, and will not interoperate well with "std" code.

> How many primitives in your interface design?

Multi-element versions of front, popFront and puts. I think this
was enough to get things working; this is the tested and proven part.

Then there's 'available' and 'free', so that it's possible to
avoid blocking. And 'allocate' and 'release', for zero-copy output
streams. But i don't remember if i've actually used these parts, i
don't think i needed them.
This is all from memory, as the last time i worked on this was a while
ago, just before i ran into:

   http://www.digitalmars.com/d/archives/digitalmars/D/dtors_in_shared_structs_fail_to_compile_157978.html

...

>>> And, 'put' primitive in output range concept doesn't support non-blocikng write.
>>> 'put' should consume *all* of given data and write it  to underlying
>>> device, then it would block.
>>
>> True, a write-as-much-as-possible-but not-more primitive is needed.
>>
>>   size_t puts(E[], size_t atleast=size_t.max);
>>
>> or something like that. (Doing it this way allows for explicit
>> non-blocking 'puts', ie '(written=puts(els, 0))==0' means EAGAIN.)
>>
>>> Therefore, whole of range concept doesn't cover non-blocking I/O.
> 
> I can agree for the signatures. but the names 'fronts' and 'puts' are a little too similar.

The names are bad, i know... If anybody has better suggestions... (and
almost any other names would be better :) )


artur