View mode: basic / threaded / horizontal-split · Log in · Help
May 18, 2012
Re: stream interfaces - with ranges
On Friday, 18 May 2012 at 15:49:23 UTC, Steven Schveighoffer 
wrote:
>>
>> I beg to differ..
>>
>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365469.aspx
>
> It still reads into an array of buffers, which are slices.  And 
> the resulting "range" looks *exactly* like a range of T[].
>
> -Steve

Uh, the resulting range can be totally discontiguous...
May 18, 2012
Re: stream interfaces - with ranges
On Fri, 18 May 2012 11:52:43 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On Friday, 18 May 2012 at 15:49:23 UTC, Steven Schveighoffer wrote:
>>>
>>> I beg to differ..
>>>
>>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365469.aspx
>>
>> It still reads into an array of buffers, which are slices.  And the  
>> resulting "range" looks *exactly* like a range of T[].
>>
>> -Steve
>
> Uh, the resulting range can be totally discontiguous...

So?  So can a range of T[].

I'm not getting your point yet...

-Steve
May 18, 2012
Re: stream interfaces - with ranges
On Friday, 18 May 2012 at 15:57:20 UTC, Steven Schveighoffer 
wrote:
> So?  So can a range of T[].
>
> I'm not getting your point yet...
>
> -Steve

Well you mentioned "There isn't an OS primitive that reads a file 
descriptor into an e.g. linked list, anything other than a slice 
would go through a translation.", but I was just pointing out 
that there is, and that it doesn't go through any translation. I 
guess you /can/ call it a "range of T[]", but then, you can 
*also* call a linked list "a range of T[]"... where each T[] has 
one element. That's not very useful tho, since their nature is 
kinda different..
May 18, 2012
Re: stream interfaces - with ranges
On Fri, 18 May 2012 12:02:16 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On Friday, 18 May 2012 at 15:57:20 UTC, Steven Schveighoffer wrote:
>> So?  So can a range of T[].
>>
>> I'm not getting your point yet...
>>
>> -Steve
>
> Well you mentioned "There isn't an OS primitive that reads a file  
> descriptor into an e.g. linked list, anything other than a slice would  
> go through a translation.", but I was just pointing out that there is,  
> and that it doesn't go through any translation. I guess you /can/ call  
> it a "range of T[]", but then, you can *also* call a linked list "a  
> range of T[]"... where each T[] has one element. That's not very useful  
> tho, since their nature is kinda different..

No, by range of T[] I mean this:

static assert(isInputRange!Range && is(ElementType!Range == T[]));

-Steve
May 18, 2012
Re: stream interfaces - with ranges
On Friday, 18 May 2012 at 16:16:21 UTC, Steven Schveighoffer 
wrote:
> No, by range of T[] I mean this:
>
> static assert(isInputRange!Range && is(ElementType!Range == 
> T[]));
>
> -Steve

Yes, I believe I understood it correctly...

In the case of ReadFileScatter, each T[] has the size of (at 
most) 1 page.

In the case of a random linked list, each T[] has the size of 1 
element.

Hence you can represent both of them as "a range of T[]", but 
really, that's just trying to fit it into a mold instead of 
creating the mold based on the actual thing.

What it *really* is is just a discontiguous buffer...
May 18, 2012
Re: stream interfaces - with ranges
On 5/18/12 8:27 AM, Steven Schveighoffer wrote:
> But be clear, I am *not* going to remove the existing stream I/O
> primitives I had for buffered i/o, I'm rather *adding* range primitives
> as well.

That sounds very promising.

Andrei
May 18, 2012
Re: stream interfaces - with ranges
On 5/18/12 8:51 AM, kenji hara wrote:
> OK. If reading bytes from underlying device failed, your 'fronts' can
> return empty slice. I understood.
> But, It is still *not efficient*. The returned slice will specifies a
> buffer controlled by underlying device. If you want to gather bytes
> into one chunk, you must copy bytes from returned slice to your chunk.
> We should reduce copying memories as much as possible.

Yah, this goes back to the fact that ranges are by definition buffered; 
there's no way to escape that.

So as I said, we need to add unbuffered primitives (e.g. "Here's a 
buffer, fill it with data). They would work with both inputs that have 
no buffering at all, and with ranges.

> And, 'put' primitive in output range concept doesn't support non-blocikng write.
> 'put' should consume *all* of given data and write it  to underlying
> device, then it would block.

Right.

Zero-copy I/O is a possibility, we need to define primitives for 
destructively transferring buffers to and from streams/ranges.

> Therefore, whole of range concept doesn't cover non-blocking I/O.

Correct. Doesn't cover zero-copy I/O either.

It's interesting to think what primitives we should define, and what 
algorithms can take advantage of them beyond just copy().

>>> I'm designing experimental IO primitives:
>>> https://github.com/9rnsr/dio

Had only time to skim it, looks very promising.

> Range concept is good abstraction if underlying container controlls
> ownership. But, in I/O we want to *move* ownership of bytes. Range is
> not designed efficiently for the purpose, IMO.

Yes, yes, yes. Perfect thinking. (What ranges are good at though is for 
algorithms to mess with.)



Andrei
May 18, 2012
Re: stream interfaces - with ranges
On Friday, 18 May 2012 at 16:38:22 UTC, Andrei Alexandrescu wrote:
>> Range concept is good abstraction if underlying container 
>> controlls
>> ownership. But, in I/O we want to *move* ownership of bytes. 
>> Range is
>> not designed efficiently for the purpose, IMO.
>
> Yes, yes, yes. Perfect thinking.

And I think the issues you brought up some time ago regarding to 
orphan ranges and non-GC allocators are also rooted in this fact, 
i.e. that the design of ranges is completely oblivious to data 
ownership concerns.

But as you said, it's a very convenient interface for algorithms, 
so…

David
May 18, 2012
Re: stream interfaces - with ranges
2012/5/19 Steven Schveighoffer <schveiguy@yahoo.com>:
> On Fri, 18 May 2012 10:39:55 -0400, kenji hara <k.hara.pg@gmail.com> wrote:
>>>> I'm designing experimental IO primitives:
>>>> https://github.com/9rnsr/dio
>
> I'm having trouble following the code, is there a place with the generated
> docs?   I'm looking for an overview to understand where to look.

I have created gh-pages:
http://9rnsr.github.com/dio/d/io_core.html

Kenji Hara
May 18, 2012
Re: stream interfaces - with ranges
On 05/18/12 17:43, kenji hara wrote:
> 2012/5/19 Artur Skawina <art.08.09@gmail.com>:
>> On 05/18/12 15:51, kenji hara wrote:
>>> OK. If reading bytes from underlying device failed, your 'fronts' can
>>> return empty slice. I understood.
>>> But, It is still *not efficient*. The returned slice will specifies a
>>> buffer controlled by underlying device. If you want to gather bytes
>>> into one chunk, you must copy bytes from returned slice to your chunk.
>>> We should reduce copying memories as much as possible.
>>
>> Depends if your input range supports zero-copy or not. IOW you avoid
>> the copy iff the range can somehow write the data directly to the caller
>> provided buffer. This can be true eg for file reads, where you can tell
>> the read(2) syscall to write into the user buffer. But what if you need to
>> buffer the stream? An intermediate buffer can become necessary anyway.
>> But, as i said before, i agree that a caller-provided-buffer-interface
>> is useful.
>>
>>   E[] fronts();
>>   void fronts(ref E[]);
>>
>> And one can be implemented in terms of the other, ie:
>>
>>  E[] fronts[] { E[] els; fronts(els); return els; }
>>  void fronts(ref E[] e) { e[] = fronts()[]; }
> 
> The flaw of your design is, the memory to store read bytes/elements is
> allocated by the lower layer.

It's a feature. :)

> E.g. If you want to construct linked list of some some elements, you
> must copy elements from returned slice to new allocated node. I think
> it is still inefficient.
> 
>> depending on which is more efficient. A range can provide
>>
>>  enum bool HasBuffer = 0 || 1;
>>
>> so that the user can pick the more suited alternative.
> 
> I think fewer primitives as possible is better design than adding
> extra/optional primitives.

If you pick just one scheme, then you will end up with an unnecessary
copy sometimes. Or using non-std APIs. Again, I'm saying *both* caller-
owned-buffer *and* range-owned-buffer interfaces should be defined.
Otherwise, code that needs decent performance will not be able to use
the pure range API, and will not interoperate well with "std" code.

> How many primitives in your interface design?

Multi-element versions of front, popFront and puts. I think this
was enough to get things working; this is the tested and proven part.

Then there's 'available' and 'free', so that it's possible to 
avoid blocking. And 'allocate' and 'release', for zero-copy output
streams. But i don't remember if i've actually used these parts, i
don't think i needed them.
This is all from memory, as the last time i worked on this was a while
ago, just before i ran into:

  http://www.digitalmars.com/d/archives/digitalmars/D/dtors_in_shared_structs_fail_to_compile_157978.html

...

>>> And, 'put' primitive in output range concept doesn't support non-blocikng write.
>>> 'put' should consume *all* of given data and write it  to underlying
>>> device, then it would block.
>>
>> True, a write-as-much-as-possible-but not-more primitive is needed.
>>
>>   size_t puts(E[], size_t atleast=size_t.max);
>>
>> or something like that. (Doing it this way allows for explicit
>> non-blocking 'puts', ie '(written=puts(els, 0))==0' means EAGAIN.)
>>
>>> Therefore, whole of range concept doesn't cover non-blocking I/O.
> 
> I can agree for the signatures. but the names 'fronts' and 'puts' are
> a little too similar.

The names are bad, i know... If anybody has better suggestions... (and
almost any other names would be better :) )


artur
1 2 3 4
Top | Discussion index | About this forum | D home