May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Steven Schveighoffer | On Friday, 18 May 2012 at 15:49:23 UTC, Steven Schveighoffer wrote:
>>
>> I beg to differ..
>>
>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365469.aspx
>
> It still reads into an array of buffers, which are slices. And the resulting "range" looks *exactly* like a range of T[].
>
> -Steve
Uh, the resulting range can be totally discontiguous...
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Mehrdad | On Fri, 18 May 2012 11:52:43 -0400, Mehrdad <wfunction@hotmail.com> wrote:
> On Friday, 18 May 2012 at 15:49:23 UTC, Steven Schveighoffer wrote:
>>>
>>> I beg to differ..
>>>
>>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365469.aspx
>>
>> It still reads into an array of buffers, which are slices. And the resulting "range" looks *exactly* like a range of T[].
>>
>> -Steve
>
> Uh, the resulting range can be totally discontiguous...
So? So can a range of T[].
I'm not getting your point yet...
-Steve
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Steven Schveighoffer | On Friday, 18 May 2012 at 15:57:20 UTC, Steven Schveighoffer wrote:
> So? So can a range of T[].
>
> I'm not getting your point yet...
>
> -Steve
Well you mentioned "There isn't an OS primitive that reads a file descriptor into an e.g. linked list, anything other than a slice would go through a translation.", but I was just pointing out that there is, and that it doesn't go through any translation. I guess you /can/ call it a "range of T[]", but then, you can *also* call a linked list "a range of T[]"... where each T[] has one element. That's not very useful tho, since their nature is kinda different..
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Mehrdad | On Fri, 18 May 2012 12:02:16 -0400, Mehrdad <wfunction@hotmail.com> wrote:
> On Friday, 18 May 2012 at 15:57:20 UTC, Steven Schveighoffer wrote:
>> So? So can a range of T[].
>>
>> I'm not getting your point yet...
>>
>> -Steve
>
> Well you mentioned "There isn't an OS primitive that reads a file descriptor into an e.g. linked list, anything other than a slice would go through a translation.", but I was just pointing out that there is, and that it doesn't go through any translation. I guess you /can/ call it a "range of T[]", but then, you can *also* call a linked list "a range of T[]"... where each T[] has one element. That's not very useful tho, since their nature is kinda different..
No, by range of T[] I mean this:
static assert(isInputRange!Range && is(ElementType!Range == T[]));
-Steve
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Steven Schveighoffer | On Friday, 18 May 2012 at 16:16:21 UTC, Steven Schveighoffer wrote:
> No, by range of T[] I mean this:
>
> static assert(isInputRange!Range && is(ElementType!Range == T[]));
>
> -Steve
Yes, I believe I understood it correctly...
In the case of ReadFileScatter, each T[] has the size of (at most) 1 page.
In the case of a random linked list, each T[] has the size of 1 element.
Hence you can represent both of them as "a range of T[]", but really, that's just trying to fit it into a mold instead of creating the mold based on the actual thing.
What it *really* is is just a discontiguous buffer...
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Steven Schveighoffer | On 5/18/12 8:27 AM, Steven Schveighoffer wrote:
> But be clear, I am *not* going to remove the existing stream I/O
> primitives I had for buffered i/o, I'm rather *adding* range primitives
> as well.
That sounds very promising.
Andrei
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to kenji hara | On 5/18/12 8:51 AM, kenji hara wrote: > OK. If reading bytes from underlying device failed, your 'fronts' can > return empty slice. I understood. > But, It is still *not efficient*. The returned slice will specifies a > buffer controlled by underlying device. If you want to gather bytes > into one chunk, you must copy bytes from returned slice to your chunk. > We should reduce copying memories as much as possible. Yah, this goes back to the fact that ranges are by definition buffered; there's no way to escape that. So as I said, we need to add unbuffered primitives (e.g. "Here's a buffer, fill it with data). They would work with both inputs that have no buffering at all, and with ranges. > And, 'put' primitive in output range concept doesn't support non-blocikng write. > 'put' should consume *all* of given data and write it to underlying > device, then it would block. Right. Zero-copy I/O is a possibility, we need to define primitives for destructively transferring buffers to and from streams/ranges. > Therefore, whole of range concept doesn't cover non-blocking I/O. Correct. Doesn't cover zero-copy I/O either. It's interesting to think what primitives we should define, and what algorithms can take advantage of them beyond just copy(). >>> I'm designing experimental IO primitives: >>> https://github.com/9rnsr/dio Had only time to skim it, looks very promising. > Range concept is good abstraction if underlying container controlls > ownership. But, in I/O we want to *move* ownership of bytes. Range is > not designed efficiently for the purpose, IMO. Yes, yes, yes. Perfect thinking. (What ranges are good at though is for algorithms to mess with.) Andrei |
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Andrei Alexandrescu | On Friday, 18 May 2012 at 16:38:22 UTC, Andrei Alexandrescu wrote:
>> Range concept is good abstraction if underlying container controlls
>> ownership. But, in I/O we want to *move* ownership of bytes. Range is
>> not designed efficiently for the purpose, IMO.
>
> Yes, yes, yes. Perfect thinking.
And I think the issues you brought up some time ago regarding to orphan ranges and non-GC allocators are also rooted in this fact, i.e. that the design of ranges is completely oblivious to data ownership concerns.
But as you said, it's a very convenient interface for algorithms, so…
David
|
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
Posted in reply to Steven Schveighoffer | 2012/5/19 Steven Schveighoffer <schveiguy@yahoo.com>: > On Fri, 18 May 2012 10:39:55 -0400, kenji hara <k.hara.pg@gmail.com> wrote: >>>> I'm designing experimental IO primitives: https://github.com/9rnsr/dio > > I'm having trouble following the code, is there a place with the generated docs? I'm looking for an overview to understand where to look. I have created gh-pages: http://9rnsr.github.com/dio/d/io_core.html Kenji Hara |
May 18, 2012 Re: stream interfaces - with ranges | ||||
---|---|---|---|---|
| ||||
On 05/18/12 17:43, kenji hara wrote: > 2012/5/19 Artur Skawina <art.08.09@gmail.com>: >> On 05/18/12 15:51, kenji hara wrote: >>> OK. If reading bytes from underlying device failed, your 'fronts' can >>> return empty slice. I understood. >>> But, It is still *not efficient*. The returned slice will specifies a >>> buffer controlled by underlying device. If you want to gather bytes >>> into one chunk, you must copy bytes from returned slice to your chunk. >>> We should reduce copying memories as much as possible. >> >> Depends if your input range supports zero-copy or not. IOW you avoid >> the copy iff the range can somehow write the data directly to the caller >> provided buffer. This can be true eg for file reads, where you can tell >> the read(2) syscall to write into the user buffer. But what if you need to >> buffer the stream? An intermediate buffer can become necessary anyway. >> But, as i said before, i agree that a caller-provided-buffer-interface >> is useful. >> >> E[] fronts(); >> void fronts(ref E[]); >> >> And one can be implemented in terms of the other, ie: >> >> E[] fronts[] { E[] els; fronts(els); return els; } >> void fronts(ref E[] e) { e[] = fronts()[]; } > > The flaw of your design is, the memory to store read bytes/elements is allocated by the lower layer. It's a feature. :) > E.g. If you want to construct linked list of some some elements, you must copy elements from returned slice to new allocated node. I think it is still inefficient. > >> depending on which is more efficient. A range can provide >> >> enum bool HasBuffer = 0 || 1; >> >> so that the user can pick the more suited alternative. > > I think fewer primitives as possible is better design than adding extra/optional primitives. If you pick just one scheme, then you will end up with an unnecessary copy sometimes. Or using non-std APIs. Again, I'm saying *both* caller- owned-buffer *and* range-owned-buffer interfaces should be defined. Otherwise, code that needs decent performance will not be able to use the pure range API, and will not interoperate well with "std" code. > How many primitives in your interface design? Multi-element versions of front, popFront and puts. I think this was enough to get things working; this is the tested and proven part. Then there's 'available' and 'free', so that it's possible to avoid blocking. And 'allocate' and 'release', for zero-copy output streams. But i don't remember if i've actually used these parts, i don't think i needed them. This is all from memory, as the last time i worked on this was a while ago, just before i ran into: http://www.digitalmars.com/d/archives/digitalmars/D/dtors_in_shared_structs_fail_to_compile_157978.html ... >>> And, 'put' primitive in output range concept doesn't support non-blocikng write. >>> 'put' should consume *all* of given data and write it to underlying >>> device, then it would block. >> >> True, a write-as-much-as-possible-but not-more primitive is needed. >> >> size_t puts(E[], size_t atleast=size_t.max); >> >> or something like that. (Doing it this way allows for explicit >> non-blocking 'puts', ie '(written=puts(els, 0))==0' means EAGAIN.) >> >>> Therefore, whole of range concept doesn't cover non-blocking I/O. > > I can agree for the signatures. but the names 'fronts' and 'puts' are a little too similar. The names are bad, i know... If anybody has better suggestions... (and almost any other names would be better :) ) artur |
Copyright © 1999-2021 by the D Language Foundation