May 16, 2012
On 05/16/12 22:58, Steven Schveighoffer wrote:
> On Wed, 16 May 2012 16:38:54 -0400, Artur Skawina <art.08.09@gmail.com> wrote:
> 
>> On 05/16/12 22:15, Steven Schveighoffer wrote:
>>> I still don't get the need to "add" this to ranges.  The streaming API works fine on its own.
>>
>> This is not an argument against a streaming API (at least not for me), but for efficient ranges. With the API above I can shift tens of gigabytes of data per second between threads. And still use the 'std' range API and everything that works with it...
> 
> But you never would want to.  Don't get me wrong, the primitives here could work for a streaming API (I haven't implemented it that way, but it could be made to work).  But the idea that it must *also* be a std.range input range makes zero sense.

Well, I do want to. For example, I can pass the produced data to *any* range consumer, it may be as efficient as mine, but will still work reasonably (I just did a quick test and the difference seems to be about 10G/s less for plain front+popFront consumer).

The goal here is: if we could agree on a standard interface then *any* producer and consumer, including the ones in the std lib could take advantage of this (optional) feature. It's not so much about function call overhead as /syscall/ and /locking/ costs. Retrieving or writing 100 elements with only one lock-unlock sequence makes a large difference.

> To me, this is as obvious as not supporting linklist[index];  Sure, it can be done, but who would ever use it?

This is not even related.

Your 'read(ref ubyte[])' approach can actually mean that one more copy of
the data is required. Think writer->range_or_stream->reader -- unless the
reader is already waiting with an empty buffer, the stream has to copy the
data to an internal buffer, which then has to be copied again when a reader
comes around. The 'slice[] = fronts' solution avoids the second copy.
Like I said, depending on the circumstances, sometimes you want one scheme,
sometimes the other. (TBH, right now i can't think of a case where i'd prefer
a non-range based approach; having the same i/f is just so convenient. But
I'm sure there's one ;) )

artur
May 16, 2012
On 05/16/2012 11:08 PM, Andrei Alexandrescu wrote:
> On 5/16/12 1:00 PM, Steven Schveighoffer wrote:
>> What I think we would end up with is a streaming API with range
>> primitives tacked on.
>>
>> - empty is clunky, but possible to implement. However, it may become
>> invalid (think of reading a file that is being appended to by another
>> process).
>> - popFront and front do not have any clear definition of what they refer
>> to. The only valid thing I can think of is bytes, and then nobody will
>> use them.
>>
>> That's hardly saying it's "range based". I refuse to believe that people
>> will be thrilled by having to 'pre-configure' each front and popFront
>> call in order to get work done. If you want to try and convince me, I'm
>> willing to listen, but so far I haven't seen anything that looks at all
>> appetizing.
>
> Where the two meet is in the notion of buffered streams. Ranges are by
> default buffered, i.e. user code can call front() several times without
> an intervening popFront() and get the same thing.  So a range has by
> definition a buffer of at least one element.
>

I don't think this necessarily holds. 'front' might be computed on the fly, as it is done for std.algorithm.map.

> That makes the range interface unsuitable for strictly UNbuffered
> streams. On the other hand, a range could no problem offer OPTIONAL
> unbuffered reads (the existence of a buffer does not preclude
> availability of unbuffered transfers).
>
> So to tie it all nicely I think we need:
>
> 1. A STRICTLY UNBUFFERED streaming abstraction
>
> 2. A notion of range that supports unbuffered transfers.
>
>
> Andrei

May 16, 2012
On 5/16/12 4:40 PM, Timon Gehr wrote:
> On 05/16/2012 11:08 PM, Andrei Alexandrescu wrote:
>> On 5/16/12 1:00 PM, Steven Schveighoffer wrote:
>>> What I think we would end up with is a streaming API with range
>>> primitives tacked on.
>>>
>>> - empty is clunky, but possible to implement. However, it may become
>>> invalid (think of reading a file that is being appended to by another
>>> process).
>>> - popFront and front do not have any clear definition of what they refer
>>> to. The only valid thing I can think of is bytes, and then nobody will
>>> use them.
>>>
>>> That's hardly saying it's "range based". I refuse to believe that people
>>> will be thrilled by having to 'pre-configure' each front and popFront
>>> call in order to get work done. If you want to try and convince me, I'm
>>> willing to listen, but so far I haven't seen anything that looks at all
>>> appetizing.
>>
>> Where the two meet is in the notion of buffered streams. Ranges are by
>> default buffered, i.e. user code can call front() several times without
>> an intervening popFront() and get the same thing. So a range has by
>> definition a buffer of at least one element.
>>
>
> I don't think this necessarily holds. 'front' might be computed on the
> fly, as it is done for std.algorithm.map.

It used to be buffered in fact but that was too much trouble. The fair thing to say here is that map relies on the implicit buffering of its input.

Andrei


May 16, 2012
On 16/05/2012 18:21, Walter Bright wrote:
<snip>
> You can have that range read from byChunk(). It's really the same thing that C's stdio does.

And what if I want it to work on ranges that don't have a byChunk method?

Stewart.
May 16, 2012
On 16/05/2012 17:48, H. S. Teoh wrote:
> On Wed, May 16, 2012 at 05:41:49PM +0100, Stewart Gordon wrote:
<snip>
>> Why would anybody want to read a large binary file _one byte at a
>> time_?
> [...]
>
> import std.range;
> byte[] readNBytes(R)(R range, size_t n)
> 	if (isInputRange!R&&  hasSlicing!R)
> {
> 	return R[0..n];
> }

What if I want it to work on ranges that don't have slicing?

Stewart.
1 2 3 4 5 6 7
Next ›   Last »