A couple of weeks ago on the next/shift convenience wrapper discussion [1], there was a nice discussion about transient ranges. It didn't come to a conclusion, so let's revie it. I just cite the best three quotes from the thread as a summary:

jmdavis:

> The reality of the matter is that ranges with a transient front tend to break if you do much more than use foreach on them. They don't even work with std.array.array

schveiguy:

> There is an implicit expectation that if you assign the front of a range to something, it's going to last forever, because many ranges do actually behave that way. But the range contract does not guarantee that, it just happens to work.

quickfur:

> isTransient can only be implemented if the range implementor defines its value. There is no way to know whether a range is transient or not just by looking at its type and the signatures of front, popFront, etc.. Having said that, though, since transient ranges are generally the exception rather than the norm, we could adopt a convention that transient ranges declare themselves thus by defining a .transient enum member or something like that, and have isTransient return false if a range doesn't have such a member.

> On a related note, there are a good number of Phobos algorithms that do not work  on transient ranges at all, because internally they (blindly) assume that assigning the value of .front to a local variable will retain its original value after popFront is called. I've fixed a couple of places where this happens, but I'm almost certain many other places haven't been fixed yet, and in some cases, cannot be fixed without restricting the algorithms to forward ranges only rather than input ranges. Some things simply cannot be implemented without copying .front somehow, and the only way to do this reliably with the current API is to require forward ranges and use save to keep track of the previous position of the range.

So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?

[1] https://github.com/dlang/phobos/pull/4010

On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?

Honestly, I don't think that supporting transient ranges is worth it. Every single range-based function would have to either test that the "transient" enum wasn't there or take transient ranges into account, and realistically, that isn't going to happen. For better or worse, we do have byLine in std.stdio, which has a transient front, but aside from the performance benefits, it's been a disaster. It's way too error-prone. We now have byLineCopy to combat that, but of course, byLine is the more obvious function and thus more likely to be used (plus it's been around longer), so a _lot_ of code is going to end up using it, and a good chunk of that code really should be using byLineCopy.

I'm of the opinion that if you want a transient front, you should just use opApply and skip ranges entirely. Allowing for front to be transient - whether you can check for it or not - simply is not worth the extra complications. I'd love it if we deprecated byLine's range functions, and made it use opApply instead and just declare transient ranges to be completely unsupported. If you want to write your code to have a transient front, you can obviously take that risk, but you're on your own.

- Jonathan M Davis

On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?
>
> Honestly, I don't think that supporting transient ranges is worth it. Every single range-based function would have to either test that the "transient" enum wasn't there or take transient ranges into account, and realistically, that isn't going to happen. For better or worse, we do have byLine in std.stdio, which has a transient front, but aside from the performance benefits, it's been a disaster. It's way too error-prone. We now have byLineCopy to combat that, but of course, byLine is the more obvious function and thus more likely to be used (plus it's been around longer), so a _lot_ of code is going to end up using it, and a good chunk of that code really should be using byLineCopy.
>
> I'm of the opinion that if you want a transient front, you should just use opApply and skip ranges entirely. Allowing for front to be transient - whether you can check for it or not - simply is not worth the extra complications. I'd love it if we deprecated byLine's range functions, and made it use opApply instead and just declare transient ranges to be completely unsupported. If you want to write your code to have a transient front, you can obviously take that risk, but you're on your own.
>
> - Jonathan M Davis

Agreed, any time I see « you're code is wrong, byLine reuses its buffer you should use byLineCopy instead » I cringe as it means that people are supposed to know implementation details. It's obviously a leaky abstraction. While its speed makes it useful nonetheless I don't think such code should be encouraged.

On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?
>
> Honestly, I don't think that supporting transient ranges is worth it.

I have personally wondered if there was a case for a TransientRange concept where the only primitives defined are `empty` and `front`.

`popFront()` is not defined because the whole point is that every single call to `front` will produce a different value.

On Saturday, 28 May 2016 at 17:27:17 UTC, Joseph Rushton Wakeling wrote:
> On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
>> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>>> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?
>>
>> Honestly, I don't think that supporting transient ranges is worth it.
>
> I have personally wondered if there was a case for a TransientRange concept where the only primitives defined are `empty` and `front`.
>
> `popFront()` is not defined because the whole point is that every single call to `front` will produce a different value.

That is a rather sound idea.

On Saturday, 28 May 2016 at 19:09:09 UTC, Stefan Koch wrote:
> On Saturday, 28 May 2016 at 17:27:17 UTC, Joseph Rushton Wakeling wrote:
>> On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
>>> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>>>> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?
>>>
>>> Honestly, I don't think that supporting transient ranges is worth it.
>>
>> I have personally wondered if there was a case for a TransientRange concept where the only primitives defined are `empty` and `front`.
>>
>> `popFront()` is not defined because the whole point is that every single call to `front` will produce a different value.
>
> That is a rather sound idea.

I like this idea too!
`opApply` implies passing in a delegate which (if I am not mistaken) implies the use of GC at the moment and isn't as easy to optimize as `while (!file.empty) { writeln(file.front) }`

On 5/28/2016 10:27 AM, Joseph Rushton Wakeling via Digitalmars-d wrote:
> On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
>> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>>> So what about the convention to explicitely declare a `.transient`
>>> enum member on a range, if the front element value can change?
>>
>> Honestly, I don't think that supporting transient ranges is worth it.
>
> I have personally wondered if there was a case for a TransientRange
> concept where the only primitives defined are `empty` and `front`.
>
> `popFront()` is not defined because the whole point is that every single
> call to `front` will produce a different value.

Um. that's wrong, just look at byLine as evidence.  It is popFront that changes the value to the 'next' one in the iteration.  Front called twice in a row does NOT change the value, and shouldn't.  I think you're misunderstanding something.

On Saturday, 28 May 2016 at 21:32:15 UTC, Brad Roberts wrote:
> On 5/28/2016 10:27 AM, Joseph Rushton Wakeling via Digitalmars-d wrote:
>> On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
>>> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>>>> So what about the convention to explicitely declare a `.transient`
>>>> enum member on a range, if the front element value can change?
>>>
>>> Honestly, I don't think that supporting transient ranges is worth it.
>>
>> I have personally wondered if there was a case for a TransientRange
>> concept where the only primitives defined are `empty` and `front`.
>>
>> `popFront()` is not defined because the whole point is that every single
>> call to `front` will produce a different value.
>
> Um. that's wrong, just look at byLine as evidence.  It is popFront that changes the value to the 'next' one in the iteration.  Front called twice in a row does NOT change the value, and shouldn't.  I think you're misunderstanding something.

I'm not talking about how byLine currently behaves.  I'm talking about a meaningful API that could be used for a range that (by design) has a transient front, that would be unambiguous in its behaviour.

On 05/28/2016 08:27 PM, Joseph Rushton Wakeling wrote:
> On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
>> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>>> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?
>>
>> Honestly, I don't think that supporting transient ranges is worth it.
> 
> I have personally wondered if there was a case for a TransientRange concept where the only primitives defined are `empty` and `front`.
> 
> `popFront()` is not defined because the whole point is that every single call to `front` will produce a different value.

I would prefer such ranges to not have `front` and return new item from `popFront` instead but yes, I would much prefer it to existing form, transient or not. It is impossible to correctly define input range without caching front which may not be always possible and may have negative performance impact. Because of that, a lot of Phobos ranges compromise `front` consistency in favor of speed up - which only seems to work because most algorithms need to access `front` once.

I believe this is biggest issue in D ranges design right now, by large margin.

On Sunday, 29 May 2016 at 11:15:19 UTC, Dicebot wrote:
> On 05/28/2016 08:27 PM, Joseph Rushton Wakeling wrote:
>> On Saturday, 28 May 2016 at 01:48:08 UTC, Jonathan M Davis wrote:
>>> On Friday, May 27, 2016 23:42:24 Seb via Digitalmars-d wrote:
>>>> So what about the convention to explicitely declare a `.transient` enum member on a range, if the front element value can change?
>>>
>>> Honestly, I don't think that supporting transient ranges is worth it.
>> 
>> I have personally wondered if there was a case for a TransientRange concept where the only primitives defined are `empty` and `front`.
>> 
>> `popFront()` is not defined because the whole point is that every single call to `front` will produce a different value.
>
> I would prefer such ranges to not have `front` and return new item from `popFront` instead but yes, I would much prefer it to existing form, transient or not. It is impossible to correctly define input range without caching front which may not be always possible and may have negative performance impact. Because of that, a lot of Phobos ranges compromise `front` consistency in favor of speed up - which only seems to work because most algorithms need to access `front` once.
>
> I believe this is biggest issue in D ranges design right now, by large margin.

+1
I think making popFront return a value for transient ranges is a sound idea. It would allow to easily distinguish between InputRange and TransientRange with very simple CT introspection. The biggest blocker is to teach the compiler to recognize TransientRange types in foreach.

Another option is to make popFront return a new range, ala slice[1..$] (like std.range.dropOne) which would have the benefit of allowing const/immutable ranges to work.