On Monday, 22 June 2020 at 22:17:43 UTC, Paul Backus wrote:
> On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
>> On 6/22/20 12:50 PM, Paul Backus wrote:
>>> 
>>> The trick used for arrays does not only apply to function calls:
>>> 
>>>      const(int[]) a = [1, 2, 3];
>>>      const(int)[] b = a; // compiles
>>
>> That's different - it's an implicit conversion.
>>
>>> IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
>>
>> What happens upon function calls is not an implicit conversion. It's a forced type change.
>
> So what you're saying is, it's even *less* principled than I thought? :)

Have you guys seen the work done here: https://gist.github.com/Biotronic/940f553fa1b4cbf80a98afe3d1970c6d ?

It is essentially the opOnCall with details on the library implementation as well.

>
> Regardless, my broader point is that once we're open to the possibility of designing a new range API, all of this can be solved without any language changes by using an API that doesn't require mutation (i.e., tail()). Surely that's a better solution than implicitly inserting calls to arbitrary user-defined code every time someone passes an argument to a function.

Can you use the tail() approach and keep the efficiency of popFront? - I.e. do you see that if tail is const it returns a newly constructed range and if tail() is mutable it can internally call a popFront links think and return a ref to itself?

On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
> On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
> One more thing before I forget - we should drop classes that are ranges. They add too much complication. The functionality would still be present by wrapping a polymorphic implementation in a struct.

I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.

So - What aspects of class-based ranges is it valuable to drop? Polymorphism/inheritance? Reference semantics? Both?

I haven't found a need for polymorphism in these ranges, just reference semantics. I realize that the Phobos experience is the opposite, that reference ranges are rarely used. Could be that my design choices could be modified to eliminate reference ranges. Could also be that approaches needed in general libraries are not always necessary in code targeting specific applications. Not trying to make hard claims about this. Just pointing out where I've found value.

--Jon

On Monday, 22 June 2020 at 22:36:05 UTC, Aliak wrote:
> On Monday, 22 June 2020 at 22:17:43 UTC, Paul Backus wrote:
>> Regardless, my broader point is that once we're open to the possibility of designing a new range API, all of this can be solved without any language changes by using an API that doesn't require mutation (i.e., tail()). Surely that's a better solution than implicitly inserting calls to arbitrary user-defined code every time someone passes an argument to a function.
>
> Can you use the tail() approach and keep the efficiency of popFront? - I.e. do you see that if tail is const it returns a newly constructed range and if tail() is mutable it can internally call a popFront links think and return a ref to itself?

In a world where tail() is a range primitive, popFront becomes an algorithm:

void popFront(R)(ref R r)
    if (isInputRangeV2!R && isMutable!R)
{
    r = r.tail;
}

Any decent optimizer should be able to turn this into a mutation. Since ranges already rely heavily on optimization to produce efficient code, I don't think it's unreasonable to rely on it here as well.

Of course, if you need more control, you are always free to specialize the algorithm for your own range type by defining a popFront method.

On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d wrote:
> On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
> > On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
> > One more thing before I forget - we should drop classes that are
> > ranges.  They add too much complication. The functionality would
> > still be present by wrapping a polymorphic implementation in a
> > struct.
> 
> I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
[...]

Yeah, I rely on class ranges on occasion too, when runtime polymorphism is required. They *are* cumbersome to use, though, I'll give you that. If we're going to remove class-based ranges, then whatever replaces them better have good support for runtime polymorphism, otherwise it's not gonna fly.

This may be more common than people think. For example, sometimes I have a range helper:

	auto myFunc(R)(R r, bool cond)
	{
		if (cond)
			return r.map!someFunc;
		else
			return r.filter!someFilter.map!someFunc;
	}

There is no way to make this compile (cond is only known at runtime), because the return statements return diverse types, except to wrap it in an InputRangeObject.

Though I only rarely need this, it's nonetheless critical functionality because sometimes, you cannot know the exact range type until runtime, and runtime polymorphism is unavoidable.  So this part of the current range API cannot be just thrown out without a viable replacement.


T

-- 
Маленькие детки - маленькие бедки.

On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
> On 6/22/20 12:50 PM, Paul Backus wrote:
>
>> IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
>
> What happens upon function calls is not an implicit conversion. It's a forced type change.

Can you expand on this ? I've never heard of this before.

On Monday, June 22, 2020 9:12:02 PM MDT H. S. Teoh via Digitalmars-d wrote:
> On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d
wrote:
> > On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
> > > On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
> > > One more thing before I forget - we should drop classes that are
> > > ranges.  They add too much complication. The functionality would
> > > still be present by wrapping a polymorphic implementation in a
> > > struct.
> >
> > I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
>
> [...]
>
> Yeah, I rely on class ranges on occasion too, when runtime polymorphism is required. They *are* cumbersome to use, though, I'll give you that. If we're going to remove class-based ranges, then whatever replaces them better have good support for runtime polymorphism, otherwise it's not gonna fly.
>
> This may be more common than people think. For example, sometimes I have a range helper:
>
>   auto myFunc(R)(R r, bool cond)
>   {
>       if (cond)
>           return r.map!someFunc;
>       else
>           return r.filter!someFilter.map!someFunc;
>   }
>
> There is no way to make this compile (cond is only known at runtime), because the return statements return diverse types, except to wrap it in an InputRangeObject.
>
> Though I only rarely need this, it's nonetheless critical functionality because sometimes, you cannot know the exact range type until runtime, and runtime polymorphism is unavoidable.  So this part of the current range API cannot be just thrown out without a viable replacement.

As far as forward ranges go, all that should be required is that the class be wrapped by a struct where the copy constructor does the equivalent of save. The class itself can then do the polymorphism like it would now. It's just that the class won't be passed around directly. But because it's wrapped in a struct, it then becomes possible to require that all forward ranges have the same copying semantics (thus allowing us to ditch save) as well as stuff like requiring that the init value be a valid, empty range (which not only is not the case right now but cannot be the case so long as we allow ranges which are classes).

That being said, it is almost always a mistake to make a range a class if it can be avoided, because the allocations for save (or the copy constructor if it replaces save) incur a huge performance hit relative to what it costs to copy a range that's just a struct (e.g. the tests for dxml which use classes for the range are _very_ slow in comparison to those that use dynamic arrays or structs). So, even wrapped in a struct, it's not a great idea to use a class for a forward range, but with problems like you describe here, AFAIK, we really don't have a better solution. It's best to avoid the need for runtime polymorphism with ranges, but it's not always possible. That being said, it should work wrapped in a struct rather than having the class exposed directly.

Now, as far as basic input ranges go, having them be classes isn't necessarily a problem, since basic input ranges are essentially reference types by their very nature (or at least, they can't be value types). If they could have value semantics, they could be forward ranges. In fact, in some ways, it would be nice to require that basic input ranges be classes, since then it would be easy to introspect a basic input range vs a forward range, and it would ensure that basic input ranges have full-on reference semantics, but it also would result in heap allocations that are not currently necessary - particularly in cases where it would currently work to use a pseudo-reference type rather than full-on reference type. So, while it would be very nice to be able to require that basic input ranges be classes, I doubt that it's actually reasonable to do so. Either way, it shouldn't be a problem to allow basic input ranges to be classes. It's just with forward ranges that it's a problem, but struct wrappers should solve that problem.

- Jonathan M Davis

On Monday, June 22, 2020 9:37:51 PM MDT Jonathan M Davis via Digitalmars-d wrote:
> Now, as far as basic input ranges go, having them be classes isn't necessarily a problem, since basic input ranges are essentially reference types by their very nature (or at least, they can't be value types). If they could have value semantics, they could be forward ranges. In fact, in some ways, it would be nice to require that basic input ranges be classes, since then it would be easy to introspect a basic input range vs a forward range, and it would ensure that basic input ranges have full-on reference semantics, but it also would result in heap allocations that are not currently necessary - particularly in cases where it would currently work to use a pseudo-reference type rather than full-on reference type. So, while it would be very nice to be able to require that basic input ranges be classes, I doubt that it's actually reasonable to do so. Either way, it shouldn't be a problem to allow basic input ranges to be classes. It's just with forward ranges that it's a problem, but struct wrappers should solve that problem.

On further reflection though, one advantage of disallowing classes as ranges entirely (thus requiring a struct wrapper even for basic input ranges) would be that we could require that the init value be a valid, empty range, which we can't do as long as we allow classes. Still, the fact that basic input ranges and forward ranges inherently have different copying semantics means that they don't have quite the same problem with classes.

- Jonathan M Davis

On Saturday, June 20, 2020 6:07:29 PM MDT Andrei Alexandrescu via Digitalmars- d wrote:
> On 6/20/20 1:02 PM, Avrina wrote:
> > Most of the algorithms in Phobos don't call front() multiple times and
> > they make a copy.
>
> Interesting. They should be easy to fix then. Got a few examples?

Part of the problem is that sometimes, you very much want to be calling front only once and storing the result, because front is calculated every time that it's called (e.g. map calculates front on every call), whereas other times, you want to call front multiple times so that you can avoid copying the return value. To make matters worse, in the case of something like map, while you might get equivalent values on each call to front, you won't necessarily get exactly the same value with each call to front (e.g. if map's predicate calls new). So, there's a strong argument to be made that it's more correct and likely more efficient to call front only once even if that means copying the return value. However, AFAIK, there isn't really consensus on what the correct approach is for generic code, and I don't know what the more common approach is in Phobos. I suppose that code could do introspection on front to see whether it returned by ref such that it called front multiple times if it returned by ref and avoided calling it multiple times if it didn't, but I rather doubt that much of anyone is going to do something like that in practice.

Obviously, if non-copyable types were to be supported, then it needs to be the norm to avoid copying the return value of front, but non-copyable types are a bit of a disaster with ranges anyway, since it's so trivial for them to result in the range itself being non-copyable (at which point, it doesn't even work with foreach), and it's very common for algorithms to assume that a range's elements are copyable. It also risks having to have code test for copyability rather than assuming that it's possible, which will add yet another stray ability that range-based code will have to test for. Personally, I don't think that supporting non-copyable types is worth it and that isInputRange should just ban them, but obviously, some folks want non-copyable types to be supported. However, isForwardRange does currently ban them (though I don't know if it's on purpose on just a happy accident). So, there's that.

- Jonathan M Davis

On 6/22/20 6:17 PM, Paul Backus wrote:
> On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
>> On 6/22/20 12:50 PM, Paul Backus wrote:
>>>
>>> The trick used for arrays does not only apply to function calls:
>>>
>>>      const(int[]) a = [1, 2, 3];
>>>      const(int)[] b = a; // compiles
>>
>> That's different - it's an implicit conversion.
>>
>>> IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
>>
>> What happens upon function calls is not an implicit conversion. It's a forced type change.
> 
> So what you're saying is, it's even *less* principled than I thought? :)

Yeah, it's a hack, but it works very well.

> Regardless, my broader point is that once we're open to the possibility of designing a new range API, all of this can be solved without any language changes by using an API that doesn't require mutation (i.e., tail()).

Using only tail() makes iteration with mutable ranges inefficient and iteration with immutable ranges difficult (must use recursion everywhere). Hardly a winner of a design contest.

>  Surely that's a better solution than implicitly inserting calls to arbitrary user-defined code every time someone passes an argument to a function.

If there is, we haven't found it.

On 6/22/20 6:54 PM, Jon Degenhardt wrote:
> On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
>> On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
>> One more thing before I forget - we should drop classes that are ranges. They add too much complication. The functionality would still be present by wrapping a polymorphic implementation in a struct.
> 
> I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
> 
> So - What aspects of class-based ranges is it valuable to drop? Polymorphism/inheritance? Reference semantics? Both?
> 
> I haven't found a need for polymorphism in these ranges, just reference semantics. I realize that the Phobos experience is the opposite, that reference ranges are rarely used. Could be that my design choices could be modified to eliminate reference ranges. Could also be that approaches needed in general libraries are not always necessary in code targeting specific applications. Not trying to make hard claims about this. Just pointing out where I've found value.

Problem is they're cumbersome to use - you need to remember to call save() in key places, and if you forget the compiler will be mum about it. If we do away with class ranges, save() can be eliminated in favor of the copy ctor.

I assume a wrapper that does define reference semantics where needed would not be difficult to write.