On 6/22/20 11:22 PM, Mathias LANG wrote:
> On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
>> On 6/22/20 12:50 PM, Paul Backus wrote:
>>
>>> IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
>>
>> What happens upon function calls is not an implicit conversion. It's a forced type change.
> 
> Can you expand on this ? I've never heard of this before.

https://run.dlang.io/is/KgeosK

Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that.

For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight.

We need to offer that chance to other ranges if we want them to enjoy a similar treatment.

On Monday, June 22, 2020 11:19:35 PM MDT Andrei Alexandrescu via Digitalmars-d wrote:
> On 6/22/20 6:54 PM, Jon Degenhardt wrote:
> > On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
> >> On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
> >> One more thing before I forget - we should drop classes that are
> >> ranges. They add too much complication. The functionality would still
> >> be present by wrapping a polymorphic implementation in a struct.
> >
> > I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
> >
> > So - What aspects of class-based ranges is it valuable to drop? Polymorphism/inheritance? Reference semantics? Both?
> >
> > I haven't found a need for polymorphism in these ranges, just reference semantics. I realize that the Phobos experience is the opposite, that reference ranges are rarely used. Could be that my design choices could be modified to eliminate reference ranges. Could also be that approaches needed in general libraries are not always necessary in code targeting specific applications. Not trying to make hard claims about this. Just pointing out where I've found value.
>
> Problem is they're cumbersome to use - you need to remember to call save() in key places, and if you forget the compiler will be mum about it. If we do away with class ranges, save() can be eliminated in favor of the copy ctor.
>
> I assume a wrapper that does define reference semantics where needed would not be difficult to write.

Except that if a forward range has reference semantics, then copying it is not equivalent to calling save. If we get rid of save in favor of using copy constructors, then we're requiring that all forward ranges have the same copying semantics and that that be that copying a forward range results in a copy that can be independently iterated over. Some internals could use references to be sure, but we get rid of the ability to have forward ranges which are actually reference types. They have to have the copying semantics of value types with regards to at least their iteration state.

Now, one of the problems that we have with ranges as things stand is that the semantics of copying a range are unspecified and can vary wildly depending on the range's type, meaning that if a range is copied, you can't ever use it again (at least in generic code), because the state of the original is unknown after mutating the copy. It could be unchanged, be in the same state as the copy, or even be in an invalid state due to the copy mutating part of its state but not all of it. So, overall, having forward ranges require that copying do what save now does would be a considerable improvement with regards to cleaning up the copying semantics.

Of course, that still leaves the problem of basic input ranges, since they can't have value semantics when copying (if they could, then they could be forward ranges). Pretty much by definition, basic input ranges are either reference types or pseudo-reference types (or the programmer was just lazy and didn't implement save). So, unfortunately, generic code that operates on both basic input ranges and forward ranges can't ever have guaranteed copying semantics. I think that we should try to nail down the copying semantics for ranges as much as we can though. Similarly, we should probably place requirements on the semantics of assigning to a range. That would almost certainly kill off RefRange, but IMHO, RefRange was a mistake anyway.

- Jonathan M Davis

On Tuesday, 23 June 2020 at 05:19:35 UTC, Andrei Alexandrescu wrote:
> On 6/22/20 6:54 PM, Jon Degenhardt wrote:
>> On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
>>> On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
>>> One more thing before I forget - we should drop classes that are ranges. They add too much complication. The functionality would still be present by wrapping a polymorphic implementation in a struct.
>> 
>> I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
>
> Problem is they're cumbersome to use - you need to remember to call save() in key places, and if you forget the compiler will be mum about it. If we do away with class ranges, save() can be eliminated in favor of the copy ctor.
>
> I assume a wrapper that does define reference semantics where needed would not be difficult to write.

A couple thoughts. First, my uses for reference ranges have so far been on non-forward input ranges, so no 'save' function. No built-in member for copying either. I'm now wondering if there is some structural correlation between these choices and my application use cases, or if it is purely accidental.

Second - If I'm designing a range intended to have reference semantics, I'd expect it to be easier to build that choice into the main implementation of the range. If the implementation needs to be in two parts, a value-based, copyable range, and a separate wrapper to define reference semantics, then I'd expect it to be more complicated. Even if the wrapper is found in the standard library.

Simple things like decisions on how to organize unit tests. Are there two sets of unit tests, one for the value-based layer, and a second for wrapped layer? (I.e. If the value based layer can never be directly instantiated, should it have standalone unit tests?). And slightly harder things, like can the value based implementation layer assume it is never instantiated as a standalone value-based range? It sounds like more design decisions to achieve the goal.

On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
> While we could have done a separation for rdmd which you struck done a while ago [1] which btw lead to the only two people interested in contributing to rdmd loosing their interest.
>
> ...
>
> [1] https://github.com/dlang/tools/pull/344

It's a bit tangential to the main thrust of discussion here, but I'd just like it on the record that the decision on that PR had nothing to do with why I didn't continue to contribute to rdmd.  I just had a lot of other things keeping me busy (and still do).

Andrei and I had a very constructive conversation at DConf later that year where he suggested, and I agreed, that the way forward with rdmd's issues was not to refactor rdmd but to bring more of its functionality inside the D frontend (of which the main things would presumably be the search for the files to compile, and the running of the compiled executable).

I don't know what if any progress has been made in that respect but it still sounds the most reasonable way to clean things up and avoid duplication of effort between the codebases.

BTW, I think it is important to listen carefully to Andrei's broader feedback here.  A lot of the reason why I got hands on with rdmd was because I was seeing a lot of what I felt were problematic development decisions, and I wanted to clean up the codebase and put in place much more rigorous tests to help avoid those problems in future.

I know it's not easy to hear feedback like this from someone who isn't hands-on at the coalface alongside you, but it really feels like far too often work gets accepted because "something must be done!" and in order to encourage the person who is stepping up to do the work, rather than asking the harder question of, is it being done the right way, and is that work really producing the right long-term outcome.

On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:
> On 6/22/20 11:22 PM, Mathias LANG wrote:
>> On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
>>> On 6/22/20 12:50 PM, Paul Backus wrote:
>>>
>>>> IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
>>>
>>> What happens upon function calls is not an implicit conversion. It's a forced type change.
>> 
>> Can you expand on this ? I've never heard of this before.
>
> https://run.dlang.io/is/KgeosK
>
> Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that.
>
> For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight.
>
> We need to offer that chance to other ranges if we want them to enjoy a similar treatment.

So it's not on function call, but it's about the type being deduced. There's quite a big difference here.

To me this makes perfect sense: This little peculiarity allows us to reduce the amount of template instantiations. E.g. the following code will only instantiate a single template:
```
void fun2(T)(const T[] a) { pragma(msg, T.stringof); }

void main() {
    immutable(int[]) a;
    fun2(a);
    int[] b;
    fun2(b);
}
```

In fact I'd love if we extend that to `auto` so the following code would work:
```
immutable(int[]) fun2() { return null; }

void main() {
    auto x = fun2();
    x = fun2(); // Currently `x` is `immutable(int[])` not `immutable(int)[]`
}
```

On 6/23/20 5:45 AM, Mathias LANG wrote:
> On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:
>> https://run.dlang.io/is/KgeosK
>>
>> Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that.
>>
>> For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight.
>>
>> We need to offer that chance to other ranges if we want them to enjoy a similar treatment.
> 
> So it's not on function call, but it's about the type being deduced. There's quite a big difference here.

Yes, I agree. It works because of the implicit conversion.

> 
> To me this makes perfect sense: This little peculiarity allows us to reduce the amount of template instantiations. E.g. the following code will only instantiate a single template:
> ```
> void fun2(T)(const T[] a) { pragma(msg, T.stringof); }
> 
> void main() {
>      immutable(int[]) a;
>      fun2(a);
>      int[] b;
>      fun2(b);
> }
> ```

No, that would happen anyway, because in that case, T is stripped of the type modifier.

It's this case where it affects the result:

void fun(T)(T a) { pragma(msg, T.stringof); }

void main()
{
   const int[] a;
   const(int)[] b;
   fun(a);
   fun(b);
}

You only get one instantiation there.

The case makes complete sense, because the head is always a different memory location (a copy). What we lack is a way to make the head mutable of any type via a type modifier, so it's a "special case" in the sense that it's possible only on T[] and T*. If we change the special case to a general case, then it naturally makes things a lot more usable, and gets rid of a lot of the complaints about const ranges.

> 
> In fact I'd love if we extend that to `auto` so the following code would work:
> ```
> immutable(int[]) fun2() { return null; }
> 
> void main() {
>      auto x = fun2();
>      x = fun2(); // Currently `x` is `immutable(int[])` not `immutable(int)[]`
> }
> ```

That would be reasonable too, IMO.

-Steve

On 6/22/20 11:12 PM, H. S. Teoh wrote:
> On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d wrote:
>> On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
>>> On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
>>> One more thing before I forget - we should drop classes that are
>>> ranges.  They add too much complication. The functionality would
>>> still be present by wrapping a polymorphic implementation in a
>>> struct.
>>
>> I have used class-based ranges quite a bit to implement ranges with
>> reference semantics. Classes are very convenient for this because the
>> implementation is structurally very similar to a struct based
>> implementation. Typically, replace 'struct' with 'final class' and
>> modify a helper function constructing the range to use 'new'. It's
>> amazingly easy.
> [...]
> 
> Yeah, I rely on class ranges on occasion too, when runtime polymorphism
> is required. They *are* cumbersome to use, though, I'll give you that.
> If we're going to remove class-based ranges, then whatever replaces them
> better have good support for runtime polymorphism, otherwise it's not
> gonna fly.
> 
> This may be more common than people think. For example, sometimes I have
> a range helper:
> 
> 	auto myFunc(R)(R r, bool cond)
> 	{
> 		if (cond)
> 			return r.map!someFunc;
> 		else
> 			return r.filter!someFilter.map!someFunc;
> 	}
> 
> There is no way to make this compile (cond is only known at runtime),
> because the return statements return diverse types, except to wrap it in
> an InputRangeObject.

You don't need polymorphism for that (necessarily), just function pointers/delegates. I've also done this kind of stuff with iopipes using tagged unions:

https://github.com/schveiguy/httpiopipe/blob/a04d87de3aa3836c07d181263c399416ba005e7c/source/iopipe/http.d#L245-L252

Such a thing is also generalized in phobos: https://dlang.org/phobos/std_range.html#choose

And can probably be extended further (not sure why the args aren't lazy).

Your example rewritten:

auto myFunc(R)(R r, bool cond)
{
     return choose(cond, r.map!someFunc, r.filter!someFilter.map!someFunc);
}

-Steve

On Tuesday, 23 June 2020 at 05:15:49 UTC, Andrei Alexandrescu wrote:
>> Regardless, my broader point is that once we're open to the possibility of designing a new range API, all of this can be solved without any language changes by using an API that doesn't require mutation (i.e., tail()).
>
> Using only tail() makes iteration with mutable ranges inefficient

I don't think it's a given that tail() is less efficient than popFront(). Here's a program where both tail() and popFront() produce the same assembly with `ldc -O2`:

https://d.godbolt.org/z/-S_JoF

Of course, if it does turn out to make a difference in some cases, mutable ranges are free to implement popFront as well. There will be a generic version in std.v2.algorithm to ensure that `mutableRange.popFront()` is always valid.

> and iteration with immutable ranges difficult (must use recursion everywhere). Hardly a winner of a design contest.

Even if we assume for the sake of argument that "recursion == difficult", I would still say that "difficult" is an improvement over "impossible".

On Monday, 22 June 2020 at 15:34:36 UTC, Seb wrote:
> The biggest gotcha of Rebindable is that it doesn't work with structs [1] which vastly limits its usability.
>
> [1] https://github.com/dlang/phobos/pull/6136

Yes, effectively what I'm looking for is "Rebindable for immutable structs." I already have this in Nullable, but it's an atrocious hack because I basically have to hit the compiler on the head until it forgets what const is. I want some way to say "yes, this type is immutable, but it's only immutable over its lifetime, and *I* control its lifetime and *I* say when it ends - and when it ends, I'm free to put other data there, such as the return value of rest()."

This is also vital for finally allowing immutable types in (key-)mutable hashmaps.

On 6/23/20 5:45 AM, Mathias LANG wrote:
> On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:
>> On 6/22/20 11:22 PM, Mathias LANG wrote:
>>> On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
>>>> On 6/22/20 12:50 PM, Paul Backus wrote:
>>>>
>>>>> IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
>>>>
>>>> What happens upon function calls is not an implicit conversion. It's a forced type change.
>>>
>>> Can you expand on this ? I've never heard of this before.
>>
>> https://run.dlang.io/is/KgeosK
>>
>> Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that.
>>
>> For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight.
>>
>> We need to offer that chance to other ranges if we want them to enjoy a similar treatment.
> 
> So it's not on function call, but it's about the type being deduced. There's quite a big difference here.

I don't understand the difference. The change $qual(T[]) -> $qual(T)[] happens specifically when a template parameter type is deduced for a function call.

> To me this makes perfect sense: This little peculiarity allows us to reduce the amount of template instantiations.

As I said before, I appreciate there's no shortage of people ready to explain things I've done back to me. In this case it's actually Walter's work. A long time ago, shortly after we introduced ranges and algorithms to Phobos, people complained (including in this forum) that there was no way to use algorithms with constant arrays. Walter and I talked about it for a while and he introduced this glorious hack, only for arrays and only for template function calls. Suddenly const arrays started to work, and nobody complained about the lack of generality of the hack. Until now I guess!

> E.g. the following code will only instantiate a single template:
> ```
> void fun2(T)(const T[] a) { pragma(msg, T.stringof); }
> 
> void main() {
>      immutable(int[]) a;
>      fun2(a);
>      int[] b;
>      fun2(b);
> }
> ```

This example is actually not illustrative. The first call involves a conversion from immutable(int[]) to immutable(int)[]. This happens even for non-templates. The second call involves a conversion from int[] to const(int[]), again a standard conversion. Neither call involves the hack discussed here, which is const(int[]) to const(int)[].

> In fact I'd love if we extend that to `auto` so the following code would work:
> ```
> immutable(int[]) fun2() { return null; }
> 
> void main() {
>      auto x = fun2();
>      x = fun2(); // Currently `x` is `immutable(int[])` not `immutable(int)[]`
> }
> ```

That'd work but would not improve the problem of working with const ranges.