On Saturday, 27 June 2015 at 00:13:08 UTC, kinke wrote:
> On Thursday, 25 June 2015 at 10:10:42 UTC, Jonathan M Davis wrote:
>> Whether a reference escapes is an orthogonal issue. The return attribute is for dealing with that. The function should be able to return by ref or not and still accept both rvalues and lvalues for its parameters. As long as the temporary variable created for holding an rvalue exists for the duration of the statement that the function call is in, it doesn't matter.
>
> When I was talking about escaping references, I didn't mean returned references; I meant storing a pointer to a ref parameter somewhere outside (global/instance variable).

That's a completely orthogonal issue to ref or auto ref. That's an @system operation (since taking the address of a local variable is @system), and it's up to you to not screw it up. scope might cover that if it were fully ironed out, since that does involve escaping, but it also might not, since it's an @system operation and thus up to you not to screw it up. Regardless, it has nothing to do with whether the function accepts lvalues and rvalues with the same parameter. You have a safety issue regardless if you're talking about taking a pointer to an argument (or to some portion of the argument), since there's no guarantee that the lifetime of the pointer is shorter than that of the argument. That depends entirely on what the argument was or what you do with the pointer, and very few variables have a global lifetime, so whether the argument passed to the ref parameter referred to a local variable is pretty much irrelevant. You have to know what you're doing with you take the address of a variable and make sure that you do it right.

Safety with regards to ref is an entirely different issue, because that's @safe code. Also, whether the function might return its ref parameter by ref is orthogonal to whether it accepted an rvalue by ref or not, since whether you want to do that or not doesn't necessarily have anything to with what's being returned from the function (rather, it has to do with what you want the function to accept), and you have a safety issue even without having any kind of ref which accepts rvalues, because all it requires is multiple layers of functions which return by ref, and you're screwed anyway. e.g.

ref int id(ref int i)
{
    return i;
}

ref int foo(ref int i)
{
    ++i;
    return id(i);
}

ref int bar()
{
    int i;
    return foo(i);
}

We introduced the return attribute (currently only with -dip25) to fix this problem. So, with that, we eliminate the safety problem with ref itself, and we can safely have auto ref accept rvalues by having it assign them to a temporary variable first. The rules with returning by ref would then be the same with auto ref as ref and require the return attribute to return a parameter. And all of this is within @safe code.

Any issues with taking pointers to arguments is firmly in @system territory and is thus completely orthogonal. Even if scope were to be implemented in a way that prevented it, it would still have nothing to do with whether the function accepted both rvalues and lvalues with the same parameter.

> In the meantime, what about making all `ref` params accept rvalues

Please, please, no. ref indicates that your intention is to mutate the argument. Having it accept rvalues completely destroys that and makes it that much harder to understand what a function is supposed to do. By having a separate attribute such as auto ref which also accepts rvalues, we cleanly separate out the code which just wants to accept both rvalues and lvalues for efficiency purposes and the code that actually wants to take its argument by ref so that it can mutate it.

- Jonathan M Davis

On Saturday, 27 June 2015 at 01:18:19 UTC, Jonathan M Davis wrote:
> On Saturday, 27 June 2015 at 00:13:08 UTC, kinke wrote:
>> When I was talking about escaping references, I didn't mean returned references; I meant storing a pointer to a ref parameter somewhere outside (global/instance variable).
>
> That's a completely orthogonal issue to ref or auto ref. That's an @system operation (since taking the address of a local variable is @system), and it's up to you to not screw it up. scope might cover that if it were fully ironed out, since that does involve escaping, but it also might not, since it's an @system operation and thus up to you not to screw it up.

The point is that with scope, it _doesn't_ need to be @system anymore. As an example where this can be useful, see:
https://github.com/D-Programming-Language/phobos/blob/41d2027c22d6f8d22edd2df688c3cfd08fb20228/std/digest/hmac.d#L252
We currently can't make this function @safe, although it is in fact memory safe.

> Regardless, it has nothing to do with whether the function accepts lvalues and rvalues with the same parameter. You have a safety issue regardless if you're talking about taking a pointer to an argument (or to some portion of the argument), since there's no guarantee that the lifetime of the pointer is shorter than that of the argument.

No, not with scope. It seems you're using the status quo to argue against what could be. But that's not valid, because the problems you mention will no longer exist with scope.

> We introduced the return attribute (currently only with -dip25) to fix this problem. So, with that, we eliminate the safety problem with ref itself, and we can safely have auto ref accept rvalues by having it assign them to a temporary variable first.

Yes this is now possible, but why in all world do you want the keyword that enables it to be `auto ref`?

> The rules with returning by ref would then be the same with auto ref as ref and require the return attribute to return a parameter. And all of this is within @safe code.
>
> Any issues with taking pointers to arguments is firmly in @system territory and is thus completely orthogonal. Even if scope were to be implemented in a way that prevented it, it would still have nothing to do with whether the function accepted both rvalues and lvalues with the same parameter.
>
>> In the meantime, what about making all `ref` params accept rvalues
>
> Please, please, no. ref indicates that your intention is to mutate the argument. Having it accept rvalues completely destroys that and makes it that much harder to understand what a function is supposed to do.

I can accept that argument, although I see cases where it's still justified. For example, if the author of the function decided it wants to mutate the argument, but you (the caller) are not interested in the new value. But still, explicitly using a temporary named `dummy` is preferable because it even documents that you're going to ignore it.

> By having a separate attribute such as auto ref which also accepts rvalues, we cleanly separate out the code which just wants to accept both rvalues and lvalues for efficiency purposes and the code that actually wants to take its argument by ref so that it can mutate it.

Okay, then you mainly want to use special syntax to opt-in to rvalue references. I simply think that `auto ref` is completely inadequate for this purpose, because its existing meaning with templates doesn't have anything to do with rvalue refs.

`scope ref` on the other hand is more fitting IMO. It's not that `scope` necessarily implies rvalue refs, but rvalue refs _do_ imply scope! If we go with DIP25 where `ref` by itself already implies that, `scope ref` would then be redundant, and therefore the syntax would be available for the special purpose of accepting rvalue refs. This has multiple advantages:

- No new keywords or @attributes.
- Rvalue refs are still opt-in, you don't get them accidentally just by writing `ref`.
- It reinforces the guarantee that the argument will not escape.
- It's usable in templates, too, with the same meaning as in normal functions.

On Saturday, 27 June 2015 at 11:10:49 UTC, Marc Schütz wrote:
> On Saturday, 27 June 2015 at 01:18:19 UTC, Jonathan M Davis wrote:
>> That's a completely orthogonal issue to ref or auto ref. That's an @system operation (since taking the address of a local variable is @system), and it's up to you to not screw it up. scope might cover that if it were fully ironed out, since that does involve escaping, but it also might not, since it's an @system operation and thus up to you not to screw it up.
>
> The point is that with scope, it _doesn't_ need to be @system anymore.
>
>> Regardless, it has nothing to do with whether the function accepts lvalues and rvalues with the same parameter. You have a safety issue regardless if you're talking about taking a pointer to an argument (or to some portion of the argument), since there's no guarantee that the lifetime of the pointer is shorter than that of the argument.
>
> No, not with scope. [...] the problems you mention will no longer exist with scope.

Exactly, that's the whole point.
The relation with lvalue/rvalue reference parameters is that the compiler should be able to statically detect and refuse escaping rvalue bugs. Escaping lvalues with potential for dangling pointers should be @system only, but preventing those bugs is a whole different beast.

Still, to make (a) taking the address of ref params, locals & globals and (b) using pointer params @safe, the compiler needs to know whether the parameter escapes or not. I tend to let the compiler figure that out in the future rather than having to use `scoped` a lot for ref and pointer params.

>> We introduced the return attribute (currently only with -dip25) to fix this problem. So, with that, we eliminate the safety problem with ref itself, and we can safely have auto ref accept rvalues by having it assign them to a temporary variable first.
>
> Yes this is now possible, but why in all world do you want the keyword that enables it to be `auto ref`?

Exactly. For a function returning a ref parameter, it doesn't matter whether it's an rvalue or lvalue. The important thing is to propagate that distinction (automatically by the compiler), to disallow binding a returned rvalue reference to an escaping ref parameter (i.e., no `escape(forward(S(1)))`).
Where the distinction between lvalues and rvalues matters for a function returning a ref parameter, as for any other function with ref params, is whether the parameter escapes (i.e., by storing a pointer somewhere). If it does, it should be `return ref`, otherwise `return scope ref` (either explicitly or inferred automatically by the compiler).

>> Any issues with taking pointers to arguments is firmly in @system territory and is thus completely orthogonal. Even if scope were to be implemented in a way that prevented it, it would still have nothing to do with whether the function accepted both rvalues and lvalues with the same parameter.

Of course it would if one made actually use of a scope system (again, either explicit or as a result of compiler escape analysis) to (a) rule out all escaping rvalue bugs and (b) possibly warn about or highlight escaping lvalue params in @system code. The main aim of the scope system would be to enhance the flexibility of @safe code though.

>>> In the meantime, what about making all `ref` params accept rvalues
>>
>> Please, please, no. ref indicates that your intention is to mutate the argument. Having it accept rvalues completely destroys that and makes it that much harder to understand what a function is supposed to do.

If that was the only reason for not accepting rvalues as `ref` params, then rvalues should have been allowed for `const ref` right from the beginning, as there's surely absolutely no intention to mutate a `const ref` argument.

> I can accept that argument, although I see cases where it's still justified. For example, if the author of the function decided it wants to mutate the argument, but you (the caller) are not interested in the new value. But still, explicitly using a temporary named `dummy` is preferable because it even documents that you're going to ignore it.

I find letting the callee decide whether the caller needs to provide an lvalue (`ref`) or not (non-templated `auto ref`), based solely on making sure the mutations are directly visible by the caller after the call, too much. I think it should be left to the caller to decide what mutations it's interested in.

But now I see why some would oppose calling that `scope ref`, as that really has nothing to do with scope or safety, that'd just be an arbitrary way of classifying ref params in 'essential output' (mutable lvalue `ref`s, mutations need to be visible by caller) and some other category (lvalue/rvalue `auto ref`s, not just for efficiency purposes). That really is an orthogonal and incompatible semantic compared to what `scope ref` is all about, and doesn't interest me at all. I find flexible @safe code and detecting escaping bugs a kazillion times more important than making sure a caller sees all mutations the callee deems essential after the call.

On Saturday, 27 June 2015 at 11:10:49 UTC, Marc Schütz wrote:
> On Saturday, 27 June 2015 at 01:18:19 UTC, Jonathan M Davis wrote:

> No, not with scope. It seems you're using the status quo to argue against what could be. But that's not valid, because the problems you mention will no longer exist with scope.

If we want to define what scope does and implement it, that's a separate issue from having a function accept both rvalues and lvalues with the same parameter. It would be trivial to have a function where you want to guarantee that no pointer to a ref parameter escapes but you still don't want to accept rvalues, because the whole point is to mutate the argument. Wanting to have a function accept lvalues and rvalues is thus fundamentally different from caring about issues with escaping. Yes, there is some overlap in terms of safety issues when it comes to returning by ref, but that's it, and the return attribute solves that problem, but whether you escape pointers to a parameter is fundamentally separate from whether a function should accept rvalues, because escaping is not the only issue, and ref itself has all of the same issues with escaping whether it accepts rvalues or not.

So, it really doesn't matter what scope might protect against in terms of escaping and what it might not if we were to fully define it (e.g. maybe it would protect against taking the address of a variable, and maybe not; that could be simply considered @system, or it could be protected against). We have to actually define it to know what it would or wouldn't do. All we have is the basic idea of what it's supposed to have been for. The _only_ thing that it applies to at this point is delegates. There's little point in arguing about what it could do until we're talking about actually defining it. But regardless, the issue of accepting rvalues by ref is orthogonal, because that is not solely an issue of escaping. It's also an issue of _why_ the argument is passed by ref in the first place.

>> We introduced the return attribute (currently only with -dip25) to fix this problem. So, with that, we eliminate the safety problem with ref itself, and we can safely have auto ref accept rvalues by having it assign them to a temporary variable first.
>
> Yes this is now possible, but why in all world do you want the keyword that enables it to be `auto ref`?

That's what auto ref was originally introduced to do. The reason that it works only with templates is because Walter misunderstood what Andrei was proposing. So, it makes sense to use it for what it was originally intended for and implement it as proposed with non-templated functions. Now, that has the downside that we can't use it with templated functions in a fashion that avoids template bloat, in which case, using a new attribute would make sense, but this _is_ what auto ref was originally for. And if we're going to use a new attribute, scope ref makes no sense precisely because the escaping issue is only part of the problem with accepting lvalues and rvalues with the same parameter, since you could easily have a function which is not supposed to escape the argument in any way, and yet it needs to be mutating the argument via ref for the caller, and it should not be accepting rvalues. So, in that case, if scope were fully ironed out, it might make sense to use scope ref when the intention was that the argument be mutated and not accept rvalues and that in not escape the function. It could also be that scope ref prevents returning by ref, since that's escaping the function, whereas when you pass an rvalue to the non-templated auto ref, you may very well want to return it by ref. So, if we were to use scope ref for this, we'd be mixing two largely orthogonal issues - escaping arguments and having a function accept both lvalues and rvalues - and it would cause us problems if and when we actually, fully defined scope.

We need to either use auto ref for this as originally intended or use a new attribute which does so so that we can use it with both templated and non-templated functions. I think that it's quite clear that scope ref does _not_ make sense in this case, and it's already been rejected by Walter and Andrei anyway.

- Jonathan M Davis

On 28 June 2015 at 08:50, Jonathan M Davis via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> [..]
>
>
> That's what auto ref was originally introduced to do. The reason that it works only with templates is because Walter misunderstood what Andrei was proposing. So, it makes sense to use it for what it was originally intended for and implement it as proposed with non-templated functions. Now, that has the downside that we can't use it with templated functions in a fashion that avoids template bloat,

Stop repeating that. It's not got anything to do with code bloat, you
can't use it because template auto ref doesn't pass rvalues by ref, it
passes them by value, which means it's not a ref in any way, and it's
not what the user wants.
I can easily get the same behaviour by not-typing 'auto ref', which I
happily do in all cases that I want something to be passed by value.
But we're not talking about by-val, we're talking about ref.

It's simple; I want to pass something by value, or I want it to pass by ref... there is no middle ground, it's an explicit binary statement about how the parameter shall be passed. I don't care about lvalues or rvalues. There is no case where I want the compiler to not do the thing I told it to. And if there were a hypothetical case - where ref-ness were dependent on some conditions, I'm confident I could easily express that with existing mechanisms in D. I don't need a weird compiler defined semantic to try and do something smart that turns out never to be what I want.


> in which case, using a new attribute would make
> sense, but this _is_ what auto ref was originally for. And if we're going to
> use a new attribute, scope ref makes no sense precisely because the escaping
> issue is only part of the problem with accepting lvalues and rvalues with
> the same parameter, since you could easily have a function which is not
> supposed to escape the argument in any way, and yet it needs to be mutating
> the argument via ref for the caller, and it should not be accepting rvalues.
> So, in that case, if scope were fully ironed out, it might make sense to use
> scope ref when the intention was that the argument be mutated and not accept
> rvalues and that in not escape the function. It could also be that scope ref
> prevents returning by ref, since that's escaping the function, whereas when
> you pass an rvalue to the non-templated auto ref, you may very well want to
> return it by ref. So, if we were to use scope ref for this, we'd be mixing
> two largely orthogonal issues - escaping arguments and having a function
> accept both lvalues and rvalues - and it would cause us problems if and when
> we actually, fully defined scope.
>
> We need to either use auto ref for this as originally intended or use a new attribute which does so so that we can use it with both templated and non-templated functions. I think that it's quite clear that scope ref does _not_ make sense in this case, and it's already been rejected by Walter and Andrei anyway.

scope ref is only related to the safety of passing temporaries by reference to functions that may retain a pointer, it has nothing to do with lvalues or rvalues; stack locals are equally unsafe right now as rvalues would be.

const is the thing you're looking for that says "this isn't to be changed", such that it makes sense to pass an rvalue.

scope was proposed as a means to address the "rvalues -> ref is
unsafe, the callee might sequester a pointer somehow" argument. If
we're not interested in scope (ie, escape analysis), then we're not
interested in ref safety in general (proposed scope applies to a much
wider set of cases than just rvalues).
That's fine, I don't really care, but then you can no longer make the
argument above that safety affects this debate, at which point you
arrive at C++ where rvalues can pass to const ref, as should be the
case here. End this auto-ref nonsense, const ref is enough.

On Sunday, 28 June 2015 at 05:46:11 UTC, Manu wrote:
> On 28 June 2015 at 08:50, Jonathan M Davis via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>>
>>> [..]
>>
>>
>> That's what auto ref was originally introduced to do. The reason that it works only with templates is because Walter misunderstood what Andrei was proposing. So, it makes sense to use it for what it was originally intended for and implement it as proposed with non-templated functions. Now, that has the downside that we can't use it with templated functions in a fashion that avoids template bloat,
>
> Stop repeating that. It's not got anything to do with code bloat, you
> can't use it because template auto ref doesn't pass rvalues by ref, it
> passes them by value, which means it's not a ref in any way, and it's
> not what the user wants.
> I can easily get the same behaviour by not-typing 'auto ref', which I
> happily do in all cases that I want something to be passed by value.
> But we're not talking about by-val, we're talking about ref.
>
> It's simple; I want to pass something by value, or I want it to pass by ref... there is no middle ground, it's an explicit binary statement about how the parameter shall be passed. I don't care about lvalues or rvalues. There is no case where I want the compiler to not do the thing I told it to. And if there were a hypothetical case - where ref-ness were dependent on some conditions, I'm confident I could easily express that with existing mechanisms in D. I don't need a weird compiler defined semantic to try and do something smart that turns out never to be what I want.

It makes no sense to pass rvalues by ref. The ref has to refer to a memory location so that a pointer can be passed underneath the hood, and rvalues don't qualify for that. A temporary variable _must_ be created in order to pass an rvalue by ref, even if the compiler is able to move the rvalue into the temporary variable rather than copy it. So, I don't understand why you would be looking to pass rvalues by ref. I can see why you would want to pass both rvalues and lvalues to the same function efficiently without caring about whether they're rvalues or lvalues, but that doesn't necessarily mean ref, and in the case of rvalues, it definitely doesn't mean ref.

The most efficient way to pass rvalues is by value, because the value can be moved or even simply constructed directly into the right place on the stack to begin with. That's why C++11 added move constructors and why D has postlblit constructors. That's also why it's no longer the advice in C++-land for folks to simply always use const& as much as possible (not that it shouldn't be used, but whether it should be used or not becomes much more complicated). D has moving built-in without the need for move constructors, so our situation is simpler, but regardless, ref really isn't the answer when it comes to rvalues. You _want_ to be passing rvalues by value. It's the lvalues that you don't want to pass by value if you're worried about efficiency, because they're the ones that are going to have to be copied. But still, if what you want is to have rvalues put into a place on the stack where they can be passed by ref and then have them passed by ref, that's what you'll get with auto ref for non-templated functions. It'll pass lvalues by ref, and it'll create temporary variables for the rvalues so that they can be passed by ref. So, as far as I can tell, auto ref for non-templated functions does exactly what you're looking for. It's just that it only works for non-templated functions, and with the templated ones, rvalues are passed efficiently - but not by ref.

- Jonathan M Davis

All values must be passed by ref because otherwise it will be leading for code bloat : we want eliminate by having only one function that accepts both rvalues and lvalues.

Thank you Jonathan, I think I (like kinke in his post above) finally understand your stance. So I guess from your POV the following would be an ideal situation (disregarding the need to avoid breaking changes):

1) `auto ref` for non-templates is required to make a function accept rvalue and lvalue refs alike.
2) `auto ref` for templates is changed to mean the same thing as 1).
3) `scope ref` prevents escaping of the reference.

Neither of `auto ref` and `scope ref` would imply the other, because you want to treat them as orthogonal. Do I understand you right?

The main problems with this is that they aren't in fact orthogonal: `auto ref` without `scope ref` almost only makes sense if you want to deliberately break something. Furthermore, `auto ref` is already taken for something else, and I'm not sure Walter would be too happy if we wanted to change its meaning for templates.

On Sunday, 28 June 2015 at 07:36:41 UTC, Jonathan M Davis wrote:
> It makes no sense to pass rvalues by ref. The ref has to refer to a memory location so that a pointer can be passed underneath the hood, and rvalues don't qualify for that.

I strongly disagree with that. This is an implementation detail. `ref` describes a specific semantic, namely that no copying (observable by a postblit being called) takes place, and that it refers to the same "thing" as the original (observable by mutations being visible on the original object, if relevant). How a value is stored is completely up to the compiler. It can be in a register, a memory location, or even - if the compiler knows the value in advance - as an immediate of a machine instruction. The same is true for passing values: the compiler is free to implement it as it wants, as long as it keeps the required semantics.

On Sunday, 28 June 2015 at 11:06:01 UTC, Marc Schütz wrote:
> On Sunday, 28 June 2015 at 07:36:41 UTC, Jonathan M Davis wrote:
>> It makes no sense to pass rvalues by ref. The ref has to refer to a memory location so that a pointer can be passed underneath the hood, and rvalues don't qualify for that.
>
> I strongly disagree with that. This is an implementation detail. `ref` describes a specific semantic, namely that no copying (observable by a postblit being called) takes place,

no, it means pass by reference.
this sounds like some convoluted C++ doublespeak definition.