On Wednesday, 24 June 2015 at 07:12:53 UTC, Iain Buclaw wrote:
> On 22 Jun 2015 08:40, "Andrei Alexandrescu via Digitalmars-d" < digitalmars-d@puremagic.com> wrote:
>>
>> On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:
>>>
>>> On 6/21/15 10:25 PM, Walter Bright wrote:
>>>>
>>>> The idea is that fun(5) would be lowered to:
>>>>
>>>>     auto tmp = 5;
>>>>     fun(tmp);
>>>
>>>
>>> I don't think that lowering is recommended - it prolongs the lifetime of the temporary through the end of the caller. But that may be actually a good thing.
>>
>>
>> On second thought - Walter's lowering, which makes the rvalue last more
> than strictly necessary, may be the most flexible of all at the cost of more resource consumption (for types that define destructors). -- Andrei
>>
>
> I think keeping the lifetime of objects strictly in the call expression is the behaviour that will give least surprise.

Call expression, or statement containing the call expression? For normal temporaries, it's the latter AFAIK.

>
> int i = 42;
> struct S { ~this() { i++; } }
>
> // auto __autoreftmp = S();
> foo(S());  // __autoreftmp
> // __dtor(__autoreftmp);
> assert(i == 43);

It would make a difference for:

    auto x = foo(S()) + foo(S());

>
> As for optimisations, I think it should be possible to assert that the reference never escapes, and use more aggressive optimisations based on that, something that is still not possible with 'scope ref' or 'in ref' parameters ...

Not possible because it's not implemented, or are there fundamental reasons against it?

On 24 June 2015 at 19:48, via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Wednesday, 24 June 2015 at 07:12:53 UTC, Iain Buclaw wrote:
>>
>> On 22 Jun 2015 08:40, "Andrei Alexandrescu via Digitalmars-d" < digitalmars-d@puremagic.com> wrote:
>>>
>>>
>>> On 6/21/15 11:31 PM, Andrei Alexandrescu wrote:
>>>>
>>>>
>>>> On 6/21/15 10:25 PM, Walter Bright wrote:
>>>>>
>>>>>
>>>>> The idea is that fun(5) would be lowered to:
>>>>>
>>>>>     auto tmp = 5;
>>>>>     fun(tmp);
>>>>
>>>>
>>>>
>>>> I don't think that lowering is recommended - it prolongs the lifetime of the temporary through the end of the caller. But that may be actually a good thing.
>>>
>>>
>>>
>>> On second thought - Walter's lowering, which makes the rvalue last more
>>
>> than strictly necessary, may be the most flexible of all at the cost of more resource consumption (for types that define destructors). -- Andrei
>>>
>>>
>>
>> I think keeping the lifetime of objects strictly in the call expression is the behaviour that will give least surprise.
>
>
> Call expression, or statement containing the call expression? For normal temporaries, it's the latter AFAIK.
>

I suppose DMD doesn't have an equivalent, but they are called BIND_EXPR in GCC, where you have a new local block with assigned temporaries that expire outside of the given bind.

>>
>> int i = 42;
>> struct S { ~this() { i++; } }
>>
>> // auto __autoreftmp = S();
>> foo(S());  // __autoreftmp
>> // __dtor(__autoreftmp);
>> assert(i == 43);
>
>
> It would make a difference for:
>
>     auto x = foo(S()) + foo(S());
>

It works like this, the last result is binded back to the lvalue.

auto x =  {
    tmpA = S();
    tmpB = S();

    foo(tmpA) + foo(tmpB);
}

>>
>> As for optimisations, I think it should be possible to assert that the reference never escapes, and use more aggressive optimisations based on that, something that is still not possible with 'scope ref' or 'in ref' parameters ...
>
>
> Not possible because it's not implemented, or are there fundamental reasons against it?

Ignoring the 'ref' part of my comment, so as we are not confusing DIP36 (which I've only just learned about). The last time I checked, the use of 'scope' and 'in' on parameters are ignored by escape analysis, so you can quite happily escape a reference as you see fit and the language will allow you to do that.

So, I cannot optimised if the language allows you to break that optimisation without warning or erroring the user.  If I did, then ultimately you'll end up seeing your programs becoming silently corrupted.

On Wednesday, 24 June 2015 at 16:39:22 UTC, kink wrote:
> On Wednesday, 24 June 2015 at 11:19:04 UTC, Jonathan M Davis wrote:
>> [...]
>>
>> 3. Add a new attribute which does what's being proposed for auto ref for non-templated functions, in which case, we can use the non-templated behavior with templates as well and thus avoid template bloat when all you want is for your templated function to accept both lvalues and rvalues. auto ref, of course, then stays exactly as it is now.
>>
>> At the moment, it seems that #2 is the most likely, and that's probably fine, but I do wonder if we'd be better off with #3, especially when you consider how much D code tends to be templated and how much code bloat auto ref is likely to generate with templated functions.
>>
>> - Jonathan M Davis
>
> If that wasn't clear before, I'm all for #3 too. Just call it `scope ref` and simplify the PR a lil' bit as suggested by Marc in an earlier post [http://forum.dlang.org/post/ricvtchihgzyisbkzcgl@forum.dlang.org].

But this has _nothing_ to do with scope, and scope ref was already rejected. The whole point of this is support having a function accept both rvalues and lvalues, not to do anything with scope.

And given that what scope does has never even been properly defined - all that the spec says about scope parameters is "references in the parameter cannot be escaped (e.g. assigned to a global variable)" - and that the only place that scope does anything at all is with delegates, trying to expand it with "scope ref" as if that were simply an extension of scope makes no sense. Before we can even consider what something like scope ref might mean, we'd have to properly define what scope means. And all we have for it is the basic idea of what it's supposed to do - none of the details - and trying to define scope ref before defining what scope means in general could totally hamstring us when properly defining scope later.

- Jonathan M Davis

On Wednesday, 24 June 2015 at 23:30:53 UTC, Jonathan M Davis wrote:
> But this has _nothing_ to do with scope, and scope ref was already rejected. The whole point of this is support having a function accept both rvalues and lvalues, not to do anything with scope.
>
> And given that what scope does has never even been properly defined - all that the spec says about scope parameters is "references in the parameter cannot be escaped (e.g. assigned to a global variable)".

Yeah right. And all we need to safely pass in rvalue references is exactly the constraint that they won't escape. And scope alone doesn't imply pass-by-ref (`in` params aren't currently passed by ref).

> [...] Before we can even consider what something like scope ref might mean, we'd have to properly define what scope means. And all we have for it is the basic idea of what it's supposed to do - none of the details - and trying to define scope ref before defining what scope means in general could totally hamstring us when properly defining scope later.

Is there a roadmap for "later"? It seems like these things always just get postponed, further postponed and never really done. What about the original argument 2 years ago, when rejecting DIP36, where Andrei explained that the idea was to make `ref` itself not escapable and delegating escaping refs to pointers? I don't see how that could be implemented without a breaking change, so I guess that may be something for D3. But we have been needing a solution for rvalue refs for years, and, as you know, are waiting since then, as `auto ref` for templates alone is simply not enough.

To quote Manu from the DIP36 thread:
> It is, without doubt, the single biggest complaint I've heard
> by virtually every programmer I've introduced D to.

+1 kazillion

On Wednesday, 24 June 2015 at 17:47:51 UTC, Namespace wrote:
> On Wednesday, 24 June 2015 at 17:45:15 UTC, Marc Schütz wrote:
>> On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis wrote:
>>> On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:
>>>> Simply give a possibility to "ref in" allowing use rvalues.
>>>
>>> That has already been rejected.
>>
>> Then we need to reconsider it. If that was several years ago, much has changed since then.
>>
>> Can you point me to that decision? Who made it, and on what basis? I can't find anything in the threads linked in http://wiki.dlang.org/DIP36
>
> Read the thread: http://forum.dlang.org/thread/ylebrhjnrrcajnvtthtt@forum.dlang.org?page=1

Thanks. There was no link to this thread on the wiki page.

But I can't really find a definite rejection in it, in the sense of someone with authority saying "This DIP is rejected". The closest I can find is Andrei making some vague statements about what is proposed being a new feature (I don't know though whether this is an argument against the DIP), and it being unsafe because scope is not implemented (which was true at the time, but we now have DIP25 as a mechanism, albeit incomplete).

If you can find anything definite, can you please add it to the wiki page? I.e. officially rejected by NN on YYYY-MM-DD (including link), summary of the arguments for rejection, etc.

On Wednesday, 24 June 2015 at 23:30:53 UTC, Jonathan M Davis wrote:
> On Wednesday, 24 June 2015 at 16:39:22 UTC, kink wrote:
>> On Wednesday, 24 June 2015 at 11:19:04 UTC, Jonathan M Davis wrote:
>>> [...]
>>>
>>> 3. Add a new attribute which does what's being proposed for auto ref for non-templated functions, in which case, we can use the non-templated behavior with templates as well and thus avoid template bloat when all you want is for your templated function to accept both lvalues and rvalues. auto ref, of course, then stays exactly as it is now.
>>>
>>> At the moment, it seems that #2 is the most likely, and that's probably fine, but I do wonder if we'd be better off with #3, especially when you consider how much D code tends to be templated and how much code bloat auto ref is likely to generate with templated functions.
>>>
>>> - Jonathan M Davis
>>
>> If that wasn't clear before, I'm all for #3 too. Just call it `scope ref` and simplify the PR a lil' bit as suggested by Marc in an earlier post [http://forum.dlang.org/post/ricvtchihgzyisbkzcgl@forum.dlang.org].
>
> But this has _nothing_ to do with scope, and scope ref was already rejected. The whole point of this is support having a function accept both rvalues and lvalues, not to do anything with scope.

It everything to do with scope, and I already explained it several times. As for the rejection, see my answer to Namespace:
http://forum.dlang.org/post/yxruhtwjllozgvpbhgik@forum.dlang.org

>
> And given that what scope does has never even been properly defined - all that the spec says about scope parameters is "references in the parameter cannot be escaped (e.g. assigned to a global variable)"

Even this current vague definition is already enough of a guarantee to allow rvalue references with it. Of course I'd be the last to object against a more detailed specification, obviously.

> - and that the only place that scope does anything at all is with delegates,

That's simply the status quo of the implementation and therefore can't be used as an argument.

> trying to expand it with "scope ref" as if that were simply an extension of scope makes no sense. Before we can even consider what something like scope ref might mean, we'd have to properly define what scope means. And all we have for it is the basic idea of what it's supposed to do - none of the details - and trying to define scope ref before defining what scope means in general could totally hamstring us when properly defining scope later.

I can assure you that it will not limit us. The very concept of borrowing/scope already requires some very specific restrictions with regards to what a function is allowed to do with a scope parameter. These same restrictions guarantee that it's allowed to pass an rvalue to it. Every working scope proposal will always guarantee that, or it wouldn't be usable.

If you still fear that it will impede us later, then at least this current proposal needs to be deferred until we have a scope proposal and have decided on it.

On Thursday, 25 June 2015 at 08:04:09 UTC, kink wrote:
> On Wednesday, 24 June 2015 at 23:30:53 UTC, Jonathan M Davis wrote:
>> But this has _nothing_ to do with scope, and scope ref was already rejected. The whole point of this is support having a function accept both rvalues and lvalues, not to do anything with scope.
>>
>> And given that what scope does has never even been properly defined - all that the spec says about scope parameters is "references in the parameter cannot be escaped (e.g. assigned to a global variable)".
>
> Yeah right. And all we need to safely pass in rvalue references is exactly the constraint that they won't escape. And scope alone doesn't imply pass-by-ref (`in` params aren't currently passed by ref).

Whether a reference escapes is an orthogonal issue. The return attribute is for dealing with that. The function should be able to return by ref or not and still accept both rvalues and lvalues for its parameters. As long as the temporary variable created for holding an rvalue exists for the duration of the statement that the function call is in, it doesn't matter. It should be perfectly fine to have the function return one of its ref arguments by ref (and thus avoid a copy), even if it was originally an rvalue, because it have been assigned to an lvalue that would be around long enough. The only issue arises when the caller then returns the return value by ref, in which case that's going to be illegal without the return attribute. But in general, the fact that a ref to an argument could be returned by ref is orthogonal to being able to pass in both lvalues and rvalues, because the rvalues all have to become lvalues to be passed in anyway, in which case, you're in exactly the same place with regards to safety that you are with normal ref.

>> [...] Before we can even consider what something like scope ref might mean, we'd have to properly define what scope means. And all we have for it is the basic idea of what it's supposed to do - none of the details - and trying to define scope ref before defining what scope means in general could totally hamstring us when properly defining scope later.
>
> Is there a roadmap for "later"? It seems like these things always just get postponed, further postponed and never really done.

Perhaps so, but just because scope hasn't properly defined yet isn't a reason to use it for something else entirely right now - especially when whether the lvalue that gets passed into the function is returned by ref is an orthogonal issue to having a type of ref which accepts an rvalue and silently converts it to an lvalue so that it can be passed by ref.

- Jonathan M Davis

On Thursday, 25 June 2015 at 08:10:06 UTC, Marc Schütz wrote:
> On Wednesday, 24 June 2015 at 17:47:51 UTC, Namespace wrote:
>> On Wednesday, 24 June 2015 at 17:45:15 UTC, Marc Schütz wrote:
>>> On Wednesday, 24 June 2015 at 09:54:01 UTC, Jonathan M Davis wrote:
>>>> On Wednesday, 24 June 2015 at 09:26:49 UTC, Temtaime wrote:
>>>>> Simply give a possibility to "ref in" allowing use rvalues.
>>>>
>>>> That has already been rejected.
>>>
>>> Then we need to reconsider it. If that was several years ago, much has changed since then.
>>>
>>> Can you point me to that decision? Who made it, and on what basis? I can't find anything in the threads linked in http://wiki.dlang.org/DIP36
>>
>> Read the thread: http://forum.dlang.org/thread/ylebrhjnrrcajnvtthtt@forum.dlang.org?page=1
>
> Thanks. There was no link to this thread on the wiki page.
>
> But I can't really find a definite rejection in it, in the sense of someone with authority saying "This DIP is rejected". The closest I can find is Andrei making some vague statements about what is proposed being a new feature (I don't know though whether this is an argument against the DIP), and it being unsafe because scope is not implemented (which was true at the time, but we now have DIP25 as a mechanism, albeit incomplete).
>
> If you can find anything definite, can you please add it to the wiki page? I.e. officially rejected by NN on YYYY-MM-DD (including link), summary of the arguments for rejection, etc.

If it helps, at dconf, two years ago, when Manu tried to repeatedly convince Walter and Andrei to have scope ref do something similar to what we're talking about auto ref doing, Walter and Andrei repeatedly shot it down. They had no interest in having scope ref as any kind of special attribute. So, as of the beginning of May 2013 (which was less than a month after what the DIP lists as its last modification date), it was being refused by Walter and Andrei in person. And if you look at its revision history, Dicebot (who is one of the two authors of the DIP) has marked it as rejected.

- Jonathan M Davis

On 06/25/2015 10:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
>
>
>> trying to expand it with "scope ref" as if that were simply an
>> extension of scope makes no sense. Before we can even consider what
>> something like scope ref might mean, we'd have to properly define what
>> scope means. And all we have for it is the basic idea of what it's
>> supposed to do - none of the details - and trying to define scope ref
>> before defining what scope means in general could totally hamstring us
>> when properly defining scope later.
>
> I can assure you that it will not limit us. The very concept of
> borrowing/scope already requires some very specific restrictions with
> regards to what a function is allowed to do with a scope parameter.
> These same restrictions guarantee that it's allowed to pass an rvalue to
> it. Every working scope proposal will always guarantee that, or it
> wouldn't be usable.
>
> If you still fear that it will impede us later, then at least this
> current proposal needs to be deferred until we have a scope proposal and
> have decided on it.

I think the arguments against allowing rvalues with scope ref are the same as those against allowing rvalues with ref, modulo accidental escaping. (It has been argued that 'ref' signifies an intent to modify, which I wouldn't necessarily agree with.)

On Thursday, 25 June 2015 at 10:10:42 UTC, Jonathan M Davis wrote:
> Whether a reference escapes is an orthogonal issue. The return attribute is for dealing with that. The function should be able to return by ref or not and still accept both rvalues and lvalues for its parameters. As long as the temporary variable created for holding an rvalue exists for the duration of the statement that the function call is in, it doesn't matter.

When I was talking about escaping references, I didn't mean returned references; I meant storing a pointer to a ref parameter somewhere outside (global/instance variable).

This snippet is a rough sketch for an approach based on `scope ref`, to prevent escaping rvalues:

<CODE>

struct S
{
    int a;
    ref S chain() return { return this; } // returns a `scope ref` for rvalues, `ref` for lvalues
}

// pointers derived from `scope ref` params can safely be passed down to scope pointer params only
void manipulateWithoutEscaping(scope ref S s) { manipulate(&s, S.sizeof); }

// doesn't escape any part of its `s` reference, hence `scope ref`
// analog to S.chain(), returns a `scope ref` for rvalues, `ref` for lvalues
ref S forward(return scope ref S s) { return s; }

// low-level manipulation functions not escaping any parts of their pointees
void manipulate(scope void* ptr, size_t size) { ptr[0..size] = 0; }
void manipulate(scope void[] data) { manipulate(data.ptr, data.length); }

struct State
{
    S* s;        // external ownership; should most likely actually be shared ownership
    void[] data; // ditto
    void initByEscaping(ref S s)
    {
        this.s = &s; // obvious
        data = (cast(void*)&s.a)[0 .. int.sizeof]; // less obvious
    }
    void manipulate() { if (s) manipulate(data); }
}

void foo()
{
    // forward rvalue ref and invoke chain() on it twice (each call forwarding the rvalue ref)
    // rvalue is destructed at the end of the statement
    forward(S(1)).chain().chain();

    // forward rvalue ref and let it be manipulated before destruction
    manipulateWithoutEscaping(forward(S(2)));

    // promote forwarded rvalue ref to lvalue
    S s3 = forward(S(3));
    // let the lvalue escape and manipulate it indirectly
    auto state = new State();
    state.initByEscaping(forward(s3)); // requires lvalue `ref`, forward() returns that for lvalue arg
    state.manipulate();
    // GC-managed `state` outlives `s3` => dangling pointers `state.s` and `state.data.ptr`!
}

</CODE>

The first observation is that a `scope ref` system would probably need to introduce scope pointers/slices too, to allow taking the address of `scope ref` params.
There'd probably be quite a few `scope` usages, cluttering code and introducing additional complexity.

I do see a point in trying to let `ref` alone and the compiler work it out. After all, that scope system would only allow us to prevent ourselves from escaping rvalues (if we cared to type 'scope'). With escape analysis, the compiler could disallow attempts to bind rvalues to escapable references, by inferring scope-ness for ref and pointer params automatically. In the future, escaping params could be disallowed in @safe functions, and otherwise require an explicit `ref` keyword when supplying the lvalue argument, hinting at the potential for dangling pointers.

In the meantime, what about making all `ref` params accept rvalues via lowering (at least for @system)? As a first step until escape analysis is sufficiently implemented and the compiler will help us out with compile errors for escaping rvalues..