On Saturday, 27 April 2013 at 03:29:10 UTC, Diggory wrote:
> I'm starting to think there does need to be new syntax if all three of the above cases are to be covered - it would work just making "ref" accept rvalues but then "move" would be the default for lvalues, and that would be confusing - but I don't think using "scope" solves any problems, as the semantics required for rvalues are orthogonal to what "scope" actually means.

I think 'scope' is yet to be fully defined. Returning a reference doesn't necessarily mean escaping unsafely. The primary goal is safety. From the perspective of the calling function, something assigned 'scope' remains in scope even if it's returned.

scope ref T gin(scope ref T a) { return a; }
void fun() {
  T t;
  gin(t); // It's still in scope
}

The return is in the outer scope still, so 'scope' is not totally out of place. But it does seem like it's pushing it, admittedly.

On Saturday, 27 April 2013 at 06:37:38 UTC, Manu wrote:
> scope ref T func(scope ref T t) { return t; }
>
> I think this solves the problem.

struct S {
  scope ref S func() { return this; }
}

Does 'scope' apply to the return value or to the hidden 'this' parameter? Or both? I think it makes a difference...

> I think 'scope' is yet to be fully defined. Returning a reference doesn't necessarily mean escaping unsafely. The primary goal is safety. From the perspective of the calling function, something assigned 'scope' remains in scope even if it's returned.

The keyword 'scope' is very well defined at the moment - it means among other things that the value cannot be returned. DIP25A depends on this being the case, changing it will break any hope of sensible semantics for safe references. I agree that rvalue references can be returned safely - that's my main point - so using "scope" to identify them makes no sense.

On Saturday, 27 April 2013 at 16:13:21 UTC, Zach the Mystic wrote:
> On Saturday, 27 April 2013 at 06:37:38 UTC, Manu wrote:
>> scope ref T func(scope ref T t) { return t; }
>>
>> I think this solves the problem.
>
> struct S {
>   scope ref S func() { return this; }
> }
>
> Does 'scope' apply to the return value or to the hidden 'this' parameter? Or both? I think it makes a difference...

Unless the meaning of "scope" is dramatically changed it doesn't make sense for it to be applied to a return value, since scope values cannot be returned. Therefore it unambiguously refers to the hidden 'this' pointer.

On Saturday, April 27, 2013 13:11:54 Namespace wrote:
> You often say that 'auto ref' can not work for non-templates.

Please quote at least part of the message that you're replying to. The threading of the various clients is often wrong, and I have no way of knowing who you're talking to here (and some people don't use threading at all). Based on the content, I'm guessing that you're talknig to me, but I really don't know.

> Why not really?
> Because the resulting code bloat is not bearable, or because it
> is technically not possible?

Because the templated solution involves creating new functions, whereas that's not going to work for non-templated functions, particularly when you consider .di files and prototypes. It also gets particularly nasty when function overriding gets added into the mix. Walter himself has said several times in the past that the solution used for auto ref and templates can't work with non-templated functions.

> Also I would like to see an answer of my overlooked question
> (just out of curiosity):
> http://forum.dlang.org/thread/kl4v8r$tkc$1@digitalmars.com?page=7#post-kyicm
> dsriwnxqiuzkaho:40forum.dlang.org

So, you're asking why the compiler can't just do a move in the rvalue case and pass by ref in the lvalue case for auto ref? The semantics are completely different. You need _one_ function, and you can't do that with one function. That would be like asking the function to be both

auto foo(int* i) {...}

and

auto foo(int i) {...}

only we're talking about ref rather than pointers (but ref is really a pointer underneat the hood). A function can only have one signature unless it's a template, and the only way that templates can have more is because they generate multiple functions (each with only one signature).

Remember that we're dealing with the C linker here, so we're pretty much restricted to what you can ultimately do in C as far as function signatures go. We have to do name mangling to do function overloading or add additional information to function signatures (such as pure or nothrow), and the only information you ultimately have is the function signature, so anything requiring messing with how the function works either must be done inside the function (where it knows nothing about the caller), or you need multiple functions, which generally means templates. To get around all of that, we'd need to create our own linker which didn't work like C's linker does, which would create its own set of problems.

- Jonathan M Davis

On 4/27/13 8:17 PM, Jonathan M Davis wrote:
> Walter himself has said several times in
> the past that the solution used for auto ref and templates can't work with
> non-templated functions.

I think he's partly wrong about that. Yes, the same mechanics won't work, but the same outcome can be made to work.

Andrei

On Saturday, April 27, 2013 21:17:07 Andrei Alexandrescu wrote:
> On 4/27/13 8:17 PM, Jonathan M Davis wrote:
> > Walter himself has said several times in
> > the past that the solution used for auto ref and templates can't work with
> > non-templated functions.
> 
> I think he's partly wrong about that. Yes, the same mechanics won't work, but the same outcome can be made to work.

Yes. I agree. It's how auto ref is implemented for templated functions which won't work, not the idea. As I've described here and elsewhere, it could be done by simply making it so that auto ref translated to ref underneath the hood with the difference that when an rvalue was passed to it, a variable would be created on the stack which the rvalue would be assigned to so that there would be an lvalue to pass to the function by ref. As long as the lifetime of the variable matches that of the statement that the function, then it should be fine.

But the question here was why we couldn't just use the same solution for auto ref as templated functions with non-templated functions, and I think that it's fairly clear that that won't work as that relies on being able to generate additional functions. It also creates a combinatorial explosion of functions, so heavily using auto ref for templated functions could create a lot of unnecessary code bloat, which is why it would be so desirable to be able to use the non-templated functions solution for auto ref with template functions (either by introducing a new attribute - either for what auto ref does for templated functions now or for the new auto ref implementation - or by doing what Timon suggested and treating the new auto ref implementation as an optimization for templated functions, but that does require the compiler to have some smarts).

- Jonathan M Davis

On Saturday, 27 April 2013 at 23:38:53 UTC, Diggory wrote:
> The keyword 'scope' is very well defined at the moment - it means among other things that the value cannot be returned.

It's only defined very briefly in the "functions" section of the  documentation. And it's only implemented for 'scope' delegates, because they get a clear benefit. Making 'ref' safe, despite how important it is, is not yet encoded in stone. DIP25A has been admitted by its author to be an aggressive solution. But there are several possible approaches to dealing with that, IMO. The most immediate is to simply force the author to mark any function which violates the rules '@trusted'. In other words, sweep all unsafe cases under the umbrella of the existing @safe-D framework. Perhaps the only reason to consider anything else is that the number of functions which will have to be marked '@trusted' will be too high.

I think it's important to decide whether '@trusted' is adequate to address 'ref' safety before getting to the next step. I even think that a full implementation of DIP25A, as is, would help to determine for people just how many functions they will be forced to mark '@trusted'. Only if '@trusted' really ends up being too blunt should attributes be considered, IMO.

That being said, 'scope' might be the right tool for the job. Or 'out' return values. I'm not under the impression that 'scope' *must* be used to solve the problem. For example, there are even more than one way to escape a parameter, and they're not all equal.

ref T func(ref T a, ref T* b) {
  return a; // Escape by return
  static T* t;
  t = &a; // Escape by global assign
  b = &a; // Escape by parameter assign
}

'scope' is only one attribute, yet there are three different types of escape here. That's why I don't want to jump in completely on 'scope' banning all three.

On Sunday, 28 April 2013 at 00:17:53 UTC, Jonathan M Davis wrote:
> On Saturday, April 27, 2013 13:11:54 Namespace wrote:
>> You often say that 'auto ref' can not work for non-templates.
>
> Please quote at least part of the message that you're replying to. The
> threading of the various clients is often wrong, and I have no way of knowing
> who you're talking to here (and some people don't use threading at all). Based
> on the content, I'm guessing that you're talknig to me, but I really don't
> know.
>
>> Why not really?
>> Because the resulting code bloat is not bearable, or because it
>> is technically not possible?
>
> Because the templated solution involves creating new functions, whereas that's
> not going to work for non-templated functions, particularly when you consider
> .di files and prototypes. It also gets particularly nasty when function
> overriding gets added into the mix. Walter himself has said several times in
> the past that the solution used for auto ref and templates can't work with
> non-templated functions.
>
>> Also I would like to see an answer of my overlooked question
>> (just out of curiosity):
>> http://forum.dlang.org/thread/kl4v8r$tkc$1@digitalmars.com?page=7#post-kyicm
>> dsriwnxqiuzkaho:40forum.dlang.org
>
> So, you're asking why the compiler can't just do a move in the rvalue case and
> pass by ref in the lvalue case for auto ref? The semantics are completely
> different. You need _one_ function, and you can't do that with one function.
> That would be like asking the function to be both
>
> auto foo(int* i) {...}
>
> and
>
> auto foo(int i) {...}
>
> only we're talking about ref rather than pointers (but ref is really a pointer
> underneat the hood). A function can only have one signature unless it's a
> template, and the only way that templates can have more is because they
> generate multiple functions (each with only one signature).
>
> Remember that we're dealing with the C linker here, so we're pretty much
> restricted to what you can ultimately do in C as far as function signatures
> go. We have to do name mangling to do function overloading or add additional
> information to function signatures (such as pure or nothrow), and the only
> information you ultimately have is the function signature, so anything
> requiring messing with how the function works either must be done inside the
> function (where it knows nothing about the caller), or you need multiple
> functions, which generally means templates. To get around all of that, we'd
> need to create our own linker which didn't work like C's linker does, which
> would create its own set of problems.
>
> - Jonathan M Davis

Sorry but you're right I meant you. Thanks for explanation.

On Sunday, 28 April 2013 at 05:19:50 UTC, Zach the Mystic wrote:
> On Saturday, 27 April 2013 at 23:38:53 UTC, Diggory wrote:
>> The keyword 'scope' is very well defined at the moment - it means among other things that the value cannot be returned.
>
> It's only defined very briefly in the "functions" section of the  documentation. And it's only implemented for 'scope' delegates, because they get a clear benefit. Making 'ref' safe, despite how important it is, is not yet encoded in stone. DIP25A has been admitted by its author to be an aggressive solution. But there are several possible approaches to dealing with that, IMO. The most immediate is to simply force the author to mark any function which violates the rules '@trusted'. In other words, sweep all unsafe cases under the umbrella of the existing @safe-D framework. Perhaps the only reason to consider anything else is that the number of functions which will have to be marked '@trusted' will be too high.
>
> I think it's important to decide whether '@trusted' is adequate to address 'ref' safety before getting to the next step. I even think that a full implementation of DIP25A, as is, would help to determine for people just how many functions they will be forced to mark '@trusted'. Only if '@trusted' really ends up being too blunt should attributes be considered, IMO.
>
> That being said, 'scope' might be the right tool for the job. Or 'out' return values. I'm not under the impression that 'scope' *must* be used to solve the problem. For example, there are even more than one way to escape a parameter, and they're not all equal.
>
> ref T func(ref T a, ref T* b) {
>   return a; // Escape by return
>   static T* t;
>   t = &a; // Escape by global assign
>   b = &a; // Escape by parameter assign
> }
>
> 'scope' is only one attribute, yet there are three different types of escape here. That's why I don't want to jump in completely on 'scope' banning all three.

OK, but I think it should be made clear that using scope for rvalue references would change the meaning of an attribute that in the near future will be very useful indeed as-is, and we will probably end up wanting to make a new attribute "@noreturn" or something if scope now means something else (TBH I would not really be against using "@noreturn" as it's somewhat clearer than "scope" but swapping one attribute for another seems like something people would be against?)

It's quite clear to me that no mathematical model for ref safety will be able to cover every possible case without seriously degrading performance. Therefore the best we can do is come up with a model which covers as many of the common situations as possible and rely on "@trusted" to handle the (hopefully small) number of cases which cannot be handled by the model.

So far the rules in DIP25A+DIP35 seem to get the closest to this ideal model, and there's no reason we can't extend the rules even further to cover even more cases in the future.

On Sunday, 28 April 2013 at 16:56:26 UTC, Diggory wrote:
> It's quite clear to me that no mathematical model for ref safety will be able to cover every possible case without seriously degrading performance. Therefore the best we can do is come up with a model which covers as many of the common situations as possible and rely on "@trusted" to handle the (hopefully small) number of cases which cannot be handled by the model.

This is my thinking also.

One rather bad thing about the '@trusted' method is that it's not the called function itself which is unsafe, but only escaping the *returned ref*, and so '@trusted' would most accurately have to attach to the *calling* function, which I think will become untenable rather quickly.

ref T func(ref T a); // Safe in and of itself

@trusted ref T fun2() {
  T b, c;
  b = func(c); // safe
  return func(c); // not safe!!
}

Since func() has no additional attributes and is invisible, there's no way at all for the compiler to know what it returns. Yet the actual unsafe action only occurs in fun2(), which makes it easy to see how unwieldy the system could get, having to mark the calling function just because it does something which *could* be unsafe.

Looking at the above case makes it seem like at least one additional attribute, located in the function signature, must be accessible to the compiler in to order to reduce the number of functions which must otherwise rather gratuitously be marked '@trusted'. Even with a single new attribute, 'out', which attaches to the function (and does not specify precisely to which parameters it refers) I suspect a huge number of frustrating cases would be taken care of. 'out', of course, simply assigns '@noreturn' to all its parameters, even if '@noreturn' itself is not allowed!