January 04, 2013
On Wednesday, 2 January 2013 at 23:33:16 UTC, Thiez wrote:
> On Wednesday, 2 January 2013 at 22:53:04 UTC, Jonathan M Davis wrote:
>> Then we're going to have to disagree, and I believe that Walter and Andrei are
>> completely with me on this one. If all of the constructs that you use are
>> @safe, then it should be _guaranteed_ that your program is memory-safe. That's
>> what @safe is for. Yes, it can be gotten around if the programmer marks
>> @system code as @trusted when it's not really memory-safe, but that's the
>> programmer's problem. @safe is not doing it's job and is completely pointless
>> if it has any holes in it beyond programmers mislabeling functions as @trusted.
>> - Jonathan M Davis
>
> Perhaps it is worth looking at Rust for this problem?

You can also look at how Algol solved this over 40 years ago: Insert a runtime check that the escaping reference does not point to the current stack frame which is about to be destroyed. The check should be very cheap at runtime but it can be deactivated in a release build for efficiency just like it is done for array indexing.

FYI Nimrod has the same problem and it's planned to prevent these cases statically with a type based alias analysis; however at least the first versions will still keep the dynamic check as these kind of static analyses cry for correctness proofs IMO.
January 04, 2013
On Friday, 4 January 2013 at 00:46:33 UTC, Araq wrote:
> On Wednesday, 2 January 2013 at 23:33:16 UTC, Thiez wrote:
>> On Wednesday, 2 January 2013 at 22:53:04 UTC, Jonathan M Davis wrote:
>>> Then we're going to have to disagree, and I believe that Walter and Andrei are
>>> completely with me on this one. If all of the constructs that you use are
>>> @safe, then it should be _guaranteed_ that your program is memory-safe. That's
>>> what @safe is for. Yes, it can be gotten around if the programmer marks
>>> @system code as @trusted when it's not really memory-safe, but that's the
>>> programmer's problem. @safe is not doing it's job and is completely pointless
>>> if it has any holes in it beyond programmers mislabeling functions as @trusted.
>>> - Jonathan M Davis
>>
>> Perhaps it is worth looking at Rust for this problem?
>
> You can also look at how Algol solved this over 40 years ago: Insert a runtime check that the escaping reference does not point to the current stack frame which is about to be destroyed. The check should be very cheap at runtime but it can be deactivated in a release build for efficiency just like it is done for array indexing.
>
> FYI Nimrod has the same problem and it's planned to prevent these cases statically with a type based alias analysis; however at least the first versions will still keep the dynamic check as these kind of static analyses cry for correctness proofs IMO.

I did suggest something like that, and it may be a good idea to implement as a debugging aid (like runtime range checking). I wonder how difficult it would be to implement?

Unfortunately, it does not help solve the @safe compile time checks.

--rt

January 04, 2013
Am 03.01.2013 00:48, schrieb Jason House:
> On Sunday, 30 December 2012 at 08:38:27 UTC, Jonathan M Davis wrote:
>> After some recent discussions relating to auto ref and const ref, I have come to the conlusion that as it stands, ref is not @safe. It's @system. And I think that we need to take a serious look at it to see what we can do to make it @safe. The problem is combining code that takes ref parameters with code that returns by ref.
> 
> The best solution I can think of is for the @safe code to require a ref return value is treated with the same care as all the function input arguments. I'll try to annotate the example code you gave to explain.
> 

+1

In other words, references returned by a function call that took any references to locals would be tainted as possibly local (in the function local data flow) and thus are not allowed to escape the scope. References derived from non-local refs could still be returned and returning references to fields from a struct method also works.

---
@safe ref int test(ref int v) {
	return v; // fine
}

@safe ref int test2() {
	int local;
	return test(local); // error: (possibly) returning ref to local
}

@safe ref int test3() {
	int local;
	int* ptr = &test(local); // fine, ptr is tainted 'local'
	return *ptr; // error: (possibly) returning ref to local
}

@safe ref int test4(ref int val) {
	return test(val); // fine, can only be a ref to the external 'val' or to a global
}
---
January 04, 2013
On Friday, 4 January 2013 at 06:30:55 UTC, Sönke Ludwig wrote:
> In other words, references returned by a function call that took any references to locals would be
> tainted as possibly local (in the function local data flow) and thus are not allowed to escape the
> scope. References derived from non-local refs could still be returned and returning references to
> fields from a struct method also works.
>
> ---
> @safe ref int test(ref int v) {
> 	return v; // fine
> }
>

v should be scope here. If not, other function have no guarantee that the reference will not escape.

> @safe ref int test2() {
> 	int local;
> 	return test(local); // error: (possibly) returning ref to local
> }
>
> @safe ref int test3() {
> 	int local;
> 	int* ptr = &test(local); // fine, ptr is tainted 'local'
> 	return *ptr; // error: (possibly) returning ref to local
> }
>
> @safe ref int test4(ref int val) {
> 	return test(val); // fine, can only be a ref to the external 'val' or to a global
> }
> ---

Given the modification mentioned above, this look like the way to go.
January 04, 2013
On Friday, 4 January 2013 at 06:30:55 UTC, Sönke Ludwig wrote:
> In other words, references returned by a function call that took any references to locals would be
> tainted as possibly local (in the function local data flow) and thus are not allowed to escape the
> scope. References derived from non-local refs could still be returned and returning references to
> fields from a struct method also works.
>
> ---
> @safe ref int test(ref int v) {
> 	return v; // fine
> }
>
> @safe ref int test2() {
> 	int local;
> 	return test(local); // error: (possibly) returning ref to local
> }
>
> @safe ref int test3() {
> 	int local;
> 	int* ptr = &test(local); // fine, ptr is tainted 'local'
> 	return *ptr; // error: (possibly) returning ref to local
> }
>
> @safe ref int test4(ref int val) {
> 	return test(val); // fine, can only be a ref to the external 'val' or to a global
> }
> ---

Trying to say that formally:

Definitions:
'Tainter function':
    A function that:
    1. takes at least one of its parameters by reference
    and
    2. returns by reference
	
'Tainting function call':
    A call to a 'tainter function' where at least one of the
    arguments passed by reference is ref to a local variable

Then the rules become:
Function may not return a reference to:
Rule 1: a function-local variable
Rule 2. a value returned by a 'tainting function call'

@safe:

ref int tfun(ref int v) { // tfun tagged 'tainter function'
    ...
}

ref int test1() {
    int local;
    return local; // error by Rule 1
}

ref int test2() {
    int local;
    return tfun(local); // error by Rule 2
}

ref int test3() {
    int local;
    int* ptr = &tfun(local); // ptr tagged 'local'
    return *ptr; // error by Rule 2
}

ref int test4(ref int val) {
    return tfun(val); // fine
}

int global;

ref int test5() {
    int local;
    int* ptr = &tfun(local); // ptr tagged 'local'
    ptr = &global; // ptr's 'local' tag removed
    return *ptr; // fine
}
January 04, 2013
On Friday, 4 January 2013 at 14:15:01 UTC, Tommi wrote:
> 'Tainting function call':
>     A call to a 'tainter function' where at least one of the
>     arguments passed by reference is ref to a local variable

I forgot to point out that the return value of a 'tainting function call' is considered to be a "reference to a function-local variable" (even if it's not in reality).
January 04, 2013
On Sunday, 30 December 2012 at 22:02:16 UTC, Jonathan M Davis wrote:
> The closest that we could get to what you suggest would be to add a new
> attribute similar to nothrow but which guarantees that the function does not
> return a ref to a parameter. So, you'd have to mark your functions that way
> (e.g. with @norefparamreturn). Maybe the compiler could infer it for templated
> ones, but this attribute would basically have to work like other inferred
> attributes and be marked manually in all other cases. Certainly, you can't
> have the compiler figuring it out for you in general, because D's compilation
> model allows the function being called to be compiled separately from (and
> potentially after) the function calling it.
>
> And when you think about what this attribute would be needed for, it gets a
> bit bizarre to have it. The _only_ time that it's applicable is when a
> function takes an argument by ref and returns the same type by ref. In all
> other cases, the compiler can guarantee it just based on the type system.

I realized just now that it's also applicable to member functions:
struct F
{
  int _i;
  ref int ser() { return _i; } // Needs to be marked as well
}

A struct's fields are implicit parameters in anything it returns.

> Honestly though, I'm inclined to argue that functions which return by ref and
> have a ref parameter of that same type just be considered @system.

Structs mess that up as well:
struct S { int i; }
ref int d(ref S s)
{
  return s.i;
}
January 04, 2013
On Friday, January 04, 2013 17:26:59 Zach the Mystic wrote:
> > Honestly though, I'm inclined to argue that functions which
> > return by ref and
> > have a ref parameter of that same type just be considered
> > @system.
> 
> Structs mess that up as well:
> struct S { int i; }
> ref int d(ref S s)
> {
> return s.i;
> }

Yes. That's a function which takes a ref and returns by ref just like I said. It's just that in this case, the ref returned isn't the full object that was passed by ref but just a portion of it. What that means is that you can't assume that the ref being returned is safe just because the type of the parameter and the return type aren't the same. But it doesn't change the statement that a function which takes a parameter by ref and returns by ref can't be considered @safe without additional constraints of some kind. It just shows why you don't have an easy way out to make many of them @safe based on the differing types involved.

- Jonathan M Davis
January 05, 2013
On Friday, 4 January 2013 at 20:20:08 UTC, Jonathan M Davis wrote:
> ... But it doesn't change the
> statement that a function which takes a parameter by ref and returns by ref
> can't be considered @safe without additional constraints of some kind. It just
> shows why you don't have an easy way out to make many of them @safe based on
> the differing types involved.

Well, I've been working on just that. I'll have it for you tomorrow, I think.
January 05, 2013
I've here formalized how I think the constraints on a non-scope ref taking and ref returning function should work. This represents a whole addition to the type system. The attribute "@outref" from my previous post has been shortened to keyword "out" (must come before parentheses). This is all I have left to say about this topic:

ref int lugs(ref int a) { return a; }
ref int h(ref int a)
{
  return lugs(a); // Okay

  int local;
  return lugs(local); // Error: the result of a function which accepts a local as non-scope ref and returns ref is treated as local and cannot be escaped unless that function is marked "out"

  int* p = &lugs(local); // Same error
}

int d;
out ref int saml(ref int a)
{
  return *(new int); // Fine
  return d; // Fine

  return a; // Error: a function marked "out" may not escape a non-scope ref parameter
}

ref int lugh(ref int a) { return a; }
out ref int druh(ref int a)
{
  return lugh(a); // Error: a function marked "out" may not escape the result of a function which accepts its non-scope ref parameter and returns a ref unless that function is also marked "out"
}

out int boops(ref int a) {} // Error: a function marked "out" must return a reference
out ref int bop(int a, in ref b, scope ref c) {} // Error: a non-member function marked "out" must accept at least one non-scope ref parameter

// "cast(out)" provides all needed flexibility:
out ref int lit(ref int a)
{
  return cast(out) a; // Not @safe

  // But with @trusted blocks, we could do:
  @trusted { return cast(out) a; } // @safe

  // And with @trusted statements, the brackets are gone:
  @trusted return cast(out) a; // @safe

  // Otherwise, this function must be marked "@trusted"
}

// You can use cast(out) anywhere:
ref int hugs(ref int a) { return a; }
ref int g(ref int a)
{
  int local;
  return cast(out) (hugs(local)); // Okay
  return cast(out) local; // Okay??
  return hugs(cast(out) local); // Won't know what hit 'em
}

// Nor did I forget about structs:
struct S
{
  int _i;
  static int _s;
  out ref int club() { return _i; } // Error: a member function marked "out" may not escape a non-static field
  out ref int trob() { return _s; } // Okay
  out ref int blub() { return cast(out) _i; } //Okay
}

struct B { int _i; ref int snub() { return _i; } }
ref int bub()
{
  B b;
  return b.snub(); // Error: the result of a local instance's non-static method which returns ref is considered local and may not be escaped unless that method is marked "out"

  int* i = &b.snub(); // Same error
}