December 30, 2012
After some recent discussions relating to auto ref and const ref, I have come to the conlusion that as it stands, ref is not @safe. It's @system. And I think that we need to take a serious look at it to see what we can do to make it @safe. The problem is combining code that takes ref parameters with code that returns by ref. Take this code for example:

ref int foo(ref int i)
{
    return i;
}

ref int bar()
{
    int i = 7;
    return foo(i);
}

ref int baz(int i)
{
    return foo(i);
}

void main()
{
    auto a = bar();
    auto b = baz(5);
}

Both bar and baz return a ref to a local variable which no longer exists. They refer to garbage. It's exactly the same problem as in

int* foo(int* i)
{
    return i;
}

int* bar()
{
    int i = 7;
    return foo(&i);
}

void main()
{
    auto a = bar();
}

However, that code is considered @system, because it's taking the address of a local variable, whereas the code using ref is considered to be @safe. But it's just as unsafe as taking the address of the local variable is. Really, it's the same thing but with differing syntax.

The question is what to do about this. The most straightforward thing is to just make ref parameters @system, but that would be horrible. With that sort of restriction, a _lot_ of code suddenly won't be able to be @safe, and it affects const ref and auto ref and anything else along those lines, so whatever solution we come up with for having auto ref with non-templated functions will almost certainly have the problem, and once that works, I'd expect it to be used pretty much by default, making most D code @system, which would be a _big_ problem.

Another possibility is to make ref imply scope, but given the transitive nature of that, that could be _really_ annoying. Maybe it's the correct solution though.

Another possibility would be to make it so that functions with a ref parameter are only @system if they also return by ref (the lack of ability to have ref variables outside of parameters and return types saves us from such a ref being squirreled away somewhere). I don't know how good or bad an idea that is. It certainly reduces how much code using ref would have to be @system, but it might not be sufficient given how much stuff like std.algorithm uses auto ref for its return types.

And maybe another solution which I can't think of at the moment would be better. But my point is that we currently have a _major_ hole in SafeD thanks to the combination of ref parameters and ref return types, and we need to find a solution.

- Jonathan M Davis


Related: http://d.puremagic.com/issues/show_bug.cgi?id=8838
December 30, 2012
On Sunday, 30 December 2012 at 08:38:27 UTC, Jonathan M Davis wrote:
> After some recent discussions relating to auto ref and const ref, I have come
> to the conlusion that as it stands, ref is not @safe. It's @system. And I
> think that we need to take a serious look at it to see what we can do to make
> it @safe. The problem is combining code that takes ref parameters with code
> that returns by ref. Take this code for example:
>
> ref int foo(ref int i)
> {
>     return i;
> }
>
> ref int bar()
> {
>     int i = 7;
>     return foo(i);
> }
>
> ref int baz(int i)
> {
>     return foo(i);
> }
>
> void main()
> {
>     auto a = bar();
>     auto b = baz(5);
> }
>
> Both bar and baz return a ref to a local variable which no longer exists. They
> refer to garbage. It's exactly the same problem as in
>

IMHO, try to return ref to local variable should be error, and such a code shouldn't be compilable
December 30, 2012
On Sunday, December 30, 2012 10:04:01 Daniel Kozak wrote:
> IMHO, try to return ref to local variable should be error, and such a code shouldn't be compilable

You can disallow that in the easy case of

ref int boo(int i)
{
    return i;
}

and in fact, that's already illegal. The problem is the wrapper function. You'd also have to disallow functions from returning ref parameters by ref. Otherwise,

ref int foo(ref int i)
{
    return i;
}

ref int baz(int i)
{
    return foo(i);
}

continues to cause problems. And making it illegal to return ref parameters by ref would be a serious problem for wrapper ranges, because they do that sort of thing all the time with front. So, that's not really going to work.

- Jonathan M Davis
December 30, 2012
On Sunday, 30 December 2012 at 09:18:30 UTC, Jonathan M Davis wrote:
> On Sunday, December 30, 2012 10:04:01 Daniel Kozak wrote:
>> IMHO, try to return ref to local variable should be error, and
>> such a code shouldn't be compilable
>
> You can disallow that in the easy case of
>
> ref int boo(int i)
> {
>     return i;
> }
>
> and in fact, that's already illegal. The problem is the wrapper function.
> You'd also have to disallow functions from returning ref parameters by ref.
> Otherwise,
>
> ref int foo(ref int i)
> {
>     return i;
> }
>
> ref int baz(int i)
> {
>     return foo(i);
> }
>
> continues to cause problems. And making it illegal to return ref parameters by
> ref would be a serious problem for wrapper ranges, because they do that sort
> of thing all the time with front. So, that's not really going to work.
>
> - Jonathan M Davis

Wouldn't it be enough to disallow functions that both take and return by ref? There would still be some limitations, but at least:

//----
@property ref T front(T)(T[] a);
//----
Would still be @safe.

It seams the only code that is unsafe always boils down to taking an argument by ref and returning it by ref...

At best, we'd (try) to only make that illegal (when we can), or (seeing things the other (safer) way around), only allow returning by ref, if the compiler is able to prove it is not also an input by ref?
December 30, 2012
On Sunday, December 30, 2012 11:04:35 monarch_dodra wrote:
> Wouldn't it be enough to disallow functions that both take and return by ref? There would still be some limitations, but at least:
> 
> //----
> @property ref T front(T)(T[] a);
> //----
> Would still be @safe.
> 
> It seams the only code that is unsafe always boils down to taking an argument by ref and returning it by ref...
> 
> At best, we'd (try) to only make that illegal (when we can), or
> (seeing things the other (safer) way around), only allow
> returning by ref, if the compiler is able to prove it is not also
> an input by ref?

The question is whether that would be too limiting. Certainly, it risks being a big problem for wrapper functions, since they may _need_ to take an argument by ref and return it by ref (or more probably, auto ref for both, but that amounts to the same thing as far as this issue goes). We could go with making such functions @system rather than @safe, but I don't know how problematic that would be. We may have no choice though, since unless you can prove that the ref being passed in will stay valid as long as the ref being passed out is used, you can't prove that that code is safe.

- Jonathan M Davis
December 30, 2012
12/30/2012 12:37 PM, Jonathan M Davis пишет:
> After some recent discussions relating to auto ref and const ref, I have come
> to the conlusion that as it stands, ref is not @safe. It's @system. And I
> think that we need to take a serious look at it to see what we can do to make
> it @safe. The problem is combining code that takes ref parameters with code
> that returns by ref. Take this code for example:

[snip]

> And maybe another solution which I can't think of at the moment would be
> better. But my point is that we currently have a _major_ hole in SafeD thanks
> to the combination of ref parameters and ref return types, and we need to find
> a solution.
>
> - Jonathan M Davis
>
>
> Related: http://d.puremagic.com/issues/show_bug.cgi?id=8838

And another one:
http://d.puremagic.com/issues/show_bug.cgi?id=9195

-- 
Dmitry Olshansky
December 30, 2012
On 30/12/2012 09:17, Jonathan M Davis wrote:
> The problem is the wrapper function.
> You'd also have to disallow functions from returning ref parameters by ref.
> Otherwise,
>
> ref int foo(ref int i)
> {
>      return i;
> }
>
> ref int baz(int i)
> {
>      return foo(i);
> }
>
> continues to cause problems. And making it illegal to return ref parameters by
> ref would be a serious problem for wrapper ranges, because they do that sort
> of thing all the time with front. So, that's not really going to work.

I think the compiler needs to be able to mark foo as a function that returns its input reference. Then, any arguments to foo that are locals should cause an error at the call site (e.g. in baz). So legal calls to foo can always be @safe.

To extend the above code:

ref int quux(ref int i)
{
    return foo(i);
}

Here the compiler already knows that foo returns its input reference. So it checks whether foo is being passed a local - no; but it also has to check if foo is passed any ref parameters of quux, which it is. The compiler now has to mark quux as a function that returns its input reference.

Works?
December 30, 2012
On Sunday, 30 December 2012 at 17:32:41 UTC, Nick Treleaven wrote:
> On 30/12/2012 09:17, Jonathan M Davis wrote:
>> The problem is the wrapper function.
>> You'd also have to disallow functions from returning ref parameters by ref.
>> Otherwise,
>>
>> ref int foo(ref int i)
>> {
>>     return i;
>> }
>>
>> ref int baz(int i)
>> {
>>     return foo(i);
>> }
>>
>> continues to cause problems. And making it illegal to return ref parameters by
>> ref would be a serious problem for wrapper ranges, because they do that sort
>> of thing all the time with front. So, that's not really going to work.
>
> I think the compiler needs to be able to mark foo as a function that returns its input reference. Then, any arguments to foo that are locals should cause an error at the call site (e.g. in baz). So legal calls to foo can always be @safe.
>
> To extend the above code:
>
> ref int quux(ref int i)
> {
>     return foo(i);
> }
>
> Here the compiler already knows that foo returns its input reference. So it checks whether foo is being passed a local - no; but it also has to check if foo is passed any ref parameters of quux, which it is. The compiler now has to mark quux as a function that returns its input reference.
>
> Works?

That seems like a promising approach. If the compiler can track where the local is being passed by ref and returned by ref, then it should be able to determine if the ref to the local is leaving the scope it was originally conceived in and issue a compiler error if it is. The idea of "tagging" the local so that it can be tracked may work well. You may still be able to hide it from the compiler using pointers, but at that point you're not @safe anymore but that should be fine because all we want to do is allow returns by ref to be proven @safe or not.

In general terms, no reference to a local should ever leave it's scope, so ultimately the compiler *has* to track the scope of any local, no matter if it is being passed by ref or not, so really this is a solution that has to be implemented one way or the other.

--rt
December 30, 2012
> Here the compiler already knows that foo returns its input reference. So it checks whether foo is being passed a local - no; but it also has to check if foo is passed any ref parameters of quux, which it is. The compiler now has to mark quux as a function that returns its input reference.
>
> Works?

If functions's source isn't available, the compiler can't know what the function does.

This could only work if this property of a function (whether it returns a reference to its ref parameter) would be part of its type. The compiler could still infer it for function literals and templates, similar to how pure works now.
December 30, 2012
On Sunday, December 30, 2012 17:32:40 Nick Treleaven wrote:
> I think the compiler needs to be able to mark foo as a function that returns its input reference. Then, any arguments to foo that are locals should cause an error at the call site (e.g. in baz). So legal calls to foo can always be @safe.
> 
> To extend the above code:
> 
> ref int quux(ref int i)
> {
>      return foo(i);
> }
> 
> Here the compiler already knows that foo returns its input reference. So it checks whether foo is being passed a local - no; but it also has to check if foo is passed any ref parameters of quux, which it is. The compiler now has to mark quux as a function that returns its input reference.
> 
> Works?

No. There's no guarantee that the compiler has access to the function's body, and the function being called could be compiled after the function which calls it. There's a reason that attribute inferrence only works with templated functions. In every other case, the programmer has to mark it. We're _not_ going to get any kind inferrence without templates. D's compilation model doesn't allow it.

The closest that we could get to what you suggest would be to add a new attribute similar to nothrow but which guarantees that the function does not return a ref to a parameter. So, you'd have to mark your functions that way (e.g. with @norefparamreturn). Maybe the compiler could infer it for templated ones, but this attribute would basically have to work like other inferred attributes and be marked manually in all other cases. Certainly, you can't have the compiler figuring it out for you in general, because D's compilation model allows the function being called to be compiled separately from (and potentially after) the function calling it.

And when you think about what this attribute would be needed for, it gets a bit bizarre to have it. The _only_ time that it's applicable is when a function takes an argument by ref and returns the same type by ref. In all other cases, the compiler can guarantee it just based on the type system.

I suppose that we could have an attribute that indicated that a function _did_ return a ref to one of its params and then have the compiler give an error if it were missing, which means that the foo function

ref int foo(ref int i)
{
    return i;
}

would end up with an error for not having the attribute, whereas a function like baz

ref int baz(int i)
{
    return foo(i);
}

would not end up with the error unless foo had the attribute on it. But that's very different from any attribute that we currently have. It would be like having a throw attribute instead of a nothrow attribute. I suppose that it is a possible solution though. I could also see an argument that the attribute should go on the parameter rather than the function, in which case you could have more fine-grained control over it, but it does complicate things further.

Honestly though, I'm inclined to argue that functions which return by ref and have a ref parameter of that same type just be considered @system. It's just way simpler. It's also more in line with how pointers to locals are handled, though because ref is far more restrictive, it should be possible to come up with a different solution (like the attribute), whereas the fact that you can squirrel away pointers to things makes it rather complicated (if not impossible) to have a solution other than simply make taking the address of a local variable @system. You can't squirrel away ref.

- Jonathan M Davis
« First   ‹ Prev
1 2 3 4 5 6 7
Top | Discussion index | About this forum | D home