View mode: basic / threaded / horizontal-split · Log in · Help
December 30, 2012
ref is unsafe
After some recent discussions relating to auto ref and const ref, I have come 
to the conlusion that as it stands, ref is not @safe. It's @system. And I 
think that we need to take a serious look at it to see what we can do to make 
it @safe. The problem is combining code that takes ref parameters with code 
that returns by ref. Take this code for example:

ref int foo(ref int i)
{
   return i;
}

ref int bar()
{
   int i = 7;
   return foo(i);
}

ref int baz(int i)
{
   return foo(i);
}

void main()
{
   auto a = bar();
   auto b = baz(5);
}

Both bar and baz return a ref to a local variable which no longer exists. They 
refer to garbage. It's exactly the same problem as in

int* foo(int* i)
{
   return i;
}

int* bar()
{
   int i = 7;
   return foo(&i);
}

void main()
{
   auto a = bar();
}

However, that code is considered @system, because it's taking the address of a 
local variable, whereas the code using ref is considered to be @safe. But it's 
just as unsafe as taking the address of the local variable is. Really, it's 
the same thing but with differing syntax.

The question is what to do about this. The most straightforward thing is to 
just make ref parameters @system, but that would be horrible. With that sort 
of restriction, a _lot_ of code suddenly won't be able to be @safe, and it 
affects const ref and auto ref and anything else along those lines, so whatever 
solution we come up with for having auto ref with non-templated functions will 
almost certainly have the problem, and once that works, I'd expect it to be 
used pretty much by default, making most D code @system, which would be a 
_big_ problem.

Another possibility is to make ref imply scope, but given the transitive 
nature of that, that could be _really_ annoying. Maybe it's the correct 
solution though.

Another possibility would be to make it so that functions with a ref parameter 
are only @system if they also return by ref (the lack of ability to have ref 
variables outside of parameters and return types saves us from such a ref 
being squirreled away somewhere). I don't know how good or bad an idea that 
is. It certainly reduces how much code using ref would have to be @system, but 
it might not be sufficient given how much stuff like std.algorithm uses auto ref 
for its return types.

And maybe another solution which I can't think of at the moment would be 
better. But my point is that we currently have a _major_ hole in SafeD thanks 
to the combination of ref parameters and ref return types, and we need to find 
a solution.

- Jonathan M Davis


Related: http://d.puremagic.com/issues/show_bug.cgi?id=8838
December 30, 2012
Re: ref is unsafe
On Sunday, 30 December 2012 at 08:38:27 UTC, Jonathan M Davis 
wrote:
> After some recent discussions relating to auto ref and const 
> ref, I have come
> to the conlusion that as it stands, ref is not @safe. It's 
> @system. And I
> think that we need to take a serious look at it to see what we 
> can do to make
> it @safe. The problem is combining code that takes ref 
> parameters with code
> that returns by ref. Take this code for example:
>
> ref int foo(ref int i)
> {
>     return i;
> }
>
> ref int bar()
> {
>     int i = 7;
>     return foo(i);
> }
>
> ref int baz(int i)
> {
>     return foo(i);
> }
>
> void main()
> {
>     auto a = bar();
>     auto b = baz(5);
> }
>
> Both bar and baz return a ref to a local variable which no 
> longer exists. They
> refer to garbage. It's exactly the same problem as in
>

IMHO, try to return ref to local variable should be error, and 
such a code shouldn't be compilable
December 30, 2012
Re: ref is unsafe
On Sunday, December 30, 2012 10:04:01 Daniel Kozak wrote:
> IMHO, try to return ref to local variable should be error, and
> such a code shouldn't be compilable

You can disallow that in the easy case of

ref int boo(int i)
{
   return i;
}

and in fact, that's already illegal. The problem is the wrapper function. 
You'd also have to disallow functions from returning ref parameters by ref. 
Otherwise,

ref int foo(ref int i)
{
   return i;
}

ref int baz(int i)
{
   return foo(i);
}

continues to cause problems. And making it illegal to return ref parameters by 
ref would be a serious problem for wrapper ranges, because they do that sort 
of thing all the time with front. So, that's not really going to work.

- Jonathan M Davis
December 30, 2012
Re: ref is unsafe
On Sunday, 30 December 2012 at 09:18:30 UTC, Jonathan M Davis 
wrote:
> On Sunday, December 30, 2012 10:04:01 Daniel Kozak wrote:
>> IMHO, try to return ref to local variable should be error, and
>> such a code shouldn't be compilable
>
> You can disallow that in the easy case of
>
> ref int boo(int i)
> {
>     return i;
> }
>
> and in fact, that's already illegal. The problem is the wrapper 
> function.
> You'd also have to disallow functions from returning ref 
> parameters by ref.
> Otherwise,
>
> ref int foo(ref int i)
> {
>     return i;
> }
>
> ref int baz(int i)
> {
>     return foo(i);
> }
>
> continues to cause problems. And making it illegal to return 
> ref parameters by
> ref would be a serious problem for wrapper ranges, because they 
> do that sort
> of thing all the time with front. So, that's not really going 
> to work.
>
> - Jonathan M Davis

Wouldn't it be enough to disallow functions that both take and 
return by ref? There would still be some limitations, but at 
least:

//----
@property ref T front(T)(T[] a);
//----
Would still be @safe.

It seams the only code that is unsafe always boils down to taking 
an argument by ref and returning it by ref...

At best, we'd (try) to only make that illegal (when we can), or 
(seeing things the other (safer) way around), only allow 
returning by ref, if the compiler is able to prove it is not also 
an input by ref?
December 30, 2012
Re: ref is unsafe
On Sunday, December 30, 2012 11:04:35 monarch_dodra wrote:
> Wouldn't it be enough to disallow functions that both take and
> return by ref? There would still be some limitations, but at
> least:
> 
> //----
> @property ref T front(T)(T[] a);
> //----
> Would still be @safe.
> 
> It seams the only code that is unsafe always boils down to taking
> an argument by ref and returning it by ref...
> 
> At best, we'd (try) to only make that illegal (when we can), or
> (seeing things the other (safer) way around), only allow
> returning by ref, if the compiler is able to prove it is not also
> an input by ref?

The question is whether that would be too limiting. Certainly, it risks being 
a big problem for wrapper functions, since they may _need_ to take an argument 
by ref and return it by ref (or more probably, auto ref for both, but that 
amounts to the same thing as far as this issue goes). We could go with making 
such functions @system rather than @safe, but I don't know how problematic 
that would be. We may have no choice though, since unless you can prove that 
the ref being passed in will stay valid as long as the ref being passed out is 
used, you can't prove that that code is safe.

- Jonathan M Davis
December 30, 2012
Re: ref is unsafe
12/30/2012 12:37 PM, Jonathan M Davis пишет:
> After some recent discussions relating to auto ref and const ref, I have come
> to the conlusion that as it stands, ref is not @safe. It's @system. And I
> think that we need to take a serious look at it to see what we can do to make
> it @safe. The problem is combining code that takes ref parameters with code
> that returns by ref. Take this code for example:

[snip]

> And maybe another solution which I can't think of at the moment would be
> better. But my point is that we currently have a _major_ hole in SafeD thanks
> to the combination of ref parameters and ref return types, and we need to find
> a solution.
>
> - Jonathan M Davis
>
>
> Related: http://d.puremagic.com/issues/show_bug.cgi?id=8838

And another one:
http://d.puremagic.com/issues/show_bug.cgi?id=9195

-- 
Dmitry Olshansky
December 30, 2012
Re: ref is unsafe
On 30/12/2012 09:17, Jonathan M Davis wrote:
> The problem is the wrapper function.
> You'd also have to disallow functions from returning ref parameters by ref.
> Otherwise,
>
> ref int foo(ref int i)
> {
>      return i;
> }
>
> ref int baz(int i)
> {
>      return foo(i);
> }
>
> continues to cause problems. And making it illegal to return ref parameters by
> ref would be a serious problem for wrapper ranges, because they do that sort
> of thing all the time with front. So, that's not really going to work.

I think the compiler needs to be able to mark foo as a function that 
returns its input reference. Then, any arguments to foo that are locals 
should cause an error at the call site (e.g. in baz). So legal calls to 
foo can always be @safe.

To extend the above code:

ref int quux(ref int i)
{
    return foo(i);
}

Here the compiler already knows that foo returns its input reference. So 
it checks whether foo is being passed a local - no; but it also has to 
check if foo is passed any ref parameters of quux, which it is. The 
compiler now has to mark quux as a function that returns its input 
reference.

Works?
December 30, 2012
Re: ref is unsafe
On Sunday, 30 December 2012 at 17:32:41 UTC, Nick Treleaven wrote:
> On 30/12/2012 09:17, Jonathan M Davis wrote:
>> The problem is the wrapper function.
>> You'd also have to disallow functions from returning ref 
>> parameters by ref.
>> Otherwise,
>>
>> ref int foo(ref int i)
>> {
>>     return i;
>> }
>>
>> ref int baz(int i)
>> {
>>     return foo(i);
>> }
>>
>> continues to cause problems. And making it illegal to return 
>> ref parameters by
>> ref would be a serious problem for wrapper ranges, because 
>> they do that sort
>> of thing all the time with front. So, that's not really going 
>> to work.
>
> I think the compiler needs to be able to mark foo as a function 
> that returns its input reference. Then, any arguments to foo 
> that are locals should cause an error at the call site (e.g. in 
> baz). So legal calls to foo can always be @safe.
>
> To extend the above code:
>
> ref int quux(ref int i)
> {
>     return foo(i);
> }
>
> Here the compiler already knows that foo returns its input 
> reference. So it checks whether foo is being passed a local - 
> no; but it also has to check if foo is passed any ref 
> parameters of quux, which it is. The compiler now has to mark 
> quux as a function that returns its input reference.
>
> Works?

That seems like a promising approach. If the compiler can track 
where the local is being passed by ref and returned by ref, then 
it should be able to determine if the ref to the local is leaving 
the scope it was originally conceived in and issue a compiler 
error if it is. The idea of "tagging" the local so that it can be 
tracked may work well. You may still be able to hide it from the 
compiler using pointers, but at that point you're not @safe 
anymore but that should be fine because all we want to do is 
allow returns by ref to be proven @safe or not.

In general terms, no reference to a local should ever leave it's 
scope, so ultimately the compiler *has* to track the scope of any 
local, no matter if it is being passed by ref or not, so really 
this is a solution that has to be implemented one way or the 
other.

--rt
December 30, 2012
Re: ref is unsafe
> Here the compiler already knows that foo returns its input 
> reference. So it checks whether foo is being passed a local - 
> no; but it also has to check if foo is passed any ref 
> parameters of quux, which it is. The compiler now has to mark 
> quux as a function that returns its input reference.
>
> Works?

If functions's source isn't available, the compiler can't know 
what the function does.

This could only work if this property of a function (whether it 
returns a reference to its ref parameter) would be part of its 
type. The compiler could still infer it for function literals and 
templates, similar to how pure works now.
December 30, 2012
Re: ref is unsafe
On Sunday, December 30, 2012 17:32:40 Nick Treleaven wrote:
> I think the compiler needs to be able to mark foo as a function that
> returns its input reference. Then, any arguments to foo that are locals
> should cause an error at the call site (e.g. in baz). So legal calls to
> foo can always be @safe.
> 
> To extend the above code:
> 
> ref int quux(ref int i)
> {
>      return foo(i);
> }
> 
> Here the compiler already knows that foo returns its input reference. So
> it checks whether foo is being passed a local - no; but it also has to
> check if foo is passed any ref parameters of quux, which it is. The
> compiler now has to mark quux as a function that returns its input
> reference.
> 
> Works?

No. There's no guarantee that the compiler has access to the function's body, 
and the function being called could be compiled after the function which calls 
it. There's a reason that attribute inferrence only works with templated 
functions. In every other case, the programmer has to mark it. We're _not_ 
going to get any kind inferrence without templates. D's compilation model 
doesn't allow it.

The closest that we could get to what you suggest would be to add a new 
attribute similar to nothrow but which guarantees that the function does not 
return a ref to a parameter. So, you'd have to mark your functions that way 
(e.g. with @norefparamreturn). Maybe the compiler could infer it for templated 
ones, but this attribute would basically have to work like other inferred 
attributes and be marked manually in all other cases. Certainly, you can't 
have the compiler figuring it out for you in general, because D's compilation 
model allows the function being called to be compiled separately from (and 
potentially after) the function calling it.

And when you think about what this attribute would be needed for, it gets a 
bit bizarre to have it. The _only_ time that it's applicable is when a 
function takes an argument by ref and returns the same type by ref. In all 
other cases, the compiler can guarantee it just based on the type system.

I suppose that we could have an attribute that indicated that a function _did_ 
return a ref to one of its params and then have the compiler give an error if 
it were missing, which means that the foo function

ref int foo(ref int i)
{
   return i;
}

would end up with an error for not having the attribute, whereas a function 
like baz

ref int baz(int i)
{
   return foo(i);
}

would not end up with the error unless foo had the attribute on it. But that's 
very different from any attribute that we currently have. It would be like 
having a throw attribute instead of a nothrow attribute. I suppose that it is 
a possible solution though. I could also see an argument that the attribute 
should go on the parameter rather than the function, in which case you could 
have more fine-grained control over it, but it does complicate things further.

Honestly though, I'm inclined to argue that functions which return by ref and 
have a ref parameter of that same type just be considered @system. It's just 
way simpler. It's also more in line with how pointers to locals are handled, 
though because ref is far more restrictive, it should be possible to come up 
with a different solution (like the attribute), whereas the fact that you can 
squirrel away pointers to things makes it rather complicated (if not 
impossible) to have a solution other than simply make taking the address of a 
local variable @system. You can't squirrel away ref.

- Jonathan M Davis
« First   ‹ Prev
1 2 3 4 5
Top | Discussion index | About this forum | D home