On Monday, 16 March 2015 at 15:25:45 UTC, Zach the Mystic wrote:
> If an RC'd struct is heap-allocated, but one of its members points to the stack, how is it ever safe to escape it?

It isn't.

> Why shouldn't the heap variable be treated as scoped too, inheriting the most restricted scope of any of its members?

See below.

> To me, the question is not how you can know that a member is scoped, so much as how you could know that it *isn't* scoped, i.e. that a sub-pointer was global while the parent was local. I think it would require a very complicated type system:
>
> struct S {
>   T* p;
> }
>
> // note the elaborate new return signature
> T* fun(return!(S.p) S x) {
>   return x.p;
> }
>
> T* run() {
>   S s;
>   s.p = new T; // s local, s.p global
>   return fun(s);
> }
>
> The above is technically safe, but the question is whether it's too complicated for the benefit. In the absence of such a complicated system, the safe default is to assume a struct is always as scoped as its most scoped member (i.e. transitive scoping).

My old proposal treated scope as a type modifier. It would have made something like your example possible. But it's now a storage class, which means that we lose information when we go through an indirection.

The solution is indeed a kind of transitivity. I describe this in the section on multiple indirections:
http://wiki.dlang.org/User:Schuetzm/scope2#Multiple_indirections

> Your idea of `scope` members would only be valid in the absence of this safe default. But even then it would be of limited usefulness, because it would prevent all uses of global references in those members, even if the parent was global. For me, it comes down to that you can't know if anything is global or local until you define an instance of it, which you can't do in the struct definition.

I think there is a misunderstanding. scope members don't really introduce anything new. They are exactly equivalent to an accessor method:

struct S {
    scope int* payload;
    scope int** indirect;
}

behaves exactly the same as:

struct S {
    private int* payload_;
    ref int* payload() return {
        return payload_;
    }
}

By the rules for multiple indirections, it's not possible to do unsafe things with them:

void foo() {
    // reading (right hand side):

    S s;
    int* p = s.payload;  // OK, `payload` lives at least as long as `s`
    p = *s.indirect;     // ditto

    {
        S s2;
        p = s2.payload;  // NOT OK, we don't know how long `payload`
                         // really lives, so we assume it is
                         // destroyed when `s2` goes out of scope
        p = *s2.indirect;// ditto
    }

    // assignment (left hand side):

    s.payload = new int; // OK, `new int` is on the heap

    int x;
    s.payload = &x;      // NOT OK, `payload` is scope by its `this`
                         // which is `s`, which outlives `x`
    *s.indirect = new int*;
                         // OK: `new int*` is on the heap
    *s.indirect = &p;    // NOT OK: accessed through indirection
                         // must assume destination exists forever
                         // => cannot contain pointer to local
}

The only difference it makes whether a member is annotated as scope or not is when it's accessed from inside a method, because in this case, the compiler doesn't know anything about the context in which the method is called.

BUt there is indeed still some confusion on my side. It's about the question whether `this` should implicitly be passed as `scope` or not. Because if it is, scope members are probably useless, because they are already implied. I think I should remove this suggestion, because it would break too much code (in @system).

On Monday, 16 March 2015 at 13:55:43 UTC, Marc Schütz wrote:
> On Monday, 16 March 2015 at 04:00:51 UTC, Zach the Mystic wrote:
>> "Functions and methods can be overloaded on scope. This allows efficient passing of RC wrappers for instance..."
>>
>> How does the compiler figure out which of the variables it's passing to the parameters are `scope` or not? Does the caller try the scoped overloads first by default, and only if there's an error tries the non-scoped overloads? If so, what causes the error?
>
> Hmm... I guess it only makes sense for postblits and destructors. I'm not sure about constructors and opAssign, so I'll leave these out for now. I've changed the wiki page accordingly.

To clarify: When a scoped variable is assigned to a location with infinite lifetime (`static`), this assignment will fail unless the type provides an assignment operator/postblit that is correctly annotated. Depending on whether `this` is implicitly passed as `scope` or not, this annotation needs to be `scope`, otherwise, conversely, the non-scope overload needs to be marked as `static`. Likewise, a scope variable without a destructor accepting scope cannot be destroyed.

I added the rules for overloading in the wiki. Now that I'm seeing clearer, I've restored the part where it says that all functions can be overloaded. No need to restrict it artificially.

On Monday, 16 March 2015 at 17:00:12 UTC, Marc Schütz wrote:
> BUt there is indeed still some confusion on my side. It's about the question whether `this` should implicitly be passed as `scope` or not. Because if it is, scope members are probably useless, because they are already implied. I think I should remove this suggestion, because it would break too much code (in @system).

I always tend to think of member functions as if they weren't:

struct S {
  T t;
  ref T fun() return {
    return t;
  }
}

In my head, I just translate fun() above to:

ref T fun(return S* __this) {
  return __this.t;
}

Therefore whatever the scope of `__this`, that's the scope of the return, just like it would be for any other parameter. Then:

S s;
s.fun();

... is really just `fun(s);` in disguise. That's why it's hard for me to grasp `scope` members, because they seem to me to be just as scope as their parent, whether global or local. How a member could be scope when the parent is global is hard for me to imagine.

On Monday, 16 March 2015 at 19:43:01 UTC, Zach the Mystic wrote:
> On Monday, 16 March 2015 at 17:00:12 UTC, Marc Schütz wrote:
>> BUt there is indeed still some confusion on my side. It's about the question whether `this` should implicitly be passed as `scope` or not. Because if it is, scope members are probably useless, because they are already implied. I think I should remove this suggestion, because it would break too much code (in @system).
>
> I always tend to think of member functions as if they weren't:
>
> struct S {
>   T t;
>   ref T fun() return {
>     return t;
>   }
> }
>
> In my head, I just translate fun() above to:
>
> ref T fun(return S* __this) {
>   return __this.t;
> }
>
> Therefore whatever the scope of `__this`, that's the scope of the return, just like it would be for any other parameter. Then:
>
> S s;
> s.fun();
>
> ... is really just `fun(s);` in disguise. That's why it's hard for me to grasp `scope` members, because they seem to me to be just as scope as their parent, whether global or local.

It works just the same:

struct S {
    private int* payload_;
    ref int* payload() return {
        return payload_;
    }
}

ref int* payload(scope ref S __this) return {
    return __this.payload_;    // well, imagine it's not private
}

Both the S.payload() and the free-standing payload() do the same thing.

From inside the functions, `return` tells us that we're allowed to a reference to our payload. From the caller's point of view, it signifies that the return value is scoped to the first argument, or `this` respectively.

To reiterate, `scope` members are just syntactical sugar for the kinds of accessor methods/functions in the example code. There's nothing special about them.

> How a member could be scope when the parent is global is hard for me to imagine.

The following is clear, right?

int* p;
scope int* borrowed = p;

That's clearly allowed, we're storing a reference to a global or GC object into a scope variable. Now let's use `S`, which contains an `int*` member:

S s;
scope S borrowed_s = s;

That's also ok. Doesn't matter whether it's the pointer itself, or something containing the pointer. And now the final step:

scope int* p2;
p2 = s.payload;          // OK
p2 = borrowed_s.payload; // also OK
static int* p3;
p3 = s.payload;          // NOT OK!

However, if `payload` were not the accessor method/function, but instead a simple (non-scope) member of `S`, that last line would be allowed, because there is nothing restricting its use. For members that the struct owns and want's to manage itself, this is not good. Therefore, we make it private and allow access to it only through accessor methods/functions that are annotated with `return`. But we could accidentally forget an annotation, and the pointer could escape. That's what the `scope` annotation on the member is fore.

Does that clear it up?

On Monday, 16 March 2015 at 20:50:46 UTC, Marc Schütz wrote:
> On Monday, 16 March 2015 at 19:43:01 UTC, Zach the Mystic wrote:
>> I always tend to think of member functions as if they weren't:
>>
>> struct S {
>>  T t;
>>  ref T fun() return {
>>    return t;
>>  }
>> }
>>
>> In my head, I just translate fun() above to:
>>
>> ref T fun(return S* __this) {
>>  return __this.t;
>> }
>>
>> Therefore whatever the scope of `__this`, that's the scope of the return, just like it would be for any other parameter. Then:
>>
>> S s;
>> s.fun();
>>
>> ... is really just `fun(s);` in disguise. That's why it's hard for me to grasp `scope` members, because they seem to me to be just as scope as their parent, whether global or local.
>
> It works just the same:
>
> struct S {
>     private int* payload_;
>     ref int* payload() return {
>         return payload_;
>     }
> }
>
> ref int* payload(scope ref S __this) return {
>     return __this.payload_;    // well, imagine it's not private
> }

More accurately,

// `return` is moved
ref int* payload(return scope ref S __this) {
   return __this.payload_;
}

I think that if you need `return` to make it safe, there's much less need for `scope`.

> Both the S.payload() and the free-standing payload() do the same thing.
>
> From inside the functions, `return` tells us that we're allowed to a reference to our payload. From the caller's point of view, it signifies that the return value is scoped to the first argument, or `this` respectively.
>
> To reiterate, `scope` members are just syntactical sugar for the kinds of accessor methods/functions in the example code. There's nothing special about them.

That's fine, but then there's the argument that syntax sugar is different from "real" functionality. To add it would require a compelling use case.

My fundamental issue with `scope` in general is that it should be the safe default, which means it doesn't really need to appear that often. If @safe is default, the compiler would force you to mark any parameter `return` when it detected such a return.

>> How a member could be scope when the parent is global is hard for me to imagine.
>
> The following is clear, right?
>
> int* p;
> scope int* borrowed = p;
>
> That's clearly allowed, we're storing a reference to a global or GC object into a scope variable. Now let's use `S`, which contains an `int*` member:
>
> S s;
> scope S borrowed_s = s;
>
> That's also ok. Doesn't matter whether it's the pointer itself, or something containing the pointer. And now the final step:
>
> scope int* p2;
> p2 = s.payload;          // OK
> p2 = borrowed_s.payload; // also OK
> static int* p3;
> p3 = s.payload;          // NOT OK!
>
> However, if `payload` were not the accessor method/function, but instead a simple (non-scope) member of `S`, that last line would be allowed, because there is nothing restricting its use.

See above. With `return` being forced on the implicit this parameter:

ref int* payload(return /*scope*/ ref S __this) { ... }

`return` covers the need for safety, unless I'm still missing something.

> For members that the struct owns and want's to manage itself, this is not good. Therefore, we make it private and allow access to it only through accessor methods/functions that are annotated with `return`. But we could accidentally forget an annotation, and the pointer could escape.

Same argument. Forgetting `return` in safe code == compiler error. I think DIP25 already does this.

On Tuesday, 17 March 2015 at 01:13:41 UTC, Zach the Mystic wrote:
> On Monday, 16 March 2015 at 20:50:46 UTC, Marc Schütz wrote:
>> It works just the same:
>>
>> struct S {
>>    private int* payload_;
>>    ref int* payload() return {
>>        return payload_;
>>    }
>> }
>>
>> ref int* payload(scope ref S __this) return {
>>    return __this.payload_;    // well, imagine it's not private
>> }
>
> More accurately,
>
> // `return` is moved
> ref int* payload(return scope ref S __this) {
>    return __this.payload_;
> }

Right, copy&paste mistake.

>
> I think that if you need `return` to make it safe, there's much less need for `scope`.
>
>> Both the S.payload() and the free-standing payload() do the same thing.
>>
>> From inside the functions, `return` tells us that we're allowed to a reference to our payload. From the caller's point of view, it signifies that the return value is scoped to the first argument, or `this` respectively.
>>
>> To reiterate, `scope` members are just syntactical sugar for the kinds of accessor methods/functions in the example code. There's nothing special about them.
>
> That's fine, but then there's the argument that syntax sugar is different from "real" functionality. To add it would require a compelling use case.
>
> My fundamental issue with `scope` in general is that it should be the safe default, which means it doesn't really need to appear that often. If @safe is default, the compiler would force you to mark any parameter `return` when it detected such a return.

Hmmm... you have a point there. On the other hand, if @safe is merely inferred (for templates), then you wouldn't get an error, but it would get inferred @system instead.

On Sunday, 15 March 2015 at 19:11:36 UTC, Marc Schütz wrote:
> On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:
>> On 15/03/2015 14:10, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
>>> Here's the new version of my scope proposal:
>>> http://wiki.dlang.org/User:Schuetzm/scope2
>>>
>>> It's still missing real-life examples, a section on the implementation,
>>> and a more formal specification, as well as a discussion of backwards
>>> compatibility. But I thought I'd show what I have, so that it can be
>>> discussed early on.
>>>
>>> I hope it will be more digestible for Walter & Andrei. It's more or less
>>> an extended version of DIP25, and avoids the need for most explicit
>>> annotations.
>>
>> I too want a scope attribute e.g. for safe slicing of static arrays, etc. I'm not sure if it's too late for scope by default though, perhaps.
>
> If we get @safe by default, we automatically get scope by default, too.

The scope storage class is a two way contract. The function promises not to escape the reference. The caller promises to ensure the storage that the reference is pointing to will remain valid for the duration of the function call. In some cases, the caller code may need to take active steps to ensure that, like keeping an otherwise temporary reference alive to prevent it from being deallocated.

But what if the pointer is null? Can this be considered to fulfill the caller's part of the deal?

Yes, the old @notnull debate again. For me, @safe by default and scope by default also suggests @notnull by default for scope references. Sorry if this opens up directions you don't want to think about at the moment...

On Wednesday, 18 March 2015 at 13:01:50 UTC, Oren Tirosh wrote:
> On Sunday, 15 March 2015 at 19:11:36 UTC, Marc Schütz wrote:
>> On Sunday, 15 March 2015 at 17:31:17 UTC, Nick Treleaven wrote:
>>> On 15/03/2015 14:10, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
>>>> Here's the new version of my scope proposal:
>>>> http://wiki.dlang.org/User:Schuetzm/scope2
>>>>
>>>> It's still missing real-life examples, a section on the implementation,
>>>> and a more formal specification, as well as a discussion of backwards
>>>> compatibility. But I thought I'd show what I have, so that it can be
>>>> discussed early on.
>>>>
>>>> I hope it will be more digestible for Walter & Andrei. It's more or less
>>>> an extended version of DIP25, and avoids the need for most explicit
>>>> annotations.
>>>
>>> I too want a scope attribute e.g. for safe slicing of static arrays, etc. I'm not sure if it's too late for scope by default though, perhaps.
>>
>> If we get @safe by default, we automatically get scope by default, too.
>
> The scope storage class is a two way contract. The function promises not to escape the reference. The caller promises to ensure the storage that the reference is pointing to will remain valid for the duration of the function call. In some cases, the caller code may need to take active steps to ensure that, like keeping an otherwise temporary reference alive to prevent it from being deallocated.
>
> But what if the pointer is null? Can this be considered to fulfill the caller's part of the deal?
>
> Yes, the old @notnull debate again. For me, @safe by default and scope by default also suggests @notnull by default for scope references. Sorry if this opens up directions you don't want to think about at the moment...

So far, null pointers haven't been a big part of the discussion. By the existing definition, a null pointer is memory safe, because it doesn't point to anything. But they are obviously a problem in their own right.

On Wednesday, 18 March 2015 at 13:01:50 UTC, Oren Tirosh wrote:
> The scope storage class is a two way contract. The function promises not to escape the reference. The caller promises to ensure the storage that the reference is pointing to will remain valid for the duration of the function call. In some cases, the caller code may need to take active steps to ensure that, like keeping an otherwise temporary reference alive to prevent it from being deallocated.
>
> But what if the pointer is null? Can this be considered to fulfill the caller's part of the deal?
>
> Yes, the old @notnull debate again. For me, @safe by default and scope by default also suggests @notnull by default for scope references. Sorry if this opens up directions you don't want to think about at the moment...

Don't be sorry, I agree with you 100%, and you stated it more clearly than i could have.

On Sunday, 15 March 2015 at 14:10:02 UTC, Marc Schütz wrote:
> Here's the new version of my scope proposal:
> http://wiki.dlang.org/User:Schuetzm/scope2
>
> It's still missing real-life examples, a section on the implementation, and a more formal specification, as well as a discussion of backwards compatibility. But I thought I'd show what I have, so that it can be discussed early on.
>
> I hope it will be more digestible for Walter & Andrei. It's more or less an extended version of DIP25, and avoids the need for most explicit annotations.

BUMP

@Walter & Andrei: What is your opinion? Is it better than the last proposal? Worse? Good enough?