On 12/17/2016 11:47 PM, Walter Bright wrote:
> So the trouble here is dmd not recognizing that f.foo() is the same as foo(f).

https://github.com/dlang/dmd/pull/6331

On Sunday, 18 December 2016 at 05:56:42 UTC, Walter Bright wrote:
> On 12/17/2016 7:56 PM, Martin Nowak wrote:
>>> On 12/17/2016 7:38 AM, Mathias Lang wrote:
>>>> For example, a stack allocator will return data that is already typed as
>>>> `scope`, and as a consequence, interior pointers will not be expressible. I'll
>>>> see if I can form an example in terms of code.
>>>
>>> It should work fine.
>>
>> What does "it" refer to here, doesn't hurt much to write "Internal pointers in
>> scope structs" or so.
>> Equally "work fine" doesn't state how the "it" is supposed to work.
>
> A stack allocator is conceptually returning a slice of a static array allocated on the stack. This works with scope.
>
> Understanding scope always goes back to understanding how it works with pointers and addresses. Adding layers of abstraction over that can be confusing by obfuscating that relationship, but rewrite the abstract as pointers and addresses, and the behavior then becomes clear.
>
> The bugs Mathias has posted have (so far) all been the result of an abstraction not conforming to how pointers and addresses work, due to a bug in dmd.
>

Thanks for expanding on that, as Martin pointed out, it helps reviewers a lot.
I don't feel the point about internal pointers to stack-allocated data have been addressed, but maybe it'll be better if I just craft a test case.

> https://github.com/dlang/dmd/pull/6331

The problem with this solution is that it forces `scope` on unrelated method.
E.g.

```
int* escape () @safe
{
    int i;
    Foo f;
    f.v = &i;
    return f.foo;
}

struct Foo
{
    int* v;
    int* foo () @safe { return null; }
}
```

Doesn't compile anymore. It might be your intention, but it needs to be clearly expressed in the DIP.
Moreover, this code is obviously correct to the reader:

```
struct Foo
{
    int* v;
    int foo () @safe { return *v; }
}
```

But won't compile. I tried with static arrays just to be sure, same outcome.

On Sunday, 18 December 2016 at 10:04:57 UTC, Walter Bright wrote:
> On 12/17/2016 11:47 PM, Walter Bright wrote:
>> So the trouble here is dmd not recognizing that f.foo() is the same as foo(f).
>
> https://github.com/dlang/dmd/pull/6331

With the two last P.R. you submitted merged, the following compiles

```
void escape () @safe
{
    Foo f;
    bar(f);
}

void bar (scope ref Foo f) @safe
{
    int[10] i;
    f.v = i;
}

struct Foo
{
    int[] v;
}
```

On 12/18/2016 3:04 AM, Mathias Lang wrote:
> The problem with this solution is that it forces `scope` on unrelated method.
> E.g.
>
> ```
> int* escape () @safe
> {
>     int i;
>     Foo f;
>     f.v = &i;
>     return f.foo;
> }
>
> struct Foo
> {
>     int* v;
>     int* foo () @safe { return null; }
> }
> ```
>
> Doesn't compile anymore.

And it shouldn't. f contains a pointer to a local, and so f must be scope. A scoped object can only be used on a scoped method, because otherwise the method can squirrel away an escaping copy.


> It might be your intention, but it needs to be clearly
> expressed in the DIP.

The rules need to be clearly expressed, but I don't know how to express every consequence of the rules.


> Moreover, this code is obviously correct to the reader:
>
> ```
> struct Foo
> {
>     int* v;
>     int foo () @safe { return *v; }
> }
> ```
>
> But won't compile. I tried with static arrays just to be sure, same outcome.

It compiles when I try it.

On 12/18/2016 3:09 AM, Mathias Lang wrote:
> With the two last P.R. you submitted merged, the following compiles
>
> ```
> void escape () @safe
> {
>     Foo f;
>     bar(f);
> }
>
> void bar (scope ref Foo f) @safe
> {
>     int[10] i;
>     f.v = i;
> }
>
> struct Foo
> {
>     int[] v;
> }
> ```

Thanks, I'll look at it tomorrow.

On Sunday, 18 December 2016 at 15:24:34 UTC, Walter Bright wrote:
>
>> Moreover, this code is obviously correct to the reader:
>>
>> ```
>> struct Foo
>> {
>>     int* v;
>>     int foo () @safe { return *v; }
>> }
>> ```
>>
>> But won't compile. I tried with static arrays just to be sure, same outcome.
>
> It compiles when I try it.

I didn't meant as a standalone example, but in the context of the previous snippet. Full code:

```
int bar () @safe
{
    int i = 42;
    Foo f;
    f.v = &i;
    return f.foo(); // Doesn't compile
}

struct Foo
{
    int* v;
    int foo () @safe { return *v; }
}
```

On 12/18/2016 8:23 AM, Mathias Lang wrote:
> I didn't meant as a standalone example, but in the context of the previous
> snippet. Full code:
>
> ```
> int bar () @safe
> {
>     int i = 42;
>     Foo f;
>     f.v = &i;
>     return f.foo(); // Doesn't compile
> }
>
> struct Foo
> {
>     int* v;
>     int foo () @safe { return *v; }
> }
> ```

That doesn't compile because the Foo could be written as:

static int* s;
struct Foo
{
    int* v;
    int foo () @safe { s = v; return *v; }
}

The compiler can't know that from the interface to Foo.foo, and so must assume the worst. The safety features are about guarantees, not having faith that the programmer didn't do something like that.

The purpose in adding the 'scope' annotation to Foo.foo is to tell the compiler that there's no hanky-panky going on in the implementation. When the compiler does compile the implementation, it can then check it against the 'scope' semantics.

On 12/18/2016 06:59 AM, Walter Bright wrote:
> On 12/17/2016 8:03 PM, Martin Nowak wrote:
>> On Saturday, 17 December 2016 at 07:29:17 UTC, Walter Bright wrote:
>>> On 12/16/2016 9:08 AM, Mathias Lang wrote:
>>>> As long as we have:
>>>> - A way to take the address of `scope` pointers
>>>
>>> Taking the address of a scope pointer is invalid, I'll fix the method
>>> you used
>>> to do it.
>>
>> What is the reasoning for disallowing that?
> 
> scope is not transitive, and there's no way to declare a scope pointer to a scope pointer.

That's not really a reason, it's a symptom of the implementation and
design ;).
Let me suggest a reason, transitive scope checking would be more complex
to implement, is it that?

>> Is dealing with second order indirections (scope int**) somehow
>> different from
>> from other escape checks?
> 
> Yes. Only the top level is checked. By disallowing taking the address of a scope variable, only the top level needs to be checked.

The biggest limitation of return ref (DIP25) was that there was no way
to obtain indirections (e.g. a named pointer variable).
Hence you'd always need to write (and execute) a.b.c.d.e, with no way to
store temporary results (tmp = a.b.c; tmp.d.e).

No we get 1, but only 1, level of indirections (`tmp = &a.b.c; tmp.d.e`). Two questions that arise from that.

- How certain are we that this is enough? In other words, why is `tmp = &a.b; tmp2 = &tmp.c; tmp2.d.e` not necessary.

- Let's imagine for a short moment, what it would mean to make scope transitive at a later point, if we'd ever find enough reasons to do so. Maximizing future options is always good.

  - Allowing `&scoped` would be simple b/c it doesn't compile atm.
  - Changing `lifetime(*p) = ∞` to `lifetime(*p) = p` later, could be
harder I guess. How restricting would it be to already add it now?
  - The requirement for `ptr = &var` that `lifetime(var) >=
reachability(ptr)` would remain the same.


Hope I fully got the point, I'm not that much into the details of DIP1000, so please correct me if that's nonsense.

On 12/18/2016 9:28 AM, Martin Nowak wrote:
> Let me suggest a reason, transitive scope checking would be more complex
> to implement, is it that?

It's a heluva lot more complicated for the user. And since it is not necessary, why pay such a large bill?


> - How certain are we that this is enough? In other words, why is `tmp =
> &a.b; tmp2 = &tmp.c; tmp2.d.e` not necessary.

It would be necessary in order to build non-trivial data structures on the stack. How necessary is that? I've never needed to in decades of programming. I don't think there are any instances in Phobos or dmd.

If one really wants to do that, there's always @system code.

Is it worth burdening everything with a feature when the use case for it is rare? I don't think so.

For example, consider 'volatile' in C++. The use case for it is rare (very rare). But there's an enormous amount of spec verbiage and thought dedicated to how to weave it into the type system, overloading rules, conversion rules, casting rules, promotion rules, mangling rules, deduction rules, on and on. It's just not worth it.


> - Let's imagine for a short moment, what it would mean to make scope
> transitive at a later point, if we'd ever find enough reasons to do so.
> Maximizing future options is always good.

This kind of annotation system was first proposed 10 years ago. I've been thinking about how to make it simpler ever since, and the original has not improved with age.

I also have a lot of experience with transitive const, and how disruptive that was, and how many years it took to work the problems out (Andrei says they're still a problem).

BTW, Rust is a system with annotations that can be applied at any level. Yet it still needs @system code for many data structures (any with cycles in it).

Rust's system also (so I've read) requires one to re-architect data structures. It's not remotely a matter of just adding annotations.

That alone makes it unacceptable for D.

D's 'return scope' system covers routine coding use. More complex things can be encapsulated in @trusted code. I believe this is the sweet spot that will make it easy to transition to, use and be effective.

---

It's true that I've repeatedly had problems explaining how it works to you, Andrei, Dicebot and Mathias. I can only infer that having a much more complex system, which scope as a type constructor would be, would be infinitely worse. I find the prospect of debugging a data structure with 'scope' sprinkled around on random edges in it to be terrifying (as well as a combinatorial template instantiation explosion problem).