On Friday, 21 October 2016 at 05:41:31 UTC, Walter Bright wrote:
> What you're looking at is I'm not very good at articulating the mental model I have for it. I don't know how to write 'formal' rules for these things.

If we're going to go through the DIP process, you're going to have to try. There's this big language enhancement in the pipeline and I still have only a tenuous understanding of what it's meant to do and an even more tenuous understanding of how it's meant to be an improvement. Your mental model needs to be stated in concise, unambiguous terms before the rest of us can catch up and be able to consent to having the language we all use modified in such a way.

The DIP process is annoying and obtuse as hell, but it does exist for a reason. Your proposals should be held to the same standards as the rest of us, because changes to the language affect all of us.

On Thursday, 20 October 2016 at 09:00:02 UTC, Walter Bright wrote:
>> I read that RC `opAssign` and `destroy` would remain unsafe b/c destroying the
>> owner might leave a dangling ref parameter in `fun(ref RCS rc, ref int ri)`.
>> It seems that unsafe assignment/destruction of RCs would be very limiting.
>> Do we at least have an idea how to tackle this problem? In the RCClass idea an
>> additional addRef/decRef call was proposed.
>> Also see https://github.com/dlang/DIPs/pull/35#issuecomment-252345548.
>
> The solution to that has been proposed and forgotten a couple of times. It is to have the compiler insert code to preemptively increment the reference count, then reassigning the RC object will not invalidate references to its internals. This is beyond the scope of DIP1000, though. DIP1000 is necessary for memory safety even without reference counting.

Thought a bit more about this.
The essence of the problem is a simple pointer aliasing problem, namely calling
    void dangling(ref A a, ref B b)
is unsafe if A is a "container" that could own a B.
This does not only apply to RC but also things like arrays with deterministic MM, unions/variants (changing type of b's pointee), or implementations of Nullable/Optional using heap allocated memory.

Fortunately any pointer aliasing through untyped memory/pointers in A (ubyte[], void*) can only result in a dangling b pointer, if doing unsafe operations on a (or incorrectly marked @trusted ones).
For pointer aliasing through typed fields of A, the compiler can detect a possible aliasing and mark the function call as unsafe.
That would allow operations on a that could free Bs to be @trustable.

This aliasing problem seems only slightly related to scope/escape analysis.
A container using scope in a @safe manner, should only escape scoped references.
So the statement of the problem is actually
  void dangling(ref A a, scope ref B b).

Copying a before the call couldn't solve all those problems, e.g. Unique isn't copyable, a union/variant could be in a reference type field of A.

But it seems that we can detect all such unsafe function calls and the problem only scratches on scope b/c we want to know whether the lifetime of b might be limited attached to a. So seems indeed fair enough to leave this aside for now.

What would be the plan for RC.opAssign? Making it @trusted after DIP1000 and implementing the aliasing detection later, @trusted opAssign but conservatively @unsafe any call with multiple references (at least one of which being scoped), or @system opAssign and changing it to @trusted later?

It seems that the aliasing detect wouldn't be too hard to implememt (even implementable as druntime template similar to std.traits.hasAliasing). Am I right that conservatively detecting the aliasing would allow a slightly limited subset of @safe RC usage?
If so going w/ @trusted RC.opAssign and the aliasing detection seems like a good milestone for 2.073.

On Friday, 21 October 2016 at 06:04:41 UTC, Walter Bright wrote:
> 3. The design utterly relies on a 'ref' value not being returned as a '*'. Any attempts to circumvent this will lose the checking. The rationale for this is a dramatic simplification of the design.

DIP1000 mentions that disallowing &var in @safe code is overly restrictive. This got me fairly confused, please help to clarify https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#escaping-via-return or my understanding of it.

On Friday, 21 October 2016 at 06:04:41 UTC, Walter Bright wrote:
>> So far you have only given answers regarding technical trivialities
>
> That is true, but we must reach mutual understanding on how those trivialities work before we can do more. So far, we have not.

I think we all agree on which overall goal we're trying to achieve, @safe deterministic memory management.
So the primitives of DIP1000 need to prove that they properly cover the reference escaping part of that goal.

1. Default initializers have infinite lifetime.
   lifetime(e.init) = ∞

   Non default initializers are treated as default init followed by an assignment.

2. The sticky lifetime axiom.

   Assignment `v = e` changes lifetime(v) to lifetime(e) if v is a pointer or reference.

3. The safe reference axiom.

   Can't assign an expression e to a variable v if that
   would change lifetime(v) so that lifetime(v) < reachability(v).

4. The conservative lifetime axiom.

   lifetime(func(args)) = min(∞, argmin_{e ∈ A}(lifetime(e)))
   with A = {a ∈ args | canAlias(returnType(func), typeof(a))}

   The lifetime of a function call expression, is the minimum of infinity and the
   lifetime of all arguments that could be referenced by the return type.
   (Shouldn't be type here b/c scope is a storage class and not part of the type)

5. The conservative escape axiom.

   The reachability of non-scope ref or pointer paramters is assumed to be ∞.

   Hence it's not allowed to assign expressions with finite lifetime to non-scope parameters, though the compiler might infer scope for parameters.

If this works out we guarantee lifetime(v) ≥ reachability(v) in @safe code.

The rest should be deducible using the rules from https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#aggregates.

The problem that remains could be stated as lifetime tunneling.

Given `v2 = func(v1)` lifetime(v2) = lifetime(v1) according to 5.
Changing lifetime(v1) by `v1 = e` can change lifetime(v2) to < reachability(v2).

On Friday, 21 October 2016 at 11:39:27 UTC, Martin Nowak wrote:
> The problem that remains could be stated as lifetime tunneling.
>
> Given `v2 = func(v1)` lifetime(v2) = lifetime(v1) according to 5.
> Changing lifetime(v1) by `v1 = e` can change lifetime(v2) to < reachability(v2).

This doesn't manifest easily because there are no named reference variables and converting references to pointers isn't allowed in @safe code.
But the void dangling(ref A a, scope ref B b) is one way to run into this.
Can we at least enforce that any value/reference returned by `func(v1)` cannot be bound to a v2 in the calling scope while still referencing v1?

On Friday, 21 October 2016 at 01:46:20 UTC, meppl wrote:
> void bar( jailed T* t) @safe {
>    func(t); // ok, compiler added hidden_scope for void func( T* t) itself
> }

As said in DIP1000 and implemented in the PR, the compiler infers scope for lambda and template function parameters. Not sure what you're proposing.
For sure we want to extend any attribute inferrence to functions, but how to make that work with separate compilation is still undecided. Talking more about inferrence would be an OT discussion though.

On Friday, 21 October 2016 at 10:11:28 UTC, Martin Nowak wrote:
> It seems that the aliasing detect wouldn't be too hard to implememt (even implementable as druntime template similar to std.traits.hasAliasing). Am I right that conservatively detecting the aliasing would allow a slightly limited subset of @safe RC usage?
> If so going w/ @trusted RC.opAssign and the aliasing detection seems like a good milestone for 2.073.

Nice side effect of http://forum.dlang.org/post/fzawzedvokbvhmdspxsh@forum.dlang.org, it's now clear that a good (not too conservative) aliasing detection is also needed for the lifetime of function return values. So we can reuse that for detecting the dangling ref problem.

On Friday, 21 October 2016 at 12:04:33 UTC, Martin Nowak wrote:
> On Friday, 21 October 2016 at 01:46:20 UTC, meppl wrote:
>> void bar( jailed T* t) @safe {
>>    func(t); // ok, compiler added hidden_scope for void func( T* t) itself
>> }
>
> As said in DIP1000 and implemented in the PR, the compiler infers scope for lambda and template function parameters. Not sure what you're proposing.
> For sure we want to extend any attribute inferrence to functions, but how to make that work with separate compilation is still undecided. Talking more about inferrence would be an OT discussion though.

I just assumed that inference for DIP1000 (for non-template private functions) is _possible_ to implement afterwards with backward compatibility. I want to know, if it is possible, because DIP1000 tells people to scatter another keyword („scope“) all over the place in the code.

On 10/21/2016 09:04 AM, Walter Bright wrote:
> On 10/20/2016 3:52 PM, Dicebot wrote:
>> So far you have only given answers regarding technical trivialities
> 
> That is true, but we must reach mutual understanding on how those trivialities work before we can do more. So far, we have not.

Well, it doesn't work that way. So far all you explanations made zero sense until I have only finally understood what your intentions are thanks to next answer ..

> 1. D has no notion of "borrowing" and describing DIP1000 in those terms is going to mislead. There is no "borrow checker".
> 
> 2. This design has nothing in common with Rust's design. Any notions brought from Rust will mislead.
> 
> 3. The design utterly relies on a 'ref' value not being returned as a '*'. Any attempts to circumvent this will lose the checking. The rationale for this is a dramatic simplification of the design.
> 
> 4. The effects of 'ref' and 'scope' are not transitive.
> 
> 5. 'scope' is ignored if applied to a type with no indirections (this is necessary to support generic code).
> 
> 6. What does 'return ref scope' mean? If the function returns by ref, then the 'return' applies to the 'ref'. Otherwise, it applies to the 'scope'.
> 
> 7. 'scope' applies to the value of a type, 'ref' applies to its address. Your example failed to apply 'scope' to the 'int*' return, because the struct had no pointer types in it.
> 
> 8. A container passed by 'ref' can return a 'ref' to the container's value or indirect contents, or by '*' to indirect contents, but it cannot return the address of the container by '*'. (This is a restatement of item 3.)

.. and means two things for me:

1) DIP1000 has to be changes a lot because it doesn't list most of those
limitation, leading to natural conclusion that borrowing can be emulated
as a user code concept. Most importantly, the statement "The design
utterly relies on a 'ref' value not being returned as a '*'. Any
attempts to circumvent this will lose the checking" is not mentioned at all.

2) The whole thing is very disappointing and won't revive dead body of @safe as I hoped it will. It won't allow to fix `Unique`, won't enable new RNG design Joe has been going, won't allow easy skipping of reference count inc/dec - the list goes on.

Leaving aside issue 2, I'd like to give a try at updating DIP1000 this weekend to be more in match with what you have just explained.

On Friday, 21 October 2016 at 06:09:36 UTC, Walter Bright wrote:
>> @safe scope int* foo (return scope ref int r) {
>>     return &r; // taking address of r is fine because returned pointer
>> can't outlive the reference
>> }
>
> Nope:
>
>   Error: function foo functions cannot be scope

Scope was supposed to apply to the return value, comare https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#scope-function-returns.

Let's maybe work with a better equivalent example.

@safe scope Klass get(return scope ref Unique!Klass);