On 10/20/2016 12:27 PM, Dicebot wrote:
> If I was to judge by available test cases only, I would have decided
> DIP1000 is waste of time and brings nothing useful to the table. They
> are terrible in showing implementation goals being low level regression
> testing artifacts.
>
> Really, do you think anyone will ever be impressed seeing code like
> `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can
> allow much more interesting things and tests should show that too, not
> just synthetic artifacts.

They are not meant to impress people, nor are they regression tests. They illustrate closing of existing safety holes.

  int* bar()
  {
    int p;
    return foo1(&p);
  }

is a currently undetected bug. DIP1000 detects it. DIP1000 also closes safety bugs that you posted to Bugzilla (listed in the PR for it).

D must have these fixes, one way or another.

Yes, the test cases are very minimal. But understanding them is necessary in order to understand more complex cases - I don't think we've gotten anywhere by using complex cases to understand the mechanics of it.

On 10/20/2016 9:32 AM, Dicebot wrote:
> On 10/16/2016 11:15 PM, Walter Bright wrote:
>> On 10/16/2016 1:03 PM, Dicebot wrote:
>>> Where does it come from?
>>
>> It comes from trying to compile code with the new safety checks, and
>> having it fail to compile and require adding annotations all over the
>> place. The idea is to infer such annotations in the obvious places.
>> 'return' was already successfully inferred in many places, this just
>> extends an existing practice.
>>
>>
>>> I don't see any mention of such deduction in
>>> https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
>>
>> It's one of those things that becomes necessary once trying to implement
>> it.
>
> Walter, there is no way we can merge your PR if it implements things not
> mentioned in DIP. If you don't want to spend time adjusting it yourself,
> please, at least notify me about what has changed. That will save both
> of us a lot of time.

Ok. But it's pretty simple. A local variable that is initialized with a scoped value is tagged with scope.


>> Inferring 'scope' and 'return' as much as possible makes @safe much more
>> palatable. All I can say is try to break it!
>
> It also makes reasoning about what your code does much harder, adding
> unacceptable amount of magic to it. I'd much prefer to rely on `auto`
> and fix rest of cases myself through deprecations than go with current
> "helpful" behavior.
>
> Such semantics are also unprecedented in D as far as I can remember.
> Before storage classes would never be inferred for variables with
> explicit type. That is highly confusing.

It's already done by DIP25, and D has been very successful in inferring @safe, @trusted, @system, pure, @nogc, and nothrow. I'd like to go even further and infer const, but that's a more complex task.


> Right now I think this is a bad decision and should be reconsidered.
> Would like to now what other developers thing though.

It's long been my belief that if lifetime annotations had to be explicit, few would be interested in using such a "bondage and discipline" language. As further evidence of that, Rust goes to great lengths to infer lifetime attributes. Annotations don't excite people.

On 10/21/2016 12:36 AM, Walter Bright wrote:
> On 10/20/2016 12:27 PM, Dicebot wrote:
>> If I was to judge by available test cases only, I would have decided DIP1000 is waste of time and brings nothing useful to the table. They are terrible in showing implementation goals being low level regression testing artifacts.
>>
>> Really, do you think anyone will ever be impressed seeing code like `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can allow much more interesting things and tests should show that too, not just synthetic artifacts.
> 
> They are not meant to impress people, nor are they regression tests. They illustrate closing of existing safety holes.
> 
>   int* bar()
>   {
>     int p;
>     return foo1(&p);
>   }
> 
> is a currently undetected bug. DIP1000 detects it. DIP1000 also closes safety bugs that you posted to Bugzilla (listed in the PR for it).

Yes, but that is not enough for such a major change. If we want to finally implement `scope`, it has to be _awesome_, plain good is not good enough.

> Yes, the test cases are very minimal. But understanding them is necessary in order to understand more complex cases - I don't think we've gotten anywhere by using complex cases to understand the mechanics of it.

Can you give a clear answer regarding borrowing snippets I am posting? Like one of:

- Such idioms are not supposed to work with DIP1000 at all (why?) and
you only want to focus on fixing existing @safe holes
- It is supposed to work, but I am doing it wrong (please show how to
write it in that case)
- It is supposed to work, but bugs need fixing
- It is supposed to work, but some parts will be implemented later
(which ones?)

So far you have only given answers regarding technical trivialities without taking actual example to the heart.

On Thursday, 20 October 2016 at 21:43:59 UTC, Walter Bright wrote:
>
> It's long been my belief that if lifetime annotations had to be explicit, few would be interested in using such a "bondage and discipline" language. As further evidence of that, Rust goes to great lengths to infer lifetime attributes. Annotations don't excite people.

explicit lifetime annotations...


isnt it possible to upgrade this scope-annotation system afterwards?


for example:  hidden scope
after dip1000 is implemented, a new storage_class gets introduced - lets say „jailed“.
If „jailed“ appears in code, the compiler scans the AST for the route the pointer takes within one module or within one package
and adds the needed hidden_scope annotations automatically.

it would look like that:
void func( T* t) @safe;  // second version of this function exists:
                         // void func( hidden_scope T* t) @safe;
void bar( jailed T* t) @safe {
   func(t); // ok, compiler added hidden_scope for void func( T* t) itself
}
how it looks like in DIP1000:
void func(scope T* t) @safe;
void bar(scope T* t) @safe {
   func(t); // ok
}

if an opaque function (of different module/package) appears, that opaque function must have the scope/jailed annotation again.

On Thursday, 20 October 2016 at 21:36:44 UTC, Walter Bright wrote:
> On 10/20/2016 12:27 PM, Dicebot wrote:
>   int* bar()
>   {
>     int p;
>     return foo1(&p);
>   }

This is a problem I had reading DIP1000. At first it decomposes language constructs into primitives then sets up rules for the primitives.
Such a mathematical approach is nice, but it isn't exercised throughout the DIP.

For example the most interesting function semantics just refer "See section dedicated to discussing methods.", but the Function section (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/DIP1000.md#functions) doesn't contain formal lifetime rules for functions.
Weirdly the Pointer section (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/DIP1000.md#pointers) seems to contain lifetime rules for function return values.
At this point the DIP regresses a bit into showing examples with short notes whether they are errors or correct.
It's somewhat difficult to extract a mental model from that.

The DIP seems to define the lifetime of function return values as the longest lifetime that can be conservatively assumed given the function arguments.

You mentioned an article and a talk, do you have any cleaner/more detailed explanation for functions @Walter?

On 10/20/2016 9:44 AM, Dicebot wrote:
> This compiles too:
>
> ```
> int* x;
>
> @safe unittest
> {
>     @safe static struct S
>     {
>         int data;
>
>         int* borrow () return scope @trusted {
>             return &this.data;
>         }
>     }
>
>     S instance;
>     x = instance.borrow();
> }
> ```
>
> Now, _this_ must be wrong, right?


It compiles because you added '@trusted' to the bad boy function, disabling the check preventing taking the address of a 'ref'. The compiler has no reason to believe that the return value of instance.borrow() has any relationship to 'instance' - it @trusted the programmer to not do that!

The practical result is that a container that is passed by ref can only control its uses by returning by ref, not by *.

On 10/20/2016 7:57 PM, Martin Nowak wrote:
> For example the most interesting function semantics just refer "See section
> dedicated to discussing methods.", but the Function section
> (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/DIP1000.md#functions)
> doesn't contain formal lifetime rules for functions.
> Weirdly the Pointer section
> (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/DIP1000.md#pointers)
> seems to contain lifetime rules for function return values.
> At this point the DIP regresses a bit into showing examples with short notes
> whether they are errors or correct.
> It's somewhat difficult to extract a mental model from that.

What you're looking at is I'm not very good at articulating the mental model I have for it. I don't know how to write 'formal' rules for these things.


> The DIP seems to define the lifetime of function return values as the longest
> lifetime that can be conservatively assumed given the function arguments.

That's right.


> You mentioned an article and a talk, do you have any cleaner/more detailed
> explanation for functions @Walter?

I haven't written either, yet, though I need to get busy with it Real Soon Now. It'll be informal, obviously, but I plan to use trivial examples to make it as clear as possible.

I hate slides with more than about 5 lines of code in them - I lose track of the presentation when I'm trying to understand the code, and I presume my audience will, too.

On 10/21/2016 08:11 AM, Walter Bright wrote:
> On 10/20/2016 9:44 AM, Dicebot wrote:
>> This compiles too:
>>
>> ```
>> int* x;
>>
>> @safe unittest
>> {
>>     @safe static struct S
>>     {
>>         int data;
>>
>>         int* borrow () return scope @trusted {
>>             return &this.data;
>>         }
>>     }
>>
>>     S instance;
>>     x = instance.borrow();
>> }
>> ```
>>
>> Now, _this_ must be wrong, right?
> 
> 
> It compiles because you added '@trusted' to the bad boy function, disabling the check preventing taking the address of a 'ref'. The compiler has no reason to believe that the return value of instance.borrow() has any relationship to 'instance' - it @trusted the programmer to not do that!

Wait what? This is in direct contradiction with what DIP1000 states and what I have been trying to write example of all this time. By existing specification annotating method with `return scope` means that lifetime of return value is same as that method `this`.

Are you saying that existing system forgets that `instance` is `this` the very moment `borrow` methods finishes? If yes, I am afraid it is a rather useless feature, much worse than one may think reading DIP1000.

> The practical result is that a container that is passed by ref can only control its uses by returning by ref, not by *.

1) Why? You still haven't answered how scope pointer is different from
ref in @safe code.
2) Does that imply that lifetime algebra as described in DIP1000 only
applies to function internal arguments and variables? As it is not
possible to neither have `ref` variables, nor take address of `ref`
return value, lifetime of return value will always be limited to single
expression.

On 10/20/2016 3:52 PM, Dicebot wrote:
> On 10/21/2016 12:36 AM, Walter Bright wrote:
>> On 10/20/2016 12:27 PM, Dicebot wrote:
>>> If I was to judge by available test cases only, I would have decided
>>> DIP1000 is waste of time and brings nothing useful to the table. They
>>> are terrible in showing implementation goals being low level regression
>>> testing artifacts.
>>>
>>> Really, do you think anyone will ever be impressed seeing code like
>>> `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can
>>> allow much more interesting things and tests should show that too, not
>>> just synthetic artifacts.
>>
>> They are not meant to impress people, nor are they regression tests.
>> They illustrate closing of existing safety holes.
>>
>>   int* bar()
>>   {
>>     int p;
>>     return foo1(&p);
>>   }
>>
>> is a currently undetected bug. DIP1000 detects it. DIP1000 also closes
>> safety bugs that you posted to Bugzilla (listed in the PR for it).
>
> Yes, but that is not enough for such a major change. If we want to
> finally implement `scope`, it has to be _awesome_, plain good is not
> good enough.

Actually, I consider it a minor change, and I *want* the fix for this bug to be as minor as possible. Furthermore, Andrei, I, Bartosz, and others have struggled with this particular problem for about 10 years. This is the best (and simplest, by far) solution we've come up with. If you can think of a better one, I'm interested.


>> Yes, the test cases are very minimal. But understanding them is
>> necessary in order to understand more complex cases - I don't think
>> we've gotten anywhere by using complex cases to understand the mechanics
>> of it.
>
> Can you give a clear answer regarding borrowing snippets I am posting?
> Like one of:
>
> - Such idioms are not supposed to work with DIP1000 at all (why?) and
> you only want to focus on fixing existing @safe holes
> - It is supposed to work, but I am doing it wrong (please show how to
> write it in that case)
> - It is supposed to work, but bugs need fixing
> - It is supposed to work, but some parts will be implemented later
> (which ones?)
>
> So far you have only given answers regarding technical trivialities

That is true, but we must reach mutual understanding on how those trivialities work before we can do more. So far, we have not.


> without taking actual example to the heart.

1. D has no notion of "borrowing" and describing DIP1000 in those terms is going to mislead. There is no "borrow checker".

2. This design has nothing in common with Rust's design. Any notions brought from Rust will mislead.

3. The design utterly relies on a 'ref' value not being returned as a '*'. Any attempts to circumvent this will lose the checking. The rationale for this is a dramatic simplification of the design.

4. The effects of 'ref' and 'scope' are not transitive.

5. 'scope' is ignored if applied to a type with no indirections (this is necessary to support generic code).

6. What does 'return ref scope' mean? If the function returns by ref, then the 'return' applies to the 'ref'. Otherwise, it applies to the 'scope'.

7. 'scope' applies to the value of a type, 'ref' applies to its address. Your example failed to apply 'scope' to the 'int*' return, because the struct had no pointer types in it.

8. A container passed by 'ref' can return a 'ref' to the container's value or indirect contents, or by '*' to indirect contents, but it cannot return the address of the container by '*'. (This is a restatement of item 3.)

On 10/20/2016 9:37 AM, Dicebot wrote:
> But this is perfectly fine:
>
> @safe scope int* foo (return scope ref int r) {
>     return &r; // taking address of r is fine because returned pointer
> can't outlive the reference
> }

Nope:

  Error: function foo functions cannot be scope

If foo was a member function, the 'scope' would be allowed and would apply to the 'this' reference.