On Thu, Dec 04, 2014 at 12:58:56PM -0800, Walter Bright via Digitalmars-d wrote:
> On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
[...]
> >I don't see where the proposal defines what exactly can be returned via scope.
> 
> The scope return value does not affect what can be returned. It affects how that return value can be used. I.e. the return value cannot be used in such a way that it escapes the lifetime of the expression.
[...]

Ahhh, this certainly clears things up a lot. You should add this explanation to the DIP.


T

-- 
Famous last words: I *think* this will work...

On 12/4/14 3:58 PM, Walter Bright wrote:
> On 12/4/2014 7:25 AM, Steven Schveighoffer wrote:
>> int* bar(scope int*);
>> scope int* foo();
>>
>> bar(foo());           // Ok, lifetime(foo()) > lifetime(bar())
>>
>> I'm trying to understand how foo can be implemented in any case. It
>> has no scope
>> ints to return, so where does it get the int from?
>
> Could be from a global variable. Or a new'd value.

Well, OK, but why do that?

>> I don't see where the proposal defines what exactly can be returned
>> via scope.
>
> The scope return value does not affect what can be returned. It affects
> how that return value can be used. I.e. the return value cannot be used
> in such a way that it escapes the lifetime of the expression.

I assumed the scope return was so you could do things like:

scope int *foo(scope int *x)
{
   return x;
}

which would be fine, I assume, right?

>> Another thing I saw early on:
>>
>> void abc() {
>>      scope int* a;
>>      int* b;
>>      scope ref int* c = a;  // Error, rule 5
>>      scope ref int* d = b;  // Ok
>>      int* i = a;            // Ok, scope is inferred for i
>>      global_ptr = d;        // Error, lifetime(d) < lifetime(global_ptr)
>>      global_ptr = i;        // Error, lifetime(i) < lifetime(global_ptr)
>>      int* j;                // Ok, scope is inferred for i
>>      global_ptr = j;        // Ok, j is not scope
>> }
>>
>> Does this mean ref can now be applied to a variable?
>
>      scope ref int p = x;
>
> is the same meaning as:
>
>      foo(scope ref int p);
>      foo(x);
>
> as far as what scope ref means. I found it convenient to use scope ref
> local variables to describe semantics. Whether we want to actually
> enable their usage hinges on if there's a point to it. Technically, it
> should work.

My question was about how this kind of allows declaring a ref variable in the middle of a function, which was never allowed before.

-Steve

On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d wrote:
> On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
> >Can we pretty please use the term "type qualifier" instead of "type constructor"? ;-)
> 
> I think "type constructor" is the more pedantically correct term, but you'll have to ask Andrei :-)

Recently there was a bug filed by Andrei himself, and a bunch of merged PRs, that renamed "type constructor" to "type qualifier". The general sense I got was that we're deprecating "type constructor" in preference for "type qualifier".


> >1) Why are scope violations only reported for @safe code?? IMO this greatly limits the usefulness of this DIP.  If I mark something as scope, I'd expect it should be enforced by the compiler regardless of @safe annotations, otherwise what's the point??
> 
> Because there are inevitably cases where you'll need to wrap unsafe code with a safe interface. By screwing this down too tightly for @system code, you'll force people to use really ugly workarounds.

Are there any use cases for bypassing scope semantics in @system code? How common are they? Maybe such code *should* look ugly -- to draw attention to the fact that something un-@safe is going on.

Or perhaps we can make cast() strip away scope? I much rather require explicit annotation for potentially unsafe operations, than to have subtle bugs creep into the code just because I forgot to mark something as @safe.


> >Currently, due to the incompleteness of @safe, it's difficult to use @safe annotations everywhere I'd like to (e.g. I need to use some un-@safe Phobos functions that really ought to be @safe, but aren't due to various reasons, like compiler limitations, etc.). This greatly limits the usefulness of this DIP, if scope is only enforced in @safe code!
> 
> Are there bug reports on this?

There are, though probably incomplete, and there's also an ongoing stream of Phobos PR's for marking things @safe that ought not to be @system.  There's a *lot* of Phobos code that needs to be cleaned up in this way before this can be workable. (Unless we abuse @trusted as a temporary workaround -- but that's not an approach I'd like to recommend!)


> >2) Is there a way to detect if something is marked as scope?
> 
> Using __traits(compiles,...) ought to work.

I see.


> >If not, how does this proposal actually enable ref-counted types? (I'm assuming a library ref-counting type here; or are we expecting further compiler enhancements for ref counting?)
> 
> The ref counting would be done by a wrapper, which implicitly converts to the wrappee by exposing a scope ref.

Just saw your other post on this topic, makes sense.

Still not sure how it would address the RC folks' request for compiler support, but I suppose that's a different topic.


> >3) What does scope mean for delegate parameters? To what does the scope apply, the delegate itself, its body, or its return value, or ...?
> >
> >	struct S {
> >		void opApply(scope void delegate(ref int) loopBody) {
> >			...
> >			// what restrictions (might) apply here w.r.t.
> >			// how loopBody can be called?
> >		}
> >	}
> 
> Hmmm, looks like a problem I didn't think of. darnit!

I think it's a fairly important issue, since making this work correctly will also open up optimization opportunities for a good chunk of delegate-based code.


> >And what would be the effect on the caller's side?
> >
> >	S s;
> >	foreach (i; s) {
> >		// what restrictions (might) apply here?
> >	}
> 
> i would get its scope inferred as necessary, so restrictions would apply as they would to a 'scope i'.

I was thinking more of what implications may hold in the loop body for references to (local) variables outside the loop, since that presents another potential optimization opportunity if we do it right.


> >4) Under "Expressions", how does scope interact with overloaded operators? Do the same rules apply to expressions that use overloaded operators as they apply to built-in operators, or do they apply as though the overloaded operators were written out in function-call syntax?  What happens if some overloaded operators take a mix of scope and non-scope parameters?
> 
> Overloaded operators are treated like calls to the functions that back them.

OK. You might want to state this explicitly in the DIP, just so it isn't left up to interpretation. :-)


> >Finally, the following isn't directly related to this DIP, since scope is intended to solve this problem, but I thought I should bring it up.  :-) In the section "escaping via return", 5 points are listed as sufficient for detecting the "return func(t);" case. The following section "scope ref" states that these 5 points are "correct" (in implementing escaping reference detection). However, isn't the following a loophole?
> >
> >	struct S {
> >		int x;
> >	}
> >	ref int func(ref S s) {
> >		return s.x;
> >	}
> >	ref T foo() {
> >		S s;
> >		return func(s); // escaping reference
> >	}
> >
> >Since S cannot implicitly convert to int, it would appear that this circumvents escaping reference detection.
> 
> No, because under this proposal, s.x is treated as if it were just s as far a scope rules are concerned.
[...]

I'm not quite sure I see how this could be implemented. When the compiler is compiling func(), say it's in a different module, there's nothing about the function that says it's illegal to return a reference to the member of an incoming argument. First of all, this isn't a template function so scope inference isn't in effect; secondly, even of it were in effect (say we write it as `func()(ref S s)` instead), it wouldn't be able to infer the return value as scope, since it's obviously escaping a reference to a function argument. So, unless I'm missing something obvious, func() will be accepted by the compiler (as it should, since we didn't mark anything as scope here).

Now when the compiler compiles foo(), it can no longer see what's inside func(), so there's no way for it to know that the ref int being returned actually comes from the ref S parameter. Furthermore, func()'s signature is not annotated with scope, so how would the compiler know where to apply scoping rules?

Unless I'm missing something, this still looks like a loophole.


T

-- 
If you compete with slaves, you become a slave. -- Norbert Wiener

On 12/04/2014 09:41 PM, Walter Bright wrote:
>
> Yes, it would be written:
>
>    scope ref T setVal(ref T t)
>    {
>       t.val = 12;
>       return t;
>    }

But when there is no scope on the argument, I could not call setVal with a local T variable.

On 12/04/2014 10:54 PM, Martin Nowak wrote:
> On 12/04/2014 09:41 PM, Walter Bright wrote:
>>
>> Yes, it would be written:
>>
>>    scope ref T setVal(ref T t)
>>    {
>>       t.val = 12;
>>       return t;
>>    }
>
> But when there is no scope on the argument, I could not call setVal with
> a local T variable.

Ah, it's inferred. Makes sense now.

On Thu, Dec 04, 2014 at 01:52:09PM -0800, H. S. Teoh via Digitalmars-d wrote:
> On Thu, Dec 04, 2014 at 01:10:31PM -0800, Walter Bright via Digitalmars-d wrote:
> > On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
> > >Can we pretty please use the term "type qualifier" instead of "type constructor"? ;-)
> > 
> > I think "type constructor" is the more pedantically correct term, but you'll have to ask Andrei :-)
> 
> Recently there was a bug filed by Andrei himself, and a bunch of merged PRs, that renamed "type constructor" to "type qualifier". The general sense I got was that we're deprecating "type constructor" in preference for "type qualifier".
[...]

FYI: https://issues.dlang.org/show_bug.cgi?id=13671


T

-- 
"A man's wife has more power over him than the state has." -- Ralph Emerson

Walter Bright:

>> This seems acceptable only if the compiler switch "-scope" implies functions to
>> be @safe by default and @system on request, because currently lot of D
>> programmers don't apply annotations like @safe to their D code. Opt-in safety
>> doesn't work well (and in D we still have the problems caused by null pointers and references).
>
> Safety by default is outside of the sco[pe (!) of this discussion. Also, you can merely put:
>
>     @safe:
>
> at the beginning of a module and it is now all safe.

I agree that @system code should allow to circumvent the part of the type system that enforces the scoping (this happens in Rust too) (but perhaps this has to happen only on scopes that gets inferred and not the ones specified explicitly).

Perhaps you can't leave the "@safe on default" issue out of the DIP69. Currently most D code I see around is not tagged with @safe. If people don't bother applying that annotation, all the work you will do to implement DIP69 will be wasted or partially wasted.

------------------

From your answer to H. S. Teoh:

> Are there bug reports on this?

I and other people have opened several bug reports on @safe, some of them are (but others are present, this is a subset):

https://issues.dlang.org/show_bug.cgi?id=13615
https://issues.dlang.org/show_bug.cgi?id=13681
https://issues.dlang.org/show_bug.cgi?id=12948
https://issues.dlang.org/show_bug.cgi?id=13607
https://issues.dlang.org/show_bug.cgi?id=12845
https://issues.dlang.org/show_bug.cgi?id=6646
https://issues.dlang.org/show_bug.cgi?id=11176
https://issues.dlang.org/show_bug.cgi?id=6333
https://issues.dlang.org/show_bug.cgi?id=13054
https://issues.dlang.org/show_bug.cgi?id=13506
https://issues.dlang.org/show_bug.cgi?id=13188


Some examples of the problems (but there are also opposite problems, like in Issue 6646):

void main() @safe {
    import std.stdio, std.algorithm, std.bigint, std.typecons, std.array;
    [1, 2].sort!("a < b", SwapStrategy.stable);
    auto r = [1, 2].sort().release;
    writeln;
    BigInt a;
    a = a + a;
    alias Foo = Tuple!int;
    Foo[] data;
    data.remove!(x => x == Foo());
    int[] b;
    auto c = b.capacity;
    b.schwartzSort!(x => x);
    const r2 = cartesianProduct([1], [1]).array;
    [Typedef!int(1)].array;
}


Output, dmd 2.067alpha:

test.d(3,11): Error: safe function 'D main' cannot call system function 'std.algorithm.sort!("a < b", cast(SwapStrategy)2, int[]).sort'
test.d(4,25): Error: safe function 'D main' cannot call system function 'std.range.SortedRange!(int[], "a < b").SortedRange.release'
test.d(5,5): Error: safe function 'D main' cannot call system function 'std.stdio.writeln!().writeln'
test.d(7,9): Error: safe function 'D main' cannot call system function 'std.bigint.BigInt.opBinary!("+", BigInt).opBinary'
test.d(10,9): Error: safe function 'D main' cannot call system function 'test.main.remove!((x) => x == Foo(), cast(SwapStrategy)2, Tuple!int[]).remove'
test.d(12,15): Error: safe function 'D main' cannot call system function 'object.capacity!int.capacity'
test.d(13,6): Error: safe function 'D main' cannot call system function 'test.main.schwartzSort!((x) => x, "a < b", cast(SwapStrategy)0, int[]).schwartzSort'
test.d(14,42): Error: safe function 'D main' cannot call system function 'std.array.array!(Result).array'
test.d(15,21): Error: safe function 'D main' cannot call system function 'std.array.array!(Typedef!(int, 0, null)[]).array'

Bye,
bearophile

On Thursday, 4 December 2014 at 14:58:47 UTC, Steven Schveighoffer wrote:
> "There can be at most one owner for any piece of data."
>
> This doesn't seem right. For GC data, the GC owns the data, that is true. But for Ref-counted data, there is more than one owner, and only when all the owners disown the data can it be destroyed.
>

The RC mechanism is the owner. Ownership is loosly defined in this DIp so that it do not close any door for future language evolution.

On Thursday, 4 December 2014 at 09:25:11 UTC, Walter Bright wrote:
> http://wiki.dlang.org/DIP69
>
> Despite its length, this is a fairly simple proposal. It adds the missing semantics for the 'scope' storage class in order to make it possible to pass a reference to a function without it being possible for it to escape.
>
> This, among other things, makes a ref counting type practical. It also makes it more practical to use other storage allocation schemes than garbage collection.
>
> It does not make scope into a type constructor, nor a general type-annotation system.
>
> It does not provide an ownership system, though it would complement one.

So as mentioned, there are various problem with this DIP :
 - rvalue are defined as having a scope that goes to the end of the statement. That mean they can never be assigned to anything as per spec.
 - It add more special casing with & (as if it wasn't enough of a mess with @property, optional () and the fact the function aren't first class). For instance *e has infinite lifetime when &(*e) is lifetime(e).

Also, considering *e has infinite lifetime, (but not e[i] ???) what is the point of scope at all ? If all indirection goes to infinite lifetime (ie GC) then lifetime only apply to local variable and it should not be able to escape these, scope or not.

During discussion, I proposed to differentiate lifetime calculation between lvalues and rvalues (which are inherently different beasts with different lifetime) and carry (or not) the scope flag with each expression.

On 12/4/2014 1:10 PM, Walter Bright wrote:
> On 12/4/2014 11:41 AM, H. S. Teoh via Digitalmars-d wrote:
>> 3) What does scope mean for delegate parameters? To what does the scope
>> apply, the delegate itself, its body, or its return value, or ...?
>>
>>     struct S {
>>         void opApply(scope void delegate(ref int) loopBody) {
>>             ...
>>             // what restrictions (might) apply here w.r.t.
>>             // how loopBody can be called?
>>         }
>>     }
>
> Hmmm, looks like a problem I didn't think of. darnit!

Turns out, 'ref' has exactly the same issue. The resolution is the same:

  alias ref int delegate() dg_t; // ref applies to return type
  void foo(dg_t dg) {
    static int i;
    ref int bar() { return i; } // ref applies to return type
    dg = &bar;
    dg() = 3;
  }

versus:

  void foo(ref int deletate() dg) { // ref applies to declaration dg
    int bar() { return 3; }
    dg = &bar;
  }

Replace 'ref' with 'scope'.