On 9/21/2014 2:11 AM, "Marc Schütz" <schuetzm@gmx.net>" wrote:
> On Sunday, 21 September 2014 at 03:39:24 UTC, Walter Bright wrote:
>> I think it's a well thought out proposal. Thanks for doing this!
>>
>> A couple thoughts:
>>
>> 1. const can be both a storage class and a type constructor. Scope is only a
>> storage class. The scope(int) syntax implies scope is a type constructor, too.
>>
>>     const int* a;  // const used as storage class
>>     const(int*) b; // const used as type constructor
>>
>> The type constructor syntax should be disallowed for const.
>
> (... disallowed for _scope_, I assume)

Yes, my mistake.


> I originally intended it to be part of the type. Ivan Timokhin pointed out
> severe problems with that [1], so I removed it from the proposal. The syntax is
> a remainder of that.
>
> But before I remove it too, I have a question: Will it still be possible to use
> the storage class syntax for members of aggregates?
>
>      struct S {
>          scope!myAllocator int* p;
>      }

Possible, but exactly how that would work remains to be seen.

On Sunday, 21 September 2014 at 11:37:19 UTC, Manu via Digitalmars-d wrote:
> On 21 September 2014 16:02, deadalnix via Digitalmars-d <
> digitalmars-d@puremagic.com> wrote:
>
>> On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:
>>
>>> On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:
>>>
>>>> What happens when a scope() thing finds it's way into generic code? If
>>>> the type
>>>> doesn't carry that information, then you end up in a situation like ref.
>>>> Have
>>>> you ever had to wrestle with ref in generic code?
>>>> ref is the biggest disaster zone in D, and I think all it's problems will
>>>> translate straight to scope if you do this.
>>>>
>>>
>>> I'm unaware of this disaster zone.
>>>
>>
>> Well it is very real. I had to duplicate bunch of code in my visitor
>> generator recently because of it. Getting generic code ref correct is very
>> tedious, error prone, and guarantees code duplication and/or various static
>> ifs all over the place.
>>
>
> It's also extremely hard to unittest; explodes the number of static if
> paths exponentially. I'm constantly finding bugs appear a year after
> writing some code because I missed some static branch paths when originally
> authoring.

If I understand you right, your problems come from the fact that sometimes in a template you want ref, and sometimes you don't.

But I think this mostly doesn't apply to scope: either you borrow things, or you don't. In particular, when you do borrow something, you're not interested in the owner your parameter has inside the caller, you just take it by scope (narrowing the lifetime). Thus there needs to be no information about it inside the callee, and you don't need different instantiations depending on it.

One special case where scope deduction might be desirable are template functions that apply predicates (delegates, lambdas) to passed-in parameters, like map and filter. For these, the scope-ness of the input range can depend on whether the predicates are able to take their parameters as scope.

On 22 September 2014 01:10, Andrei Alexandrescu via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 9/21/14, 4:27 AM, Manu via Digitalmars-d wrote:
>
>> On 21 September 2014 16:02, deadalnix via Digitalmars-d <digitalmars-d@puremagic.com <mailto:digitalmars-d@puremagic.com>> wrote:
>>
>>     On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:
>>
>>         On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:
>>
>>             What happens when a scope() thing finds it's way into
>>             generic code? If the type
>>             doesn't carry that information, then you end up in a
>>             situation like ref.. Have
>>             you ever had to wrestle with ref in generic code?
>>             ref is the biggest disaster zone in D, and I think all it's
>>             problems will
>>             translate straight to scope if you do this.
>>
>>
>>         I'm unaware of this disaster zone.
>>
>>
>>     Well it is very real. I had to duplicate bunch of code in my visitor
>>     generator recently because of it. Getting generic code ref correct
>>     is very tedious, error prone, and guarantees code duplication and/or
>>     various static ifs all over the place.
>>
>>
>> It's also extremely hard to unittest; explodes the number of static if paths exponentially.. I'm constantly finding bugs appear a year after writing some code because I missed some static branch paths when originally authoring.
>>
>
> Is this because of problems with ref's definition, or a natural consequence of supporting ref parameters? -- Andrei
>

It's all because ref is not part of the type.
You can't capture ref with typeof() or templates, you can't make ref
locals, it's hard to find if something is ref or not (detection is
different than everything else), etc.
The nature of it not being a type leads to static if's in every template
that ref appears, which must detect if things are ref (which is awkward),
and produce multiple paths for a ref and not-ref version. If we're dealing
with arguments, this might lead to num-arguments^^2 paths.

The only practical conclusion I (and others too) have reached, is to eventually give up and invent Ref!T, but then we arrive at a new world of problems. It's surprisingly hard to write a transparent Ref template which interacts effectively with other generic code, and no 3rd party library will support it.

On 22 September 2014 13:19, Walter Bright via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 9/21/2014 4:27 AM, Manu via Digitalmars-d wrote:
>
>> It's also extremely hard to unittest; explodes the number of static if
>> paths
>> exponentially. I'm constantly finding bugs appear a year after writing
>> some code
>> because I missed some static branch paths when originally authoring.
>>
>
> If you throw -cov while running unittests, it'll give you a report on which code was executed and which wasn't. Very simple and useful.
>

It is a useful tool, but you can see how going to great lengths to write this explosion of paths is a massive pain in the first place, let alone additional overhead to comprehensively test that it works... it should never have been a problem to start with.

You may argue that I didn't test my code effectively. I argue that my code should never have existed in the first place. It's wildly unsanitary, and very difficult to maintain; I can rarely understand the complexity of ref handling code looking back after some months.

This was my very first complaint about D, on day 1... 6 years later, I'm still struggling with it on a daily basis.

Here's some code I've been struggling with lately, tell me you think this
is right:
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L286
<- this one only deals with the return value. is broken, need to
special-case properties that receive arguments by ref, since you can't pass
rvalue->ref
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L115
<- Ref!RT workaround, because I need a ref local
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L172
<- Ref!T again, another instance where I need a ref local
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L180
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L129 <-
special case handling for Ref!T that I could eliminate if ref was part of
the type. Note: 3rd party code never has this concession...
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L448 <- more
invasion of Ref!, because getValue may or may not return ref, which would
be lost beyond this point, and it's not practical to static-if duplicate
this entire function with another version that returns ref.
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L157 <-
special-case function, again because ref isn't part of the type
There are many more instances of these throughout this code.

This is just one example, and not even a particularly bad one (I've had
worse), because there are never multiple args involved. These sorts of
things come up on most projects I've worked on.
The trouble with these examples, is that you can't try and imagine a direct
substitution of the static branches and Ref!T with 'ref T' if ref were part
of the type. This problem has deeply interfered with the code, and API
concessions have been made throughout to handle it. If ref were part of the
type, this code would be significantly re-worked and simplified. There
would probably be one little part somewhere that did logic on T, alias with
or without 'ref' as part of the type, and the cascading noise would mostly
disappear.

Add to all of that that I still can't pass an rvalue to a ref function (5
years later!!) >_<

It's also worth noting that, with regard to 'ref', the above code only _just_ suits my needs. It's far from bug-free; there are places in there where I was dealing with ref, but it got so complicated, and I didn't actually make front-end use of the case in my project, that I gave up and ignored those cases... which is not really ideal considering this is a fairly popular library.

I have spent days trying to get this right. If I were being paid hourly, I think I would have burned something like $2000.

On 22 September 2014 19:22, via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> On Sunday, 21 September 2014 at 11:37:19 UTC, Manu via Digitalmars-d wrote:
>
>> On 21 September 2014 16:02, deadalnix via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
>>
>>  On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:
>>>
>>>  On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:
>>>>
>>>>  What happens when a scope() thing finds it's way into generic code? If
>>>>> the type
>>>>> doesn't carry that information, then you end up in a situation like
>>>>> ref.
>>>>> Have
>>>>> you ever had to wrestle with ref in generic code?
>>>>> ref is the biggest disaster zone in D, and I think all it's problems
>>>>> will
>>>>> translate straight to scope if you do this.
>>>>>
>>>>>
>>>> I'm unaware of this disaster zone.
>>>>
>>>>
>>> Well it is very real. I had to duplicate bunch of code in my visitor
>>> generator recently because of it. Getting generic code ref correct is
>>> very
>>> tedious, error prone, and guarantees code duplication and/or various
>>> static
>>> ifs all over the place.
>>>
>>>
>> It's also extremely hard to unittest; explodes the number of static if
>> paths exponentially. I'm constantly finding bugs appear a year after
>> writing some code because I missed some static branch paths when
>> originally
>> authoring.
>>
>
> If I understand you right, your problems come from the fact that sometimes in a template you want ref, and sometimes you don't.
>
> But I think this mostly doesn't apply to scope: either you borrow things, or you don't. In particular, when you do borrow something, you're not interested in the owner your parameter has inside the caller, you just take it by scope (narrowing the lifetime). Thus there needs to be no information about it inside the callee, and you don't need different instantiations depending on it.
>
> One special case where scope deduction might be desirable are template functions that apply predicates (delegates, lambdas) to passed-in parameters, like map and filter. For these, the scope-ness of the input range can depend on whether the predicates are able to take their parameters as scope.
>

Application to scope will be identical to ref. A function that returns or receives scope that is inserted into generic code must have that property cascaded outwards appropriately. If typeof() or alias loses 'scope', then it will all go tits-up.

On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d wrote:
> Application to scope will be identical to ref. A function that returns or
> receives scope that is inserted into generic code must have that property
> cascaded outwards appropriately. If typeof() or alias loses 'scope', then
> it will all go tits-up.

For receiving it's not necessary, because whether or not the argument is scoped, the function can always borrow it. The lifetime of its parameter is narrower than what it gets passed.

For return values, the situation is a bit different: They can of course not be assigned to non-scoped variables. But the solution for this simple: the generic code needs to use scope, too. A function that returns scope does so for a reason after all. This will work even if the return value of the called function turns out not to be scoped for this particular instantiation. And all this is an implementation of the generic code, it won't bleed outside, unless the generic code wants to return the scoped value. In this case, simply apply the same technique, just one lever higher.

I don't see this as a problem for (new) code written with scope in mind. For existing code, of course some adjustments are necessary, but the same is true if you change existing code to be const correct, for example, or to be compatible with `shared`.

On 22 September 2014 22:14, via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d wrote:
>
>> Application to scope will be identical to ref. A function that returns or receives scope that is inserted into generic code must have that property cascaded outwards appropriately. If typeof() or alias loses 'scope', then it will all go tits-up.
>>
>
> For receiving it's not necessary, because whether or not the argument is scoped, the function can always borrow it. The lifetime of its parameter is narrower than what it gets passed.
>

It's particularly common in D to produce templates that wrap functions. If the wrapper doesn't propagate scope outwards, then it can no longer be called by a caller who borrowed arguments which are to be forwarded to the function being called. Likewise for return values.

For return values, the situation is a bit different: They can of course not
> be assigned to non-scoped variables. But the solution for this simple: the generic code needs to use scope, too.


This is precisely the problem with ref...
Are you saying that ALL generic code needs to be 'scope' always? That's not
semantically correct.

A function that returns scope does so for a reason after all.


And the generic code can't know what it is. That knowledge must be encoded in the type system.

This will work even if the return value of the called function turns out
> not to be scoped for this particular instantiation. And all this is an implementation of the generic code, it won't bleed outside, unless the generic code wants to return the scoped value. In this case, simply apply the same technique, just one lever higher.
>

I can't see the solution you're trying to ilustrate, can you demonstrate?

On Monday, 22 September 2014 at 11:20:57 UTC, Manu via Digitalmars-d wrote:
> It is a useful tool, but you can see how going to great lengths to write
> this explosion of paths is a massive pain in the first place, let alone
> additional overhead to comprehensively test that it works... it should
> never have been a problem to start with.

Hmm... even if the code is syntactically succinct, it doesn't necessarily mean lower complexity or that it requires less testing. You provided an example yourself: you have generic code, which works for values, but not for references. You need a lot of testing not because the features have different syntax, but because they work differently, so code, which works for one thing, may not work for another.

On Monday, 22 September 2014 at 12:37:47 UTC, Manu via Digitalmars-d wrote:
> On 22 September 2014 22:14, via Digitalmars-d <digitalmars-d@puremagic.com>
> wrote:
>
>> On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d wrote:
>>
>>> Application to scope will be identical to ref. A function that returns or
>>> receives scope that is inserted into generic code must have that property
>>> cascaded outwards appropriately. If typeof() or alias loses 'scope', then
>>> it will all go tits-up.
>>>
>>
>> For receiving it's not necessary, because whether or not the argument is
>> scoped, the function can always borrow it. The lifetime of its parameter is
>> narrower than what it gets passed.
>>
>
> It's particularly common in D to produce templates that wrap functions.
> If the wrapper doesn't propagate scope outwards, then it can no longer be
> called by a caller who borrowed arguments which are to be forwarded to the
> function being called. Likewise for return values.

You have a point there.

>
> For return values, the situation is a bit different: They can of course not
>> be assigned to non-scoped variables. But the solution for this simple: the
>> generic code needs to use scope, too.
>
>
> This is precisely the problem with ref...
> Are you saying that ALL generic code needs to be 'scope' always? That's not
> semantically correct.
>

To be clear, I am referring to the implementation, the actual code of the generic functions, not to its signature. The signature of course needs to match the semantics of the generic function.

I also over-generalized when I said that the return value cannot be assigned to non-scope. It can theoretically depend on the input, though I'm not sure whether it's a good idea to allow this:

    scope!a T scopeFunc(scope T a, scope T b);

    T* genericFunc(T)(T* input1, T* input2) {
        ...
        // this is fine in theory: input1 points to GC or global data
        // (because it's not designated as scope)
        string temp = scopeFunc(input1, input2);
        ...
        return temp;
    }

Evidently, this generic function cannot accept scoped pointers, thus it can't take advantage of the fact that scopeFunc() does. It's therefore a good idea, to make any generic (and non-generic, too) function take its parameters by scope if at all possible:

    scope!input1 T* genericFunc(T)(scope T* input1, scope T* input2) {
        ...
        scope temp = scopeFunc(input1, input2);
        ...
        return temp;
    }

This second version of the function will work with scope and non-scope inputs alike. More importantly, it doesn't depend on whether it's allowed to assign a scope return value to non-scope if its owners aren't scoped (which I'd like to avoid).

Now, `genericFunc()` in turn returns a scoped reference, so any other generic code that calls it must again be treated in the same way. Everything else would be unsafe, after all. But note that this only goes as far as an actual scoped value is returned up the call-chain. Once you stop doing so (because you only need to call the scope-returning functions internally for intermediate results, for example), returning scope would no longer be necessary. It still makes sense for these higher-up functions to _accept_ scope, of course, if it's possible.

Of course, this is only true as long as the generic function knows about the semantics of `scopeFunc()`. Once you're trying to wrap functions (as alias predicates, opDispatch), there needs to be another solution. I'm not sure what this could be though. I see now why you mentioned ref. But the problem is not restricted to ref and scope, it would also apply to UDAs. Maybe, because it is a more general problem independent of scope, the solution needs to be a more general one, too.

As far as I can see, there's always a variadic template parameter involved (which is actually a list of aliases in most cases, right?). Would it work if aliases would forward their storage classes, too? Thinking about it, this seems natural, because aliases mean "pass by name".

> > A function that returns scope does so for a reason after all.
>
>
> And the generic code can't know what it is. That knowledge must be encoded
> in the type system.
>
> This will work even if the return value of the called function turns out
>> not to be scoped for this particular instantiation. And all this is an
>> implementation of the generic code, it won't bleed outside, unless the
>> generic code wants to return the scoped value. In this case, simply apply
>> the same technique, just one lever higher.
>>
>
> I can't see the solution you're trying to ilustrate, can you demonstrate?

I hope that the examples above illustrate what I mean. Of course, this doesn't solve the "perfect forwarding" problem, which should maybe be treated separately.

Maybe you can give counter examples too, if you think it doesn't work.

On 22 September 2014 23:38, Kagamin via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On Monday, 22 September 2014 at 11:20:57 UTC, Manu via Digitalmars-d wrote:
>
>> It is a useful tool, but you can see how going to great lengths to write this explosion of paths is a massive pain in the first place, let alone additional overhead to comprehensively test that it works... it should never have been a problem to start with.
>>
>
> Hmm... even if the code is syntactically succinct, it doesn't necessarily mean lower complexity or that it requires less testing. You provided an example yourself: you have generic code, which works for values, but not for references. You need a lot of testing not because the features have different syntax, but because they work differently, so code, which works for one thing, may not work for another.
>

Eliminating static branches containing different code has a very
significant reduction in complexity. It's also DRY.
I don't think I provided that example... although it's certainly true that
there are semantic differences that may lead to distinct code paths, it is
my experience that in the majority of cases, if I just had the ref-ness as
part of the type, the rest would follow naturally. I have never encountered
a situation where I would feel hindered by ref as part of the type.
I think it's also easier to get from ref in the type to the raw type than
the reverse (which we must do now); We are perfectly happy with Unqual!T
and things like that.