On 23 September 2014 01:00, via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> On Monday, 22 September 2014 at 12:37:47 UTC, Manu via Digitalmars-d wrote:
>
>> On 22 September 2014 22:14, via Digitalmars-d <
>> digitalmars-d@puremagic.com>
>> wrote:
>>
>>  On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d
>>> wrote:
>>>
>>>  Application to scope will be identical to ref. A function that returns
>>>> or
>>>> receives scope that is inserted into generic code must have that
>>>> property
>>>> cascaded outwards appropriately. If typeof() or alias loses 'scope',
>>>> then
>>>> it will all go tits-up.
>>>>
>>>>
>>> For receiving it's not necessary, because whether or not the argument is
>>> scoped, the function can always borrow it. The lifetime of its parameter
>>> is
>>> narrower than what it gets passed.
>>>
>>>
>> It's particularly common in D to produce templates that wrap functions. If the wrapper doesn't propagate scope outwards, then it can no longer be called by a caller who borrowed arguments which are to be forwarded to the function being called. Likewise for return values.
>>
>
> You have a point there.
>

It's massive. Trust me, when you're fabricating functions from introspecting other functions, you NEED all these details in the type. If they're not part of the types, then you need to craft lots of junk code to detect that information explicitly, and then you need to branch out n^^2 distinct paths (massive DRY violation) to handle all the combinations. Imagine if 'const' was a storage class in the context of generic code... the practicality of that would be identical to 'ref', except that const appears everywhere, and ref appears rarely (probably because people tend to avoid it, because it's broken).

For return values, the situation is a bit different: They can of course not
>>
>>> be assigned to non-scoped variables. But the solution for this simple:
>>> the
>>> generic code needs to use scope, too.
>>>
>>
>>
>> This is precisely the problem with ref...
>> Are you saying that ALL generic code needs to be 'scope' always? That's
>> not
>> semantically correct.
>>
>>
> To be clear, I am referring to the implementation, the actual code of the generic functions, not to its signature. The signature of course needs to match the semantics of the generic function.
>

So...?


I also over-generalized when I said that the return value cannot be
> assigned to non-scope. It can theoretically depend on the input, though I'm not sure whether it's a good idea to allow this:
>
>     scope!a T scopeFunc(scope T a, scope T b);
>
>     T* genericFunc(T)(T* input1, T* input2) {
>         ...
>         // this is fine in theory: input1 points to GC or global data
>         // (because it's not designated as scope)
>         string temp = scopeFunc(input1, input2);
>         ...
>         return temp;
>     }
>
> Evidently, this generic function cannot accept scoped pointers, thus it can't take advantage of the fact that scopeFunc() does. It's therefore a good idea, to make any generic (and non-generic, too) function take its parameters by scope if at all possible:
>
>     scope!input1 T* genericFunc(T)(scope T* input1, scope T* input2) {
>         ...
>         scope temp = scopeFunc(input1, input2);
>         ...
>         return temp;
>     }
>
> This second version of the function will work with scope and non-scope inputs alike. More importantly, it doesn't depend on whether it's allowed to assign a scope return value to non-scope if its owners aren't scoped (which I'd like to avoid).
>

We arrive at yet another case of "it should have been that way from the
start" wrt 'scope'.
The baggage of annotation, and the lack of annotation to existing code is a
pretty big pill to swallow.
If it were just part of the type, there would be no problem, T would
already be 'scope T' in the cases where you expect. I can't see any
disadvantages there.

Now, `genericFunc()` in turn returns a scoped reference, so any other
> generic code that calls it must again be treated in the same way. Everything else would be unsafe, after all. But note that this only goes as far as an actual scoped value is returned up the call-chain. Once you stop doing so (because you only need to call the scope-returning functions internally for intermediate results, for example), returning scope would no longer be necessary. It still makes sense for these higher-up functions to _accept_ scope, of course, if it's possible.
>
> Of course, this is only true as long as the generic function knows about the semantics of `scopeFunc()`. Once you're trying to wrap functions (as alias predicates, opDispatch), there needs to be another solution. I'm not sure what this could be though. I see now why you mentioned ref. But the problem is not restricted to ref and scope, it would also apply to UDAs. Maybe, because it is a more general problem independent of scope, the solution needs to be a more general one, too.
>

I think UDA's are clearly distinct from ref and scope. UDA's can attribute
types.
I strongly believe that the problem is the notion of a 'storage class',
it's a faulty concept. It has never yet proven itself be what I've ever
wanted in any case I'm aware of.
Your proposal even implies changes to the concept as it is; like being able
to create a local that is 'scope'. Is that a recognition of existing
problems? I've been asking for 'ref' local's for half a decade...


As far as I can see, there's always a variadic template parameter involved
> (which is actually a list of aliases in most cases, right?). Would it work if aliases would forward their storage classes, too?


I'll say, no. It's equally important that is(), typeof(), and type aliasing
all work too.
I also think this kinda undermines the notion of a storage class in
principle...



> Thinking about it, this seems natural, because aliases mean "pass by name".
>

Types can't be passed to alias parameters. Alias refers to a symbol, not a type.

Even if this were to be jigged somehow, I think it really just kicks the can forward, and ref/scope is lost somewhere else. I can't imagine ANY situation where I would ever want that information to be lost (unless it was deliberate, like Unqual!), so what's the end goal? We end up in a situation where it's properly passed along everywhere that the information is currently lost... and we have the same thing as if it were just part of the type in the first place?

 > A function that returns scope does so for a reason after all.
>>
>>
>> And the generic code can't know what it is. That knowledge must be encoded in the type system.
>>
>> This will work even if the return value of the called function turns out
>>
>>> not to be scoped for this particular instantiation. And all this is an implementation of the generic code, it won't bleed outside, unless the generic code wants to return the scoped value. In this case, simply apply the same technique, just one lever higher.
>>>
>>>
>> I can't see the solution you're trying to ilustrate, can you demonstrate?
>>
>
> I hope that the examples above illustrate what I mean. Of course, this doesn't solve the "perfect forwarding" problem, which should maybe be treated separately.
>

This would be another band-aid to a core problem. D already has plenty of
these.
I hear this "perfect forwarding" concept thrown around, and I think it's
another faulty concept. I rarely want 'perfect' forwarding... why would I
be forwarding in the first place if it's 'perfect' (there are cases, but
not so common)? I almost always want *imperfect* forwarding; that is, some
small detail(/s) about the forwarding are manipulated. I think this is the
primary case where storage class falls apart in concept.


Maybe you can give counter examples too, if you think it doesn't work.
>

It's complex and time consuming to do so. The situations where it all breaks down are often fairly complex (probably why 'ref' as a storage class seems like an okay idea at face value, although I still don't understand the advantage conceptually), and they tend to appear when you don't expect it. My examples with ref above are all practically applicable to scope too though.

Let's turn this around... Why the complexity? Why would you make the change to your proposal to make 'scope' something else outside of the type system? What is the advantage to that complexity. D has no structured method for dealing with that sort of meta, we only have types. Beyond that, it's just spaghetti, as we learn from ref.

Marc Schütz:

> http://wiki.dlang.org/User:Schuetzm/scope

If a mutable argument of a function is tagged as unique, the type system guarantees that there are no other references to it. So can a function 'foo' like this be "strongly pure"?


int[] foo(unique int[] a) pure {
    a[0]++;
    return a;
}

Bye,
bearophile

On 9/22/2014 4:20 AM, Manu via Digitalmars-d wrote:
> On 22 September 2014 13:19, Walter Bright via Digitalmars-d
> <digitalmars-d@puremagic.com <mailto:digitalmars-d@puremagic.com>> wrote:
>
>     On 9/21/2014 4:27 AM, Manu via Digitalmars-d wrote:
>
>         It's also extremely hard to unittest; explodes the number of static if paths
>         exponentially. I'm constantly finding bugs appear a year after writing
>         some code
>         because I missed some static branch paths when originally authoring.
>
>
>     If you throw -cov while running unittests, it'll give you a report on which
>     code was executed and which wasn't. Very simple and useful.
>
>
> It is a useful tool, but you can see how going to great lengths to write this
> explosion of paths is a massive pain in the first place, let alone additional
> overhead to comprehensively test that it works... it should never have been a
> problem to start with.

There are two separate issues here - the first is knowing whether or not the code is unittested. -cov solves that issue. The second is having the multiple code paths in the first place.

Have you tried auto ref?

I don't really know what the code you show is supposed to do from a high level. My first impression is you are trying to write it like you'd write C++ code. Perhaps there's a more D idiomatic way of doing it that doesn't lead to the tangle you've got.

BTW, ref (as you know) is part of the type in C++. However, I can vouch for it being a special case everywhere in C++, and is a horrifying quagmire of strange edge cases. That's why it's not part of the type in D.

Manu, once again your posts have the message embedded twice in them, making for very large posts. What's happening is the posts have the text in plain text, then the text again in HTML.

Please configure your news editor to only produce the plain text messages. The HTML versions are ignored and uselessly consume bandwidth and disk space.

On Tuesday, 23 September 2014 at 09:17:48 UTC, bearophile wrote:
> Marc Schütz:
>
>> http://wiki.dlang.org/User:Schuetzm/scope
>
> If a mutable argument of a function is tagged as unique, the type system guarantees that there are no other references to it. So can a function 'foo' like this be "strongly pure"?
>
>
> int[] foo(unique int[] a) pure {
>     a[0]++;
>     return a;
> }

I think so. But note that `unique` is not part of my proposal, I merely used it in the example. I think it could be implemented relatively easily, because DMD internally already has a concept of uniqueness that is used for converting things to immutable implicitly. But this would be a different proposal.

On 9/23/14 5:17 AM, bearophile wrote:
> Marc Schütz:
>
>> http://wiki.dlang.org/User:Schuetzm/scope
>
> If a mutable argument of a function is tagged as unique, the type system
> guarantees that there are no other references to it. So can a function
> 'foo' like this be "strongly pure"?
>
>
> int[] foo(unique int[] a) pure {
>      a[0]++;
>      return a;
> }

I don't think so. Strong pure function optimizations would not work for something like:

auto x = foo(a) ~ foo(a);

Which for a strong pure function could be optimized to:

auto r = foo(a);
auto x = r ~ r;

-Steve

On Tuesday, 23 September 2014 at 09:46:17 UTC, Walter Bright wrote:
> Have you tried auto ref?

For some purposes, auto ref does the wrong thing. Whether you get a reference depends on whether you pass an lvalue or an rvalue. But some templates need to take either a struct by reference, or a class/interface (already being a reference) by value.

This is the case deadalnix mentioned further up in this thread. I don't know whether it applies to Manu's code, too. AFAIUI he's more concerned about forwarding parameters in wrapper types.

On Tuesday, 23 September 2014 at 10:16:26 UTC, Marc Schütz wrote:
> On Tuesday, 23 September 2014 at 09:17:48 UTC, bearophile wrote:
>> Marc Schütz:
>>
>>> http://wiki.dlang.org/User:Schuetzm/scope
>>
>> If a mutable argument of a function is tagged as unique, the type system guarantees that there are no other references to it. So can a function 'foo' like this be "strongly pure"?
>>
>>
>> int[] foo(unique int[] a) pure {
>>    a[0]++;
>>    return a;
>> }
>
> I think so.

Ok, I take it back ;-) Steven is right. It is however the case that this function's return value would still be unique.

On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
> Ok, I take it back ;-) Steven is right. It is however the case that this
> function's return value would still be unique.

Yes, it could be unique. I haven't read this thread really, so I don't know what has been proposed, but looking at the snippet, wouldn't you have to tag the return value? You tagged the parameter with unique.

-Steve

On Monday, 22 September 2014 at 15:54:23 UTC, Manu via Digitalmars-d wrote:
> We arrive at yet another case of "it should have been that way from the
> start" wrt 'scope'.
> The baggage of annotation, and the lack of annotation to existing code is a
> pretty big pill to swallow.
> If it were just part of the type, there would be no problem, T would
> already be 'scope T' in the cases where you expect. I can't see any
> disadvantages there.

But it has very different semantics. You cannot expect the same code to work for scope and non-scope alike. And it is to be expected  that you have to adjust existing code if you want to take advantage of a new feature. If by "it should have been that way from the start" you mean that scope should be the default, well yes, but we're in the same situation with immutable by default, pure by default, @safe by default, nothrow by default, ... That ship has sailed, unfortunately.

> I think UDA's are clearly distinct from ref and scope. UDA's can attribute
> types.
> I strongly believe that the problem is the notion of a 'storage class',
> it's a faulty concept. It has never yet proven itself be what I've ever
> wanted in any case I'm aware of.
> Your proposal even implies changes to the concept as it is; like being able
> to create a local that is 'scope'. Is that a recognition of existing
> problems? I've been asking for 'ref' local's for half a decade...

That you can't declare ref locals is not inherent in the concept of storage class. You can already today declare scope locals, it just doesn't have an effect. And static is a storage class, too.

> This would be another band-aid to a core problem. D already has plenty of
> these.
> I hear this "perfect forwarding" concept thrown around, and I think it's
> another faulty concept. I rarely want 'perfect' forwarding... why would I
> be forwarding in the first place if it's 'perfect' (there are cases, but
> not so common)? I almost always want *imperfect* forwarding; that is, some
> small detail(/s) about the forwarding are manipulated. I think this is the
> primary case where storage class falls apart in concept.

That is a straw man. Of course, perfect forwarding needs to allow for imperfect forwarding, too. It would indeed be not useful if you couldn't inspect and modify the types (and storage classes) of your parameters.

>
>
> Maybe you can give counter examples too, if you think it doesn't work.
>>
>
> It's complex and time consuming to do so. The situations where it all
> breaks down are often fairly complex (probably why 'ref' as a storage class
> seems like an okay idea at face value, although I still don't understand
> the advantage conceptually), and they tend to appear when you don't expect
> it. My examples with ref above are all practically applicable to scope too
> though.

A concrete example would still be very helpful to understand your point of view. I.e., not only the concrete wrapper type/function, but also how you would want to use it, and why it wouldn't work without scope being part of the type.

>
> Let's turn this around... Why the complexity? Why would you make the change
> to your proposal to make 'scope' something else outside of the type system?

Well, I think Ivan gave an excellent example (ElementType) why it should be separate from the type. Type modifiers currently only deal with mutability (and the related shared). There can be some nasty surprises if we add another concept there.

> What is the advantage to that complexity. D has no structured method for
> dealing with that sort of meta, we only have types. Beyond that, it's just
> spaghetti, as we learn from ref.

Then it's better to introduce such a method, IMO.