On 12/12/2014 4:07 PM, deadalnix wrote:
> On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:
>> How not?
>
> scope a = new Stuff();
> scope b = a;
>
> auto rc1 = RC(a);
> auto rc2 = RC(b);
>
> // Enjoy the free when it comes !

What's the type signature of RC ?

On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright wrote:
> On 12/12/2014 4:07 PM, deadalnix wrote:
>> On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:
>>> How not?
>>
>> scope a = new Stuff();
>> scope b = a;
>>
>> auto rc1 = RC(a);
>> auto rc2 = RC(b);
>>
>> // Enjoy the free when it comes !
>
> What's the type signature of RC ?

RCWrapper!T RC(T)(scope T t) or somethign along these lines.

On 12/12/2014 4:59 PM, deadalnix wrote:
> On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright wrote:
>> On 12/12/2014 4:07 PM, deadalnix wrote:
>>> On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote:
>>>> How not?
>>>
>>> scope a = new Stuff();
>>> scope b = a;
>>>
>>> auto rc1 = RC(a);
>>> auto rc2 = RC(b);
>>>
>>> // Enjoy the free when it comes !
>>
>> What's the type signature of RC ?
>
> RCWrapper!T RC(T)(scope T t) or somethign along these lines.

And how does RC assign it to the wrapper? (It will fail to compile, and so won't be a safety hole.)

On 13 December 2014 at 08:14, Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:
>>>
>>> I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do.
>>
>>
>> I've said so many times, it's the single greatest regular frustration
>> I encounter, by far.
>> That is of course a relative measure. It's not the 'cornerstone of
>> everything I do'; I don't start discussions about things that are not
>> broken and otherwise painless.
>> It is the thing that is *the most broken*, and leads to the most edge
>> cases, general bloat, and text mixins.
>> It comes up the most frequently, and by that metric alone, I consider
>> it highest priority on my list.
>>
>> I've also said many times before, it possibly stems from the fact that
>> one of the key cornerstones of almost everything I do in D is interact
>> with other languages.
>> This is a practical reality, I can't write all my code in D, and have
>> mountains of existing code to interact with.
>> Boilerplate is the result. D has powerful systems to automate
>> boilerplate, like a carrot dangling right in front of my face, but
>> it's almost always thwarted by ref, and almost exclusively so.
>>
>> My recent work updating LuaD to support all the features I required
>> wasted about 80% of my time dealing with ref issues.
>> In my past C++ bindings solutions, much code, which was otherwise
>> simple, readable, and elegant, turned into a mess of text mixins,
>> almost exclusively because ref is broken.
>
>
> You've said this before, many times, but what is lacking is an explanation of WHY. What is the pattern, and why do you need that pattern? You say you don't like auto ref because it doesn't give you exact control, but what are the cases where auto ref is wrong?

I did just give some examples, I'll repeat; auto ref fails when the function is extern.

It is also wrong that when I pass an int or float, it is passed by ref instead of by value... that is never what I want!

What do you get when you take a pointer of a function with an auto-ref
arg? ...an error, because it's not actually a function!
So in a context where I'm dealing with functions and function pointers
(very, very frequent), I suddenly have something that's not a function
find it's way into my meta and I have to special-case the hell out of
it.
The solution in this case is to wrap it in a non-auto-ref shim with
the ref-ness explicitly stated in the way I expect... which is in
itself another problem, because 'ref' is not part of the type! So the
annoying de-auto-ref-ing shim must be a text mixin with some very
complex, practically unreadable, definitely unmaintainable logic
surrounding it. It's insanity on top of insanity!

I also run the invisible risk of generating 2^num_args instances of the code for functions containing auto-ref args.

When I'm writing a function that's not a template, I intend, and
expect, to write a function that's _not a template_.
Templates and functions are different things. I think it's a massive
mistake to have created a way to write a template that looks nothing
like a template.

auto-ref is not, and never has been a tool I have wanted. I don't have any use for auto-ref, and it has only proven to make an already severe problem worse. I've tried to use it before in some instances, but it made ref's out of int's and floats, so even in the rare potentially useful cases, I had to reject it.

At the time, you introduced auto-ref as a 'solution' to an issue that
I was the primary complainant (although the solution was mainly pushed
by Andrei iirc?). I absolutely rejected it then, and said it would
have the disastrous effect of setting ref in stone, which I was
already very critical about. I reject it even more now that I have had
some experience with it.
Who was the customer that you designed it for? (It wasn't me!)
What issue was it designed to address?


>>> Consider a ref counted type, RC!T. If scope were transitive, then you
>>> could
>>> not have, say, a tree where the edges were RC!T. I.e., the payload of an
>>> RC
>>> type should not be forced to be scope.
>>
>>
>> I'm not sure I quite visualise this correctly...
>
>
> struct Tree {
>    RefCount!(Tree*) left;
>    RefCount!(Tree*) right;
>    ...
> }

... I don't think I'd ever have a use for this code.
I've been using trees for a long time, and I can't imagine a situation
where each node in a tree would want to be ref counted.
It sounds like a disaster for performance, and I can't imagine any
advantage it would offer?
Perhaps in some complex graph structure... but I expect that sort of
thing would be far more specialised.

I can see common cases where nodes may contain a refcounted object as
node data, but that's different than the situation you demonstrate
here.
I'll need to think about how that situation would be affected in this case.


>> So you're saying that a pointer itself shouldn't be able to escape a call tree, but the thing it points to should be able to escape just fine?
>
>
> Yes.
>
>
>> It feels like that kinda defeats the purpose...
>
>
> You're arguing that a data structure with only one access point is the only kind of data structure in use. With transitive scope, such a data structure would be the only one possible!

Surely the possibility remains for such a data structure that is NOT scope... but I think I can see your point.

There is a common case, ie, refcounting, where we want to be able to
elide rc fiddling, and that may be the sole call for scope in those
cases.
It seems like that situation calls for something like head-scope. I
think transitive scope has a different application, useful in
different situations.

It seems a problem that a struct that contains a single pointer, which is passed by value, can't have scope applied under your proposal(?). I use this pattern in almost every interaction with other languages that I encounter. Some external API which operates via opaque pointer, wants to be wrapped in such a way that it maintains it's behaviour as a reference type, but also gains convenient member-style access. It's kinda making a pseudo-class out of something that's not a class, and scope needs to support this.


>> I guess you're seeing a situation where 'scope' is almost exclusively
>> useful as a mechanism to tell the RC that it doesn't need to worry
>> about ref-fiddling, and the thing we're passing isn't interested in
>> scope restrictions at any level other than that RC optimisation?
>> I guess I can see your angle... but my reaction is if that's all you
>> are concerned about, maybe scope is the wrong tool for eliding ref
>> fiddling in that case :/
>
>
> 'scope' is a way to say that this use of a pointer does not escape this scope. That is incredibly useful.
>
> Recall my statements that pervasive refcounting is a terrible performance problem because of all the inc/dec? Knowing that references cannot escape means an inc/dec pair can be elided.

I know this. I'm not arguing against any proposition that we need
scope to make RC practical. I've been arguing in favour of that for at
least as long as I've been batting for the ARC team, and even then
some more, because it would also address rvalue->ref, which is another
massive pain point!
But there are more things than pointers which need to be protected
against escaping their scope, in particular, things that contain
pointers...

Maybe there's a compromise. If we say scope isn't 'transitive', but it
is transitive when it comes to aggregates?
Ie, you can apply scope to a by-val struct, and all aggregate members
have scope applied.
It is not transitive in the sense that it does not follow through
pointer members, but any such pointer members may not escape as if it
were a pointer argument alone.

This is unlike any other behaviour in D, but without that, I think this proposal fails to suit my most common use cases.

On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:
> I did just give some examples, I'll repeat; auto ref fails when the
> function is extern.

Don't make it extern, then.


> It is also wrong that when I pass an int or float, it is passed by ref
> instead of by value... that is never what I want!

If there's source to the function, it'll often be inlined which will remove the indirection.


> What do you get when you take a pointer of a function with an auto-ref
> arg? ...an error, because it's not actually a function!
> So in a context where I'm dealing with functions and function pointers
> (very, very frequent), I suddenly have something that's not a function
> find it's way into my meta and I have to special-case the hell out of
> it.

Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates?


> The solution in this case is to wrap it in a non-auto-ref shim with
> the ref-ness explicitly stated in the way I expect... which is in
> itself another problem, because 'ref' is not part of the type! So the
> annoying de-auto-ref-ing shim must be a text mixin with some very
> complex, practically unreadable, definitely unmaintainable logic
> surrounding it. It's insanity on top of insanity!
>
> I also run the invisible risk of generating 2^num_args instances of
> the code for functions containing auto-ref args.

I wonder what is the need for the code that you are writing.


> When I'm writing a function that's not a template, I intend, and
> expect, to write a function that's _not a template_.
> Templates and functions are different things. I think it's a massive
> mistake to have created a way to write a template that looks nothing
> like a template.

A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think "template", think "compile-time argument to a function". I convinced Andrei to never mention "template" in his D book, as it brings forth all kinds of connotations and expectations that impede understanding what D templates actually are. I think the result was successful.


> auto-ref is not, and never has been a tool I have wanted. I don't have
> any use for auto-ref, and it has only proven to make an already severe
> problem worse. I've tried to use it before in some instances, but it
> made ref's out of int's and floats, so even in the rare potentially
> useful cases, I had to reject it.

If it's a rare useful case, why is it a pervasive problem?


> At the time, you introduced auto-ref as a 'solution' to an issue that
> I was the primary complainant (although the solution was mainly pushed
> by Andrei iirc?). I absolutely rejected it then, and said it would
> have the disastrous effect of setting ref in stone, which I was
> already very critical about. I reject it even more now that I have had
> some experience with it.
> Who was the customer that you designed it for? (It wasn't me!)
> What issue was it designed to address?

I still have no idea what code you are developing that you need to send ints and function pointers to the same argument of a function template, yet you don't use function templates. Nor do I understand what pattern you need that simply must mix up ref and value parameters, and why that pattern appears pervasively in your code.


>> struct Tree {
>>     RefCount!(Tree*) left;
>>     RefCount!(Tree*) right;
>>     ...
>> }
>
> ... I don't think I'd ever have a use for this code.

You have no use for tree structures?


> I've been using trees for a long time, and I can't imagine a situation
> where each node in a tree would want to be ref counted.

You have more than one parent of a node. You never write data structures like that? dmd uses such pervasively (Type and Expression).


> It sounds like a disaster for performance, and I can't imagine any
> advantage it would offer?

How do you propose to manage the memory explicitly otherwise? Use a GC? :-)


> I can see common cases where nodes may contain a refcounted object as
> node data, but that's different than the situation you demonstrate
> here.

That would fail if scope were transitive.


> But there are more things than pointers which need to be protected
> against escaping their scope, in particular, things that contain
> pointers...

Solve it in the general case (this proposal) and RC now works.


> Maybe there's a compromise. If we say scope isn't 'transitive', but it
> is transitive when it comes to aggregates?

One thing I've tried very hard to make work in D is have basic types / aggregages / arrays be interchangeable.


> Ie, you can apply scope to a by-val struct, and all aggregate members
> have scope applied.
> It is not transitive in the sense that it does not follow through
> pointer members, but any such pointer members may not escape as if it
> were a pointer argument alone.
>
> This is unlike any other behaviour in D, but without that, I think
> this proposal fails to suit my most common use cases.

I have no idea what your use cases are, but when you're explicitly managing memory, every damn pointer needs to be carefully evaluated as to what exactly it's pointing two and who owns it. Scope is not a magic solution to this, neither is shared_ptr, neither are Rust's annotations.

The only scheme that absolves the user of having to deal with this is - GC.

Walter Bright:

>>> struct Tree {
>>>    RefCount!(Tree*) left;
>>>    RefCount!(Tree*) right;
>>>    ...
>>> }
>>
>> ... I don't think I'd ever have a use for this code.
>
> You have no use for tree structures?

Giving a reference counter to every pointer in a binary tree sounds a bit too much.

Bye,
bearophile

On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via Digitalmars-d wrote:

> Templates and functions are different things. I think it's a massive
> mistake to have created a way to write a template that looks nothing
> like a template.

That is a misconception spread by C++. Templates are pure functions applied at compile time. Function templates are functions partially applied at compile time.

On 2014-12-13 01:09, Walter Bright wrote:

> The proposal provides escape proof passing of arguments. This is
> necessary in order to make rc safe, for example.
>
> What are you looking for?

Passing references to stack allocated data to functions safely is one idea that comes to mind.

-- 
/Jacob Carlborg

On 2014-12-13 06:11, Walter Bright wrote:

> Don't make it extern, then.

He's said a couple of time he's interfacing with C and/or C++ code.

> Why are function pointers and ints going to the same argument of a
> function? I thought you weren't using templates?

I think he meant taking the address of a function with auto ref.

-- 
/Jacob Carlborg

On 12/13/2014 8:01 AM, Jacob Carlborg wrote:
> On 2014-12-13 01:09, Walter Bright wrote:
>
>> The proposal provides escape proof passing of arguments. This is
>> necessary in order to make rc safe, for example.
>>
>> What are you looking for?
>
> Passing references to stack allocated data to functions safely is one idea that
> comes to mind.

'scope ref' does that.