On 12/14/2014 5:44 PM, Dicebot wrote:
> I am looking for a tool to prevent escaping references to user-defined entities
> / resources from specific scope. RC case is not interesting at all to me though
> I recognize the potential benefits of tweaking the scope system to help with it.
> But there is a big difference between tweak for one specific use case or
> designing feature around it (so it becomes almost useless in other cases).
>
> I have already provided you an example of code I want to be enforced with scope
> and two major issues with existing proposal that make it lacking. This example
> is akin to litmus test for any scope proposal - if it doesn't allow to express
> such design, that proposal is simply not good enough.
>
> In case you have forgotten, I am reminding about two critical points that are
> necessary to make it fly:
>
> 1) inout analog for scope to be able to deduce borrowship of return values based
> on borrowship of input arguments. Current system is conservative and results in
> template bloat (complicated by the fact that it is storage class thus not
> actually part of a type)

1. All inout actually does is reduce code copy/pasta, it is not critical.
2. This is what scope inference is all about.


> 2) at least optional transitivity to be able to express to protect with scope
> data referenced by slice or owned linked list referenced from root node.

1. that won't work unless scope is a type constructor
2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper)


> In your tree example I would have never wanted scope protection of one specific
> node of such tree - but a transitive scope protection of whole tree view
> available through on of node pointers/references. It doesn't matter who and how
> owns the data for borrowship implementation - only thing that matters that _it
> is not me_.

As I explained to Manu, transitive scope makes things like doing a symbol table lookup, then putting a pointer to the found symbol in an AST, not workable.

On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote:
> 1. All inout actually does is reduce code copy/pasta, it is not critical.

Being forced to duplicate every single function in two flavors to actually make scope system usable? This is as much critical as it can be.

> 2. This is what scope inference is all about.

Which only works with templates and lack of scope on arguments does not affect function body -> templates are not necessary, same as inout.

>> 2) at least optional transitivity to be able to express to protect with scope
>> data referenced by slice or owned linked list referenced from root node.
>
> 1. that won't work unless scope is a type constructor

I know and this is why I am leaning toward it being qualifier despite all related issues.

> 2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper)

It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example.

>> In your tree example I would have never wanted scope protection of one specific
>> node of such tree - but a transitive scope protection of whole tree view
>> available through on of node pointers/references. It doesn't matter who and how
>> owns the data for borrowship implementation - only thing that matters that _it
>> is not me_.
>
> As I explained to Manu, transitive scope makes things like doing a symbol table lookup, then putting a pointer to the found symbol in an AST, not workable.

This sounds totally against my understanding of scope. I want scope exactly to prohibit such actions. However it is possible in slightly modified way - where you don't directly insert pointer to AST but use public method of AST control structure that checks if supplied scope pointer belongs to list of nodes it owns  and casts away scope after that.

On 12/14/2014 7:41 PM, Dicebot wrote:
> On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote:
>> 1. All inout actually does is reduce code copy/pasta, it is not critical.
>
> Being forced to duplicate every single function in two flavors to actually make
> scope system usable? This is as much critical as it can be.

C++ seems to do fine without it for const. It's a convenience feature.


>> 2. This is what scope inference is all about.
>
> Which only works with templates and lack of scope on arguments does not affect
> function body -> templates are not necessary, same as inout.

It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-)

I do view inference as something we need to extend to more functions.


>>> 2) at least optional transitivity to be able to express to protect with scope
>>> data referenced by slice or owned linked list referenced from root node.
>>
>> 1. that won't work unless scope is a type constructor
>
> I know and this is why I am leaning toward it being qualifier despite all
> related issues.

That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward.


>> 2. it can be achieved by using wrappers that only allow by-scope references to
>> their data (RC is an example of such a wrapper)
> It is viral approach and backwards incompatible one - we can't change signatures
> of Phobos functions to return Wrapped!(char[]) instead of char[] for example.
> While theoretically possible I can't call this approach practical - not until
> see some convincing application example.

But you can change signatures to add annotations?

You're going to get one or the other.


> This sounds totally against my understanding of scope. I want scope exactly to
> prohibit such actions. However it is possible in slightly modified way - where
> you don't directly insert pointer to AST but use public method of AST control
> structure that checks if supplied scope pointer belongs to list of nodes it
> owns  and casts away scope after that.

Wrappers do that, too.

I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is to make it easy to adapt the behavior of a type by wrapping it in a struct, rather than build new behavior into the language.

On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:
> C++ seems to do fine without it for const. It's a convenience feature.

C++ const does not really restrict or affect anything "for real", it is non-existent feature. `scope` as proposed would result in inability to store result of predicates if those are ever to accept scope data - unless you defined two versions for each function that may possibly accept scope data with absolutely identical body.

>>> 2. This is what scope inference is all about.
>>
>> Which only works with templates and lack of scope on arguments does not affect
>> function body -> templates are not necessary, same as inout.
>
> It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-)

This is not really answering my objections but side-stepping. I do support attribute inference. I would actually support full attribute inference in the language (and help convincing Don about it ;)) if it was introduced as foundation of the language and not arbitrary hack (== rethink the way .di interfaces and static libraries are defined)

> I do view inference as something we need to extend to more functions.

That would help with many issues but it need careful design to be well-accepted. How about teaming up to do a DIP about it at some point where this discussion is over? :P

>>>> 2) at least optional transitivity to be able to express to protect with scope
>>>> data referenced by slice or owned linked list referenced from root node.
>>>
>>> 1. that won't work unless scope is a type constructor
>>
>> I know and this is why I am leaning toward it being qualifier despite all
>> related issues.
>
> That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward.

Yes, I know and this why I am leaning towards that path but willing to accept any storage-class based solution - as long as it fits my basic criteria. I wonder if some hybrid approach is possible - for example, keep it storage class but pretend anything transitively accessible though it as if it was of scope storage class on its own. But that quickly gets into "exceptions for special cases" land :(

>>> 2. it can be achieved by using wrappers that only allow by-scope references to
>>> their data (RC is an example of such a wrapper)
>> It is viral approach and backwards incompatible one - we can't change signatures
>> of Phobos functions to return Wrapped!(char[]) instead of char[] for example.
>> While theoretically possible I can't call this approach practical - not until
>> see some convincing application example.
>
> But you can change signatures to add annotations?

Such annotation would only prevent from compiling code which was already undefined by documentation. Changing return type (which wasn't Voldemort type before) may break any program that has lines akin to `char[] data = foo()` - perfectly legal working code.

>> This sounds totally against my understanding of scope. I want scope exactly to
>> prohibit such actions. However it is possible in slightly modified way - where
>> you don't directly insert pointer to AST but use public method of AST control
>> structure that checks if supplied scope pointer belongs to list of nodes it
>> owns  and casts away scope after that.
>
> Wrappers do that, too.
>
> I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is to make it easy to adapt the behavior of a type by wrapping it in a struct, rather than build new behavior into the language.

Problem is that our UDT are simply not that good enough yet. Wrappers are fine in general but I want to wrap for less common cases and support more common ones natively. I suppose this is your intention too but we disagree on what is actually the common case and what is the special case.

On Sunday, 14 December 2014 at 23:27:15 UTC, Walter Bright wrote:
>> I think I said that clearly: "What do you get when you take a pointer
>> of a function with an auto-ref
>> arg? ...an error, because it's not actually a function!"
>
> What's baffling to me is why even declare a function pointer parameter as auto-ref? I don't use floating point to index strings, either, that doesn't mean floating point is a failed concept.

That's not what he's talking about. My understanding is that Manu is talking about something like this:

void foo(auto ref someConcreteType) { ... }

auto fooP = &foo; //error, foo is not a function

On Monday, 15 December 2014 at 01:44:27 UTC, Dicebot wrote:
> On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright wrote:
>> The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example.
>>
>> What are you looking for?
>
> I am looking for a tool to prevent escaping references to user-defined entities / resources from specific scope. RC case is not interesting at all to me though I recognize the potential benefits of tweaking the scope system to help with it. But there is a big difference between tweak for one specific use case or designing feature around it (so it becomes almost useless in other cases).
>
> I have already provided you an example of code I want to be enforced with scope and two major issues with existing proposal that make it lacking. This example is akin to litmus test for any scope proposal - if it doesn't allow to express such design, that proposal is simply not good enough.
>
> In case you have forgotten, I am reminding about two critical points that are necessary to make it fly:
>
> 1) inout analog for scope to be able to deduce borrowship of return values based on borrowship of input arguments. Current system is conservative and results in template bloat (complicated by the fact that it is storage class thus not actually part of a type)

Is it be possible to make `inout` a more general tool for type erasure? I don't think yet another ad-hoc mechanism should be introduced.

>
> 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node.
>
> ------
>
> In your tree example I would have never wanted scope protection of one specific node of such tree - but a transitive scope protection of whole tree view available through on of node pointers/references. It doesn't matter who and how owns the data for borrowship implementation - only thing that matters that _it is not me_.

I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that).

The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping.

Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO.

On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote:
> I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that).
>
> The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping.
>
> Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO.

Actually I think in neither case children should be declared as scope members. Instead those should be declared as private members but methods that return pointers/references to children would be declared to return (transitive) scope pointers/references.

For me "scopeness" is a property of "view", not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available.

On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote:
> On 12/14/2014 7:41 PM, Dicebot wrote:
>> Being forced to duplicate every single function in two flavors to actually make
>> scope system usable? This is as much critical as it can be.
>
> C++ seems to do fine without it for const. It's a convenience feature.
>

It's more than that. It has the potential (well, could have) to be used for type erasure, which is a useful and worthwhile concept by itself.

>> I know and this is why I am leaning toward it being qualifier despite all
>> related issues.
>
> That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward.

To be honest, I think one reason for those bugs is the awkward implementation. I had a look at mtype.c to see how difficult it would be to add another type modifier. It's no fun - the current implementation requires a dozen groups of helper functions to be written for every possible combination of type modifiers. Therefore there are helpers like `makeConst`, `makeImmutable`, `makeShared`, `makeSharedConst`, `makeWild` (this is `inout`), `makeWildConst`, `makeSharedWild`, `makeSharedWildConst`, and `makeMutable`. There are also some monster switch statements in the same fashion.

No wonder this is error prone. I suspect that with a cleaner implementation most of these problems will go away.

On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote:
> On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote:
>> I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that).
>>
>> The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping.
>>
>> Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO.
>
> Actually I think in neither case children should be declared as scope members. Instead those should be declared as private members but methods that return pointers/references to children would be declared to return (transitive) scope pointers/references.

I see this as the same thing, conceptually (module the "transitive" in in your parentheses). It's just a different mechanism. Scope members are just a short-cut instead of writing an accessor method that returns scope.

>
> For me "scopeness" is a property of "view", not object itself - this also makes ownership method of actual data irrelevant.

Yes, and this would be true for scope members just the same. (If the aggregate "owns" it's scope members, it will however get in trouble when it tries to free() them, therefore your mechanism is more appropriate in this case.)

> Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available.

Agreed.

On Saturday, 13 December 2014 at 20:45:53 UTC, Walter Bright wrote:
> On 12/13/2014 2:10 AM, bearophile wrote:
>> Walter Bright:
>>
>>>>> struct Tree {
>>>>>   RefCount!(Tree*) left;
>>>>>   RefCount!(Tree*) right;
>>>>>   ...
>>>>> }
>>>>
>>>> ... I don't think I'd ever have a use for this code.
>>>
>>> You have no use for tree structures?
>>
>> Giving a reference counter to every pointer in a binary tree sounds a bit too much.
>
> DMD, for example, uses ASTs where the nodes have multiple parents.

It is a DAG then.