Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
i think the current full closure implementation should take a redesign.

My suggestion:
Let D have 2 kind of delegate closures (I don't know how to call this correctly)
local delegates and heap delegates

local delegate are that from D1 without modifiction.
They can access every local variable in the surrounding scope, no heap allocation if the addess is taken. It is expected the adress is not used outside the allowed scope.

heap delegates
- can only access variables from the surrounding scope, if they are marked as 'const', 'invariant' or 'final'. Which means, they are not expected to changed after initialization.
- with instantiation of the delegate, a heap allocated frame is used to store a copy of the reference 'final' varialbles for the delegate.
That means in case of a loop, the delegate get a new heap allocation with each iteration.

foreach( element; container ){
  const c = element;
  logLater( new { writefln(c); });
}

Reply to Frank,

> Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043 i
> think the current full closure implementation should take a redesign.
> 
> My suggestion:
> Let D have 2 kind of delegate closures (I don't know how to call this
> correctly)
> local delegates and heap delegates
> local delegate are that from D1 without modifiction.
> They can access every local variable in the surrounding scope, no heap
> allocation if the addess is taken. It is expected the adress is not
> used
> outside the allowed scope.
> heap delegates
> - can only access variables from the surrounding scope, if they are
> marked as 'const', 'invariant' or 'final'. Which means, they are not
> expected to changed after initialization.
> - with instantiation of the delegate, a heap allocated frame is used
> to
> store a copy of the reference 'final' varialbles for the delegate.
> That means in case of a loop, the delegate get a new heap allocation
> with each iteration.
> foreach( element; container ){
> const c = element;
> logLater( new { writefln(c); });
> }

Sounds reasonable. I like that it makes the semantics of referenced variables explicit.

However I would amend it by allowing access to non const (etc.) function arguments (which would be copied on function entry) and variables who's scope is exactly that of the function.

void fn(int thisArg)
{
 int andThis;
 {
   int butNotThis;
 }
}

These would greatly increase the practicality of the design without adding ambiguities. 

On second thought, the function args might be bit of a trick. I could live without them.

BCS wrote:
> Reply to Frank,
> 
>> Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043 i
>> think the current full closure implementation should take a redesign.
>>
>> My suggestion:
>> Let D have 2 kind of delegate closures (I don't know how to call this
>> correctly)
>> local delegates and heap delegates
>> local delegate are that from D1 without modifiction.
>> They can access every local variable in the surrounding scope, no heap
>> allocation if the addess is taken. It is expected the adress is not
>> used
>> outside the allowed scope.
>> heap delegates
>> - can only access variables from the surrounding scope, if they are
>> marked as 'const', 'invariant' or 'final'. Which means, they are not
>> expected to changed after initialization.
>> - with instantiation of the delegate, a heap allocated frame is used
>> to
>> store a copy of the reference 'final' varialbles for the delegate.
>> That means in case of a loop, the delegate get a new heap allocation
>> with each iteration.
>> foreach( element; container ){
>> const c = element;
>> logLater( new { writefln(c); });
>> }
> 
> Sounds reasonable. I like that it makes the semantics of referenced variables explicit.
> 
> However I would amend it by allowing access to non const (etc.) function arguments (which would be copied on function entry) and variables who's scope is exactly that of the function.
> 
> void fn(int thisArg)
> {
>  int andThis;
>  {
>    int butNotThis;
>  }
> }
> 
> These would greatly increase the practicality of the design without adding ambiguities.
> On second thought, the function args might be bit of a trick. I could live without them.

Ewww, -- changing the scope of the variable should never change its behavior, IMO.

Reply to Robert,

> BCS wrote:
> 
>> [...] allowing access to non const (etc.)
>> function arguments (which would be copied on function entry) and
>> variables who's scope is exactly that of the function.
>> 
>
> Ewww, -- changing the scope of the variable should never change its
> behavior, IMO.
> 

That is a good point, but how else you you get a real closure that has mutable state?

int delegate() Seq()
{
  int at = 0;
  int Next() { return at++; }
  return &Next;
}

BCS wrote:
> 
> That is a good point, but how else you you get a real closure that has mutable state?
> 
> int delegate() Seq()
> {
>   int at = 0;
>   int Next() { return at++; }
>   return &Next;
> }
> 
> 

Err.... just have a closure like D has now?... one that allows access to any visible variable?

-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D

Frank Benoit wrote:
> Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
> i think the current full closure implementation should take a redesign.
> 
> My suggestion:
> Let D have 2 kind of delegate closures (I don't know how to call this correctly)
> local delegates and heap delegates
> 
> local delegate are that from D1 without modifiction.
> They can access every local variable in the surrounding scope, no heap allocation if the addess is taken. It is expected the adress is not used outside the allowed scope.
> 

Refresh my memory on why would that be wanted. Is it because the compiler incorrectly heap allocates some variables in a situation that is statically verifiable that such allocation wasn't necessary?
Or is it more of a programmer's help, to avoid him/her making mistakes and writing code that inadvertently causes locals to be heap allocated?


> heap delegates
> - can only access variables from the surrounding scope, if they are marked as 'const', 'invariant' or 'final'. Which means, they are not expected to changed after initialization.
> - with instantiation of the delegate, a heap allocated frame is used to store a copy of the reference 'final' varialbles for the delegate.
> That means in case of a loop, the delegate get a new heap allocation with each iteration.
> 
> foreach( element; container ){
>   const c = element;
>   logLater( new { writefln(c); });
> }
> 
> 

Why would we want such version of "heap delegates" that are more restrictive in power than D's current full closures?



-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D

Bruno Medeiros schrieb:
> Frank Benoit wrote:
>> Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
>> i think the current full closure implementation should take a redesign.
>>
>> My suggestion:
>> Let D have 2 kind of delegate closures (I don't know how to call this correctly)
>> local delegates and heap delegates
>>
>> local delegate are that from D1 without modifiction.
>> They can access every local variable in the surrounding scope, no heap allocation if the addess is taken. It is expected the adress is not used outside the allowed scope.
>>
> 
> Refresh my memory on why would that be wanted. Is it because the compiler incorrectly heap allocates some variables in a situation that is statically verifiable that such allocation wasn't necessary?
> Or is it more of a programmer's help, to avoid him/her making mistakes and writing code that inadvertently causes locals to be heap allocated?
> 

the D1 nested functions or anonymous classes can access local variable from the surrounding scope. local vars are located on the stack. if the reference to the nested function of anonymous class is escaping the local scope (by storing it somewhere or passing it or returning it) and the surrounding scope if left, the variables are no more valid. Running the nested function or anonymous class accessing those variable will result in crashes.

So the D1 feature is very good in performance, but it should be taken care or the livetime of the accessed variables.

The D2 "full closure" feature allocated the whole stack frame on the stack, if the compile detects the "take address from nested function".

> 
>> heap delegates
>> - can only access variables from the surrounding scope, if they are marked as 'const', 'invariant' or 'final'. Which means, they are not expected to changed after initialization.
>> - with instantiation of the delegate, a heap allocated frame is used to store a copy of the reference 'final' varialbles for the delegate.
>> That means in case of a loop, the delegate get a new heap allocation with each iteration.
>>
>> foreach( element; container ){
>>   const c = element;
>>   logLater( new { writefln(c); });
>> }
>>
>>
> 
> Why would we want such version of "heap delegates" that are more restrictive in power than D's current full closures?
> 

The current D2 approach of simply allocating the whole stack frame looks good on the first sight, but IMHO it is not sufficient.
- it breaks the old semantic
- it always do heap allocation, but the old semantic is an important D feature.
- it make invariant/const data change value, if used in a loop.

See the link to the bug report.

This is why i think, for the case of "heap delegates" the access should be restricted to constant data, and that data should be copied to the delegate, instead of heap-ing the surrounding scope stack frame.

Frank Benoit wrote:
> Bruno Medeiros schrieb:
>> Frank Benoit wrote:
>>> Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
>>> i think the current full closure implementation should take a redesign.
>>>
>>> My suggestion:
>>> Let D have 2 kind of delegate closures (I don't know how to call this correctly)
>>> local delegates and heap delegates
>>>
>>> local delegate are that from D1 without modifiction.
>>> They can access every local variable in the surrounding scope, no heap allocation if the addess is taken. It is expected the adress is not used outside the allowed scope.
>>>
>>
>> Refresh my memory on why would that be wanted. Is it because the compiler incorrectly heap allocates some variables in a situation that is statically verifiable that such allocation wasn't necessary?
>> Or is it more of a programmer's help, to avoid him/her making mistakes and writing code that inadvertently causes locals to be heap allocated?
>>
> 
> the D1 nested functions or anonymous classes can access local variable from the surrounding scope. local vars are located on the stack. if the reference to the nested function of anonymous class is escaping the local scope (by storing it somewhere or passing it or returning it) and the surrounding scope if left, the variables are no more valid. Running the nested function or anonymous class accessing those variable will result in crashes.
> 
> So the D1 feature is very good in performance, but it should be taken care or the livetime of the accessed variables.
> 
> The D2 "full closure" feature allocated the whole stack frame on the stack, if the compile detects the "take address from nested function".
> 

Agh, I didn't think it would heap allocate that easily. That's the most conservative approach, and it "indeed puts a serious burden on the use of scoped closures".
So I agree we either need:
* a way to specify scoped closures/delegates.
* a smarter compiler that better detects scoped delegates. I'd say it doesn't have to be 100% accurate, but should be better than the simple heuristic "take address from nested function".



>>
>>> heap delegates
>>> - can only access variables from the surrounding scope, if they are marked as 'const', 'invariant' or 'final'. Which means, they are not expected to changed after initialization.
>>> - with instantiation of the delegate, a heap allocated frame is used to store a copy of the reference 'final' varialbles for the delegate.
>>> That means in case of a loop, the delegate get a new heap allocation with each iteration.
>>>
>>> foreach( element; container ){
>>>   const c = element;
>>>   logLater( new { writefln(c); });
>>> }
>>>
>>>
>>
>> Why would we want such version of "heap delegates" that are more restrictive in power than D's current full closures?
>>
> 
> The current D2 approach of simply allocating the whole stack frame looks good on the first sight, but IMHO it is not sufficient.
> - it breaks the old semantic
D2 breaks with a lot of things. And I fully agree that D shouldn't be bound to D1.0 compatibility (at least not on such early D stage)

> - it always do heap allocation, but the old semantic is an important D feature.
Your "heap closures" also do heap allocation. (although they may deferred to only when it's strictly needed, such as when the closure delegate is evaluated/created, instead of when the outer variable is declared.

> - it make invariant/const data change value, if used in a loop.
> 
> See the link to the bug report.
Yeah, but that's a bug, it should go away.



-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D

Reply to Bruno,

> BCS wrote:
> 
>> That is a good point, but how else you you get a real closure that
>> has mutable state?
>> 
>> int delegate() Seq()
>> {
>> int at = 0;
>> int Next() { return at++; }
>> return &Next;
>> }
> Err.... just have a closure like D has now?... one that allows access
> to any visible variable?
> 

Franks local delegates won't work because they aren't valid after the other function returns, his heap delegates won't work because the variables they can access can't be altered.

Reply to Bruno,

>> - it make invariant/const data change value, if used in a loop.
>> 
>> See the link to the bug report.
> 
> Yeah, but that's a bug, it should go away.
> 

This is a proposal on how to make it go away. It's a design bug not a compiler bug.