All,
  I ran into a small problem regarding the use of anon delegates, and what their context (.ptr) is set to.  If anyone can share any insight as to if this is a bug or not, I would appreciate it.

In short: I'm attempting to create delegates on the fly and use them elsewhere.  After many strange errors, I narrowed it down to the following case.  Note the assertions at the end:

abstract class Foobar{
    Fn getFn();
}
alias void delegate() Fn;

auto foo = new class Foobar{
    int x;
    Fn getFn(){
	return { x = x };
    }
};
auto fn = foo.getFn();
assert((&foo.getFn).ptr is foo); // passes
assert(fn.ptr is foo); // fails (ptr does *not* point to the object)


So by this, anon delegates are not the same as actual object delegates.  Is this a bug, or is this intentional?

"When comparing with nested functions, the function form is analogous to static or non-nested functions, and the delegate form is analogous to non-static nested functions. In other words, a delegate literal can access stack variables in its enclosing function, a function literal cannot."

As you can see, the docs are as clear as mud.  I suppose a "non-static nested function" wouldn't know about the 'this' pointer, nor care - but then why encumber it with the 'delegate' keyword?


-- 
- EricAnderton at yahoo

Pragma wrote:
> All,
>   I ran into a small problem regarding the use of anon delegates, and what their context (.ptr) is set to.  If anyone can share any insight as to if this is a bug or not, I would appreciate it.
> 
> In short: I'm attempting to create delegates on the fly and use them elsewhere.  After many strange errors, I narrowed it down to the following case.  Note the assertions at the end:
> 
> abstract class Foobar{
>     Fn getFn();
> }
> alias void delegate() Fn;
> 
> auto foo = new class Foobar{
>     int x;
>     Fn getFn(){
>     return { x = x };
>     }
> };

That class has a member function returning a delegate. That returned delegate has the stack frame of said member function as its context pointer and should thus not be called after it returns.
Basically, the return value is useless as it can't safely be called.

> auto fn = foo.getFn();

'fn' now contains said delegate, with the context pointer set to an invalidated stackframe.

> assert((&foo.getFn).ptr is foo); // passes

(&foo.getFn) is a delegate with context foo and function pointer Foo.getFn.

> assert(fn.ptr is foo); // fails (ptr does *not* point to the object)

fn.ptr is the invalid stack frame of the call to foo.getFn(), not foo itself.

> So by this, anon delegates are not the same as actual object delegates.  Is this a bug, or is this intentional?

Intentional.

> "When comparing with nested functions, the function form is analogous to static or non-nested functions, and the delegate form is analogous to non-static nested functions. In other words, a delegate literal can access stack variables in its enclosing function, a function literal cannot."
> 
> As you can see, the docs are as clear as mud.  I suppose a "non-static nested function" wouldn't know about the 'this' pointer, nor care - but then why encumber it with the 'delegate' keyword?

It *does* know about 'this'. However, its context pointer is a stack frame (that happens to store 'this' in your case), not 'this' itself.


There are simply several 'types' of delegates. One is a delegate that has a class or struct (or perhaps even a union) instance as its context pointer. These are created by (&instance.memberFunction).

Another 'type' has a stack frame as its context pointer. These are created by (&nestedFunction) or { foo(); }. They should not be called after the function call (of the function directly containing their definition) in which they were created returns.


Spec reference: http://www.digitalmars.com/d/function.html#closures

Reply to Pragma,

> All,
> I ran into a small problem regarding the use of anon delegates, and
> what their context (.ptr) is set to.  If anyone
> can share any insight as to if this is a bug or not, I would
> appreciate it.
> In short: I'm attempting to create delegates on the fly and use them
> elsewhere.  After many strange errors, I narrowed it down to the
> following case.  Note the assertions at the end:
> 
> abstract class Foobar{
> Fn getFn();
> }
> alias void delegate() Fn;
> 
> auto foo = new class Foobar{
> int x;
> Fn getFn(){
> return { x = x };
> }
> };
> auto fn = foo.getFn();
> assert((&foo.getFn).ptr is foo); // passes
> assert(fn.ptr is foo); // fails (ptr does *not* point to the object)
> So by this, anon delegates are not the same as actual object
> delegates.  Is this a bug, or is this intentional?
> 
> "When comparing with nested functions, the function form is analogous
> to static or non-nested functions, and the delegate form is analogous
> to non-static nested functions. In other words, a delegate literal can
> access stack variables in its enclosing function, a function literal
> cannot."
> 
> As you can see, the docs are as clear as mud.  I suppose a "non-static
> nested function" wouldn't know about the 'this' pointer, nor care -
> but then why encumber it with the 'delegate' keyword?
> 


Really, some BIG RED TEXT needs to be added to the doc in  few places. This is one of them. Anon delegates are totally invalid after the function they are declared within returns. Any statement of the form:

return {...};

is ALWAYS invalid. This has gotten about a dozen people in the last week or so and, as you said. "the docs are as clear as mud" on this. Anon delegate in any scope use the current function's frame pointer as the context. Once the function returns, this more than likely is junk.

BCS wrote:
> Reply to Pragma,
> 
>> All,
>> I ran into a small problem regarding the use of anon delegates, and
>> what their context (.ptr) is set to.  If anyone
>> can share any insight as to if this is a bug or not, I would
>> appreciate it.
>> In short: I'm attempting to create delegates on the fly and use them
>> elsewhere.  After many strange errors, I narrowed it down to the
>> following case.  Note the assertions at the end:
>>
>> abstract class Foobar{
>> Fn getFn();
>> }
>> alias void delegate() Fn;
>>
>> auto foo = new class Foobar{
>> int x;
>> Fn getFn(){
>> return { x = x };
>> }
>> };
>> auto fn = foo.getFn();
>> assert((&foo.getFn).ptr is foo); // passes
>> assert(fn.ptr is foo); // fails (ptr does *not* point to the object)
>> So by this, anon delegates are not the same as actual object
>> delegates.  Is this a bug, or is this intentional?
>>
>> "When comparing with nested functions, the function form is analogous
>> to static or non-nested functions, and the delegate form is analogous
>> to non-static nested functions. In other words, a delegate literal can
>> access stack variables in its enclosing function, a function literal
>> cannot."
>>
>> As you can see, the docs are as clear as mud.  I suppose a "non-static
>> nested function" wouldn't know about the 'this' pointer, nor care -
>> but then why encumber it with the 'delegate' keyword?
>>
> 
> 
> Really, some BIG RED TEXT needs to be added to the doc in  few places. This is one of them. Anon delegates are totally invalid after the function they are declared within returns. Any statement of the form:
> 
> return {...};
> 
> is ALWAYS invalid. This has gotten about a dozen people in the last week or so and, as you said. "the docs are as clear as mud" on this. Anon delegate in any scope use the current function's frame pointer as the context. Once the function returns, this more than likely is junk.
> 
> 

Thanks for the clarification.  I wanted to be sure that I wasn't out of my mind.

This is really disappointing.  I was working on a runtime Spirit engine (as a counterpart to your own library) that would use inline delegates as valid parse expressions:

auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args parsed"); };

It works fine for this kind of stuff, but once you graduate beyond there and want to bind terminals to class members, you need the ability to perform filtering and aritmetic on them.  But once you do that, you find that your stack frame is all out of whack. :(

// This construct is used to help create a context for binding and manipulating parse terms.
// create() is called within LambdaExpr's constructor to create and cache the Expr to be used
// at this stage of the parser.  Of course, the inline delegates are invalid once this returns.

/** 'factor' rule from the infamous 'infix calculator' BNF grammar. */
auto factor = new class LambdaExpr{
    real result,term;
    Expr create(){
        auto posFactor   = T('+') >> ws >> term[&term] >> { result += term; };
        auto negFactor   = T('-') >> ws >> term[&term] >> { result +- term; };
        return             ws >> term[&result] >> -+(ws >> (posFactor | negFactor)) >> { return result; };
    }
};

It would be nice if things so succinct would work as I expected.  Instead I have to break all the delegates out into proper class methods, which bloats the code and runs contrary to the whole point of having a parser framework like this.


The ability to declare *first class* delegates as anonymous and inline like the above is just too tempting to leave alone.  I suppose the only way to have your cake and eat it too is to preserve the stack somehow.  Maybe this is a job for stack threads?

-- 
- EricAnderton at yahoo

Pragma wrote:
[snip]
> It would be nice if things so succinct would work as I expected.  Instead I have to break all the delegates out into proper class methods, which bloats the code and runs contrary to the whole point of having a parser framework like this.
> 
> 
> The ability to declare *first class* delegates as anonymous and inline like the above is just too tempting to leave alone.  I suppose the only way to have your cake and eat it too is to preserve the stack somehow.  Maybe this is a job for stack threads?

Maybe we could get Walter to implement a slight change to such delegates: instead of getting the innermost context pointer, they should get the outermost possible context pointer that still contains all the variables/members used?
So then in your case they would get the 'this' pointer, in nested functions they could get the stackframe of the outer function if they didn't reference variables local to the inner function, and in member functions of (non-static) nested classes they could even contain a copy of the 'outer' pointer of the nested class, so only the outer class is referenced if that's all that's needed.

I have no idea how difficult this would be to implement, but I think this will go a long way towards making delegates more usable.

The only disadvantage would be that the validity of a delegate after a function call returns would depend on the code it contains, which may be a source of tricky bugs...

An alternative would be the explicit context syntax I somewhere today (can't find it anymore :( ). Your example delegate would then look something like 'this.{ x = x; }'. Not as pretty, but still much better than having to define a new member function for this operator, don't you think?

Reply to Pragma,

[...]
> Thanks for the clarification.  I wanted to be sure that I wasn't out
> of my mind.
> 
> This is really disappointing.  I was working on a runtime Spirit
> engine (as a counterpart to your own library) that would use inline
> delegates as valid parse expressions:

Cool!

> 
> auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
> parsed"); };
> 
> It works fine for this kind of stuff, but once you graduate beyond
> there and want to bind terminals to class members, you need the
> ability to perform filtering and aritmetic on them.  But once you do
> that, you find that your stack frame is all out of whack. :(

I tend to use structs for that kind of thing. They entail less overhead than classes.

[...]
> It would be nice if things so succinct would work as I expected.
> Instead I have to break all the delegates out into proper class
> methods, which bloats the code and runs contrary to the whole point of
> having a parser framework like this.

I've said it befor, What is needed is to allow delegate leterals to expicetly state there context.

auto c = new C;
return c.(char[] foo){retrun this.toString() ~ foo;}

Another thing would be non static init for anon structs. This and a few others would make for some neat code.

int delegate(int,int,int) Bar(int i, int j, int k)
{
  return new struct {
     int a = i;
     int b = j;
     int c = k;
     }.(int m, int n, int o){ return a*m + b*n + c*o;};
}

> 
> The ability to declare *first class* delegates as anonymous and inline
> like the above is just too tempting to leave alone.  I suppose the
> only way to have your cake and eat it too is to preserve the stack
> somehow.

I guess that is more or less what the above does.

(Where's the 2.0 wish list?  <g> )

You might try somthing like this (warning it has been called "evil"), this vertion dosn't work but it can actualy be made to by using structs and a little refactoring. The basic idea is use structs with delegate as members to build operatio trees. In this case I used it for the reduction, you'd could using it for parsing as well.

p.s. It realy is evil, you have been warned.


   // a base type for derivation
class Type{}
   // a templated derived type to store things in
class TypeT(T) : Type
{
   public T val;
   static TypeT!(T) opCall(T v)
   {
       auto ret = new TypeT!(T);
       ret.val = v;
       return ret;
   }
}
/*
Doc ::=
    A:a B:b C:c    = new Fig(a,b,c)
   | D:d E:e        = new Fig(d,e)
*/
TypeT!(Fig) delegate() Parse_Doc(inout char[] from)
{
   char[] tmp;

       // copy from
   tmp = from;
   if(auto a = Parse_A(tmp)) // try to parse an "a"
   if(auto b = Parse_B(tmp)) // try to parse an "b"
   if(auto c = Parse_C(tmp)) // try to parse an "c"
   {    // if all's good
           // update what's been parsed
       from = tmp;
       return &(
               // make an array from the args
           [cast(Type delegate())a,b,c]
               // copy it (force off stack)
           .dup
           )
               // form delegate from array
               // and function literal                           .function TypeT!(Fig)(Type delegate()[] args)
           {
                   // literal calls given code
               auto ret =
                   new Fig(
                       // call args for values
                   cast(a.typeof)args[0]().val,
                   cast(b.typeof)args[1]().val,
                   cast(b.typeof)args[2]().val);

                   // place in object and return
               return TypeT!(ret.typeof)(ret);
           };
   }

       // restore tmp
   tmp = from;

       // blah, blah, blah
   if(auto d = Parse_D(tmp))
   if(auto e = Parse_E(tmp))
   {
       from = tmp;
       return &([cast(Type delegate())d,e].dup)
           .function TypeT!(Fig)(Type delegate()[] args)
           {
               auto ret =
                   new Fig(
               cast(d.typeof)args[0]().val,
               cast(e.typeof)args[1]().val);
               return TypeT!(ret.typeof)(ret);
           };
   }
   return null;
}
/*
A ::=
   "hello":v    = v
*/
TypeT!(char[]) delegate() Parse_A(inout char[] from)
{
   char[] tmp;

   tmp=from;
       // check for a "hello"
   if(from[0.."hello".length] == "hello")
   {
           // update from
       from = from["hello".length..$];
           // return delegate returning "hello" in obj
       return &("hello".dup).function TypeT!(char[])()
           {
               auto ret =
                   from[0.."hello".length];
               return TypeT!(ret.typeof)(ret);
           };
   }

   return null;
}
/// ........

BCS wrote:
> Reply to Pragma,
>>
>> auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
>> parsed"); };
>>
>> It works fine for this kind of stuff, but once you graduate beyond
>> there and want to bind terminals to class members, you need the
>> ability to perform filtering and aritmetic on them.  But once you do
>> that, you find that your stack frame is all out of whack. :(
> 
> I tend to use structs for that kind of thing. They entail less overhead than classes.

I would have done the same, but I'm making heavy use of polymorphism to get the operators to work as expected.  However, that could be refactored into a struct-oriented design with some clever templates and mixins to emulate inheritance.

> 
> [...]
>> It would be nice if things so succinct would work as I expected.
>> Instead I have to break all the delegates out into proper class
>> methods, which bloats the code and runs contrary to the whole point of
>> having a parser framework like this.
> 
> I've said it befor, What is needed is to allow delegate leterals to expicetly state there context.
> 
> auto c = new C;
> return c.(char[] foo){retrun this.toString() ~ foo;}
> 
> Another thing would be non static init for anon structs. This and a few others would make for some neat code.
> 
> int delegate(int,int,int) Bar(int i, int j, int k)
> {
>   return new struct {
>      int a = i;
>      int b = j;
>      int c = k;
>      }.(int m, int n, int o){ return a*m + b*n + c*o;};
> }

I read the other thread here, in D.learn, on this concept.  I like the ability to anonymously declare any construct, but the ability to add literal methods/members to those is where it really shines.  I've been using similar techniques in Javascript programming, and it completely transforms the way you approach solving particular problems.  As Walter has written a compliant JS engine, you'd think he's be wise to the advantages here?  Maybe there's something else blocking this kind of syntax from inclusion into D.

> 
>>
>> The ability to declare *first class* delegates as anonymous and inline
>> like the above is just too tempting to leave alone.  I suppose the
>> only way to have your cake and eat it too is to preserve the stack
>> somehow.
> 
> I guess that is more or less what the above does.
> 
> (Where's the 2.0 wish list?  <g> )
> 

You mean, we dont' have one yet?!


-- 
- EricAnderton at yahoo

Pragma wrote:
> BCS wrote:
> 
>> Reply to Pragma,
>>
>>>
>>> auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
>>> parsed"); };
>>>
>>> It works fine for this kind of stuff, but once you graduate beyond
>>> there and want to bind terminals to class members, you need the
>>> ability to perform filtering and aritmetic on them.  But once you do
>>> that, you find that your stack frame is all out of whack. :(
>>
>>
>> I tend to use structs for that kind of thing. They entail less overhead than classes.
> 
> 
> I would have done the same, but I'm making heavy use of polymorphism to get the operators to work as expected.  However, that could be refactored into a struct-oriented design with some clever templates and mixins to emulate inheritance.

Ah, you must be doing something more complicated than I was thinking of. I often find myself using the pattern of a struct that is used in exactly one place:

struct S
{
	int i;
	int j(){return i}
}

auto s = new S;
s.i=2;
return &s.j

this can be done with a class, but it has some more overhead.

class C
{
	int i;
	this(int k){i=k;}
	int j(){return i}
}

return &(new C(1)).j

>>
>> (Where's the 2.0 wish list?  <g> )
>>
> 
> You mean, we dont' have one yet?!
> 

Not that I know of :-|

BCS wrote:
> Pragma wrote:
>> BCS wrote:
>>
>>> Reply to Pragma,
>>>
>>>>
>>>> auto rule = ws >> '(' >> +Argument >> ')' >> { writefln("args
>>>> parsed"); };
>>>>
>>>> It works fine for this kind of stuff, but once you graduate beyond
>>>> there and want to bind terminals to class members, you need the
>>>> ability to perform filtering and aritmetic on them.  But once you do
>>>> that, you find that your stack frame is all out of whack. :(
>>>
>>>
>>> I tend to use structs for that kind of thing. They entail less overhead than classes.
>>
>>
>> I would have done the same, but I'm making heavy use of polymorphism to get the operators to work as expected.  However, that could be refactored into a struct-oriented design with some clever templates and mixins to emulate inheritance.
> 
> Ah, you must be doing something more complicated than I was thinking of. 

Yea, sorry to talk so much about something that is basically vaporware.  I've learned to be cautious of releasing stuff without having completely worked the kinks out of it myself - if it's at all useful, poor quality offerings breed a contemptible support situation for everyone involved (especially for me).

But since the cat's out of the bag, I'll explain what I'm talking about.  Basically, it's something like this (paraphrased for clarity):

abstract class Expr{
  Expr opShr(Expr expr){ return new AndGroup(this,expr); }
  Expr opShr(String value){ return opShr(new Terminal(value)); }

  Expr opOr(Expr expr){ return new OrGroup(this,expr); }
  Expr opOr(String value){ return opOr(new Terminal(value)); }

  Expr opPlus(Expr expr){ return new Multiple(this); }
  Expr opNeg(Expr expr){ return new Optional(this); }

  bool Parse(IScanner scanner);
}
class Terminal: Expr{}
class Optional: Expr{}
class Multiple: Expr{}
class AndGroup: Expr{}
class OrGroup: Expr{}

... and so on, plus a few shims to get some expressions to work.

So that:
Expr ws =  /* defined elsewhere */
Expr rule = ws >> 'hello' >> ws >> 'world';

is equivalent to:
Expr rule = new AndGroup(
  ws,
  new AndGroup(
    new Terminal('hello'),
    new AndGroup(
      ws,
      new Terminal('world')
    )
  )
);

The various operators are overridden in the sub-classes so optimizations can be made.  For instance, AndGroup and OrGroup will maintain a list of terms such that the redundant nesting in the above never happens.  Now, to bring it back to where I started, I had some methods on Expr that did this:

Expr opShr(void delegate() predicate){ return opShr(new PredicateExpr(predicate)); }
Expr opOr(void delegate() predicate){ return opOr(new PredicateExpr(predicate)); }

... but that's dangerous for the reasons we discussed earlier.

FWIW, another limitation I ran into is that D offers no static operator overloads in the global scope, so we can't get operators to generate templates in the same way that Spirit can in C++.  Granted, that's a small limitation, but a 100% static parser /might/ help somewhat.

> I often find myself using the pattern of a struct that is used in exactly one place:
> 
> struct S
> {
>     int i;
>     int j(){return i}
> }
> 
> auto s = new S;
> s.i=2;
> return &s.j
> 
> this can be done with a class, but it has some more overhead.
> 
> class C
> {
>     int i;
>     this(int k){i=k;}
>     int j(){return i}
> }
> 
> return &(new C(1)).j
> 
>>>
>>> (Where's the 2.0 wish list?  <g> )
>>>
>>
>> You mean, we dont' have one yet?!
>>
> 
> Not that I know of :-|


-- 
- EricAnderton at yahoo