Hello there,

I'm new to D and experimenting with closures. I know the basics of how compilers translate things, especially in C++. So I appreciate the difficulty of implementing closures correctly and been wondering if it really works. I found a place where dmd 2 appears to fail, but I don't know whether that's a bug or just not supported:

import std.stdio;

struct Foo
{
  int a = 7;
  int bar() { return a; }

  int delegate() makedelegate() {
    int abc() { return a; }
    return &abc;
  }
}

void call(int delegate() dg)
{
  writefln("foo: %d", dg());
}

int delegate() makedelegate1()
{
  int x = 27;
  int abc() { return x; }
  return &abc;
}

int delegate() makedelegate2()
{
  Foo f;
  int abc() { return f.a; }
  return &abc;
}

int delegate() makedelegate3()
{
  Foo f;
  return &f.bar;
}

int delegate() makedelegate4b(ref Foo f)
{
  int abc() { return f.a; }
  return &abc;
}

int delegate() makedelegate4()
{
  Foo f;
  return makedelegate4b(f);
}


void main(string[] args)
{
  // On dmd v2.029, linux build, this...

  call(makedelegate1()); // ...works: 27
  call(makedelegate2()); // ...works: 7
  call(makedelegate3()); // ...doesn't work: 134518855
  call(makedelegate4()); // ...doesn't work: 134518947

  Foo f;
  call(&f.bar); // ...works: 7
}

In case 4 the reference is explicit, so it's somehow easier to see that something dangerous is being done, but in case 3, D seems to make it too easy to shoot yourself in the foot.

Is there a resource discussing these issues?

On Thu, Jul 16, 2009 at 8:02 AM, Jens<jens-theisen-tmp01@gmx.de> wrote:
> Hello there,
>
> I'm new to D and experimenting with closures. I know the basics of how compilers translate things, especially in C++. So I appreciate the difficulty of implementing closures correctly and been wondering if it really works. I found a place where dmd 2 appears to fail, but I don't know whether that's a bug or just not supported:
>
> import std.stdio;
>
> struct Foo
> {
>  int a = 7;
>  int bar() { return a; }
>
>  int delegate() makedelegate() {
>    int abc() { return a; }
>    return &abc;
>  }
> }
>
> void call(int delegate() dg)
> {
>  writefln("foo: %d", dg());
> }
>
> int delegate() makedelegate1()
> {
>  int x = 27;
>  int abc() { return x; }
>  return &abc;
> }
>
> int delegate() makedelegate2()
> {
>  Foo f;
>  int abc() { return f.a; }
>  return &abc;
> }
>
> int delegate() makedelegate3()
> {
>  Foo f;
>  return &f.bar;
> }
>
> int delegate() makedelegate4b(ref Foo f)
> {
>  int abc() { return f.a; }
>  return &abc;
> }
>
> int delegate() makedelegate4()
> {
>  Foo f;
>  return makedelegate4b(f);
> }
>
>
> void main(string[] args)
> {
>  // On dmd v2.029, linux build, this...
>
>  call(makedelegate1()); // ...works: 27
>  call(makedelegate2()); // ...works: 7
>  call(makedelegate3()); // ...doesn't work: 134518855
>  call(makedelegate4()); // ...doesn't work: 134518947
>
>  Foo f;
>  call(&f.bar); // ...works: 7
> }
>
> In case 4 the reference is explicit, so it's somehow easier to see that something dangerous is being done, but in case 3, D seems to make it too easy to shoot yourself in the foot.
>
> Is there a resource discussing these issues?

Yeah, you've found it :)

I think what's going on here is that the compiler will *only* allocate closures for nested functions.  However allocating a closure for a delegate of a value object would be a nice addition.

Reply to Jarrett,

> I think what's going on here is that the compiler will *only* allocate
> closures for nested functions. However allocating a closure for a
> delegate of a value object would be a nice addition.
> 

This will quickly devolve into the general escape analysts problem and here there be dragons. I think the correct solution is to say it's unsupported by calling this an escaping reference bug in the user code.

On Thu, Jul 16, 2009 at 1:24 PM, BCS<ao@pathlink.com> wrote:
> Reply to Jarrett,
>
>> I think what's going on here is that the compiler will *only* allocate closures for nested functions. However allocating a closure for a delegate of a value object would be a nice addition.
>>
>
> This will quickly devolve into the general escape analysts problem and here there be dragons. I think the correct solution is to say it's unsupported by calling this an escaping reference bug in the user code.

At least for non-ref-param value types, I don't think it's unreasonable to say that &obj.func should allocate a closure.  I mean, it's the same as saying

int delegate() dg;
dg.funcptr = &typeof(obj).func;
dg.ptr = &obj;

and the last line there would be illegal to return under normal circumstances anyway.

But you're probably right that it might just be easier to disallow it
and force people to write { return f.func(); } instead.  :)

On Thu, 16 Jul 2009 13:34:27 -0400, Jarrett Billingsley <jarrett.billingsley@gmail.com> wrote:

> On Thu, Jul 16, 2009 at 1:24 PM, BCS<ao@pathlink.com> wrote:
>> Reply to Jarrett,
>>
>>> I think what's going on here is that the compiler will *only* allocate
>>> closures for nested functions. However allocating a closure for a
>>> delegate of a value object would be a nice addition.
>>>
>>
>> This will quickly devolve into the general escape analysts problem and here
>> there be dragons. I think the correct solution is to say it's unsupported by
>> calling this an escaping reference bug in the user code.
>
> At least for non-ref-param value types, I don't think it's
> unreasonable to say that &obj.func should allocate a closure.  I mean,
> it's the same as saying
>
> int delegate() dg;
> dg.funcptr = &typeof(obj).func;
> dg.ptr = &obj;
>
> and the last line there would be illegal to return under normal
> circumstances anyway.
>
> But you're probably right that it might just be easier to disallow it
> and force people to write { return f.func(); } instead.  :)

What about syntax to force closure behavior (on or off)?  Not that I have any to suggest, but if the compiler is going to remain ignorant about escape analysis, then we should be able to supply the intelligence, and hacking an extra wrapper function to force behavior seems... well, hackish :)

-Steve

Jens:
>  // On dmd v2.029, linux build, this...
>   call(makedelegate1()); // ...works: 27
>   call(makedelegate2()); // ...works: 7
>   call(makedelegate3()); // ...doesn't work: 134518855
>   call(makedelegate4()); // ...doesn't work: 134518947
>   Foo f;
>   call(&f.bar); // ...works: 7

I have run your code with DMD v2.031 on Windows, and it prints:
foo: 27
foo: 7
foo: 7
foo: 7
foo: 7

That sounds too much good, probably the right values are left in the stack. So I have added some printf() that return an int and show better the problems:

import std.stdio: writefln, printf;

struct Foo {
    int a = 7;
    int bar() { return a; }

    int delegate() makeDelegate() {
        printf("filler");
        int abc() { return a; }
        printf("filler");
        return &abc;
    }
}

void call(int delegate() dg) {
    writefln("foo: %d", dg());
}

int delegate() makeDelegate1() {
    int x = 27;
    int abc() { return x; }
    return &abc;
}

int delegate() makeDelegate2() {
    Foo f;
    int abc() { return f.a; }
    return &abc;
}

int delegate() makeDelegate3() {
    Foo f;
    return &f.bar;
}

int delegate() makeDelegate4b(ref Foo f) {
    int abc() { return f.a; }
    printf("filler");
    return &abc;
}

int delegate() makeDelegate4() {
    Foo f;
    printf("filler");
    return makeDelegate4b(f);
}


void main() {
    auto d1 = makeDelegate1();
    printf("filler");
    call(d1);

    auto d2 = makeDelegate2();
    printf("filler");
    call(d2);

    auto d3 = makeDelegate3();
    printf("filler");
    call(d3);

    auto d4 = makeDelegate4();
    printf("filler");
    call(d4);

    Foo f;
    call(&f.bar);
}

Now the output is:
fillerfoo: 27
fillerfoo: 7
fillerfoo: 4354187
fillerfillerfillerfoo: 7
foo: 7

It shows some difference still compared to your output. So are some things improved between 2.029 and 2.031?

I don't know much about where and when to add "scope" to avoid creation of closures. Maybe Andrei's book will help me understand/learn better :-)

Bye,
bearophile

On Thu, Jul 16, 2009 at 2:22 PM, Steven Schveighoffer<schveiguy@yahoo.com> wrote:
>
> What about syntax to force closure behavior (on or off)?  Not that I have any to suggest, but if the compiler is going to remain ignorant about escape analysis, then we should be able to supply the intelligence, and hacking an extra wrapper function to force behavior seems... well, hackish :)

There already is, at least for turning closures off.  A common design pattern in D1 is to pass the address of a nested function as a callback.  This is convenient and performant:

void putChar(char c)
{
    write(c);
}

std.format.doFormat(&putChar, blahblahblah); // or so

If D2 were to allocate a closure for &putChar, suddenly your formatting function that's called several hundred times a second starts eating memory.  The solution is to put 'scope' on the parameter:

void doFormat(scope void delegate(char) dg, blahblah) { .. }

DMD2 will not allocate a closure for the initial code if 'scope' is present, and if removed, it will.

On Thu, Jul 16, 2009 at 2:33 PM, bearophile<bearophileHUGS@lycos.com> wrote:
> Jens:
>>  // On dmd v2.029, linux build, this...
>>   call(makedelegate1()); // ...works: 27
>>   call(makedelegate2()); // ...works: 7
>>   call(makedelegate3()); // ...doesn't work: 134518855
>>   call(makedelegate4()); // ...doesn't work: 134518947
>>   Foo f;
>>   call(&f.bar); // ...works: 7
>
> I have run your code with DMD v2.031 on Windows, and it prints:
> foo: 27
> foo: 7
> foo: 7
> foo: 7
> foo: 7

Strange.  I use 2.031 on Windows too, and I get strange values for the third and fourth items, as expected.

On Thu, 16 Jul 2009 14:56:28 -0400, Jarrett Billingsley <jarrett.billingsley@gmail.com> wrote:

> On Thu, Jul 16, 2009 at 2:22 PM, Steven
> Schveighoffer<schveiguy@yahoo.com> wrote:
>>
>> What about syntax to force closure behavior (on or off)?  Not that I have
>> any to suggest, but if the compiler is going to remain ignorant about escape
>> analysis, then we should be able to supply the intelligence, and hacking an
>> extra wrapper function to force behavior seems... well, hackish :)
>
> There already is, at least for turning closures off.

Yes I'm aware.   But that has some limitations.

For example, if I truly want my delegate to be a variable, I have to decide which delegate to use at the time I'm calling the function.  I can't store the delegate and then call the function later.

Also, forcing a closure for the reasons discussed in this thread would be useful.

I think the philosophy that D has on say const would be beneficial here:

1. When the compiler can prove the code is correct, allow it.  e.g. implicit cast of a const value type to a mutable value type.
2. When the compiler can't prove, do the safest thing.  e.g. e.g. const could be a pointer to mutable, but compiler isn't sure, so don't allow implicit cast.
3. When the user knows more about the situation than the compiler, allow override.  e.g. explicit cast away const is allowed, or explicit cast to immutable.

The same thing could be applied to closures:

1. Calling a function with a delegate where the arg is scope is allowed.
2. Assigning a delegate to a variable causes a closure.
3. The user can specify that a delegate does not need a closure.

-Steve