The following code does not invoke S.~this. If I change `struct S` to `class S` - it does. Memory consumption remains constant, meaning memory is collected, but destructors are not called.

import std.stdio;
import std.conv;

struct S {
    string s;

    ~this() {
        writeln("~S");
    }
}

void main() {
    auto i = 0;
    S[] arr;

    while (true) {
        arr ~= S("hello " ~ text(i++));
        if (arr.length == 1_000_000) {
            writeln(&arr[8888], " = ", arr[8888].s);
            arr.length = 0;
        }
    }
}


Is it a bug? How can I effectively implement RAII with this behavior?

The situation is, I allocate resources for my users and return them a handle. I can't control what my users do with this handle, they might as well append it to a dynamic array/insert to AA, and remove it eventually.
Also, the handle might be shared between several logical components, so it's not that one of them can explicitly finalize it.

Any workaround for this? Perhaps disallow certain types be allocated by the GC (I would actually want this feature very much)?


-tomer

On Mon, 10 Nov 2014 08:10:08 +0000
Tomer Filiba via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> The following code does not invoke S.~this. If I change `struct S` to `class S` - it does. Memory consumption remains constant, meaning memory is collected, but destructors are not called.
> 
> import std.stdio;
> import std.conv;
> 
> struct S {
>      string s;
> 
>      ~this() {
>          writeln("~S");
>      }
> }
> 
> void main() {
>      auto i = 0;
>      S[] arr;
> 
>      while (true) {
>          arr ~= S("hello " ~ text(i++));
>          if (arr.length == 1_000_000) {
>              writeln(&arr[8888], " = ", arr[8888].s);
>              arr.length = 0;
>          }
>      }
> }
> 
> 
> Is it a bug? How can I effectively implement RAII with this behavior?
it's a bug, it was recently filled and Walter (afair) made a PR with
fix, but it's not yet merged.

> Any workaround for this?
made your own array implementation which manually calls dtors. or wait for next DMD release with this bug fixed. ;-)

Thanks for the info, ketmar.

By the way, what do you guys think of adding @nogc to structs, in which case they cannot be instantiated by the GC?

e.g.,

@nogc struct S {...}
auto s = new S(); // does not compile
S s;              // all is well


-tomer

On Monday, 10 November 2014 at 08:28:03 UTC, ketmar via Digitalmars-d wrote:
> On Mon, 10 Nov 2014 08:10:08 +0000
> Tomer Filiba via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
>> The following code does not invoke S.~this. If I change `struct S` to `class S` - it does. Memory consumption remains constant, meaning memory is collected, but destructors are not called.
>> 
>> import std.stdio;
>> import std.conv;
>> 
>> struct S {
>>      string s;
>> 
>>      ~this() {
>>          writeln("~S");
>>      }
>> }
>> 
>> void main() {
>>      auto i = 0;
>>      S[] arr;
>> 
>>      while (true) {
>>          arr ~= S("hello " ~ text(i++));
>>          if (arr.length == 1_000_000) {
>>              writeln(&arr[8888], " = ", arr[8888].s);
>>              arr.length = 0;
>>          }
>>      }
>> }
>> 
>> 
>> Is it a bug? How can I effectively implement RAII with this behavior?
> it's a bug, it was recently filled and Walter (afair) made a PR with
> fix, but it's not yet merged.
>
>> Any workaround for this?
> made your own array implementation which manually calls dtors. or wait
> for next DMD release with this bug fixed. ;-)

On 11/10/14 3:10 AM, Tomer Filiba wrote:
> The following code does not invoke S.~this. If I change `struct S` to
> `class S` - it does. Memory consumption remains constant, meaning memory
> is collected, but destructors are not called.
>
> import std.stdio;
> import std.conv;
>
> struct S {
>      string s;
>
>      ~this() {
>          writeln("~S");
>      }
> }
>
> void main() {
>      auto i = 0;
>      S[] arr;
>
>      while (true) {
>          arr ~= S("hello " ~ text(i++));
>          if (arr.length == 1_000_000) {
>              writeln(&arr[8888], " = ", arr[8888].s);
>              arr.length = 0;
>          }
>      }
> }
>
>
> Is it a bug? How can I effectively implement RAII with this behavior?

Only classes call dtors from the GC. Structs do not. There are many hairy issues with structs calling dtors from GC. Most struct dtors expect to be called synchronously, and are not expecting to deal with multithreading issues.

Note that structs inside classes WILL call dtors.

For example, imagine you have a struct that increments/decrements a reference count:

struct refCounter(T)
{
   T t; // assume this to be a reference type
   this(T t) { t.inc();}
   ~this() {if(t !is null) t.dec();}
}

Of course, you would need more machinery to deal with copies, but let's just assume that is also implemented.

You can see how this would be some kind of useful guard using RAII on the stack.

Now, imagine you wanted to put this on the GC heap, and the GC would call struct dtors. And let's say the program is multi-threaded. First, the memory referred to by t isn't guaranteed to be alive, it could have already been finalized and freed. Second, the thread that calls the destructor may not be the thread that owns t. This means that two threads could potentially be calling t.inc() or t.dec() at the same time, leading to race conditions.

Note, even D's std.stdio.File is not able to deal with this properly.

> The situation is, I allocate resources for my users and return them a
> handle. I can't control what my users do with this handle, they might as
> well append it to a dynamic array/insert to AA, and remove it eventually.
> Also, the handle might be shared between several logical components, so
> it's not that one of them can explicitly finalize it.

Is the resource a GC resource? If so, don't worry about it. If not, use a class to wrap the resource. At this point, that is all that D supports.

> Any workaround for this? Perhaps disallow certain types be allocated by
> the GC (I would actually want this feature very much)?

That would be a nice feature I think.

-Steve

Am Mon, 10 Nov 2014 09:13:09 +0000
schrieb "Tomer Filiba" <tomerfiliba@gmail.com>:

> Thanks for the info, ketmar.
> 
> By the way, what do you guys think of adding @nogc to structs, in which case they cannot be instantiated by the GC?
> 
> e.g.,
> 
> @nogc struct S {...}
> auto s = new S(); // does not compile
> S s;              // all is well
> 
> 
> -tomer

That collides with the use of @nogc on S to mean that this struct itself wont use the GC, but may be used in a GC context, like as part of a class allocated on the GC heap. Well, in general it means "does not call GC functions".

-- 
Marco

On 10.11.2014 15:19, Steven Schveighoffer wrote:
>
> Now, imagine you wanted to put this on the GC heap, and the GC would
> call struct dtors. And let's say the program is multi-threaded. First,
> the memory referred to by t isn't guaranteed to be alive, it could have
> already been finalized and freed. Second, the thread that calls the
> destructor may not be the thread that owns t. This means that two
> threads could potentially be calling t.inc() or t.dec() at the same
> time, leading to race conditions.

So, would you object to actually call the destructor for GC collected structs? I don't think that threading problems in the implmentation of the destructor should prohibit this.

The reference count in your example also doesn't work for heap allocated structs that are never collected or for structs inside classes.

The pull request is almost ready to be merged, please chime in: https://github.com/D-Programming-Language/druntime/pull/864

On 11/11/14 2:46 PM, Rainer Schuetze wrote:
>
>
> On 10.11.2014 15:19, Steven Schveighoffer wrote:
>>
>> Now, imagine you wanted to put this on the GC heap, and the GC would
>> call struct dtors. And let's say the program is multi-threaded. First,
>> the memory referred to by t isn't guaranteed to be alive, it could have
>> already been finalized and freed. Second, the thread that calls the
>> destructor may not be the thread that owns t. This means that two
>> threads could potentially be calling t.inc() or t.dec() at the same
>> time, leading to race conditions.
>
> So, would you object to actually call the destructor for GC collected
> structs? I don't think that threading problems in the implmentation of
> the destructor should prohibit this.
>
> The reference count in your example also doesn't work for heap allocated
> structs that are never collected or for structs inside classes.
>

I think in general, it's a problem of the expectation of where structs will live. It's obvious we know classes will be on the heap, so we can write those dtors accordingly. Most struct dtors are written for when the struct is on the stack.

The way around this is to have 2 functions for destruction -- as Tango does. One is called during synchronous destruction (i.e. when a struct goes out of scope, or when destroy is called), and the other is called during both synchronous and asynchronous destruction (when the GC is collecting).

But even that solution does not allow the struct that I wrote to properly deal with the GC. If the struct has a reference to GC memory, it CANNOT access it during GC destruction to decrement the count, as the memory may be gone.

It is why all the claims that we can accomplish what we want with reference counting backed by the GC all never seem to satisfy my doubts.

> The pull request is almost ready to be merged, please chime in:
> https://github.com/D-Programming-Language/druntime/pull/864

At this point, I am not super-concerned about this. I cannot think of any bullet-proof way to ensure that struct dtors for structs that were meant only for stack variables can be called correctly from the GC. This pull doesn't change that, and it does have some nice new features that we do need for other reasons.

In other words, putting a struct in the GC heap that was written to be scope-destroyed is an error before and after this pull. Before the pull, the dtor doesn't run, which is wrong, and after the pull the dtor may cause race issues, which is wrong. So either way, it's wrong :)

I also am strapped for free cycles to review such PRs. I trust you guys know what you are doing :)

-Steve

On 10/11/14 16:19, Steven Schveighoffer wrote:
>
> Only classes call dtors from the GC. Structs do not. There are many
> hairy issues with structs calling dtors from GC. Most struct dtors
> expect to be called synchronously, and are not expecting to deal with
> multithreading issues.
>
> Note that structs inside classes WILL call dtors.

How is this any different? If one should not be allowed, how is the other okay?

Shachar

On 11/11/14 22:41, Steven Schveighoffer wrote:
>
> At this point, I am not super-concerned about this. I cannot think of
> any bullet-proof way to ensure that struct dtors for structs that were
> meant only for stack variables can be called correctly from the GC.
Isn't "structs meant only for stack variables" a semantic thing? The D compiler cannot possibly know. Shouldn't that be the programmer's choice?

> This
> pull doesn't change that, and it does have some nice new features that
> we do need for other reasons.
>
> In other words, putting a struct in the GC heap that was written to be
> scope-destroyed is an error before and after this pull. Before the pull,
> the dtor doesn't run, which is wrong, and after the pull the dtor may
> cause race issues, which is wrong. So either way, it's wrong :)
I disagree.

The first is wrong. The second is a corner case the programmer needs to be aware of, and account for. The difference is that, in the first case, the programmer is left with no tools to fix the problem, while in the second case this is simply a bug in the program (which, like I said in another email, also happens with the current implementation when the struct is inside a class).

In other words, the second case exposes a second (more direct and more likely to be handled) path to an already existing problem, while the first puts the programmer up against a new problem with no work around.

Shachar

On Wednesday, 12 November 2014 at 04:59:33 UTC, Shachar Shemesh wrote:
> On 10/11/14 16:19, Steven Schveighoffer wrote:
>>
>> Only classes call dtors from the GC. Structs do not. There are many
>> hairy issues with structs calling dtors from GC. Most struct dtors
>> expect to be called synchronously, and are not expecting to deal with
>> multithreading issues.
>>
>> Note that structs inside classes WILL call dtors.
>
> How is this any different? If one should not be allowed, how is the other okay?

Supposedly, a struct destructor will only access resources that the struct itself manages. As long as that's the case, it will be safe. In practice, there's still a lot that can go wrong.