Hello everyone,

I was writing some unit tests and I also wanted to test that in certain cases object references are properly removed everywhere so that GC can collect them in order to make sure there is no memory leak. While trying to achieve this I learned that objects are not always collected at the time I would expect them to, so I documented current behavior and some tips.


/**
 * Compiled with:
 *   DMD32 D Compiler v2.060
 *   Copyright (c) 1999-2012 by Digital Mars written by Walter Bright
 *   Documentation: http://www.dlang.org/index.html
 */

module program;

import std.stdio;
import std.string;
import core.memory;

void main () {

  writeln("// ---------- beginning of main");

  mixin(writeAndExec(4, q{

    GC.collect();
    new a("a1");
    // GC will collect a1
    GC.collect();

    auto a2 = new a("a2");
    // GC will collect a2 because a2 will not be used in the future
    GC.collect();

    auto a3 = new a("a3");
    // GC will not collect a3 because a3 will be used in the future
    // tip: make some interaction with the object if you don't want
    // it to get collected before the end of scope GC.collect();
    GC.collect();
    a3.i = 1;

    {
      auto a4 = new a("a4");
      // GC will collect a4 because a4 will not be used in the future
      GC.collect();
    }

    {
      auto a5 = new a("a5");
      a5.i = 1;
      // maybe GC could collect a5 because a5 will be used
      // in the future, but it has been used and for some
      // (probably implementation) reasons GC does not collect it
      GC.collect();
    }

    ({
      auto a6 = new a("a6");
      a6.i = 1;
      // same as previous GC.collect (for a5)
      GC.collect();
    })();

    // maybe GC could collect a5 because a5 has gone out of scope
    // but for some (probably implementation) reasons GC does not
    // collect it, however, it will collect a6 that was declared
    // and used in a function call
    // tip: use function call (like a6 case) for a scope if you
    // want GC.collect(); to collect objects declared inside it
    GC.collect();

  }));

  writeln("// ---------- end of main");

}

class a {

  string id;
  int i;

  this (string id) {
    this.id = id;
    writeln("// ! ", this.id, " is now constructed");
  }

  ~this () {
    writeln("// ! ", this.id, " is now destructed");
  }

}

string writeAndExec (uint ident, string code) {
  string result = "";
  foreach (line; code.splitLines())
    result ~= "writeln(`" ~ line[line.length > ident ? ident : 0 .. $] ~ "`);\n" ~ line ~ "\n";
  return result;
}

On 11/06/2012 03:27 AM, luka8088 wrote:

> I was writing some unit tests and I also wanted to test that in certain
> cases object references are properly removed everywhere so that GC can
> collect them in order to make sure there is no memory leak. While trying
> to achieve this I learned that objects are not always collected at the
> time I would expect them to, so I documented current behavior and some
> tips.

Thanks for the analysis and the tips but they all depend on observations made by a particular test program on a particular version of a particular compiler. It is conceivable that even the same GC algorithm can behave differently in another program.

In fact, some objects may never be collected even if there are no more references to them.

In comparison, destroy() and scoped() do call the destructors at precise times.

Ali

On Tuesday, 6 November 2012 at 11:27:25 UTC, luka8088 wrote:
> Hello everyone,
>
> I was writing some unit tests and I also wanted to test that in certain cases object references are properly removed everywhere so that GC can collect them in order to make sure there is no memory leak. While trying to achieve this I learned that objects are not always collected at the time I would expect them to, so I documented current behavior and some tips.

Those are some sample cases which all fit into the general rule: on collection the stack is scanned and everything that looks like a pointer to live heap is used to mark heap objects as live, then the heap objects are scanned and marked transitively. Unmarked objects get collected. In each case described in the original post the decision whether some value will be collected or not depends on whether the local variable is still on the stack and not overwritten by some newer stack variable. As a consequence:
If it's going to be used later, it's still live on stack.
If it's not going to be used later, it all depends on whether some other variable defined later is placed by compiler in the same stack slot, and whether that new variable was initialized already.
If it was used in a function called and returned, it's not in the active stack, so it should be collected. And so on.

On 6.11.2012 18:00, Ali Çehreli wrote:
> On 11/06/2012 03:27 AM, luka8088 wrote:
>
>  > I was writing some unit tests and I also wanted to test that in certain
>  > cases object references are properly removed everywhere so that GC can
>  > collect them in order to make sure there is no memory leak. While trying
>  > to achieve this I learned that objects are not always collected at the
>  > time I would expect them to, so I documented current behavior and some
>  > tips.
>
> Thanks for the analysis and the tips but they all depend on observations
> made by a particular test program on a particular version of a
> particular compiler. It is conceivable that even the same GC algorithm
> can behave differently in another program.

Yes, but it seems that we can in general say that the following code will never fail... or am I wrong ?

import core.memory;

class a {
  static int totalRefCount = 0;
  this () { totalRefCount++; }
  ~this () { totalRefCount--; }
}

void main () {
  assert(a.totalRefCount == 0);
  ({
    auto a1 = new a();
    assert(a.totalRefCount == 1);
  })();
  GC.collect();
  assert(a.totalRefCount == 0);
}

>
> In fact, some objects may never be collected even if there are no more
> references to them.

Can you give me an example ?

>
> In comparison, destroy() and scoped() do call the destructors at precise
> times.

Yes, but the main reason for doing this is testing for memory leaks so this cases are not much of a concern =)

>
> Ali
>

Luka

On 6.11.2012 18:02, thedeemon wrote:
> On Tuesday, 6 November 2012 at 11:27:25 UTC, luka8088 wrote:
>> Hello everyone,
>>
>> I was writing some unit tests and I also wanted to test that in
>> certain cases object references are properly removed everywhere so
>> that GC can collect them in order to make sure there is no memory
>> leak. While trying to achieve this I learned that objects are not
>> always collected at the time I would expect them to, so I documented
>> current behavior and some tips.
>
> Those are some sample cases which all fit into the general rule: on
> collection the stack is scanned and everything that looks like a pointer
> to live heap is used to mark heap objects as live, then the heap objects
> are scanned and marked transitively. Unmarked objects get collected. In
> each case described in the original post the decision whether some value
> will be collected or not depends on whether the local variable is still
> on the stack and not overwritten by some newer stack variable. As a
> consequence:
> If it's going to be used later, it's still live on stack.
> If it's not going to be used later, it all depends on whether some other
> variable defined later is placed by compiler in the same stack slot, and
> whether that new variable was initialized already.
> If it was used in a function called and returned, it's not in the active
> stack, so it should be collected. And so on.

Thx for the more detailed implementation explanation =)

On 11/06/2012 12:00 PM, luka8088 wrote:

> Yes, but it seems that we can in general say that the following code
> will never fail... or am I wrong ?
>
> import core.memory;
>
> class a {
> static int totalRefCount = 0;
> this () { totalRefCount++; }
> ~this () { totalRefCount--; }
> }
>
> void main () {
> assert(a.totalRefCount == 0);
> ({
> auto a1 = new a();
> assert(a.totalRefCount == 1);
> })();
> GC.collect();

The current definition of GC.collect() explicitly says that its definition may change:

  http://dlang.org/phobos/core_memory.html#collect

"the meaning of this may change based on the garbage collector implementation".

> assert(a.totalRefCount == 0);
> }
>
>>
>> In fact, some objects may never be collected even if there are no more
>> references to them.
>
> Can you give me an example ?

I can't come up with an example but the spec is clear on that issue:

  http://dlang.org/class.html#destructors

"The garbage collector is not guaranteed to run the destructor for all unreferenced objects. Furthermore, the order in which the garbage collector calls destructors for unreference objects is not specified. This means that when the garbage collector calls a destructor for an object of a class that has members that are references to garbage collected objects, those references may no longer be valid. This means that destructors cannot reference sub objects."

Ali

On 6.11.2012 21:59, Ali Çehreli wrote:
> On 11/06/2012 12:00 PM, luka8088 wrote:
>
>  > Yes, but it seems that we can in general say that the following code
>  > will never fail... or am I wrong ?
>  >
>  > import core.memory;
>  >
>  > class a {
>  > static int totalRefCount = 0;
>  > this () { totalRefCount++; }
>  > ~this () { totalRefCount--; }
>  > }
>  >
>  > void main () {
>  > assert(a.totalRefCount == 0);
>  > ({
>  > auto a1 = new a();
>  > assert(a.totalRefCount == 1);
>  > })();
>  > GC.collect();
>
> The current definition of GC.collect() explicitly says that its
> definition may change:
>
> http://dlang.org/phobos/core_memory.html#collect
>
> "the meaning of this may change based on the garbage collector
> implementation".
>
>  > assert(a.totalRefCount == 0);
>  > }
>  >
>  >>
>  >> In fact, some objects may never be collected even if there are no more
>  >> references to them.
>  >
>  > Can you give me an example ?
>
> I can't come up with an example but the spec is clear on that issue:
>
> http://dlang.org/class.html#destructors
>
> "The garbage collector is not guaranteed to run the destructor for all
> unreferenced objects. Furthermore, the order in which the garbage
> collector calls destructors for unreference objects is not specified.
> This means that when the garbage collector calls a destructor for an
> object of a class that has members that are references to garbage
> collected objects, those references may no longer be valid. This means
> that destructors cannot reference sub objects."
>
> Ali
>

I see, so is there some alternative for memory leak tests ?

Luka

On Nov 6, 2012, at 12:59 PM, Ali Çehreli <acehreli@yahoo.com> wrote:
> 
> I can't come up with an example but the spec is clear on that issue:
> 
>  http://dlang.org/class.html#destructors
> 
> "The garbage collector is not guaranteed to run the destructor for all unreferenced objects. Furthermore, the order in which the garbage collector calls destructors for unreference objects is not specified. This means that when the garbage collector calls a destructor for an object of a class that has members that are references to garbage collected objects, those references may no longer be valid. This means that destructors cannot reference sub objects."

This clause is mostly intended to say that the GC may not detect that every unreferenced object is truly unreferenced.  It's also possible, however, that the GC could delay finalizing collected objects until some later time.