On Mon, Oct 08, 2007 at 09:04:27AM -0700, David Brown wrote:

> I've been developing an application on x86_64 (dgcc) without any problems.
> Yesterday, I tried building it on x86 to discover that it has a memory leak
> on that platform.  The leak is rather significant, and the program quickly
> exhausts memory and is killed.
>
> Any suggestions from the list on how to debug/fix this?

Thanks to the posts and suggestions on fixing zlib.  I've tried patching
zlib to use ubyte[] instead of void[].  Unfortunately, it doesn't really
fix things.

I seem to have something else in my program that is making lots of stray
pointers for the GC to follow.  My program consistently leaks memory until
it is killed by running out of address space (or I kill it because it is
trying to swap my machine to death).

I can make it much better by manually 'deleting' some large buffers, but
there shouldn't be any references to them, so it is only something invalid
pointing to them that is keeping them.  That it happens much more slowly on
x86_64 also suggests this, since fewer random pointers would point to a
valid block.

Any ideas on how to find this?  For a C program, I would use valgrind, but
I'm not sure what to do here.

There was some mention of a problem with the flags changing when resizing
an array.  Might this be a possible problem?

My other thought is to allocate something large, with std.c.stdlib.malloc
and put some special checks in the GC to trace anything down that it thinks
is pointing in to my, I guess I'd call it a memory honey pot.  It might be
some work to track these down, but it's a possibility.

Thanks,
David

David Brown wrote:
> On Mon, Oct 08, 2007 at 09:04:27AM -0700, David Brown wrote:
> 
>> I've been developing an application on x86_64 (dgcc) without any problems.
>> Yesterday, I tried building it on x86 to discover that it has a memory leak
>> on that platform.  The leak is rather significant, and the program quickly
>> exhausts memory and is killed.
>>
>> Any suggestions from the list on how to debug/fix this?
> 
> Thanks to the posts and suggestions on fixing zlib.  I've tried patching
> zlib to use ubyte[] instead of void[].  Unfortunately, it doesn't really
> fix things.
> 
> I seem to have something else in my program that is making lots of stray
> pointers for the GC to follow.  My program consistently leaks memory until
> it is killed by running out of address space (or I kill it because it is
> trying to swap my machine to death).
> 
> I can make it much better by manually 'deleting' some large buffers, but
> there shouldn't be any references to them, so it is only something invalid
> pointing to them that is keeping them.  That it happens much more slowly on
> x86_64 also suggests this, since fewer random pointers would point to a
> valid block.
> 
> Any ideas on how to find this?  For a C program, I would use valgrind, but
> I'm not sure what to do here.
> 
> There was some mention of a problem with the flags changing when resizing
> an array.  Might this be a possible problem?
> 
> My other thought is to allocate something large, with std.c.stdlib.malloc
> and put some special checks in the GC to trace anything down that it thinks
> is pointing in to my, I guess I'd call it a memory honey pot.  It might be
> some work to track these down, but it's a possibility.

I'm not sure how to do it either, but we definitely could use some debugging tools to help figure out what memory is hanging around and who the GC thinks is referring to it.

I think I may have a memory leak too, since when I run my program in a loop, things get slower and slower.

--bb

On Mon, Oct 08, 2007 at 10:23:17PM -0700, David Brown wrote:

> I seem to have something else in my program that is making lots of stray
> pointers for the GC to follow.  My program consistently leaks memory until
> it is killed by running out of address space (or I kill it because it is
> trying to swap my machine to death).

I think I found my problem.  I have something like this:

  class Foo {
    ubyte[20] hash;
    ...
  }

and a _lot_ of these are alive at any given time.  The hash is a sha1 hash,
and so tends to be evenly distributed.

It appears that heap objects only have a single flag indicating whether or
not they have pointers, and since classes do have pointers, the GC will
look at the "pointers" in the hash to see if they hit anything.  Build up a
reasonable set of these and something will point to almost everything.

It's going to take me a little while to change the code to not do this (it
isn't quite as simple as my example).  I was concerned about having 20
bytes living in its own gc allocation being wasteful, but leaking nearly
everything is much more wasteful :-)

Dave

David Brown wrote:
> On Mon, Oct 08, 2007 at 10:23:17PM -0700, David Brown wrote:
> 
>> I seem to have something else in my program that is making lots of stray
>> pointers for the GC to follow.  My program consistently leaks memory until
>> it is killed by running out of address space (or I kill it because it is
>> trying to swap my machine to death).
> 
> I think I found my problem.  I have something like this:
> 
>   class Foo {
>     ubyte[20] hash;
>     ...
>   }
> 
> and a _lot_ of these are alive at any given time.  The hash is a sha1 hash,
> and so tends to be evenly distributed.
> 
> It appears that heap objects only have a single flag indicating whether or
> not they have pointers, and since classes do have pointers, the GC will

Yeah, it's just a single bit currently. The vtable and monitor pointers don't count though (the first points to static memory and the second to malloc()ed (non-gc) memory), so class bodies only gets tagged as containing pointers if it has explicit pointer/reference/dyn. array/assoc.array members.

> look at the "pointers" in the hash to see if they hit anything.  Build up a
> reasonable set of these and something will point to almost everything.
> 
> It's going to take me a little while to change the code to not do this (it
> isn't quite as simple as my example).  I was concerned about having 20
> bytes living in its own gc allocation being wasteful, but leaking nearly
> everything is much more wasteful :-)

A precise GC (or at least something more in that direction) would really help in these kinds of situations...

David Brown wrote:
> On Mon, Oct 08, 2007 at 10:23:17PM -0700, David Brown wrote:
> 
>> I seem to have something else in my program that is making lots of stray
>> pointers for the GC to follow.  My program consistently leaks memory until
>> it is killed by running out of address space (or I kill it because it is
>> trying to swap my machine to death).
> 
> I think I found my problem.  I have something like this:
> 
>   class Foo {
>     ubyte[20] hash;
>     ...
>   }
> 
> and a _lot_ of these are alive at any given time.  The hash is a sha1 hash,
> and so tends to be evenly distributed.
> 
> It appears that heap objects only have a single flag indicating whether or
> not they have pointers, and since classes do have pointers, the GC will
> look at the "pointers" in the hash to see if they hit anything.  Build up a
> reasonable set of these and something will point to almost everything.

Hmm.  That could be what's happening to me too.  I think I might have some classes with a mix of pointers and floating point data.  A dynamic array in the class shouldn't cause a problem, right?  Just a static one?

What's the best indicator of how much memory the gc thinks is actually being used?  Right now I'm using
        std.gc.fullCollect();
        std.gc.GCStats stats;
        std.gc.getStats(stats);
        writefln("%s", stats.usedsize);
plus looking at the Windows task manager mem usage for the process.
The .usedsize stat seems a little suspect, though.  A simple test program that allocs big arrays in a loop (storing pointers to those arrays outside the loop) reports usedsize of zero.

--bb

Frits van Bommel wrote:
> David Brown wrote:
>> On Mon, Oct 08, 2007 at 10:23:17PM -0700, David Brown wrote:
>>
>>> I seem to have something else in my program that is making lots of stray
>>> pointers for the GC to follow.  My program consistently leaks memory until
>>> it is killed by running out of address space (or I kill it because it is
>>> trying to swap my machine to death).
>>
>> I think I found my problem.  I have something like this:
>>
>>   class Foo {
>>     ubyte[20] hash;
>>     ...
>>   }
>>
>> and a _lot_ of these are alive at any given time.  The hash is a sha1 hash,
>> and so tends to be evenly distributed.
>>
>> It appears that heap objects only have a single flag indicating whether or
>> not they have pointers, and since classes do have pointers, the GC will
> 
> Yeah, it's just a single bit currently. The vtable and monitor pointers don't count though (the first points to static memory and the second to malloc()ed (non-gc) memory), so class bodies only gets tagged as containing pointers if it has explicit pointer/reference/dyn. array/assoc.array members.
> 
>> look at the "pointers" in the hash to see if they hit anything.  Build up a
>> reasonable set of these and something will point to almost everything.
>>
>> It's going to take me a little while to change the code to not do this (it
>> isn't quite as simple as my example).  I was concerned about having 20
>> bytes living in its own gc allocation being wasteful, but leaking nearly
>> everything is much more wasteful :-)
> 
> A precise GC (or at least something more in that direction) would really help in these kinds of situations...

I think there's another big potential source of leaks:  Associative Arrays.

A node in an AA has two pointers and a hash value, so the gc will treat all those hash values as pointers too.  I think that's probably what's killing me, because I use a lot of AA's.

--bb

Bill Baxter wrote:
> Frits van Bommel wrote:
>> David Brown wrote:
>>> On Mon, Oct 08, 2007 at 10:23:17PM -0700, David Brown wrote:
>>>
>>>> I seem to have something else in my program that is making lots of stray
>>>> pointers for the GC to follow.  My program consistently leaks memory until
>>>> it is killed by running out of address space (or I kill it because it is
>>>> trying to swap my machine to death).
>>>
>>> I think I found my problem.  I have something like this:
>>>
>>>   class Foo {
>>>     ubyte[20] hash;
>>>     ...
>>>   }
>>>
>>> and a _lot_ of these are alive at any given time.  The hash is a sha1 hash,
>>> and so tends to be evenly distributed.
>>>
>>> It appears that heap objects only have a single flag indicating whether or
>>> not they have pointers, and since classes do have pointers, the GC will
>>
>> Yeah, it's just a single bit currently. The vtable and monitor pointers don't count though (the first points to static memory and the second to malloc()ed (non-gc) memory), so class bodies only gets tagged as containing pointers if it has explicit pointer/reference/dyn. array/assoc.array members.
>>
>>> look at the "pointers" in the hash to see if they hit anything.  Build up a
>>> reasonable set of these and something will point to almost everything.
>>>
>>> It's going to take me a little while to change the code to not do this (it
>>> isn't quite as simple as my example).  I was concerned about having 20
>>> bytes living in its own gc allocation being wasteful, but leaking nearly
>>> everything is much more wasteful :-)
>>
>> A precise GC (or at least something more in that direction) would really help in these kinds of situations...
> 
> I think there's another big potential source of leaks:  Associative Arrays.
> 
> A node in an AA has two pointers and a hash value, so the gc will treat all those hash values as pointers too.  I think that's probably what's killing me, because I use a lot of AA's.
> 
> --bb


Here's a simple test case:

module memunion;
import std.stdio;
import std.gc;

// Just an ordinary AA with a lot of values.
class BigAA
{
    int[int] aa;
    this() {
        for(int i=0;i<1000;i++) {
            aa[i] = i;
        }
    }
}

void main()
{
    int nloops = 10_000;
    auto b = new BigAA[100];

    for(int i=0; i<nloops; ++i)
    {
        // Create some AAs (overwriting old ones)
        foreach(ref v; b) { v = new BigAA; }

        // See how we're doing
        std.gc.GCStats stats;
        std.gc.fullCollect();
        std.gc.getStats(stats);
        writefln("Loop %-5s - poolsize=%-10s   %s Mbytes  (%s KB)",
                 i, stats.poolsize,
                 stats.usedsize/1024/1024,
                 stats.usedsize/1024);

    }
}


With phobos, mem usage goes up pretty steadily.

--bb

On Wed, Oct 10, 2007 at 03:56:13AM +0900, Bill Baxter wrote:

> I think there's another big potential source of leaks:  Associative Arrays.
>
> A node in an AA has two pointers and a hash value, so the gc will treat all those hash values as pointers too.  I think that's probably what's killing me, because I use a lot of AA's.

Hmm, the keys of my hash were all stored in an AA.  At least the AA only
has the first 4 bytes of the hash rather than the whole thing.

I had started writing a more memory-efficient data structure than AA, but
found the AA to work fine, at least on x86_64.  Perhaps I need to rethink
that.

One approach I've seen to handling this is to have the compiler split heap
objects, and have in the header how much of the beginning of the block has
pointers.  It wouldn't work for D structs, which must be in the same order,
but classes can be reordered by the compiler.

Dave