I haven't dealt directly with the garbage collector much before, but just used delete and scope to deal with deallocation.  And left some of the minor allocations to be freed by the GC.

Now I want to work more closely with the GC.  Can anyone explain to me why fullCollect doesn't seem to do anything here?  This example just keeps allocating memory, until the GC seems to kick in to stop it from growing.  It stabilizes at 426MB.  The same example using Tango instead stops at 597MB.

---
import std.gc;

void main()
{
    int[] a;

    for (;;) {
        a = new int[1024 * 1024 * 10];
        a = null;
        assert(a.ptr is null);
        fullCollect();
    }
}
---

Removing the call to fullCollect() doesn't change the behavior.  I expected fullCollect to collect as much memory as it can, and then reuse it for the next allocation.  Am I on the wrong track here?

torhu wrote:
> I haven't dealt directly with the garbage collector much before, but just used delete and scope to deal with deallocation.  And left some of the minor allocations to be freed by the GC.
> 
> Now I want to work more closely with the GC.  Can anyone explain to me why fullCollect doesn't seem to do anything here?  This example just keeps allocating memory, until the GC seems to kick in to stop it from growing.  It stabilizes at 426MB.  The same example using Tango instead stops at 597MB.

When I run the test on Tango, memory juse jumps to 467k and stays steady there.  Doing some quick math with a calculator shows an array 1024^2*10 integers should occupy ~409k of memory, so the rest is just program overhead.  For what it's worth, running this app on Tango without the call to gc.collect() brings memory use up to 594k.  I'd have to debug the GC to figure out why.

> Removing the call to fullCollect() doesn't change the behavior.  I expected fullCollect to collect as much memory as it can, and then reuse it for the next allocation.  Am I on the wrong track here?

Nope.  That's what it should do.  But be aware that the current GC allocates memory in pages and those pages are devoted to a specific size of data.  And once so assigned, I don't think the GC ever attempts to reclaim empty pages if another size allocation is required.  It doesn't matter in this case because anything over a page (4k) is the same "size" as far as the GC is concerned, but it may matter in an app that allocates a ton of small objects and discards them, then allocates a ton of larger objects and discards them, etc.


Sean

Sean Kelly wrote:
> When I run the test on Tango, memory juse jumps to 467k and stays steady there.  Doing some quick math with a calculator shows an array 1024^2*10 integers should occupy ~409k of memory, so the rest is just program overhead.  For what it's worth, running this app on Tango without the call to gc.collect() brings memory use up to 594k.  I'd have to debug the GC to figure out why.

Uh... but 1024^2*10 ints is 40 MB exactly.  Are you talking about a different test program?

> 
>> Removing the call to fullCollect() doesn't change the behavior.  I expected fullCollect to collect as much memory as it can, and then reuse it for the next allocation.  Am I on the wrong track here?
> 
> Nope.  That's what it should do.  But be aware that the current GC allocates memory in pages and those pages are devoted to a specific size of data.  And once so assigned, I don't think the GC ever attempts to reclaim empty pages if another size allocation is required.  It doesn't matter in this case because anything over a page (4k) is the same "size" as far as the GC is concerned, but it may matter in an app that allocates a ton of small objects and discards them, then allocates a ton of larger objects and discards them, etc.


Are you on Windows or Linux?  I'm on windows, and I'm not seeing the behavior you describe.  The test I posted allocates the same amount of memory over and over again.  But the GC doesn't seem to start reusing it until somewhere around the eleventh time through the loop.  Until then it just grows.

Just for the record, the number I'm looking at is 'private bytes' in process explorer, which is the same as 'VM size' in task manager.  Which shows allocated heap memory, and probably some other stuff.  And I use dmd 1.007.

torhu wrote:
> I haven't dealt directly with the garbage collector much before, but just used delete and scope to deal with deallocation.  And left some of the minor allocations to be freed by the GC.
> 
> Now I want to work more closely with the GC.  Can anyone explain to me why fullCollect doesn't seem to do anything here?  This example just keeps allocating memory, until the GC seems to kick in to stop it from growing.  It stabilizes at 426MB.  The same example using Tango instead stops at 597MB.
> 
> ---
> import std.gc;
> 
> void main()
> {
>     int[] a;
> 
>     for (;;) {
>         a = new int[1024 * 1024 * 10];
>         a = null;
>         assert(a.ptr is null);
>         fullCollect();
>     }
> }
> ---
> 
> Removing the call to fullCollect() doesn't change the behavior.  I expected fullCollect to collect as much memory as it can, and then reuse it for the next allocation.  Am I on the wrong track here?

Not being exactly an expert on garbage collection myself, I did encounter the same behaviour which could be changed with a call to std.gc.minimize. This led me to think fullCollect does in fact not always clean up all unused memory, but just forces a GC scan, which may decide there is no need to collect if enough memory is available. Can anybody confirm this?

Lutger wrote:
> torhu wrote:
>> Removing the call to fullCollect() doesn't change the behavior.  I expected fullCollect to collect as much memory as it can, and then reuse it for the next allocation.  Am I on the wrong track here?
> 
> Not being exactly an expert on garbage collection myself, I did encounter the same behaviour which could be changed with a call to std.gc.minimize. This led me to think fullCollect does in fact not always clean up all unused memory, but just forces a GC scan, which may decide there is no need to collect if enough memory is available. Can anybody confirm this?

I tried minimize, but now the memory usage goes even higher, up to 469 MB instead of 426.  So none of these functions seem to do what the docs say.  Same goes for tango.core.Memory.gc.collect().

torhu wrote:
> Sean Kelly wrote:
>> When I run the test on Tango, memory juse jumps to 467k and stays steady there.  Doing some quick math with a calculator shows an array 1024^2*10 integers should occupy ~409k of memory, so the rest is just program overhead.  For what it's worth, running this app on Tango without the call to gc.collect() brings memory use up to 594k.  I'd have to debug the GC to figure out why.
> 
> Uh... but 1024^2*10 ints is 40 MB exactly.  Are you talking about a different test program?

No, you're right.  I misread the output to contain one more zero than it does.

>>> Removing the call to fullCollect() doesn't change the behavior.  I expected fullCollect to collect as much memory as it can, and then reuse it for the next allocation.  Am I on the wrong track here?
>>
>> Nope.  That's what it should do.  But be aware that the current GC allocates memory in pages and those pages are devoted to a specific size of data.  And once so assigned, I don't think the GC ever attempts to reclaim empty pages if another size allocation is required.  It doesn't matter in this case because anything over a page (4k) is the same "size" as far as the GC is concerned, but it may matter in an app that allocates a ton of small objects and discards them, then allocates a ton of larger objects and discards them, etc.
> 
> 
> Are you on Windows or Linux?  I'm on windows, and I'm not seeing the behavior you describe.  The test I posted allocates the same amount of memory over and over again.  But the GC doesn't seem to start reusing it until somewhere around the eleventh time through the loop.  Until then it just grows.

Windows.  The app jumps to 467k almost immediately and stays there.  I haven't tried stepping through the loop, so it may just be happening too fast for me to see.

> Just for the record, the number I'm looking at is 'private bytes' in process explorer, which is the same as 'VM size' in task manager.  Which shows allocated heap memory, and probably some other stuff.  And I use dmd 1.007.

Thanks, I'll check that number to be sure.


Sean

Sean Kelly wrote:
> Windows.  The app jumps to 467k almost immediately and stays there.  I haven't tried stepping through the loop, so it may just be happening too fast for me to see.

I assume you mean 467MB?  Anyway, the point is not how fast it happens.  The problem is that it happens at all.  I expected to be able to use the garbage collector to keep the test app's memory at ~40MB.  If std.gc.minimize doesn't do that, then how is it different from std.gc.fullCollect?

By the way, the limit for how much memory size grows when running the test app depends on the size of the array.  So the GC seems to set a limit based on the app's typical allocation sizes.  Which makes sense and probably keeps allocation speeds up.

It might be that I should start viewing the GC more as a leak preventor than a simpler way of keeping memory usage down.  Simpler than explicit deallocation, that is.  The GC might decide to let memory usage grow to what the programmer or the user consider unreasonable sizes.  So basically, you should always use delete or scope to free memory, since when the GC does it, it will be a bit late.

Unfortunately this is unlikely to help but..

> This example just keeps allocating memory,  until the GC seems to kick
> in to stop it from growing. It stabilizes at 426MB.
Using plain phobos, my testing stabilises at ~51MB. When minimize was used instead of fullCollect the memory usage stabilised at ~93MB

This is with 512MB of total RAM

 - Paul

Paul Findlay wrote:
> Unfortunately this is unlikely to help but..

No, it's very interesting to hear what results other people get.

> Using plain phobos, my testing stabilises at ~51MB. When minimize was used instead of fullCollect the memory usage stabilised at ~93MB
> 
> This is with 512MB of total RAM

Ok, I have 896 MB.  I have a feeling that doesn't affect the GC, but I can't be sure.

I use dmd 1.007 on winxp sp2.  What are you using?

On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:

> I haven't dealt directly with the garbage collector much before, but just used delete and scope to deal with deallocation.  And left some of the minor allocations to be freed by the GC.
> 
> Now I want to work more closely with the GC.  Can anyone explain to me why fullCollect doesn't seem to do anything here?

There is quite a large portion of the fullCollect source which is commented out in phobos, mainly in the section which reclaims RAM (I think).

-- 
Derek Parnell
Melbourne, Australia
"Justice for David Hicks!"
skype: derek.j.parnell