Hi,

This code seems to leak memory, as the memory isn't reclaimed:

//Test memory here: low
{
	auto b = Array!(bool)();
	b.length = 1024 * 1024 * 128 * 8;
	//Test memory here: high
}
//Test memory here: high

Am I missing something about how Array(T) (and RefCounted) works, or is this
really a bug?

Thank you!

On 08/01/2011 05:56, %u wrote:
> {
> 	auto b = Array!(bool)();
> 	b.length = 1024 * 1024 * 128 * 8;
> 	//Test memory here: high
> }

What method are you using to test the memory?  I'm puzzled that you've put a comment there rather than the code you're actually using.

If you run this code twice, does the memory usage double?

Stewart.

> What method are you using to test the memory?
> I'm puzzled that you've put a comment there rather than the code you're actually
using.

I'm not using code, I'm checking the working set of my process in Task Manager, and through every iteration, it adds 128 MB.


> If you run this code twice, does the memory usage double?

Yes. I ran this code:
{
	auto b = Array!(bool)();
	b.length = 1024 * 1024 * 128 * 8;
}
{
	auto b = Array!(bool)();
	b.length = 1024 * 1024 * 128 * 8;
}
and Task Manager showed two increases of 128-MB.


Thank you!

Sorry to bump this up, but is RefCounted(T) really leaking, or am I missing something? I would like to use this in my program, and I'm curious as to why no one responded, since if it's actually leaking, it would be an important issue.

Thanks!

On Wednesday, January 12, 2011 15:29:51 %u wrote:
> Sorry to bump this up, but is RefCounted(T) really leaking, or am I missing something? I would like to use this in my program, and I'm curious as to why no one responded, since if it's actually leaking, it would be an important issue.

There probably aren't all that many people who saw your post, and out of those who did, there are probably very few - if any - who have actually done much with RefCounted. It's fairly new.

There's at least one major bug on Array at the moment ( http://d.puremagic.com/issues/show_bug.cgi?id=4942 ). There are also several bugs having to do with destructors at the moment which could be causing you problems.

Now, even assuming that you're not seeing any problem with a destructor-related bug and that you're not hitting a known bug with Array, there are three things that you need to be aware of which would likely show high memory usage regardless:

1. Array uses an array internally, and there is some caching that goes on with regards to arrays that has to do with appending. This means that if you're dealing with large arrays, you could have several which haven't been garbage collected yet simply because they're cached. Steven Schveighoffer has talked about it in several posts, and he has done some work to improve the situation, but I'm not sure that any of it has been in a release yet.

2. The garbage collector does not currently run in its own thread. IIUC, it only gets run when you try and allocate memory. So, if you allocate a bunch of memory, and then you never try and allocate memory again, no memory will be collected, regardless of whether it's currently being used or not.

3. As I understand it, the current garbage collector _never_ gives memory back to the OS. It will reclaim memory that you're not referencing any longer so that it doesn't necessarily need to go grab more memory from the OS when you try and allocate something, but once the garbage collector has gotten a block of memory from the OS, it doesn't give it back. So, currently you will _never_ see the memory usage of a D program go down, unless you're explicitly using malloc and free instead of GC-allocated memory.

So, if you really want to be testing for leaks, you probably should testing for short-lived small arrays in a loop with lots of iterations or something similar. Testing a couple of large arrays will almost certainly mean that the memory usage will be high and that it won't drop. On the other hand, if you keep creating small arrays in a loop where they have no references outside the loop and could be collected after they go out of scope, then you have lots of arrays for the garbage collector to collect, and if it fails to properly collect them, _then_ you'll see the memory usage continue to rise and show that there's a leak. But at this point, using a few large arrays is almost certainly going to look like it's leaking.

I'm sure that at some point D's garbage collector will improve so that these issues don't exist or at least or definitely diminished, but fixing the GC is not exactly at the top of the TODO list, so it's not currently as performant as would be nice. Once more important stuff has been fixed though, it'll get its turn.

- Jonathan M Davis

On Wed, 12 Jan 2011 19:29:30 -0500, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Wednesday, January 12, 2011 15:29:51 %u wrote:
>> Sorry to bump this up, but is RefCounted(T) really leaking, or am I missing
>> something? I would like to use this in my program, and I'm curious as to
>> why no one responded, since if it's actually leaking, it would be an
>> important issue.
>
> There probably aren't all that many people who saw your post, and out of those
> who did, there are probably very few - if any - who have actually done much with
> RefCounted. It's fairly new.
>
> There's at least one major bug on Array at the moment (
> http://d.puremagic.com/issues/show_bug.cgi?id=4942 ). There are also several
> bugs having to do with destructors at the moment which could be causing you
> problems.
>
> Now, even assuming that you're not seeing any problem with a destructor-related
> bug and that you're not hitting a known bug with Array, there are three things
> that you need to be aware of which would likely show high memory usage
> regardless:
>
> 1. Array uses an array internally, and there is some caching that goes on with
> regards to arrays that has to do with appending. This means that if you're
> dealing with large arrays, you could have several which haven't been garbage
> collected yet simply because they're cached. Steven Schveighoffer has talked
> about it in several posts, and he has done some work to improve the situation,
> but I'm not sure that any of it has been in a release yet.

No, there is no release yet, but the code is checked into svn.  But Array doesn't use D appending anyways.

>
> 2. The garbage collector does not currently run in its own thread. IIUC, it only
> gets run when you try and allocate memory. So, if you allocate a bunch of
> memory, and then you never try and allocate memory again, no memory will be
> collected, regardless of whether it's currently being used or not.

>
> 3. As I understand it, the current garbage collector _never_ gives memory back
> to the OS. It will reclaim memory that you're not referencing any longer so that
> it doesn't necessarily need to go grab more memory from the OS when you try and
> allocate something, but once the garbage collector has gotten a block of memory
> from the OS, it doesn't give it back. So, currently you will _never_ see the
> memory usage of a D program go down, unless you're explicitly using malloc and
> free instead of GC-allocated memory.

Um... Array acutally uses malloc and free to allocate its data.

But even so, malloc and free have the same property where they don't always give back memory to the OS.  IIUC, Linux can only change the size of memory it wants, it cannot free pages in the middle of the block.

-Steve

Steven Schveighoffer Wrote:

> But even so, malloc and free have the same property where they don't always give back memory to the OS.  IIUC, Linux can only change the size of memory it wants, it cannot free pages in the middle of the block.
> 
> -Steve

Disclaimer: I don't know what I am talking about.

I think this is correct, the program isn't responsible for reclaiming the memory, that is what the OS does. If you don't have an OS then you don't have anything to return the memory to, so it just becomes free memory. Modern operating systems aren't going to take their memory back until it is needed (wast of cycles).

What I observed using Linux and $ free; each section would result in a reduction of free memory and an increase in buffered data. This suggests to me that the OS doesn't want the memory yet.

Tracking memory in a modern OS is not easy, and this is probably why no one wanted to make a statement on what was really happening. As I said I don't know if this is what is happening, but it usually isn't as straight forward as checking memory usage.

On Thu, 13 Jan 2011 12:40:04 -0500, Jesse Phillips <jessekphillips+D@gmail.com> wrote:

> Steven Schveighoffer Wrote:
>
>> But even so, malloc and free have the same property where they don't
>> always give back memory to the OS.  IIUC, Linux can only change the size
>> of memory it wants, it cannot free pages in the middle of the block.
>>
>> -Steve
>
> Disclaimer: I don't know what I am talking about.
>
> I think this is correct, the program isn't responsible for reclaiming the memory, that is what the OS does. If you don't have an OS then you don't have anything to return the memory to, so it just becomes free memory. Modern operating systems aren't going to take their memory back until it is needed (wast of cycles).
>
> What I observed using Linux and $ free; each section would result in a reduction of free memory and an increase in buffered data. This suggests to me that the OS doesn't want the memory yet.
>
> Tracking memory in a modern OS is not easy, and this is probably why no one wanted to make a statement on what was really happening. As I said I don't know if this is what is happening, but it usually isn't as straight forward as checking memory usage.

I think all memory is allocated/deallocated from the OS via the sbrk/brk system call:


       brk()  and  sbrk()  change  the  location  of  the program break, which
       defines the end of the process's data segment (i.e., the program  break
       is the first location after the end of the uninitialized data segment).
       Increasing the program break has the effect of allocating memory to the
       process; decreasing the break deallocates memory.

So you can only ever add to the *end* of memory, and you can only ever deallocate from the *end*.  And the OS doesn't ever just jump in and claim memory, you have to tell it that you are deallocating memory.

Which means, if you say allocated 100MB, and wanted to deallocate the first 99MB, you still couldn't release any back to the OS.

A moving GC would allow for more memory to be freed, but we aren't there yet.

Of course, I could be completely wrong about all this, I've never really used sbrk or brk :)

-Steve

Thanks very nice info, just two guys babbling about things they've only read I guess, but you seem much better informed.

Steven Schveighoffer Wrote:

> I think all memory is allocated/deallocated from the OS via the sbrk/brk system call:
> 
> 
>         brk()  and  sbrk()  change  the  location  of  the program break,
> which
>         defines the end of the process's data segment (i.e., the program
> break
>         is the first location after the end of the uninitialized data
> segment).
>         Increasing the program break has the effect of allocating memory to
> the
>         process; decreasing the break deallocates memory.
> 
> So you can only ever add to the *end* of memory, and you can only ever deallocate from the *end*.  And the OS doesn't ever just jump in and claim memory, you have to tell it that you are deallocating memory.
> 
> Which means, if you say allocated 100MB, and wanted to deallocate the first 99MB, you still couldn't release any back to the OS.
> 
> A moving GC would allow for more memory to be freed, but we aren't there yet.
> 
> Of course, I could be completely wrong about all this, I've never really used sbrk or brk :)
> 
> -Steve

> Tracking memory in a modern OS is not easy, and this is probably why no one
wanted to make a statement on what was really happening.

The issue is that the memory *is* leaking -- it's because the struct destructor is simply not getting called. If I call free() manually, the memory usage decreases normally, so it's not a measurement problem.

Furthermore, this doesn't seem to be an Array(T)-related bug at all -- it seems that pretty much *any* struct with a destructor will not have its destructor called on exit. In fact, after reading the language specifications, it seems like the glossary contradicts itself: it defines Plain Old Data as referring "to a struct that [...] has no destructor. D structs are POD."

By definition, if D structs were POD, then they could not have any destructors. It seems like the language contradicts itself, and the compiler only *sometimes* calls struct destructors.

Any ideas? Is this a bug?

And thank you for all your great responses! :)