View mode: basic / threaded / horizontal-split · Log in · Help
November 28, 2012
Apparent problem with GC not collecting on Windows
A friend of mine reports that the memory usage of the following program 
grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 
home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).

If you are on Windows, could you please say whether you have the same 
problem. (I don't have Windows.) The following is how we reproduce:

Generate a file of about 1G by running the following program:

import std.stdio;

void foo()
{
    auto file = File("one_gigabyte_file", "w");
    auto data = new ubyte[](100 * 1024 * 1024);

    foreach (i; 0 .. 10) {
        file.rawWrite(data);
    }
}

void main()
{
    foo();
}

That part is not the problem. The problem is when reading such a large 
file by chunks. Could you please run the following program and report 
whether the memory consumption of it is increasing continuously:

import std.stdio;

void foo()
{
    auto file = File("one_gigabyte_file", "r");
    auto data = new ubyte[](100 * 1024 * 1024);

    foreach (i; 0 .. 10) {
        file.rawRead(data);
    }
}

void main()
{
    for (size_t i; true; ++i) {
        writeln(i);
        foo();
    }
}

The program gets terminated by a core.exception.OutOfMemoryError. (Same 
problem with std.stream.BufferedFile.)

I don't see any problem under Linux.

Thank you,
Ali
November 28, 2012
Re: Apparent problem with GC not collecting on Windows
11/28/2012 10:51 PM, Ali Çehreli пишет:
> A friend of mine reports that the memory usage of the following program
> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>
> If you are on Windows, could you please say whether you have the same
> problem. (I don't have Windows.) The following is how we reproduce:

[snip]

> The program gets terminated by a core.exception.OutOfMemoryError. (Same
> problem with std.stream.BufferedFile.)

Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 
running 32-bit app.


-- 
Dmitry Olshansky
November 28, 2012
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
> 11/28/2012 10:51 PM, Ali Çehreli пишет:
>> A friend of mine reports that the memory usage of the following program
>> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>>
>> If you are on Windows, could you please say whether you have the same
>> problem. (I don't have Windows.) The following is how we reproduce:
>
> [snip]
>
>> The program gets terminated by a core.exception.OutOfMemoryError. (Same
>> problem with std.stream.BufferedFile.)
>
> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
> running 32-bit app.
>
>

Isn't this a known problem with a conservative GC on 32bit?  The GC sees 
something that *Could* be a reference, and refuses to collect that data.
November 28, 2012
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 11:11 AM, 1100110 wrote:
> On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
>> 11/28/2012 10:51 PM, Ali Çehreli пишет:
>>> A friend of mine reports that the memory usage of the following program
>>> grows continuously when compiled with dmd 2.060 and run on Windows 
7 sp1
>>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>>>
>>> If you are on Windows, could you please say whether you have the same
>>> problem. (I don't have Windows.) The following is how we reproduce:
>>
>> [snip]
>>
>>> The program gets terminated by a core.exception.OutOfMemoryError. (Same
>>> problem with std.stream.BufferedFile.)
>>
>> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
>> running 32-bit app.
>>
>>
>
> Isn't this a known problem with a conservative GC on 32bit? The GC sees
> something that *Could* be a reference, and refuses to collect that data.

I think it is more like an integer value looking like a pointer to 
something. I don't see what the GC may be confused with in this case.

I've still created a bug:

  http://d.puremagic.com/issues/show_bug.cgi?id=9094

Ali
November 28, 2012
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 02:53 PM, Ali Çehreli wrote:
> On 11/28/2012 11:11 AM, 1100110 wrote:
>  > On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
>  >> 11/28/2012 10:51 PM, Ali Çehreli пишет:
>  >>> A friend of mine reports that the memory usage of the following
> program
>  >>> grows continuously when compiled with dmd 2.060 and run on Windows
> 7 sp1
>  >>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>  >>>
>  >>> If you are on Windows, could you please say whether you have the same
>  >>> problem. (I don't have Windows.) The following is how we reproduce:
>  >>
>  >> [snip]
>  >>
>  >>> The program gets terminated by a core.exception.OutOfMemoryError.
> (Same
>  >>> problem with std.stream.BufferedFile.)
>  >>
>  >> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
>  >> running 32-bit app.
>  >>
>  >>
>  >
>  > Isn't this a known problem with a conservative GC on 32bit? The GC sees
>  > something that *Could* be a reference, and refuses to collect that data.
>
> I think it is more like an integer value looking like a pointer to
> something. I don't see what the GC may be confused with in this case.
>
> I've still created a bug:
>
> http://d.puremagic.com/issues/show_bug.cgi?id=9094
>
> Ali
>

That's what I meant. =P
November 28, 2012
Re: Apparent problem with GC not collecting on Windows
1. Can be solved using allocators 
http://dlang.org/memory.html#newdelete and 
http://dlang.org/class.html#ClassAllocator (here deprecated)?
2. Why with
class Too
{
private uint[] pp;
   this(int s)
   {
      pp = new unit[s];
   }

alias pp this;
}

Only 3 iterations before out of memory exception?

3. Why it helps?

foreach (i; 0 .. 10) {
        file.rawRead(data);
    }

GC.free(data.ptr);


Win 8 Pro, 64 bit.
November 29, 2012
Re: Apparent problem with GC not collecting on Windows
On Wednesday, 28 November 2012 at 22:02:35 UTC, Michael wrote:
> Only 3 iterations before out of memory exception?

Because you used uint instead of ubyte, array is bigger, memory 
exhausts faster.

> 3. Why it helps?
> GC.free(data.ptr);

Initial leak happened because for some reason array allocated in 
previous iteration was not collected by GC when allocating new 
one, so the new one was allocated in another space growing the 
heap. If you place GC.free the array gets removed from heap on 
each iteration and each new allocation reuses the same memory, 
heap doesn't grow.
November 29, 2012
Re: Apparent problem with GC not collecting on Windows
> Because you used uint instead of ubyte, array is bigger, memory 
> exhausts faster.
Oh, I see.

>> 3. Why it helps?
>> GC.free(data.ptr);
>
> Initial leak happened because for some reason array allocated 
> in previous iteration was not collected by GC when allocating 
> new one, so the new one was allocated in another space growing 
> the heap. If you place GC.free the array gets removed from heap 
> on each iteration and each new allocation reuses the same 
> memory, heap doesn't grow.
If we do this manually it's works, but automatically is broken?
November 29, 2012
Re: Apparent problem with GC not collecting on Windows
On 11/29/2012 12:06 PM, Michael wrote:
>> Because you used uint instead of ubyte, array is bigger, memory
>> exhausts faster.
> Oh, I see.
>
>>> 3. Why it helps?
>>> GC.free(data.ptr);
>>
>> Initial leak happened because for some reason array allocated in
>> previous iteration was not collected by GC when allocating new one, so
>> the new one was allocated in another space growing the heap. If you
>> place GC.free the array gets removed from heap on each iteration and
>> each new allocation reuses the same memory, heap doesn't grow.
> If we do this manually it's works, but automatically is broken?
>

Nothing is "broken." GC.free would have been applied by the GC only if 
there have been no more references to the allocated block of memory.

The fact is, dmd uses a conservative GC. The issue is due to the 
combination of conservative GC, 32-bit address space, and a large chunk 
of memory. When that happens, it is very likely that any other value in 
the system, including an innocent int, has the risk of looking like a 
reference into that memory. For that reason the GC keeps that memory 
block allocated.

The risk of that happening is grossly reduced in a 64-bit address space. 
Similarly, allocating a much smaller buffer helps as well.

Alternatively, we can use a better GC or a runtime that has a better GC.

Ali
November 30, 2012
Re: Apparent problem with GC not collecting on Windows
11/30/2012 12:36 AM, Ali Çehreli пишет:
>
> On 11/29/2012 12:06 PM, Michael wrote:
>  >> Because you used uint instead of ubyte, array is bigger, memory
>  >> exhausts faster.
>  > Oh, I see.
>  >
>  >>> 3. Why it helps?
>  >>> GC.free(data.ptr);
>  >>
>  >> Initial leak happened because for some reason array allocated in
>  >> previous iteration was not collected by GC when allocating new one, so
>  >> the new one was allocated in another space growing the heap. If you
>  >> place GC.free the array gets removed from heap on each iteration and
>  >> each new allocation reuses the same memory, heap doesn't grow.
>  > If we do this manually it's works, but automatically is broken?
>  >
>
> Nothing is "broken." GC.free would have been applied by the GC only if
> there have been no more references to the allocated block of memory.
>
> The fact is, dmd uses a conservative GC. The issue is due to the
> combination of conservative GC, 32-bit address space, and a large chunk
> of memory. When that happens, it is very likely that any other value in
> the system, including an innocent int, has the risk of looking like a
> reference into that memory. For that reason the GC keeps that memory
> block allocated.
>

I'd just throw in that we have a (almost) precise GC that is used by  at 
least one large project (the VisualD apparently). Though there were some 
problems with it. Anyway I'd expect to see it in upstream by 2.062 at 
least. It should help the cases like this tremendously.

P.S. Either way the manual memory management (and reuse) is the way to 
go with big allocations.

-- 
Dmitry Olshansky
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home