November 28, 2012
A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).

If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce:

Generate a file of about 1G by running the following program:

import std.stdio;

void foo()
{
    auto file = File("one_gigabyte_file", "w");
    auto data = new ubyte[](100 * 1024 * 1024);

    foreach (i; 0 .. 10) {
        file.rawWrite(data);
    }
}

void main()
{
    foo();
}

That part is not the problem. The problem is when reading such a large file by chunks. Could you please run the following program and report whether the memory consumption of it is increasing continuously:

import std.stdio;

void foo()
{
    auto file = File("one_gigabyte_file", "r");
    auto data = new ubyte[](100 * 1024 * 1024);

    foreach (i; 0 .. 10) {
        file.rawRead(data);
    }
}

void main()
{
    for (size_t i; true; ++i) {
        writeln(i);
        foo();
    }
}

The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.)

I don't see any problem under Linux.

Thank you,
Ali
November 28, 2012
11/28/2012 10:51 PM, Ali Çehreli пишет:
> A friend of mine reports that the memory usage of the following program
> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>
> If you are on Windows, could you please say whether you have the same
> problem. (I don't have Windows.) The following is how we reproduce:

[snip]

> The program gets terminated by a core.exception.OutOfMemoryError. (Same
> problem with std.stream.BufferedFile.)

Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app.


-- 
Dmitry Olshansky
November 28, 2012
On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
> 11/28/2012 10:51 PM, Ali Çehreli пишет:
>> A friend of mine reports that the memory usage of the following program
>> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>>
>> If you are on Windows, could you please say whether you have the same
>> problem. (I don't have Windows.) The following is how we reproduce:
>
> [snip]
>
>> The program gets terminated by a core.exception.OutOfMemoryError. (Same
>> problem with std.stream.BufferedFile.)
>
> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
> running 32-bit app.
>
>

Isn't this a known problem with a conservative GC on 32bit?  The GC sees something that *Could* be a reference, and refuses to collect that data.
November 28, 2012
On 11/28/2012 11:11 AM, 1100110 wrote:
> On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
>> 11/28/2012 10:51 PM, Ali Çehreli пишет:
>>> A friend of mine reports that the memory usage of the following program
>>> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
>>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>>>
>>> If you are on Windows, could you please say whether you have the same
>>> problem. (I don't have Windows.) The following is how we reproduce:
>>
>> [snip]
>>
>>> The program gets terminated by a core.exception.OutOfMemoryError. (Same
>>> problem with std.stream.BufferedFile.)
>>
>> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
>> running 32-bit app.
>>
>>
>
> Isn't this a known problem with a conservative GC on 32bit? The GC sees
> something that *Could* be a reference, and refuses to collect that data.

I think it is more like an integer value looking like a pointer to something. I don't see what the GC may be confused with in this case.

I've still created a bug:

  http://d.puremagic.com/issues/show_bug.cgi?id=9094

Ali

November 28, 2012
On 11/28/2012 02:53 PM, Ali Çehreli wrote:
> On 11/28/2012 11:11 AM, 1100110 wrote:
>  > On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
>  >> 11/28/2012 10:51 PM, Ali Çehreli пишет:
>  >>> A friend of mine reports that the memory usage of the following
> program
>  >>> grows continuously when compiled with dmd 2.060 and run on Windows
> 7 sp1
>  >>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
>  >>>
>  >>> If you are on Windows, could you please say whether you have the same
>  >>> problem. (I don't have Windows.) The following is how we reproduce:
>  >>
>  >> [snip]
>  >>
>  >>> The program gets terminated by a core.exception.OutOfMemoryError.
> (Same
>  >>> problem with std.stream.BufferedFile.)
>  >>
>  >> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
>  >> running 32-bit app.
>  >>
>  >>
>  >
>  > Isn't this a known problem with a conservative GC on 32bit? The GC sees
>  > something that *Could* be a reference, and refuses to collect that data.
>
> I think it is more like an integer value looking like a pointer to
> something. I don't see what the GC may be confused with in this case.
>
> I've still created a bug:
>
> http://d.puremagic.com/issues/show_bug.cgi?id=9094
>
> Ali
>

That's what I meant. =P
November 28, 2012
1. Can be solved using allocators http://dlang.org/memory.html#newdelete and http://dlang.org/class.html#ClassAllocator (here deprecated)?
2. Why with
class Too
{
private uint[] pp;
   this(int s)
   {
      pp = new unit[s];
   }

alias pp this;
}

Only 3 iterations before out of memory exception?

3. Why it helps?

foreach (i; 0 .. 10) {
        file.rawRead(data);
    }

GC.free(data.ptr);


Win 8 Pro, 64 bit.


November 29, 2012
On Wednesday, 28 November 2012 at 22:02:35 UTC, Michael wrote:
> Only 3 iterations before out of memory exception?

Because you used uint instead of ubyte, array is bigger, memory exhausts faster.

> 3. Why it helps?
> GC.free(data.ptr);

Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow.


November 29, 2012
> Because you used uint instead of ubyte, array is bigger, memory exhausts faster.
Oh, I see.

>> 3. Why it helps?
>> GC.free(data.ptr);
>
> Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow.
If we do this manually it's works, but automatically is broken?

November 29, 2012
On 11/29/2012 12:06 PM, Michael wrote:
>> Because you used uint instead of ubyte, array is bigger, memory
>> exhausts faster.
> Oh, I see.
>
>>> 3. Why it helps?
>>> GC.free(data.ptr);
>>
>> Initial leak happened because for some reason array allocated in
>> previous iteration was not collected by GC when allocating new one, so
>> the new one was allocated in another space growing the heap. If you
>> place GC.free the array gets removed from heap on each iteration and
>> each new allocation reuses the same memory, heap doesn't grow.
> If we do this manually it's works, but automatically is broken?
>

Nothing is "broken." GC.free would have been applied by the GC only if there have been no more references to the allocated block of memory.

The fact is, dmd uses a conservative GC. The issue is due to the combination of conservative GC, 32-bit address space, and a large chunk of memory. When that happens, it is very likely that any other value in the system, including an innocent int, has the risk of looking like a reference into that memory. For that reason the GC keeps that memory block allocated.

The risk of that happening is grossly reduced in a 64-bit address space. Similarly, allocating a much smaller buffer helps as well.

Alternatively, we can use a better GC or a runtime that has a better GC.

Ali

November 30, 2012
11/30/2012 12:36 AM, Ali Çehreli пишет:
>
> On 11/29/2012 12:06 PM, Michael wrote:
>  >> Because you used uint instead of ubyte, array is bigger, memory
>  >> exhausts faster.
>  > Oh, I see.
>  >
>  >>> 3. Why it helps?
>  >>> GC.free(data.ptr);
>  >>
>  >> Initial leak happened because for some reason array allocated in
>  >> previous iteration was not collected by GC when allocating new one, so
>  >> the new one was allocated in another space growing the heap. If you
>  >> place GC.free the array gets removed from heap on each iteration and
>  >> each new allocation reuses the same memory, heap doesn't grow.
>  > If we do this manually it's works, but automatically is broken?
>  >
>
> Nothing is "broken." GC.free would have been applied by the GC only if
> there have been no more references to the allocated block of memory.
>
> The fact is, dmd uses a conservative GC. The issue is due to the
> combination of conservative GC, 32-bit address space, and a large chunk
> of memory. When that happens, it is very likely that any other value in
> the system, including an innocent int, has the risk of looking like a
> reference into that memory. For that reason the GC keeps that memory
> block allocated.
>

I'd just throw in that we have a (almost) precise GC that is used by  at least one large project (the VisualD apparently). Though there were some problems with it. Anyway I'd expect to see it in upstream by 2.062 at least. It should help the cases like this tremendously.

P.S. Either way the manual memory management (and reuse) is the way to go with big allocations.

-- 
Dmitry Olshansky
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home