http://d.puremagic.com/issues/show_bug.cgi?id=11021

           Summary: Huge GC leak leads to crash; GC uses 7x more memory
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: blocker
          Priority: P2
         Component: druntime
        AssignedTo: nobody@puremagic.com
        ReportedBy: temtaime@gmail.com


--- Comment #0 from Temtaime <temtaime@gmail.com> 2013-09-13 07:09:22 PDT ---
import core.memory;
import std.traits;

void main() {
    while(true) {
        GC.collect();
        new ubyte[40 * 1024 * 1024];
    }
}

Application's memory usage grows and it crashes with message: core.exception.OutOfMemoryError

If i comment import std.traits; everything goes OK, but now application uses 300 MM of memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021


monarchdodra@gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |monarchdodra@gmail.com


--- Comment #1 from monarchdodra@gmail.com 2013-09-13 09:54:03 PDT ---
Are you, by any chance, on win32?

What is your metric for "GC uses 7x more memory"?

Also, you say: "but now application uses 300 MM of memory." But is that a problem?

I don't know what to make about your "import" issue, as I can't reproduce on linux. I'll try on windows.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #2 from monarchdodra@gmail.com 2013-09-13 10:01:26 PDT ---
(In reply to comment #1)
> Are you, by any chance, on win32?

I can reproduce on win32, all version of dmd.

> What is your metric for "GC uses 7x more memory"?
> 
> Also, you say: "but now application uses 300 MM of memory." But is that a problem?

Never mind, I understood what you meant.

> I don't know what to make about your "import" issue, as I can't reproduce on linux. I'll try on windows.

I don't know how you did it, but I run out of memory regardless.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #3 from Temtaime <temtaime@gmail.com> 2013-09-13 10:07:59 PDT ---
Yes, it's on win32.

I think app should use about 50 mb as i'm allocating 40 mb array, not 300.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021


Orvid King <blah38621@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |blah38621@gmail.com


--- Comment #4 from Orvid King <blah38621@gmail.com> 2013-09-13 10:10:20 PDT ---
(In reply to comment #3)
> Yes, it's on win32.
> 
> I think app should use about 50 mb as i'm allocating 40 mb array, not 300.

I would actually expect it to use 90mb, your allocating a 40mb array inside a loop, the previous 40mb array will still be referenced when you go to allocate the new 40mb array, meaning that it can't be collected if the second allocation triggers a GC.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #5 from Temtaime <temtaime@gmail.com> 2013-09-13 10:12:56 PDT ---
In loop i calling collect, so gc should frees the memory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #6 from Orvid King <blah38621@gmail.com> 2013-09-13 10:19:48 PDT ---
(In reply to comment #5)
> In loop i calling collect, so gc should frees the memory.

You should really be clearing the existing array rather than allocating a new one, but that's beside the point. When you call the GC manually, the array is still referenced in local variable, if you were to reset that local variable before calling the GC, then it should collect it correctly.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #7 from monarchdodra@gmail.com 2013-09-13 10:38:13 PDT ---
(In reply to comment #4)
> I would actually expect it to use 90mb, your allocating a 40mb array inside a loop, the previous 40mb array will still be referenced when you go to allocate the new 40mb array, meaning that it can't be collected if the second allocation triggers a GC.

(In reply to comment #6)
> You should really be clearing the existing array rather than allocating a new one, but that's beside the point. When you call the GC manually, the array is still referenced in local variable, if you were to reset that local variable before calling the GC, then it should collect it correctly.

He keeps no references to the array, so it should be collected immediatly on each call to GC.collect(). OR at least, there is no reason for it not to be collected.

That said, I'm unsure "GC.collect()" promises it'll acutally *do* anything,
it's more of a hint "now would be a good time to run a collection" (AFAIK).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021


Dmitry Olshansky <dmitry.olsh@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmitry.olsh@gmail.com


--- Comment #8 from Dmitry Olshansky <dmitry.olsh@gmail.com> 2013-09-13 11:51:16 PDT ---
(In reply to comment #0)
> import core.memory;
> import std.traits;
> 
> void main() {
>     while(true) {
>         GC.collect();
>         new ubyte[40 * 1024 * 1024];
>     }
> }
> 
> Application's memory usage grows and it crashes with message: core.exception.OutOfMemoryError
> 
> If i comment import std.traits; everything goes OK, but now application uses 300 MM of memory.

32bits? Allocating large blocks in a loop?
Good luck with that... sooner or later they would be pinned by something that
looks like a pointer to GC.  We are back to false pointers and precise
collection topic.

How std.traits affects that is interesting thing, maybe inflates executable size. Anyway this is game of chance so any slight change may avoid it or bring it back.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11021



--- Comment #9 from monarchdodra@gmail.com 2013-09-13 12:07:28 PDT ---
(In reply to comment #8)
> 32bits? Allocating large blocks in a loop?
> Good luck with that... sooner or later they would be pinned by something that
> looks like a pointer to GC.  We are back to false pointers and precise
> collection topic.
> 
> How std.traits affects that is interesting thing, maybe inflates executable size. Anyway this is game of chance so any slight change may avoid it or bring it back.

2 things I don't understand:

1. New initializes, so the block contains nothing but 0x00's. How could that contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as pointer?

2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking for new ubytes, which is a non-pointer type. Why would the GC scan any of that at all anywas?

If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32
runs forever...

--------

TLDR, I don't see how the false pointer issue in 32 bit environment would produce this failure.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------