http://d.puremagic.com/issues/show_bug.cgi?id=1804

           Summary: Severe GC leaks with repetitive array allocations
           Product: D
           Version: 1.026
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla@digitalmars.com
        ReportedBy: webmaster@villagersonline.com


Linux (Fedora 6), 64 bit

The following program will run out of memory when the allocations reach about 8 MB.  The problem happens on *both* 1.026 and 2.010

FYI: I tried running fullCollect() periodically, just in case...no luck.

  import std.stdio;
  void main()
  {
    for(int i=1; ; i++)
    {
      auto tmp = new char[i*1024];
      if(i % 1_000 == 0)
        writefln("%d iterations, roughly %.1f MB", i,
tmp.length/(1024*1024.0));
    }
  }


-- 

d-bugmail@puremagic.com wrote:
> 
> FYI: I tried running fullCollect() periodically, just in case...no luck.
> 
>   import std.stdio;
>   void main()
>   {
>     for(int i=1; ; i++)
>     {
>       auto tmp = new char[i*1024];
>       if(i % 1_000 == 0)
>         writefln("%d iterations, roughly %.1f MB", i,
> tmp.length/(1024*1024.0));
>     }
>   }

Interestingly, this app crashes in the same place if 'tmp' is deleted on each iteration.  If I had to guess, I'd say that empty pools aren't being added to the free list after collections, even though there's code in place that's supposed to do this.  Perhaps it's just pools dedicated to big allocations that are the problem.  This portion of the GC isn't really my area of expertise, but I'll submit a patch if I can figure this one out.

http://d.puremagic.com/issues/show_bug.cgi?id=1804





------- Comment #2 from webmaster@villagersonline.com  2008-01-24 17:18 -------
This came up in a program where I had a large array, containing all of the prime numbers through some value.  As I found more primes, I would append to the array.  This works just fine, often (meaning that *sometimes* buffers are getting correctly collected), but after we hit some point the memory usage would spike because, whenever an append operation required a copy, the old array wasn't getting GC'd.

I originally theorized that the prime numbers in the array were happening to match addresses of arrays (and that the bug was that the GC was scanning non-pointer memory).  But that doesn't play out, exactly.  My gut is still that there is a false pointer out there somewhere.  Are the memory allocator metadata structures scanned by the GC?  Maybe, after lots of allocations, we by random get metadata that looks like a pointer, which pins some memory....and then it gradually builds from there.  Dunno.

RECREATE UPDATE:

Update the writefln line to also trace the address:
   writefln("%d iterations, roughly %.1f MB at %x", i, ..... ,
cast(uint)tmp.ptr);

Then change the loop to start at 1014, 1015, or 1016 (maybe others) and the 1st buffer to be allocated will only get garbage collected once: after it is allocated the 2nd time, it doesn't appear that it is ever collected again.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1804





------- Comment #3 from sean@f4.ca  2008-02-05 12:48 -------
Okay, I think I know what's going on.  For big allocations, the GC will look for a pre-existing set of contiguous pages within a single pool to hold the block.  If one cannot be found, then it creates a new pool of the requested size to hold the block.  However, in this app the allocation size just continuously increases, so as time goes on the GC is unable to re-use existing pools and so allocates successively larger pools to hold the requested block. The problem is that all these old pools which were only used once are held by the GC for future use, so the total memory used by the application steadily increases, with most of this memory going completely unused.

The simplest fix in this case would be for the GC to always release pools obtained for big allocations back to the OS when they are no longer needed. This should address the pathological case demonstrated in this test app.  I'm going to make this change in Tango and see if it helps.


-- 

I've created a ticket for this in Tango:

http://www.dsource.org/projects/tango/ticket/878

The fix is done and will be checked in once DSource is back online.  The diff is pretty succinct, but I'd be happy to supply a patch for Phobos if that would help.

http://d.puremagic.com/issues/show_bug.cgi?id=1804





------- Comment #5 from webmaster@villagersonline.com  2008-03-31 12:34 -------
FYI: I hit this problem on Phobos, not Tango.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1804





------- Comment #6 from sean@invisibleduck.org  2008-04-03 18:20 -------
Sure.  But the same problem existed in Tango so I thought I'd diagnose it there and report my findings to save Walter some legwork with Phobos.  Just trying to be helpful here :-)


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1804


Martin Krejcirik <mk@krej.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |mk@krej.cz
         Resolution|                            |FIXED


--- Comment #7 from Martin Krejcirik <mk@krej.cz> 2013-04-01 20:17:06 CEST ---
Looks like it has been fixed somewhere between v1.056 and current v1.076.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=1804


Martin Krejcirik <mk@krej.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
          Component|DMD                         |Phobos
            Version|1.026                       |D1 & D2
         Resolution|FIXED                       |
         OS/Version|Linux                       |Windows


--- Comment #8 from Martin Krejcirik <mk@krej.cz> 2013-06-01 14:12:20 CEST ---
I'm reopening this, because I noticed the problem (core.exception.OutOfMemoryError after about 20000 iterations) still exists on Windows XP 32bit with latest dmd 2.063. Works ok on Linux though.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------