http://d.puremagic.com/issues/show_bug.cgi?id=1980

           Summary: memory allocated via double[][] is not being freed by
                    the GC
           Product: D
           Version: 2.012
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla@digitalmars.com
        ReportedBy: markusle@gmail.com


Hi,

For a data processing application I need to read a large number
of data sets from disk. Due to their size, they have to be read and
processed sequentially, i.e. in pseudocode

int main()
{
    while (some condition)
    {
         double[][] myLargeDataset = read_data();
         process_data(myLargeDataset);
         // free all memory here otherwise next cycle will
         // run out of memory
     }

   return 0;
}

Now, the "problem" is the fact that each single data-set saturates
system memory and hence I need to make sure that all memory
is freed after each process_data step is complete. Unfortunately,
using dmd-2.012 I have not been able to achieve this. Whatever
I do (including nothing, i.e., letting the GC do its job), the resulting
binary keeps accumulating memory and crashing shortly after.
I've tried deleting the array, setting it to null, setting the array
lengths to 0, and manually forcing the GC to collect, to no avail.
Hence, is there something I am doing terribly wrong or is this a
bug in dmd?

Thanks much,
Markus


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1980





------- Comment #1 from markusle@gmail.com  2008-04-08 18:57 -------
Below is a complete piece of code that still exhibits this issue. The file "data_random.dat" simply contains a single row of random ints (about 30M filesize). Each iteration allocated additional heap and I don't seem to be able to free foo no matter what.

import std.stream;
import std.stdio;
import std.contracts;
import std.gc;


double[][] parse(BufferedFile inputFile)
{

  double[][] array;
  foreach(char[] line; inputFile)
  {
    double[] temp;

    foreach(string item; std.string.split(assumeUnique(line)))
    {
       temp ~= std.string.atof(item);
    }

    array ~= temp;
  }

  /* rewind for next round */
  inputFile.seekSet(0);

  return array;
}



int main()
{
  BufferedFile inputFile = new BufferedFile("data_random.dat");

  while(1)
  {
    double[][] foo = parse(inputFile);
  }

  return 1;
}


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1980





------- Comment #2 from wbaxter@gmail.com  2008-04-08 19:27 -------
The test program works fine for me on Windows.
I tried it with
  DMD/Phobos 1.028,
  DMD/Tango/Tangobos 1.028, and
  DMD/Phobos 2.012.

Although the data file I tested it with was only 50K.  That could make a difference if it's being caused by "false pointers".  Or it could be a Linux-only bug, I suppose.

Maybe for the lazy you could also modify your program to generate a 30MB file full of random numbers.  :-)


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1980





------- Comment #3 from markusle@gmail.com  2008-04-08 19:49 -------
Thanks much for testing Bill!

I just ran both the 32bit DMD executable and the
64bit GDC (v. 0.24) version of the above code
on my opteron box. The 32bit DMD version leaks memory
whereas the GDC binary works just fine and properly
free its memory.

Thanks,
Markus



-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1980





------- Comment #4 from webmaster@villagersonline.com  2008-04-09 08:01 -------
It's possible that this might be related to issue 1804 (http://d.puremagic.com/issues/show_bug.cgi?id=1804)


--