I'm writing a simple program which uses merge sort to sort very large text files. I split the text file into several temporary files which are sorted and merged together.

The trouble I'm having is that the BufferedFile class seems to have a memory leak. The memory usage of the process slowly goes up until it throws an out of memory exception. I can fix the memory leak by switching to the File class, but that makes the program much slower.

Calling GC.collect() didn't help, and neither did explicitly deleting the
BufferedFile object.

On 8/28/11 1:00 , Leon wrote:
> I'm writing a simple program which uses merge sort to sort very large text
> files. I split the text file into several temporary files which are sorted and
> merged together.
>
> The trouble I'm having is that the BufferedFile class seems to have a memory
> leak. The memory usage of the process slowly goes up until it throws an out of
> memory exception. I can fix the memory leak by switching to the File class,
> but that makes the program much slower.
>
> Calling GC.collect() didn't help, and neither did explicitly deleting the
> BufferedFile object.

Thats very interesting!
Perhaps you can have a look at: https://github.com/gizmomogwai/performance
There I ran into performanceproblems using BufferedFile.
One of the best solutions I found till now is using mmfile as in https://github.com/gizmomogwai/performance/blob/master/src/d/readbytes5.d. Perhaps this could be of help for you.

I dont know how you have to access the data in the files, but perhaps the interface of mmfile is good enough for you!


christian

On 28.08.2011 01:00, Leon wrote:
> I'm writing a simple program which uses merge sort to sort very large text
> files. I split the text file into several temporary files which are sorted and
> merged together.
>
> The trouble I'm having is that the BufferedFile class seems to have a memory
> leak. The memory usage of the process slowly goes up until it throws an out of
> memory exception. I can fix the memory leak by switching to the File class,
> but that makes the program much slower.
>
> Calling GC.collect() didn't help, and neither did explicitly deleting the
> BufferedFile object.

You could try just using the functions in std.stdio instead.