Thread overview | |||||
---|---|---|---|---|---|
|
August 27, 2011 Memory leak with BufferedFile? | ||||
---|---|---|---|---|
| ||||
I'm writing a simple program which uses merge sort to sort very large text files. I split the text file into several temporary files which are sorted and merged together. The trouble I'm having is that the BufferedFile class seems to have a memory leak. The memory usage of the process slowly goes up until it throws an out of memory exception. I can fix the memory leak by switching to the File class, but that makes the program much slower. Calling GC.collect() didn't help, and neither did explicitly deleting the BufferedFile object. |
August 29, 2011 Re: Memory leak with BufferedFile? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Leon | On 8/28/11 1:00 , Leon wrote: > I'm writing a simple program which uses merge sort to sort very large text > files. I split the text file into several temporary files which are sorted and > merged together. > > The trouble I'm having is that the BufferedFile class seems to have a memory > leak. The memory usage of the process slowly goes up until it throws an out of > memory exception. I can fix the memory leak by switching to the File class, > but that makes the program much slower. > > Calling GC.collect() didn't help, and neither did explicitly deleting the > BufferedFile object. Thats very interesting! Perhaps you can have a look at: https://github.com/gizmomogwai/performance There I ran into performanceproblems using BufferedFile. One of the best solutions I found till now is using mmfile as in https://github.com/gizmomogwai/performance/blob/master/src/d/readbytes5.d. Perhaps this could be of help for you. I dont know how you have to access the data in the files, but perhaps the interface of mmfile is good enough for you! christian |
August 29, 2011 Re: Memory leak with BufferedFile? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Leon | On 28.08.2011 01:00, Leon wrote:
> I'm writing a simple program which uses merge sort to sort very large text
> files. I split the text file into several temporary files which are sorted and
> merged together.
>
> The trouble I'm having is that the BufferedFile class seems to have a memory
> leak. The memory usage of the process slowly goes up until it throws an out of
> memory exception. I can fix the memory leak by switching to the File class,
> but that makes the program much slower.
>
> Calling GC.collect() didn't help, and neither did explicitly deleting the
> BufferedFile object.
You could try just using the functions in std.stdio instead.
|
Copyright © 1999-2021 by the D Language Foundation