I'm trying to generate images and after displaying them I want to save them to a png file. I use parallelism to generate them asynchronously with the gui-thread to avoid blocking (don't know if this is really relevant).

If I add a single line in the method where I generate the image that creates a class which allocates a pretty big array, slowly but surely the memory starts to rise until it ends in a spectacular out of memory exception. I always thought the garbage collector would deal with it.

I tried reducing it but failed. Maybe I should let dustmite bite in it a
bit later.
How can I manually fix this? Is this a known bug in the garbage collector?

btw, this code also caused segfaults on my  64 bit laptop on linux. How can I avoid this?

main class: https://dl.dropboxusercontent.com/u/15024434/picturegenerator/generalised.d

zip file with everything ready to go to compile: https://dl.dropboxusercontent.com/u/15024434/picturegenerator/picturegenerator.zip

Am 11.05.2013 01:18, schrieb maarten van damme:
> I'm trying to generate images and after displaying them I want to save
> them to a png file. I use parallelism to generate them asynchronously
> with the gui-thread to avoid blocking (don't know if this is really
> relevant).
>
> If I add a single line in the method where I generate the image that
> creates a class which allocates a pretty big array, slowly but surely
> the memory starts to rise until it ends in a spectacular out of memory
> exception. I always thought the garbage collector would deal with it.
>
> I tried reducing it but failed. Maybe I should let dustmite bite in it a
> bit later.
> How can I manually fix this? Is this a known bug in the garbage collector?
>
> btw, this code also caused segfaults on my  64 bit laptop on linux.
> How can I avoid this?
>
> main class:
> https://dl.dropboxusercontent.com/u/15024434/picturegenerator/generalised.d
>
> zip file with everything ready to go to compile:
> https://dl.dropboxusercontent.com/u/15024434/picturegenerator/picturegenerator.zip

Try allocating the big array with:

import core.memory;

size_t imageSize = 1024*1024*3;
auto imageData[] = (cast(ubyte*)GC.malloc(imageSize , GC.BlkAttr.NO_SCAN)[0..imageSize];

Which will result in the big memory block not beeing scanned by the GC which should reduce the number of false positives during collection.

If this doesn't work, use malloc / free of the c standard library to allocate and free the image data.

Kind Regards
Benjamin Thaut

I tried using gc.malloc and gc.free but got an invalidmemoryoperation
exception when doing gc.free(data.ptr);
Simply using gc.malloc like you proposed, the memory keeps growing. Is this
a bug in the d garbage collector?

What's really strange though, is that the destructor of trueimagecolor (the
class allocating the buffer) is getting called everytime...
Does this mean that a destructor getting called still is no garantee that
all data is getting free'd?



2013/5/12 Benjamin Thaut <code@benjamin-thaut.de>

> Am 11.05.2013 01:18, schrieb maarten van damme:
>
>  I'm trying to generate images and after displaying them I want to save
>> them to a png file. I use parallelism to generate them asynchronously with the gui-thread to avoid blocking (don't know if this is really relevant).
>>
>> If I add a single line in the method where I generate the image that creates a class which allocates a pretty big array, slowly but surely the memory starts to rise until it ends in a spectacular out of memory exception. I always thought the garbage collector would deal with it.
>>
>> I tried reducing it but failed. Maybe I should let dustmite bite in it a
>> bit later.
>> How can I manually fix this? Is this a known bug in the garbage collector?
>>
>> btw, this code also caused segfaults on my  64 bit laptop on linux. How can I avoid this?
>>
>> main class:
>> https://dl.dropboxusercontent.**com/u/15024434/**
>> picturegenerator/generalised.d<https://dl.dropboxusercontent.com/u/15024434/picturegenerator/generalised.d>
>>
>> zip file with everything ready to go to compile: https://dl.dropboxusercontent.**com/u/15024434/**picturegenerator/** picturegenerator.zip<https://dl.dropboxusercontent.com/u/15024434/picturegenerator/picturegenerator.zip>
>>
>
> Try allocating the big array with:
>
> import core.memory;
>
> size_t imageSize = 1024*1024*3;
> auto imageData[] = (cast(ubyte*)GC.malloc(**imageSize ,
> GC.BlkAttr.NO_SCAN)[0..**imageSize];
>
> Which will result in the big memory block not beeing scanned by the GC which should reduce the number of false positives during collection.
>
> If this doesn't work, use malloc / free of the c standard library to allocate and free the image data.
>
> Kind Regards
> Benjamin Thaut
>

On Sunday, 12 May 2013 at 11:27:15 UTC, maarten van damme wrote:
> I tried using gc.malloc and gc.free but got an invalidmemoryoperation
> exception when doing gc.free(data.ptr);
> Simply using gc.malloc like you proposed, the memory keeps growing. Is this
> a bug in the d garbage collector?
>
> What's really strange though, is that the destructor of trueimagecolor (the
> class allocating the buffer) is getting called everytime...
> Does this mean that a destructor getting called still is no garantee that
> all data is getting free'd?
>
>

Calling GC.free should not be neccessary as the data will still be collected.  The destructor of TrueImageColor beeing called does not mean the Array with the Image Data gets collected. Are you Sure that TrueColorImage is the only class holding a reference to the data? Do you take a slice of the data somewhere?


>
> 2013/5/12 Benjamin Thaut <code@benjamin-thaut.de>
>
>> Am 11.05.2013 01:18, schrieb maarten van damme:
>>
>>  I'm trying to generate images and after displaying them I want to save
>>> them to a png file. I use parallelism to generate them asynchronously
>>> with the gui-thread to avoid blocking (don't know if this is really
>>> relevant).
>>>
>>> If I add a single line in the method where I generate the image that
>>> creates a class which allocates a pretty big array, slowly but surely
>>> the memory starts to rise until it ends in a spectacular out of memory
>>> exception. I always thought the garbage collector would deal with it.
>>>
>>> I tried reducing it but failed. Maybe I should let dustmite bite in it a
>>> bit later.
>>> How can I manually fix this? Is this a known bug in the garbage collector?
>>>
>>> btw, this code also caused segfaults on my  64 bit laptop on linux.
>>> How can I avoid this?
>>>
>>> main class:
>>> https://dl.dropboxusercontent.**com/u/15024434/**
>>> picturegenerator/generalised.d<https://dl.dropboxusercontent.com/u/15024434/picturegenerator/generalised.d>
>>>
>>> zip file with everything ready to go to compile:
>>> https://dl.dropboxusercontent.**com/u/15024434/**picturegenerator/**
>>> picturegenerator.zip<https://dl.dropboxusercontent.com/u/15024434/picturegenerator/picturegenerator.zip>
>>>
>>
>> Try allocating the big array with:
>>
>> import core.memory;
>>
>> size_t imageSize = 1024*1024*3;
>> auto imageData[] = (cast(ubyte*)GC.malloc(**imageSize ,
>> GC.BlkAttr.NO_SCAN)[0..**imageSize];
>>
>> Which will result in the big memory block not beeing scanned by the GC
>> which should reduce the number of false positives during collection.
>>
>> If this doesn't work, use malloc / free of the c standard library to
>> allocate and free the image data.
>>
>> Kind Regards
>> Benjamin Thaut

This is ridiculous. Your solution appears to keep the memory somewhat constant around 20mb-28mb untill it suddenly runs out of memory. Literally out of nowhere.

I have no idea what's causing everything, so I decided to get rid of the window altogether and to try saving the images instead of displaying + saving them. Now I run in "Internal error: ..\ztc\cgcs.c 343"...

D is really dissapointing here. Maybe it's the gc having fun, maybe it's something else, no way to know for sure. Having to give up on displaying it altogether runs into internal error's.

maarten van damme:

> I have no idea what's causing everything, so I decided to get rid of the
> window altogether and to try saving the images instead of displaying +
> saving them. Now I run in "Internal error: ..\ztc\cgcs.c 343"...

Once minimized, this may be fit for D Bugzilla.

Bye,
bearophile

Am 12.05.2013 21:05, schrieb maarten van damme:
> This is ridiculous. Your solution appears to keep the memory somewhat
> constant around 20mb-28mb untill it suddenly runs out of memory.
> Literally out of nowhere.
>
> I have no idea what's causing everything, so I decided to get rid of the
> window altogether and to try saving the images instead of displaying +
> saving them. Now I run in "Internal error: ..\ztc\cgcs.c 343"...
>
> D is really dissapointing here. Maybe it's the gc having fun, maybe it's
> something else, no way to know for sure. Having to give up on displaying
> it altogether runs into internal error's.

As D is a relative new lagnauge stuff like this can happen. It would be great if you could reduce the compiler crash to a minimal test case using dustmite: https://github.com/CyberShadow/DustMite
A description how to use it is aviable in the wiki. If you successfully reduced the compiler error please fill in a bug report at http://d.puremagic.com/issues/

-- 
Kind Regards
Benjamin Thaut

aren't you guys all getting tons of internal error's as soon as you combine dynamic arrays with static arrays? It seems as if these things are completely seperate things, but their syntax sugests a more closely related connection. I really find it confusing...

So, after reducing, I am very certain that at least one problem comes from
generating a png, the code used is here available :
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/png.d
seeing as this is pure d, am I right in assuming there should never be any
memory leaks when using this?

The reduced loop that only generates images is here :

TrueColorImage i= new TrueColorImage(width,height);
PNG* png;
double[][4] curtrans;
while(true){
curtrans=generateTransformationMatrix();

for(int y=0;y<height;y++)
for(int x=0;x<width;x++)
i.data[(y*width+x)*4..y*width+x)*4+4]=colorify(applyTransformation(transformXY(x,y),curtrans)).dup[0..3]
~ 255;
 // and finally write the data to a png file
png = pngFromImage(i);
//std.file.write("images/"~toHexString(md5Of(curtrans))~".png",
writePng(png));
}

if you comment out "png = pngFromImage(i)" the program appears to not blow
up over time. I think the ice come from assiging a slice of a dynamic array
to a slice of a static array (hence the .dup). (I'll try reducing it with
dustmite)



2013/5/12 Benjamin Thaut <code@benjamin-thaut.de>

> Am 12.05.2013 21:05, schrieb maarten van damme:
>
>> This is ridiculous. Your solution appears to keep the memory somewhat constant around 20mb-28mb untill it suddenly runs out of memory. Literally out of nowhere.
>>
>> I have no idea what's causing everything, so I decided to get rid of the window altogether and to try saving the images instead of displaying + saving them. Now I run in "Internal error: ..\ztc\cgcs.c 343"...
>>
>> D is really dissapointing here. Maybe it's the gc having fun, maybe it's something else, no way to know for sure. Having to give up on displaying it altogether runs into internal error's.
>>
>
> As D is a relative new lagnauge stuff like this can happen. It would be
> great if you could reduce the compiler crash to a minimal test case using
> dustmite: https://github.com/**CyberShadow/DustMite<https://github.com/CyberShadow/DustMite>
> A description how to use it is aviable in the wiki. If you successfully
> reduced the compiler error please fill in a bug report at
> http://d.puremagic.com/issues/
>
> --
> Kind Regards
> Benjamin Thaut
>

Instead of the .dup you can use the slice operator []. Although this really should not cause a compiler crash if you forgett to do so.

int[3] staticArray;
int[] normalArray = staticArray[];


I'm not seeing something fundamentally wrong with your code right now. I will most likely have some more time to look at it tomorrow. Does it use any plattform specific features? (E.g. does it work on windows?)

Having a GC does not mean you can not have memory leaks. If you manage to keep the memory referenced by some hard rooted reference the GC is not allowed to free it and you will get a memory leak. I'm also currently not aware of any tools for D that will help you find memory leaks.

Kind Regards
Benjamin Thaut

But seeing as the memory only starts growing when I put the png line in, and that I keep deleting the old class and replacing it for a new instance, shouldn't the gc clean up everything as nothing can still be referencing to that data?

I am working on windows, on linux my original code segfaulted, I don't know about the reduced version though. Afaik it doesn't use a platform dependent feature.


2013/5/13 Benjamin Thaut <code@benjamin-thaut.de>

> Instead of the .dup you can use the slice operator []. Although this really should not cause a compiler crash if you forgett to do so.
>
> int[3] staticArray;
> int[] normalArray = staticArray[];
>
>
> I'm not seeing something fundamentally wrong with your code right now. I will most likely have some more time to look at it tomorrow. Does it use any plattform specific features? (E.g. does it work on windows?)
>
> Having a GC does not mean you can not have memory leaks. If you manage to keep the memory referenced by some hard rooted reference the GC is not allowed to free it and you will get a memory leak. I'm also currently not aware of any tools for D that will help you find memory leaks.
>
> Kind Regards
> Benjamin Thaut
>
>