Dear D user, i have this code:

import core.sys.windows.windows;

real[] a;
while(1) {
  a.length=4096*4096;
  a=null;
		
  Sleep(2000);
}

It allocates memory, but its never gets freed, just keep going up, and after 10 secs, memoryexception is thrown. I checked the pointer it is GC safe (attributes is 10). Am i missing something?

TY: collerblade

Also forget to mention, that im using DMD 2.066.1.
If i alloc memory with GC.malloc(..), it is handled correcty (and freed).
Why GC doesn't frees dynamic arrays?

i tested malloc, and it also run out of mem, except when i manually free the pointer.
There is no GC im DMD?

On 1/22/15 8:44 AM, collerblade wrote:
> Dear D user, i have this code:
>
> import core.sys.windows.windows;
>
> real[] a;
> while(1) {
>    a.length=4096*4096;
>    a=null;
>
>    Sleep(2000);
> }
>
> It allocates memory, but its never gets freed, just keep going up, and
> after 10 secs, memoryexception is thrown. I checked the pointer it is GC
> safe (attributes is 10). Am i missing something?

Are you compiling 64-bit code or 32-bit? It makes a huge difference.

A real is going to be 16-byte aligned. This means real[4096*4096] is allocating 1/4GB per allocation. This consumes 1/16th of your total address space on 32-bit address space.

The GC in D is conservative. This means that on 32-bit architecture, any 4-byte segment scanned by the GC (whether it's a pointer or not) that happens to point into the array you allocated is going to keep that memory pinned. Because any 4-byte garbage has a 1:16 chance of pointing there, odds are good that this will happen. Then it keeps happening, you eventually run out of memory.

Now, some possible explanation of why GC.malloc might work while this does not:

GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes through the append code. Because extending length is considered an appending event, it *over-allocates* by a certain factor, expecting you to keep appending. (this is how appending can be amortized O(1) performance). This means the amount of memory allocated is larger than 1/4GB. Therefore the chance of failing by setting a.length is going to be higher than with using GC.malloc directly, but it could fail with either case.

Answers?

1. Manually free such a large array when done with it.
2. Do not allocate such large arrays.
3. Use 64-bit code generation (-m64 on DMD windows, even if you are on Win64) if possible.

-Steve

On Thursday, 22 January 2015 at 14:19:10 UTC, Steven Schveighoffer wrote:
> On 1/22/15 8:44 AM, collerblade wrote:
>> Dear D user, i have this code:
>>
>> import core.sys.windows.windows;
>>
>> real[] a;
>> while(1) {
>>   a.length=4096*4096;
>>   a=null;
>>
>>   Sleep(2000);
>> }
>>
>> It allocates memory, but its never gets freed, just keep going up, and
>> after 10 secs, memoryexception is thrown. I checked the pointer it is GC
>> safe (attributes is 10). Am i missing something?
>
> Are you compiling 64-bit code or 32-bit? It makes a huge difference.
>
> A real is going to be 16-byte aligned. This means real[4096*4096] is allocating 1/4GB per allocation. This consumes 1/16th of your total address space on 32-bit address space.
>
> The GC in D is conservative. This means that on 32-bit architecture, any 4-byte segment scanned by the GC (whether it's a pointer or not) that happens to point into the array you allocated is going to keep that memory pinned. Because any 4-byte garbage has a 1:16 chance of pointing there, odds are good that this will happen. Then it keeps happening, you eventually run out of memory.
>
> Now, some possible explanation of why GC.malloc might work while this does not:
>
> GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes through the append code. Because extending length is considered an appending event, it *over-allocates* by a certain factor, expecting you to keep appending. (this is how appending can be amortized O(1) performance). This means the amount of memory allocated is larger than 1/4GB. Therefore the chance of failing by setting a.length is going to be higher than with using GC.malloc directly, but it could fail with either case.
>
> Answers?
>
> 1. Manually free such a large array when done with it.
> 2. Do not allocate such large arrays.
> 3. Use 64-bit code generation (-m64 on DMD windows, even if you are on Win64) if possible.
>
> -Steve

TY for your answer.
I checked malloc and it is the same. It runs out if memory too.
Also tried with int[]->result is the same.
Memory is initialized to 0-s, so there is no pointer pointing to memory. At least at my side. Are u saiying, that most likely other GC allocated memory points to my arrays? Hm thats maybe true. If thats a case i have to manage memory myself.
Also what about scpoed alloc?

scoped int[] a=new int[whatever_big_size];

It sould free memory but it does not :( :(.

D is soo convinient. i never wanna use malloc again :D :D :D

On 1/22/15 9:40 AM, collerblade wrote:
> On Thursday, 22 January 2015 at 14:19:10 UTC, Steven Schveighoffer wrote:
>> On 1/22/15 8:44 AM, collerblade wrote:
>>> Dear D user, i have this code:
>>>
>>> import core.sys.windows.windows;
>>>
>>> real[] a;
>>> while(1) {
>>>   a.length=4096*4096;
>>>   a=null;
>>>
>>>   Sleep(2000);
>>> }
>>>
>>> It allocates memory, but its never gets freed, just keep going up, and
>>> after 10 secs, memoryexception is thrown. I checked the pointer it is GC
>>> safe (attributes is 10). Am i missing something?
>>
>> Are you compiling 64-bit code or 32-bit? It makes a huge difference.
>>
>> A real is going to be 16-byte aligned. This means real[4096*4096] is
>> allocating 1/4GB per allocation. This consumes 1/16th of your total
>> address space on 32-bit address space.
>>
>> The GC in D is conservative. This means that on 32-bit architecture,
>> any 4-byte segment scanned by the GC (whether it's a pointer or not)
>> that happens to point into the array you allocated is going to keep
>> that memory pinned. Because any 4-byte garbage has a 1:16 chance of
>> pointing there, odds are good that this will happen. Then it keeps
>> happening, you eventually run out of memory.
>>
>> Now, some possible explanation of why GC.malloc might work while this
>> does not:
>>
>> GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes
>> through the append code. Because extending length is considered an
>> appending event, it *over-allocates* by a certain factor, expecting
>> you to keep appending. (this is how appending can be amortized O(1)
>> performance). This means the amount of memory allocated is larger than
>> 1/4GB. Therefore the chance of failing by setting a.length is going to
>> be higher than with using GC.malloc directly, but it could fail with
>> either case.
>>
>> Answers?
>>
>> 1. Manually free such a large array when done with it.
>> 2. Do not allocate such large arrays.
>> 3. Use 64-bit code generation (-m64 on DMD windows, even if you are on
>> Win64) if possible.
>>
>> -Steve
>
> TY for your answer.

Before I respond, first question to answer is if you are using 32-bit compiled code. If not, then it's not typical for what you are doing to cause problems.

> I checked malloc and it is the same. It runs out if memory too.
> Also tried with int[]->result is the same.
> Memory is initialized to 0-s, so there is no pointer pointing to memory.
> At least at my side. Are u saiying, that most likely other GC allocated
> memory points to my arrays? Hm thats maybe true. If thats a case i have
> to manage memory myself.

Most likely it's a stack variable or global/TLS variable. Those are scanned too :)

Also note that reals are NOT initialized to 0, but NaN.

> Also what about scpoed alloc?
>
> scoped int[] a=new int[whatever_big_size];
>
> It sould free memory but it does not :( :(.

I would not use this, I think it's scheduled for deprecation, or at least the meaning is scheduled to change.

Note that a must go out of scope pointing at the array that was allocated in order to free the memory. You can't set it to null, because then nothing would be freed!

-Steve

Steven Schveighoffer:

>> scoped int[] a=new int[whatever_big_size];
>> ...
> I would not use this, I think it's scheduled for deprecation, or at least the meaning is scheduled to change.

As D introduces some limited static management of memory ownership, the semantics of scoped arrays could be fixed.

Bye,
bearophile

Thank all of u,
im going to use malloc.