Windows task manager shows constantly increasing memory usage of my D application, yet there is ZERO allocation done by it at the time. I have made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get notified when an array or a class object in constructed.
What could be the source of rapidly climbing (hundred kilobytes per second) memory usage?

Am 13.10.2013 14:08, schrieb develop32:
> Windows task manager shows constantly increasing memory usage of my D
> application, yet there is ZERO allocation done by it at the time. I have
> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
> notified when an array or a class object in constructed.
> What could be the source of rapidly climbing (hundred kilobytes per
> second) memory usage?

If your programm is a 32-bit application the GC most likely leaks memory because it thinks its still referenced. Try compilling your application for 64-bit. If the problem goes away it's not your fault.
A possible workaround for this situation is to allocate large blocks of memory which you know will not contain any pointers with GC.malloc manually and specifying the "don't scan" flag.

If the problem still exists in a 64-bit application you most likely continue to allocate memory that remains referenced. Make sure you don't have any infinitly growing arrays or other containers that still reference the memory you allocated.

-- 
Kind Regards
Benjamin Thaut

On Sunday, 13 October 2013 at 13:18:47 UTC, Benjamin Thaut wrote:
> Am 13.10.2013 14:08, schrieb develop32:
>> Windows task manager shows constantly increasing memory usage of my D
>> application, yet there is ZERO allocation done by it at the time. I have
>> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
>> notified when an array or a class object in constructed.
>> What could be the source of rapidly climbing (hundred kilobytes per
>> second) memory usage?
>
> If your programm is a 32-bit application the GC most likely leaks memory because it thinks its still referenced. Try compilling your application for 64-bit. If the problem goes away it's not your fault.
> A possible workaround for this situation is to allocate large blocks of memory which you know will not contain any pointers with GC.malloc manually and specifying the "don't scan" flag.
>
> If the problem still exists in a 64-bit application you most likely continue to allocate memory that remains referenced. Make sure you don't have any infinitly growing arrays or other containers that still reference the memory you allocated.

Thanks, will try 64-bit when possible.
No, there are no allocations in my code, I have set breakpoints wherever I can throughout it and there are no allocations being made every frame (its a video game). I don't use built-in dynamic array directly, have my own wrapper for it.
My question is, are there any more places in druntime that I can change so that I will get a notification when allocation occurs, besides _d_allocmemory?

On Sunday, 13 October 2013 at 13:18:47 UTC, Benjamin Thaut wrote:
> Am 13.10.2013 14:08, schrieb develop32:
>> Windows task manager shows constantly increasing memory usage of my D
>> application, yet there is ZERO allocation done by it at the time. I have
>> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
>> notified when an array or a class object in constructed.
>> What could be the source of rapidly climbing (hundred kilobytes per
>> second) memory usage?
>
> If your programm is a 32-bit application the GC most likely leaks memory because it thinks its still referenced. Try compilling your application for 64-bit. If the problem goes away it's not your fault.
> A possible workaround for this situation is to allocate large blocks of memory which you know will not contain any pointers with GC.malloc manually and specifying the "don't scan" flag.
>
> If the problem still exists in a 64-bit application you most likely continue to allocate memory that remains referenced. Make sure you don't have any infinitly growing arrays or other containers that still reference the memory you allocated.

I've added GC.collect and GC.minimize() at the end of each frame, looks like the growth has stopped, so its not a 32-bit problem yet.

On Sunday, 13 October 2013 at 13:46:13 UTC, develop32 wrote:
> My question is, are there any more places in druntime that I can change so that I will get a notification when allocation occurs, besides _d_allocmemory?

I set breakpoints on gc_malloc and gc_qalloc too. I'm pretty sure they catch more than d_allocmemory.

qalloc in particular catches accidental usage of array literals. Yes, they always allocate at runtime unless specifically marked static... i'd love for that to change!

Am 13.10.2013 15:52, schrieb develop32:
> On Sunday, 13 October 2013 at 13:18:47 UTC, Benjamin Thaut wrote:
>> Am 13.10.2013 14:08, schrieb develop32:
>>> Windows task manager shows constantly increasing memory usage of my D
>>> application, yet there is ZERO allocation done by it at the time. I have
>>> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
>>> notified when an array or a class object in constructed.
>>> What could be the source of rapidly climbing (hundred kilobytes per
>>> second) memory usage?
>>
>> If your programm is a 32-bit application the GC most likely leaks
>> memory because it thinks its still referenced. Try compilling your
>> application for 64-bit. If the problem goes away it's not your fault.
>> A possible workaround for this situation is to allocate large blocks
>> of memory which you know will not contain any pointers with GC.malloc
>> manually and specifying the "don't scan" flag.
>>
>> If the problem still exists in a 64-bit application you most likely
>> continue to allocate memory that remains referenced. Make sure you
>> don't have any infinitly growing arrays or other containers that still
>> reference the memory you allocated.
>
> I've added GC.collect and GC.minimize() at the end of each frame, looks
> like the growth has stopped, so its not a 32-bit problem yet.

Well the GC doesn't kick in always. It only collects memory when its neccessary (a certain limit is reached). For a game it is usually a good idea to run the GC every frame to avoid the ocasional long pause times that otherwise occur. Or you could work around the GC entierly.

Kind Regards
Benjamin Thaut

-- 
Kind Regards
Benjamin Thaut

On Sunday, 13 October 2013 at 14:01:09 UTC, Adam D. Ruppe wrote:
> On Sunday, 13 October 2013 at 13:46:13 UTC, develop32 wrote:
>> My question is, are there any more places in druntime that I can change so that I will get a notification when allocation occurs, besides _d_allocmemory?
>
> I set breakpoints on gc_malloc and gc_qalloc too. I'm pretty sure they catch more than d_allocmemory.
>
> qalloc in particular catches accidental usage of array literals. Yes, they always allocate at runtime unless specifically marked static... i'd love for that to change!

Thank you, that helped to find it.
Its as shame that usage of static arrays doesn't prevent heap allocation.

On Sunday, 13 October 2013 at 14:13:39 UTC, Benjamin Thaut wrote:
> Am 13.10.2013 15:52, schrieb develop32:
>> On Sunday, 13 October 2013 at 13:18:47 UTC, Benjamin Thaut wrote:
>>> Am 13.10.2013 14:08, schrieb develop32:
>>>> Windows task manager shows constantly increasing memory usage of my D
>>>> application, yet there is ZERO allocation done by it at the time. I have
>>>> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
>>>> notified when an array or a class object in constructed.
>>>> What could be the source of rapidly climbing (hundred kilobytes per
>>>> second) memory usage?
>>>
>>> If your programm is a 32-bit application the GC most likely leaks
>>> memory because it thinks its still referenced. Try compilling your
>>> application for 64-bit. If the problem goes away it's not your fault.
>>> A possible workaround for this situation is to allocate large blocks
>>> of memory which you know will not contain any pointers with GC.malloc
>>> manually and specifying the "don't scan" flag.
>>>
>>> If the problem still exists in a 64-bit application you most likely
>>> continue to allocate memory that remains referenced. Make sure you
>>> don't have any infinitly growing arrays or other containers that still
>>> reference the memory you allocated.
>>
>> I've added GC.collect and GC.minimize() at the end of each frame, looks
>> like the growth has stopped, so its not a 32-bit problem yet.
>
> Well the GC doesn't kick in always. It only collects memory when its neccessary (a certain limit is reached). For a game it is usually a good idea to run the GC every frame to avoid the ocasional long pause times that otherwise occur. Or you could work around the GC entierly.
>
> Kind Regards
> Benjamin Thaut

Indeed, I tried to work around it and use manual memory management a year ago, but everything felt so ugly that I decided to throw that and go the idiomatic way.
I'm bit surprised that even with GC running every frame I haven't noticed a drop in performance, even when using more than 1 gb RAM.

Am 13.10.2013 16:23, schrieb develop32:
> On Sunday, 13 October 2013 at 14:13:39 UTC, Benjamin Thaut wrote:
>> Am 13.10.2013 15:52, schrieb develop32:
>>> On Sunday, 13 October 2013 at 13:18:47 UTC, Benjamin Thaut wrote:
>>>> Am 13.10.2013 14:08, schrieb develop32:
>>>>> Windows task manager shows constantly increasing memory usage of my D
>>>>> application, yet there is ZERO allocation done by it at the time. I
>>>>> have
>>>>> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
>>>>> notified when an array or a class object in constructed.
>>>>> What could be the source of rapidly climbing (hundred kilobytes per
>>>>> second) memory usage?
>>>>
>>>> If your programm is a 32-bit application the GC most likely leaks
>>>> memory because it thinks its still referenced. Try compilling your
>>>> application for 64-bit. If the problem goes away it's not your fault.
>>>> A possible workaround for this situation is to allocate large blocks
>>>> of memory which you know will not contain any pointers with GC.malloc
>>>> manually and specifying the "don't scan" flag.
>>>>
>>>> If the problem still exists in a 64-bit application you most likely
>>>> continue to allocate memory that remains referenced. Make sure you
>>>> don't have any infinitly growing arrays or other containers that still
>>>> reference the memory you allocated.
>>>
>>> I've added GC.collect and GC.minimize() at the end of each frame, looks
>>> like the growth has stopped, so its not a 32-bit problem yet.
>>
>> Well the GC doesn't kick in always. It only collects memory when its
>> neccessary (a certain limit is reached). For a game it is usually a
>> good idea to run the GC every frame to avoid the ocasional long pause
>> times that otherwise occur. Or you could work around the GC entierly.
>>
>> Kind Regards
>> Benjamin Thaut
>
> Indeed, I tried to work around it and use manual memory management a
> year ago, but everything felt so ugly that I decided to throw that and
> go the idiomatic way.
> I'm bit surprised that even with GC running every frame I haven't
> noticed a drop in performance, even when using more than 1 gb RAM.

Did you profile how much time the GC takes every frame?
For me that was 8-9 ms every frame. That was not acceptable for my case.

-- 
Kind Regards
Benjamin Thaut

On Sunday, 13 October 2013 at 14:31:13 UTC, Benjamin Thaut wrote:
> Am 13.10.2013 16:23, schrieb develop32:
>> On Sunday, 13 October 2013 at 14:13:39 UTC, Benjamin Thaut wrote:
>>> Am 13.10.2013 15:52, schrieb develop32:
>>>> On Sunday, 13 October 2013 at 13:18:47 UTC, Benjamin Thaut wrote:
>>>>> Am 13.10.2013 14:08, schrieb develop32:
>>>>>> Windows task manager shows constantly increasing memory usage of my D
>>>>>> application, yet there is ZERO allocation done by it at the time. I
>>>>>> have
>>>>>> made a hook inside the rt/lifetime.d for _d_allocmemory, and I do get
>>>>>> notified when an array or a class object in constructed.
>>>>>> What could be the source of rapidly climbing (hundred kilobytes per
>>>>>> second) memory usage?
>>>>>
>>>>> If your programm is a 32-bit application the GC most likely leaks
>>>>> memory because it thinks its still referenced. Try compilling your
>>>>> application for 64-bit. If the problem goes away it's not your fault.
>>>>> A possible workaround for this situation is to allocate large blocks
>>>>> of memory which you know will not contain any pointers with GC.malloc
>>>>> manually and specifying the "don't scan" flag.
>>>>>
>>>>> If the problem still exists in a 64-bit application you most likely
>>>>> continue to allocate memory that remains referenced. Make sure you
>>>>> don't have any infinitly growing arrays or other containers that still
>>>>> reference the memory you allocated.
>>>>
>>>> I've added GC.collect and GC.minimize() at the end of each frame, looks
>>>> like the growth has stopped, so its not a 32-bit problem yet.
>>>
>>> Well the GC doesn't kick in always. It only collects memory when its
>>> neccessary (a certain limit is reached). For a game it is usually a
>>> good idea to run the GC every frame to avoid the ocasional long pause
>>> times that otherwise occur. Or you could work around the GC entierly.
>>>
>>> Kind Regards
>>> Benjamin Thaut
>>
>> Indeed, I tried to work around it and use manual memory management a
>> year ago, but everything felt so ugly that I decided to throw that and
>> go the idiomatic way.
>> I'm bit surprised that even with GC running every frame I haven't
>> noticed a drop in performance, even when using more than 1 gb RAM.
>
> Did you profile how much time the GC takes every frame?
> For me that was 8-9 ms every frame. That was not acceptable for my case.

Currently it is 3 ms, would love to drop that to one.
Though I'm using a very slow laptop, can't play any AAA game released in last 5 years.