Thread overview |
---|
July 24, 2017 Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Hi guys, I have a question on how to free large arrays in D after they're no longer needed. Let's say I have this: SomeKey[] getKeys() { SomeKey[] n; foreach(batching stuff...) { SomeKey newstuff = db.select!(SomeKey[])(...); // gets around 6M of these from db redis.send("newItems", newStuff.length); n ~= newStuff; destroy(newStuff); } return n; } void main() { SomeKey[] myarray = getKeys(); ulong[string] mappedValues; foreach(i, item; myArray) { mappedValues[item.name] = i; } // No longer need myarray, and we need the memory back destory(myarray); GC.collect; // Another memory intensive operation below } This seems to work for me to a point, but I notice my program still holds a lot in memory even after certain memory intensive arrays should have been collected. Is this just a hit and miss with the GC or am I doing something wrong? Uses around 40% less memory forcing the GC and cleaning arrays but seems like a lot is left over, especially for threads that have already finished(I've expect a thread to free all it's memory). I'm not sharing anything across threads besides a config struct. This is a simplified version of my app. At the moment when at around 9GB at it's peak instead of 19GB like before. |
July 24, 2017 Re: Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Posted in reply to Clinton | Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free |
July 24, 2017 Re: Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Posted in reply to rikki cattermole | rikki cattermole wrote:
> Tell it to free[0] the array directly.
>
> ```D
> T[] array;
>
> GC.free(array.ptr);
or just `delete arr;`. it is marked as "deprecated" in changelog, but who cares? it works.
|
July 24, 2017 Re: Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Posted in reply to rikki cattermole | On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:
> Stuff the GC.
> You don't need it to care about collecting (or destroying for that matter).
>
> Tell it to free[0] the array directly.
>
> ```D
> T[] array;
>
> GC.free(array.ptr);
> ```
>
> Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it.
>
> [0] http://dlang.org/phobos/core_memory.html#.GC.free
Thanks! I didn't know I could do that with the GC enabled.
|
July 24, 2017 Re: Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Posted in reply to rikki cattermole | On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:
> Stuff the GC.
> You don't need it to care about collecting (or destroying for that matter).
>
> Tell it to free[0] the array directly.
>
> ```D
> T[] array;
>
> GC.free(array.ptr);
> ```
>
> Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it.
>
> [0] http://dlang.org/phobos/core_memory.html#.GC.free
BTW, how would I do this with associative arrays?
|
July 24, 2017 Re: Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Posted in reply to Clinton | Clinton wrote:
> On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:
>> Stuff the GC.
>> You don't need it to care about collecting (or destroying for that matter).
>>
>> Tell it to free[0] the array directly.
>>
>> ```D
>> T[] array;
>>
>> GC.free(array.ptr);
>> ```
>>
>> Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it.
>>
>> [0] http://dlang.org/phobos/core_memory.html#.GC.free
>
> BTW, how would I do this with associative arrays?
you can't. you can `.clear` 'em, though, but this won't immediately free the internal storage.
|
July 24, 2017 Re: Cleaning/Releasing large arrays from memory | ||||
---|---|---|---|---|
| ||||
Posted in reply to ketmar | On Monday, 24 July 2017 at 18:01:19 UTC, ketmar wrote:
> Clinton wrote:
>
>> On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:
>>> Stuff the GC.
>>> You don't need it to care about collecting (or destroying for that matter).
>>>
>>> Tell it to free[0] the array directly.
>>>
>>> ```D
>>> T[] array;
>>>
>>> GC.free(array.ptr);
>>> ```
>>>
>>> Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it.
>>>
>>> [0] http://dlang.org/phobos/core_memory.html#.GC.free
>>
>> BTW, how would I do this with associative arrays?
>
> you can't. you can `.clear` 'em, though, but this won't immediately free the internal storage.
Meh.. Okay, well using these two should make things a lot better. Thanks for the help and clearing up my confusion.
|
Copyright © 1999-2021 by the D Language Foundation