Permalink
Reply
Hi,
we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer.
First guess was some memory leak so we've added periodic call to:
GC.collect();
GC.minimize();
malloc_trim(0);
And when called very often (ie 500ms) service memory stays stable and doesn't grow, so it doesn't look as a memory leak (or at least not that fast to explain that service can go from 90MB to 900MB in 2 days).
But this is very bad for performance so we've prolonged the interval for example to 30 seconds and now memory still goes up (not that dramatically but still).
Stats of the GC when it growed up between 30s are these (logged after GC.collect() and GC.minimize():
GC.stats: usedSize=9994864, freeSize=92765584, total=102760448
GC.stats: usedSize=11571456, freeSize=251621120, total=263192576
Before the grow it has 88MB free space from 98MB total allocated.
After 30s it has 239MB free from 251MB allocated.
So it wastes a lot of free space which it can't return back to OS for some reason.
Can these numbers be caused by memory fragmentation? There is probably a lot of small allocations (postgresql query, result processing and REST API json serialization).
Only explanation that makes some sense is that in some operation there is required memory allocations that can't be fulfilled with current memory pool (ie due to the memory fragmentation in it) and then it allocates some data in new memory segment that can't be returned afterwards as it still holds the 'live' data. But that should be freed too at some point and GC should minimize (if another request doesn'cause allocation in the 'wrong' page again).
But still, the amount of used vs free memory seems wrong, its a whooping 95% of free space that can't be minimized :-o. I have a problem imagining some fragmentation in it.
Are there any tools that can help diagnosing this more?
Also note that malloc_trim is not called from the GC itself and as internally used malloc handles it's own memory pool it has it's own quirks with returning unused memory back to the OS (it does it only on free() in some cases).
See for example: http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html
Behavior of malloc can be controlled with mallopt with:
M_TRIM_THRESHOLD
When the amount of contiguous free memory at the top of the heap grows sufficiently large, free(3) employs sbrk(2) to release this memory back to the system. (This can be useful in programs that continue to execute
for a long period after freeing a significant amount of memory.) The M_TRIM_THRESHOLD parameter specifies the minimum size (in bytes) that this block of memory must reach before sbrk(2) is used to trim the heap.
The default value for this parameter is 128*1024.
But the default is 128kB free heap memory block for trim to activate but when malloc_trim is called manually, a much larger block of memory is often returned. Thats puzzling too :)
