I am writing a web application using vibe.d (not sure whether that is relevant or not), and have run into a memory leak. I wrote the following code to try and replicate the problem.

import std.algorithm;
import std.range;
import std.format;
import std.stdio;
import core.thread;
import core.memory;

auto f(R)(R r) {
 return format("%s", r);
}

void main()
{
 while(true)
 {
  writeln("Starting");
  {
   auto str = f(iota(100000000).map!(x=>x+1));
  }
  writeln("Done");
  GC.collect();
  Thread.sleep( dur!("msecs")( 30000 ) );
 }
}

 It doesn't replicate the problem but it still doesn't behave as I would expect. I would expect the memory usage of this code to grow and shrink. However, I find that the memory usage grows to about 1.5GB and never decreases. Is there something I am not understanding?

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
> I am writing a web application using vibe.d (not sure whether that is relevant or not), and have run into a memory leak. I wrote the following code to try and replicate the problem.
>
> [...]

I now compiled the same code above with ldc2 and it is leaking.
Any suggestions?

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:

> I find that the memory usage grows to about 1.5GB and never decreases. Is there something I am not understanding?

How are you measuring that? GC.collect() does not necessarily release the pages to the OS. For that, there's the GC.minimize().

On Saturday, 27 June 2020 at 11:00:58 UTC, Stanislav Blinov wrote:
> On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
>
>> I find that the memory usage grows to about 1.5GB and never decreases. Is there something I am not understanding?
>
> How are you measuring that? GC.collect() does not necessarily release the pages to the OS. For that, there's the GC.minimize().

I am measuring the memory usage using top from the command line.
GC.minimize() does seem to stop the leak. But it doesn't explain why
the program isn't releasing essentially all the memory between calls
to f (it using around 2GB ram all the time). Is there a way of achieving that?

On Saturday, 27 June 2020 at 11:11:38 UTC, James Gray wrote:

>
> I am measuring the memory usage using top from the command line.
> GC.minimize() does seem to stop the leak. But it doesn't explain why
> the program isn't releasing essentially all the memory between calls
> to f (it using around 2GB ram all the time). Is there a way of achieving that?

It's not a leak. The GC allocates memory as it needs it and holds on to it. When something is collected, the GC can reuse then released memory when it needs it.

On Saturday, 27 June 2020 at 11:11:38 UTC, James Gray wrote:

> I am measuring the memory usage using top from the command line.
> GC.minimize() does seem to stop the leak.

That is not a memory leak. That's the allocator keeping pages for itself to not have to go to the kernel every time you allocate.

> But it doesn't explain why the program isn't releasing essentially all the memory between calls to f (it using around 2GB ram all the time).

Allocators usually don't do that. They keep (at least some) memory mapped to make allocations more efficient.

> Is there a way of achieving that?

I would think collect + minimize should do the trick. Just keep in mind that that's grossly inefficient.

On 27/6/20 13:21, Stanislav Blinov wrote:
> 
> I would think collect + minimize should do the trick. Just keep in mind that that's grossly inefficient.

If you are using linux, have in mind that the memory is often not returned to the OS even after a (libc) free. If you check with tools like `top`, it'll still show as assigned to the process.

What I had to do (both in D and in C/C++) was to call malloc_trim [1] manually to have the memory actually sent back to the OS.

[1]: https://man7.org/linux/man-pages/man3/malloc_trim.3.html

On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote:

> If you are using linux, have in mind that the memory is often not returned to the OS even after a (libc) free.

That's a good observation. Although a GC implementation is not required to actually use malloc, so depending on that falls into "know what you're doing" territory :)

On Saturday, 27 June 2020 at 12:07:19 UTC, Stanislav Blinov wrote:
> On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote:
>
>> If you are using linux, have in mind that the memory is often not returned to the OS even after a (libc) free.
>
> That's a good observation. Although a GC implementation is not required to actually use malloc, so depending on that falls into "know what you're doing" territory :)

Thanks for the help, but unfortunately it isn't stopping memory usage growing in the original app. I will try and build a minimal example. In the meantime perhaps someone can suggest how I might figure out what is going on. Repeating the same action is giving memory usage growth as follows. 1.7GB first time (which now drops to about 1GB), then 2.7GB dropping to about 2GB and so on.

On Saturday, 27 June 2020 at 12:53:01 UTC, James Gray wrote:
> On Saturday, 27 June 2020 at 12:07:19 UTC, Stanislav Blinov wrote:
>> On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote:
>>
>>> [...]
>>
>> That's a good observation. Although a GC implementation is not required to actually use malloc, so depending on that falls into "know what you're doing" territory :)
>
> Thanks for the help, but unfortunately it isn't stopping memory usage growing in the original app. I will try and build a minimal example. In the meantime perhaps someone can suggest how I might figure out what is going on. Repeating the same action is giving memory usage growth as follows. 1.7GB first time (which now drops to about 1GB), then 2.7GB dropping to about 2GB and so on.

Which eventually results in mac os running out of memory.