On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
> have run into a memory leak

Something seems really off indeed. I've run this on Win64 with DMD (2.092) and LDC (1.22), without any extra cmdline options:

-----
import core.memory;
import core.stdc.stdio;
import std.range;
import std.format;

auto f(R)(R r) { return format("%s", r); }

int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); }

void printGCStats()
{
    const stats = GC.stats;
    const used = toMB(stats.usedSize);
    const free = toMB(stats.freeSize);
    const total = toMB(stats.usedSize + stats.freeSize);
    printf("  GC stats: %dM used, %dM free, %dM total\n", used, free, total);
}

void main()
{
    printGCStats();

    while (true)
    {
        puts("Starting");
        string str = f(iota(100_000_000));
        printf("  string size: %dM\n", toMB(str.length));
        str = null;
        GC.collect();
        printGCStats();
    }
}
-----

Output with DMD (no change with the precise GC via `--DRT-gcopt=gc:precise`):
-----
  GC stats: 0M used, 1M free, 1M total
Starting
  string size: 943M
  GC stats: 1168M used, 1139M free, 2306M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
-----

With LDC:
-----
  GC stats: 0M used, 1M free, 1M total
Starting
  string size: 943M
  GC stats: 1168M used, 1139M free, 2306M total
Starting
  string size: 943M
  GC stats: 2335M used, 1288M free, 3623M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
-----

Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.

=> https://issues.dlang.org/show_bug.cgi?id=20983

On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
> On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
>> have run into a memory leak
>
> Something seems really off indeed. I've run this on Win64 with DMD (2.092) and LDC (1.22), without any extra cmdline options:
>
> -----
> import core.memory;
> import core.stdc.stdio;
> import std.range;
> import std.format;
>
> auto f(R)(R r) { return format("%s", r); }
>
> int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); }
>
> void printGCStats()
> {
>     const stats = GC.stats;
>     const used = toMB(stats.usedSize);
>     const free = toMB(stats.freeSize);
>     const total = toMB(stats.usedSize + stats.freeSize);
>     printf("  GC stats: %dM used, %dM free, %dM total\n", used, free, total);
> }
>
> void main()
> {
>     printGCStats();
>
>     while (true)
>     {
>         puts("Starting");
>         string str = f(iota(100_000_000));
>         printf("  string size: %dM\n", toMB(str.length));
>         str = null;
>         GC.collect();
>         printGCStats();
>     }
> }
> -----
>
> Output with DMD (no change with the precise GC via `--DRT-gcopt=gc:precise`):
> -----
>   GC stats: 0M used, 1M free, 1M total
> Starting
>   string size: 943M
>   GC stats: 1168M used, 1139M free, 2306M total
> Starting
>   string size: 943M
>   GC stats: 1168M used, 2456M free, 3623M total
> Starting
>   string size: 943M
>   GC stats: 1168M used, 2456M free, 3623M total
> Starting
>   string size: 943M
>   GC stats: 1168M used, 2456M free, 3623M total
> Starting
>   string size: 943M
>   GC stats: 1168M used, 2456M free, 3623M total
> -----
>
> With LDC:
> -----
>   GC stats: 0M used, 1M free, 1M total
> Starting
>   string size: 943M
>   GC stats: 1168M used, 1139M free, 2306M total
> Starting
>   string size: 943M
>   GC stats: 2335M used, 1288M free, 3623M total
> Starting
>   string size: 943M
>   GC stats: 2335M used, 2605M free, 4940M total
> Starting
>   string size: 943M
>   GC stats: 2335M used, 2605M free, 4940M total
> Starting
>   string size: 943M
>   GC stats: 2335M used, 2605M free, 4940M total
> -----
>
> Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.

Thank you for doing this. I hope my example doesn't obscure what you show here.
(I borrowed some of your code).

I have produced something which essentially reproduces my problem.

-----------
import std.range;
import std.algorithm;
import std.format;
import std.stdio;
import core.thread;
import core.memory;


struct Node {
 Node* next;
 Node* prev;
 ulong val;
}

Node* insertAfter(Node* cur, ulong val) {
 Node* node = new Node;
 if (cur != null) {
  node.next = cur.next;
  node.prev = cur;
  cur.next = node;
  node.next.prev = node;
 }
 else {
  node.next = node;
  node.prev = node;
 }
 node.val = val;
 return node;
}

int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); }

void printGCStats()
{
 const stats = GC.stats;
 const used = toMB(stats.usedSize);
 const free = toMB(stats.freeSize);
 const total = toMB(stats.usedSize + stats.freeSize);
 writef("  GC stats: %dM used, %dM free, %dM total\n", used, free, total);
}


void main()
{
 while(true)
 {
  printGCStats();
  writeln("Starting");
  Node* dll;
  dll = iota(200000000).fold!((c,x)=>insertAfter(c,x))(dll);
  writef("Last element %s\n", dll.val);
  dll = null;
  writeln("Done");
  GC.collect();
  GC.minimize();
  Thread.sleep( dur!("msecs")( 10000 ) );
 }
}
----------
With DMD this produces:

  GC stats: 0M used, 0M free, 0M total
Starting
Last element 199999999
Done
  GC stats: 6104M used, 51M free, 6155M total
Starting
Last element 199999999
Done
  GC stats: 12207M used, 28M free, 12235M total

With LDC2 this produces:

  GC stats: 0M used, 0M free, 0M total
Starting
Last element 199999999
Done
  GC stats: 6104M used, 51M free, 6155M total
Starting
Last element 199999999
Done
  GC stats: 12207M used, 28M free, 12235M total

On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:

> Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.

Hrm... What happens if you call collect() twice?

On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote:
> On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
>
>> Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.
>
> Hrm... What happens if you call collect() twice?

Nothing changes, even when collecting 5 times at the end of each iteration. In the filed testcase, I've extracted the stack ref to a dedicated function, so that there really shouldn't be any refs on the stack (this is unoptimized code after all...).

On Saturday, 27 June 2020 at 16:03:12 UTC, kinke wrote:
> On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote:

>> Hrm... What happens if you call collect() twice?
>
> Nothing changes, even when collecting 5 times at the end of each iteration. In the filed testcase, I've extracted the stack ref to a dedicated function, so that there really shouldn't be any refs on the stack (this is unoptimized code after all...).

Here on Linux, the double collection results in this output:

  GC stats: 0M used, 0M free, 0M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total

On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
> On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
>> [...]
>
> Thank you for doing this. I hope my example doesn't obscure what you show here.
> (I borrowed some of your code).
>
> [...]

In case it helps, setting all the next and previous pointers in the link list to null allows the garbage collector to collect in the above code.

On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
> I have produced something which essentially reproduces my problem.

What is the problem? Do you have a leak or you want to know how GC works?

On Tuesday, 30 June 2020 at 06:16:26 UTC, Kagamin wrote:
> On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
>> I have produced something which essentially reproduces my problem.
>
> What is the problem? Do you have a leak or you want to know how GC works?

I have managed to resolve my problem (which was a memory leak). My code uses a large data structure similar to a link list and the garbage collector was not collecting it. However,
if I set all the "links" between the nodes in the data structure to null it is then collected.