June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to James Gray | On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
> have run into a memory leak
Something seems really off indeed. I've run this on Win64 with DMD (2.092) and LDC (1.22), without any extra cmdline options:
-----
import core.memory;
import core.stdc.stdio;
import std.range;
import std.format;
auto f(R)(R r) { return format("%s", r); }
int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); }
void printGCStats()
{
const stats = GC.stats;
const used = toMB(stats.usedSize);
const free = toMB(stats.freeSize);
const total = toMB(stats.usedSize + stats.freeSize);
printf(" GC stats: %dM used, %dM free, %dM total\n", used, free, total);
}
void main()
{
printGCStats();
while (true)
{
puts("Starting");
string str = f(iota(100_000_000));
printf(" string size: %dM\n", toMB(str.length));
str = null;
GC.collect();
printGCStats();
}
}
-----
Output with DMD (no change with the precise GC via `--DRT-gcopt=gc:precise`):
-----
GC stats: 0M used, 1M free, 1M total
Starting
string size: 943M
GC stats: 1168M used, 1139M free, 2306M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
-----
With LDC:
-----
GC stats: 0M used, 1M free, 1M total
Starting
string size: 943M
GC stats: 1168M used, 1139M free, 2306M total
Starting
string size: 943M
GC stats: 2335M used, 1288M free, 3623M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
-----
Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.
|
June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to kinke | => https://issues.dlang.org/show_bug.cgi?id=20983 |
June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to kinke | On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote: > On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote: >> have run into a memory leak > > Something seems really off indeed. I've run this on Win64 with DMD (2.092) and LDC (1.22), without any extra cmdline options: > > ----- > import core.memory; > import core.stdc.stdio; > import std.range; > import std.format; > > auto f(R)(R r) { return format("%s", r); } > > int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); } > > void printGCStats() > { > const stats = GC.stats; > const used = toMB(stats.usedSize); > const free = toMB(stats.freeSize); > const total = toMB(stats.usedSize + stats.freeSize); > printf(" GC stats: %dM used, %dM free, %dM total\n", used, free, total); > } > > void main() > { > printGCStats(); > > while (true) > { > puts("Starting"); > string str = f(iota(100_000_000)); > printf(" string size: %dM\n", toMB(str.length)); > str = null; > GC.collect(); > printGCStats(); > } > } > ----- > > Output with DMD (no change with the precise GC via `--DRT-gcopt=gc:precise`): > ----- > GC stats: 0M used, 1M free, 1M total > Starting > string size: 943M > GC stats: 1168M used, 1139M free, 2306M total > Starting > string size: 943M > GC stats: 1168M used, 2456M free, 3623M total > Starting > string size: 943M > GC stats: 1168M used, 2456M free, 3623M total > Starting > string size: 943M > GC stats: 1168M used, 2456M free, 3623M total > Starting > string size: 943M > GC stats: 1168M used, 2456M free, 3623M total > ----- > > With LDC: > ----- > GC stats: 0M used, 1M free, 1M total > Starting > string size: 943M > GC stats: 1168M used, 1139M free, 2306M total > Starting > string size: 943M > GC stats: 2335M used, 1288M free, 3623M total > Starting > string size: 943M > GC stats: 2335M used, 2605M free, 4940M total > Starting > string size: 943M > GC stats: 2335M used, 2605M free, 4940M total > Starting > string size: 943M > GC stats: 2335M used, 2605M free, 4940M total > ----- > > Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore. Thank you for doing this. I hope my example doesn't obscure what you show here. (I borrowed some of your code). I have produced something which essentially reproduces my problem. ----------- import std.range; import std.algorithm; import std.format; import std.stdio; import core.thread; import core.memory; struct Node { Node* next; Node* prev; ulong val; } Node* insertAfter(Node* cur, ulong val) { Node* node = new Node; if (cur != null) { node.next = cur.next; node.prev = cur; cur.next = node; node.next.prev = node; } else { node.next = node; node.prev = node; } node.val = val; return node; } int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); } void printGCStats() { const stats = GC.stats; const used = toMB(stats.usedSize); const free = toMB(stats.freeSize); const total = toMB(stats.usedSize + stats.freeSize); writef(" GC stats: %dM used, %dM free, %dM total\n", used, free, total); } void main() { while(true) { printGCStats(); writeln("Starting"); Node* dll; dll = iota(200000000).fold!((c,x)=>insertAfter(c,x))(dll); writef("Last element %s\n", dll.val); dll = null; writeln("Done"); GC.collect(); GC.minimize(); Thread.sleep( dur!("msecs")( 10000 ) ); } } ---------- With DMD this produces: GC stats: 0M used, 0M free, 0M total Starting Last element 199999999 Done GC stats: 6104M used, 51M free, 6155M total Starting Last element 199999999 Done GC stats: 12207M used, 28M free, 12235M total With LDC2 this produces: GC stats: 0M used, 0M free, 0M total Starting Last element 199999999 Done GC stats: 6104M used, 51M free, 6155M total Starting Last element 199999999 Done GC stats: 12207M used, 28M free, 12235M total |
June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to kinke | On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
> Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.
Hrm... What happens if you call collect() twice?
|
June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to Stanislav Blinov | On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote:
> On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
>
>> Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore.
>
> Hrm... What happens if you call collect() twice?
Nothing changes, even when collecting 5 times at the end of each iteration. In the filed testcase, I've extracted the stack ref to a dedicated function, so that there really shouldn't be any refs on the stack (this is unoptimized code after all...).
|
June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to kinke | On Saturday, 27 June 2020 at 16:03:12 UTC, kinke wrote: > On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote: >> Hrm... What happens if you call collect() twice? > > Nothing changes, even when collecting 5 times at the end of each iteration. In the filed testcase, I've extracted the stack ref to a dedicated function, so that there really shouldn't be any refs on the stack (this is unoptimized code after all...). Here on Linux, the double collection results in this output: GC stats: 0M used, 0M free, 0M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total |
June 27, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to James Gray | On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
> On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
>> [...]
>
> Thank you for doing this. I hope my example doesn't obscure what you show here.
> (I borrowed some of your code).
>
> [...]
In case it helps, setting all the next and previous pointers in the link list to null allows the garbage collector to collect in the above code.
|
June 30, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to James Gray | On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
> I have produced something which essentially reproduces my problem.
What is the problem? Do you have a leak or you want to know how GC works?
|
July 20, 2020 Re: Garbage collection | ||||
---|---|---|---|---|
| ||||
Posted in reply to Kagamin | On Tuesday, 30 June 2020 at 06:16:26 UTC, Kagamin wrote:
> On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
>> I have produced something which essentially reproduces my problem.
>
> What is the problem? Do you have a leak or you want to know how GC works?
I have managed to resolve my problem (which was a memory leak). My code uses a large data structure similar to a link list and the garbage collector was not collecting it. However,
if I set all the "links" between the nodes in the data structure to null it is then collected.
|
Copyright © 1999-2021 by the D Language Foundation