February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
 Posted in reply to rikki cattermole | On Saturday, 6 February 2021 at 09:42:38 UTC, rikki cattermole wrote:
> On 06/02/2021 3:32 PM, frame wrote:
>> [...]
>
> This won't do anything.
>
>> [...]
>
> Don't forget to stdout.flush; Otherwise stuff can get caught in the buffer before erroring out.
>
>> [...]
>
> Turn on the precise GC, 32bit is a bit too small of a range and you can get false positives like in this case (at least looks like it).
For reference, how does one turn on precise GC?
| |||
Permalink
ReplyFebruary 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
 Posted in reply to Imperatorn | On Saturday, 6 February 2021 at 11:20:18 UTC, Imperatorn wrote: > On Saturday, 6 February 2021 at 09:42:38 UTC, rikki cattermole wrote: >> On 06/02/2021 3:32 PM, frame wrote: >>> [...] >> >> This won't do anything. >> >>> [...] >> >> Don't forget to stdout.flush; Otherwise stuff can get caught in the buffer before erroring out. >> >>> [...] >> >> Turn on the precise GC, 32bit is a bit too small of a range and you can get false positives like in this case (at least looks like it). > > For reference, how does one turn on precise GC? https://dlang.org/spec/garbage.html#gc_config Strange things happens: - precise scanning dont change result - OOM @ same round 27 --DRT-gcopt=help wont show used gc implementation, also cleanup type not printed - maxPoolSize:N dont limit total size of GC - in gc:manual mode GC.collect() not releasing memory When i print free GC memory, it seems to memory leaking writefln("Usage: %.2f MiB (free %.2f MiB) / collected: %d", (cast(double) GC.stats.usedSize) / 1_048_576, (cast(double) GC.stats.freeSize) / 1_048_576, GC.profileStats.numCollections); stdout.flush(); | |||
February 07, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
 Posted in reply to Siemargl | On 07/02/2021 12:38 AM, Siemargl wrote: > On Saturday, 6 February 2021 at 11:20:18 UTC, Imperatorn wrote: >> On Saturday, 6 February 2021 at 09:42:38 UTC, rikki cattermole wrote: >>> On 06/02/2021 3:32 PM, frame wrote: >>>> [...] >>> >>> This won't do anything. >>> >>>> [...] >>> >>> Don't forget to stdout.flush; Otherwise stuff can get caught in the buffer before erroring out. >>> >>>> [...] >>> >>> Turn on the precise GC, 32bit is a bit too small of a range and you can get false positives like in this case (at least looks like it). >> >> For reference, how does one turn on precise GC? > > https://dlang.org/spec/garbage.html#gc_config > > Strange things happens: > - precise scanning dont change result - OOM @ same round 27 Okay, its still seeing something is alive then. > --DRT-gcopt=help wont show used gc implementation, also cleanup type not printed https://github.com/dlang/druntime/blob/master/src/core/gc/config.d#L36 > - maxPoolSize:N dont limit total size of GC Shouldn't change anything, except make OOM happen faster. > - in gc:manual mode GC.collect() not releasing memory https://github.com/dlang/druntime/blob/master/src/gc/impl/manual/gc.d#L84 > > When i print free GC memory, it seems to memory leaking > writefln("Usage: %.2f MiB (free %.2f MiB) / collected: %d", > (cast(double) GC.stats.usedSize) / 1_048_576, > (cast(double) GC.stats.freeSize) / 1_048_576, GC.profileStats.numCollections); > stdout.flush(); I've compiled and ran it under ldc. Dmd in 32bit mode is certainly doing something that the GC doesn't appreciate. I can reproduce it with -m32mscoff as well. So yeah dmd specific all right. | |||
February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
 Posted in reply to rikki cattermole | On Saturday, 6 February 2021 at 13:30:03 UTC, rikki cattermole wrote: > Okay, its still seeing something is alive then. That's why I used the scope guard. I know it shouldn't have any effect but I want to give the GC an extra hint ;) > I've compiled and ran it under ldc. Dmd in 32bit mode is certainly doing something that the GC doesn't appreciate. I can reproduce it with -m32mscoff as well. So yeah dmd specific all right. The sad story never ends. But seriously are there no runtime tests before releasing the next DMD? The GC is a main feature of D and such things give a bad impression. | |||
February 07, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
Posted in reply to frame | On 07/02/2021 4:22 AM, frame wrote: > On Saturday, 6 February 2021 at 13:30:03 UTC, rikki cattermole wrote: > >> Okay, its still seeing something is alive then. > > That's why I used the scope guard. I know it shouldn't have any effect but I want to give the GC an extra hint ;) The GC shouldn't be aware of the scope guard. It expands out into a try finally block. >> I've compiled and ran it under ldc. Dmd in 32bit mode is certainly doing something that the GC doesn't appreciate. I can reproduce it with -m32mscoff as well. So yeah dmd specific all right. > > The sad story never ends. > > But seriously are there no runtime tests before releasing the next DMD? The GC is a main feature of D and such things give a bad impression. Nah, this is old. It is also bad D code. Allocate up front and then set. T[] buffer; buffer.length = 1_000_000; foreach(i, v; source[0 .. 1_000_000]) { buffer[i] = someOp(v); } This will be significantly faster, as it won't require allocating more than once and will prevent heap fragmentation which 32bit is severely affected by (hence why precise GC is important for testing on this target). | |||
February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
Posted in reply to rikki cattermole | On Saturday, 6 February 2021 at 15:45:47 UTC, rikki cattermole wrote: > The GC shouldn't be aware of the scope guard. It expands out into a try finally block. But .length = 0 should. > Nah, this is old. It is also bad D code. > > Allocate up front and then set. I agree but it has to work anyway. New users do not ask about how they can write friendly GC code. > This will be significantly faster, as it won't require allocating more than once and will prevent heap fragmentation which 32bit is severely affected by (hence why precise GC is important for testing on this target). Ah, yes. I read about it in another thread. This is a specific problem for any GC that work on the same way, not only in D. But if it's a known problem, the default architecture should be 64bit where things just work. 32bit should be an opt-in then. Default settings should work out of the box. If not - it's bad for reputation of the language. | |||
February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
 Posted in reply to frame | On Saturday, 6 February 2021 at 17:50:18 UTC, frame wrote: > On Saturday, 6 February 2021 at 15:45:47 UTC, rikki cattermole wrote: > > Default settings should work out of the box. If not - it's bad for reputation of the language. Given that 32-bit has been the default on Windows for D's entire lifetime, I don't expect this is so common of an issue. More harmful would be requiring Visual Studio to compile with the default settings. That said, dub already uses -m64 by default on Windows. And the goal is to enable 64-bit in dmd on Windows by default. That's the reason the MinGW-based link libraries and the LDC linker were added to the distribution, so that it will work out of the box. I don't know what the timetable is supposed to be, but at some point after that is rock solid, the switch to 64-bit by default will be made. | |||
February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
Posted in reply to frame | On Saturday, 6 February 2021 at 17:50:18 UTC, frame wrote:>
> But .length = 0 should.
>
What do you expect it to do in this case?
| |||
February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
Posted in reply to Mike Parker | On Saturday, 6 February 2021 at 19:10:14 UTC, Mike Parker wrote:
> On Saturday, 6 February 2021 at 17:50:18 UTC, frame wrote:
>> On Saturday, 6 February 2021 at 15:45:47 UTC, rikki cattermole wrote:
>
>>
>> Default settings should work out of the box. If not - it's bad for reputation of the language.
>
> Given that 32-bit has been the default on Windows for D's entire lifetime, I don't expect this is so common of an issue. More harmful would be requiring Visual Studio to compile with the default settings.
>
> That said, dub already uses -m64 by default on Windows. And the goal is to enable 64-bit in dmd on Windows by default. That's the reason the MinGW-based link libraries and the LDC linker were added to the distribution, so that it will work out of the box. I don't know what the timetable is supposed to be, but at some point after that is rock solid, the switch to 64-bit by default will be made.
This example seems been as a corner uglyness of AA realisation.
64bit heal a problem, but uses about 500Mb of RAM continuosly for this simple example, so its only patching sinkin' ship :-(
| |||
February 06, 2021 Re: Minimize GC memory footprint | ||||
|---|---|---|---|---|
| ||||
Posted in reply to Mike Parker | On Saturday, 6 February 2021 at 19:10:14 UTC, Mike Parker wrote:
> On Saturday, 6 February 2021 at 17:50:18 UTC, frame wrote:
Sorry, i forgot mem leak. Or maybe i incorrect understand Gc counters
So log
Usage: 698.46 MiB (free 187.42 MiB) / collected: 14 Round: 12 Usage: 759.72 MiB (free 184.16 MiB) / collected: 15 ... Usage: 802.00 MiB (free 202.88 MiB) / collected: 16 Usage: 871.38 MiB (free 197.50 MiB) / collected: 17 Usage: 935.64 MiB (free 200.24 MiB) / collected: 18 Usage: 995.86 MiB (free 210.02 MiB) / collected: 19 Usage: 1071.70 MiB (free 207.18 MiB) / collected: 20 Usage: 1139.22 MiB (free 215.66 MiB) / collected: 21 Usage: 1214.83 MiB (free 219.05 MiB) / collected: 22 Usage: 1297.88 MiB (free 218.00 MiB) / collected: 23 Usage: 1378.43 MiB (free 222.45 MiB) / collected: 24 Usage: 1416.72 MiB (free 184.16 MiB) / collected: 25 Usage: 1459.00 MiB (free 229.88 MiB) / collected: 26 Usage: 1501.28 MiB (free 187.60 MiB) / collected: 27 Usage: 1543.55 MiB (free 236.32 MiB) / collected: 28 Usage: 1585.83 MiB (free 194.05 MiB) / collected: 29 Usage: 1628.11 MiB (free 245.77 MiB) / collected: 30 Round: 28 Usage: 1670.39 MiB (free 203.49 MiB) / collected: 31 Round: 29
So GC.used is growing, but GC.free is stable
| |||
Copyright © 1999-2021 by the D Language Foundation