Thread overview
[Issue 3284] snn linked programs never release memory back to the OS
Aug 20, 2016
Cauterite
Aug 21, 2016
Cauterite
Aug 22, 2016
Sobirari Muhomori
Aug 22, 2016
Cauterite
Aug 22, 2016
Sobirari Muhomori
Nov 01, 2021
Ate Eskola
Jan 19, 2023
Walter Bright
Jan 19, 2023
Vladimir Panteleev
June 09, 2015
https://issues.dlang.org/show_bug.cgi?id=3284

Andrei Alexandrescu <andrei@erdani.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|D1 & D2                     |D2

--
August 20, 2016
https://issues.dlang.org/show_bug.cgi?id=3284

Cauterite <cauterite@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |cauterite@gmail.com

--- Comment #2 from Cauterite <cauterite@gmail.com> ---
I highly suspect this issue has already been resolved. Here's a simple test:

import core.memory;
void main() {
    // allocate and free lots of 10MB arrays
    foreach (_; 0 .. 1000) {
        auto x = GC.calloc(10_000_000, 1);
        GC.free(x);
        x = null;
    };

    import std.c.stdio;
    printf("done\n"); getchar();
};

if you remove the `GC.free(x)` the working set will grow to >1GB. if you leave
it in, memory usage is normal ~15MB or so.
so the GC is definitely releasing pages back to the OS when it deallocates.

And before you ask, yes I am linking with SNN.lib

--
August 21, 2016
https://issues.dlang.org/show_bug.cgi?id=3284

Cauterite <cauterite@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #3 from Cauterite <cauterite@gmail.com> ---
reopen if there's still a way to trigger this bug

--
August 22, 2016
https://issues.dlang.org/show_bug.cgi?id=3284

--- Comment #4 from Sobirari Muhomori <dfj1esp02@sneakemail.com> ---
As I understand, the test is as follows:

import core.memory, core.stdc.stdio;
void main()
{
    void*[100] arrays;

    // allocate and free lots of 10MB arrays
    foreach (ref x; arrays)
    {
        x = GC.calloc(10_000_000, 1);
    }

    foreach (ref x; arrays)
    {
        GC.free(x);
        x = null;
    }

    puts("must have a small working set here");
    getchar();
}

(didn't test)
i.e. the working set never shrinks, so your best strategy is not let it ever
grow.

--
August 22, 2016
https://issues.dlang.org/show_bug.cgi?id=3284

Cauterite <cauterite@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |---

--- Comment #5 from Cauterite <cauterite@gmail.com> ---
My mistake; your adjusted test does in fact leave a massive working set. I think I misunderstood the original bug report, because when you call GC.minimize() it does successfully reduce working set to normal size.

So the exact problem then is that the GC doesn't call minimize() automatically
when it is appropriate. Currently, minimize() is only ever called when an
allocation fails.

Ideally the GC should minimize during collection whenever the amount of unused reserved memory reaches some threshold. With my limited knowledge of the GC's internals this sounds like a simple patch, so I might give it a crack soon. Lest this bug remain open for 7 whole years.

--
August 22, 2016
https://issues.dlang.org/show_bug.cgi?id=3284

--- Comment #6 from Sobirari Muhomori <dfj1esp02@sneakemail.com> ---
The original description probably complains about C malloc too - worth checking.

--
November 01, 2021
https://issues.dlang.org/show_bug.cgi?id=3284

Ate Eskola <Ajieskola@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Ajieskola@gmail.com
          Component|phobos                      |druntime

--- Comment #7 from Ate Eskola <Ajieskola@gmail.com> ---
The function in question reside in the core namespace, so reclassifying as a DRuntime issue.

--
January 19, 2023
https://issues.dlang.org/show_bug.cgi?id=3284

--- Comment #8 from Walter Bright <bugzilla@digitalmars.com> ---
snn.lib uses the Win32 HeapAlloc and HeapFree routines for malloc/free:

https://github.com/DigitalMars/dmc/blob/master/src/HEAP32/malloc.c#L22

--
January 19, 2023
https://issues.dlang.org/show_bug.cgi?id=3284

Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |INVALID

--- Comment #9 from Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> ---
I think this needs more focus/clarity of what is broken and needs to be fixed. Is it the C functions or the GC?

Note that we don't use the libc allocators in the GC, we use the OS APIs directly.

Also worth noting that heap allocators, whether new (GC), malloc (libc), or
HeapAlloc (OS), are all vulnerable to fragmentation. Programs can only release
memory back to the OS if the entire page is free.

It's possible that we no longer release memory to the OS after a GC cycle, because in many applications any released memory is going to be immediately requested again. Applications which require memory in bursts are comparatively rare. I recall that we no longer reserve memory from the OS - though it was a thing we could do and it aligned with the GC design, it was not useful in any measurable way, so it was removed.

--