On Monday, 12 April 2021 at 20:50:49 UTC, Per Nordlöw wrote:

What is a more aggressive collection compared to a normal collection? Unfortunately we still have no move support in D's GC because of its impreciseness.

On Monday, 12 April 2021 at 07:03:02 UTC, Sebastiaan Koppe wrote:

I knew this must be a more common problem :)

What I've found in the meantime:

• nice writeup of how GC actually works by Vladimir Panteleev - https://thecybershadow.net/d/Memory_Management_in_the_D_Programming_Language.pdf
  • described tool (https://github.com/CyberShadow/Diamond) would be very helpfull, but I assume it's for D1 and based on some old druntime fork :(
• we've implemented log rotation using std.zlib (by just foreach (chunk; fin.byChunk(4096).map!(x => c.compress(x))) fout.rawWrite(chunk);)
  • oh boy, don't use std.zlib.Compress that way, it allocates each chunk and for a large files it creates large GC memory peaks that sometimes doesn't go down
  • rewritten using direct etc.c.zlib completely out of GC
• currently testing with --DRT-gcopt=incPoolSize:0 as otherwise allocated page size multiplies with number of allocated pools * 3MB by default
• profile-gc is not much helpfull in this case as it only prints total allocated memory for each allocation on the application exit and as it's a long running service using many various libraries it's just hundreds of lines :)
  • I've considered to fork the process periodically, terminate it and rename the created profile statistics to at least see the differences between the states, but still not sure if it would help much
• as I understand the GC it uses different algorithm for small allocations and for large objects
  • small (<=2048)
    • it categorizes objects to fixed set of used sizes and for each uses whole memory page as bucket with free list from which it reserves memory on request
    • when the bucket is full, new page is allocated and allocations are provided from that
  • big - similar, but it allocates N pages as a pool

So If I understand it correctly when for example vibe-d initializes new fiber on some request, it's handled and fiber can be discarded it can easily lead to a scenario when fiber itself is allocated in one page, it's filled up during the request processing so new page is allocated and when cleaning, bucket with fiber cannot be cleaned up as it's added to a TaskFiber pool (with a fixed size). This way fiber's bucket would never be freed and easily never be used anymore during the application lifetime.

I'm not so sure if pages of small objects (or large) that are not completely empty can be reused as a new bucket or only free pages can be reused.

Does anyone has some insight of this?

Some kind of GC memory dump and analyzer tool as mentioned Diamond would be of tremendous help to diagnose this..

On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote:

You could try to get the stack traces of the allocating calls via eBPF. Maybe that leads to some new insights.

On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote:

As now it's possible to inject own GC to runtime, I've tried to compile debug version of the GC implementation and use that to check some debugging printouts with regard to small allocations.

The code I've used to simulate the behavior (with --DRT-gcopt=incPoolSize:0):

// fill 1st pool (minsize is 1MB so it's 256 pages, two small objects in each
ubyte[][512] pool1;
foreach (i; 0..pool1.length) pool1[i] = new ubyte[2042];

// no more space in pool1 so new one is allocated and also filled
ubyte[][512] pool2;
foreach (i; 0..pool2.length) pool2[i] = new ubyte[2042];

// free up first small object from first pool first page
pool1[0] = null;
GC.collect();

// allocate another small object
pool1[0] = new ubyte[2042];

And shortened result is:

// first pool allocations
GC::malloc(gcx = 0x83f6a0, size = 2044 bits = a, ti = TypeInfo_h)
Gcx::allocPage(bin = 13)
Gcx::allocPage(bin = 13)
Gcx::newPool(npages = 1, minPages = 256)
Gcx::allocPage(bin = 13)
  => p = 0x7f709e75c000

// second pool allocations
GC::malloc(gcx = 0x83f6a0, size = 2044 bits = a, ti = TypeInfo_h)
Gcx::allocPage(bin = 13)
Gcx.fullcollect()
preparing mark.
        scan stacks.
        scan roots[]
        scan ranges[]
                0x51e4e0 .. 0x53d580
        free'ing
        free'd 0 bytes, 0 pages from 1 pools
        recovered small pages = 0
Gcx::allocPage(bin = 13)
Gcx::newPool(npages = 1, minPages = 256)
Gcx::allocPage(bin = 13)
  => p = 0x7f709e65c000

// collection of first item of first pool first bin
GC.fullCollect()
Gcx.fullcollect()
preparing mark.
        scan stacks.
        scan roots[]
        scan ranges[]
                0x51e4e0 .. 0x53d580
        free'ing
        collecting 0x7f709e75c000
        free'd 0 bytes, 0 pages from 2 pools
        recovered small pages = 0

// and allocation to test where it'll allocate
GC::malloc(gcx = 0x83f6a0, size = 2044 bits = a, ti = TypeInfo_h)
  recover pool => 0x83d1a0
  => p = 0x7f709e75c000

So if I'm not mistaken, bins in pools are reused when there is a free space (ast last allocation reuses address from first item in first bin in first pool).

I've also tested big allocations with same result.

But if this is so, I don't understand why in our case there is 95% free space in the GC and it still grows from time to time.

At least rate of grow is minimized with --DRT-gcopt=incPoolSize:0

Would have to log and analyze the behavior more with a custom GC build..k

On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote:

The Hunt Framework is also suffering from this. We are trying to make a simple example to illustrate it.

On Wednesday, 14 April 2021 at 12:47:22 UTC, Heromyth wrote:

I think this is a rather serious problem for a language saying "GC is fine" 🌈.

Is there an issue for this?

On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote:

I've used bpftrace to do some of that stuff:
https://theartofmachinery.com/2019/04/26/bpftrace_d_gc.html

On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote:

I am wondering about your overall system design. Are there any "shared" data held in the REST API server which are not in the PostgreSQL database? I mean: If there are two potentially concurrent REST API calls is it necessary that these are handled both by the same operating system process?

On Wednesday, 14 April 2021 at 12:47:22 UTC, Heromyth wrote:

The bug has been fixed now[1]. It's because that a moudle named ArrayList wrapped the Array in std.container.array. We use a dynamic array instead.

We are not sure why this happens.

[1] https://github.com/huntlabs/hunt-extra/commit/903f3b4b529097013254342fdcfac6ba5543b401

On Wednesday, 3 November 2021 at 06:26:49 UTC, Heromyth wrote:

Hmm, interesting