Jump to page: 1 2
Thread overview
GC issue? List.pool overwritten by allocated object
2 days ago
kinke
4 hours ago
Denis Feklushkin
2 days ago
Ali Çehreli
3 days ago

Hi!

It seems I have encountered a bug that is hard to understand and fix without knowlenge of the GC internals. But I have some code that reproduces the problem well. I made a branch so that everyone can try it (see below)

Usual (not very beautiful, yes) code that I do for fun. During run it creates and destroys various objects, everything is as usual, it does nothing strange, no manipulations with the GC, except collect() called once or twice, and I also often call destroy(). Also no multithreading, but Vulkan API is used and it implicitly creates threads. On sucessful run code displays window with two rotating pictures.

For small objects my code regularly and deterministically gets into a situation when at some point the value of core.internal.gc.impl.conservative.gc.List.pool pointer is overwritten by garbage. Using gdb I tracked that after appropriate List.pool is created and written, at some time this piece of memory is overwritten by a newly allocated D object. As result, garbage value of List.pool is used at next gc.Gcx.smallAlloc() call and SIGSEGV occurs.

(For tracking I used gdb option "set scheduler-locking on" - it seems that this is what makes List* address the same every time, which makes debugging much easier.)

I tried to turn on --d-debug=INVARIANT --d-debug=SENTINEL --d-debug=MEMSTOMP for druntime. All these options confirming the problem. Sometimes issue shifted either to a newly added GC invariant as assert error, assert(*sentinel_pre(p) == SENTINEL_PRE) error, or problem manifests itself not immediately after launch, but after a few seconds of the application's operation when allocating object. But it still repeats every time - that is, this is not a heisenbug.

Perhaps all this is the result of an error somewhere else, which results in this behavior. That is, if some my code (or third party) corrupts something that affects to allocation? But it seems that I do not do any hacks, any manipulations with pointers, etc.

Everything is reproduced on DMD and LDC. I use LDC for debugging because it is easy to switch between different druntimes in it.

I couldn't reduce code to highlight issue. So here is how to reproduce:

$ git clone --branch=move_to_ldc2 https://github.com/denizzzka/pukan.git
(ensure you are on commit f7e5293cdeb14da911bc337e281378b92ca39f25)
$ cd pukan #important!
$ dub run

For now I tested my code only on Linux, so it might not work in Windows at all.
Issue is reproduceable on druntime supplied with:
DMD64 D Compiler v2.111.0
LDC 1.40.1

3 days ago
On 13/05/2025 3:31 AM, Denis Feklushkin wrote:
> Perhaps all this is the result of an error somewhere else, which results in this behavior. That is, if some my code (or third party) corrupts something that affects to allocation? But it seems that I do not do any hacks, any manipulations with pointers, etc.

This is pretty heavily used code in druntime, my immediate thought is what is your code doing to cause this (I didn't see anything obvious)?

Try using ldc's address sanitizer to see if that finds something.

Otherwise try dividing and conquering to find what triggers it down to the statement.

3 days ago
On Monday, 12 May 2025 at 15:40:01 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 13/05/2025 3:31 AM, Denis Feklushkin wrote:
>> Perhaps all this is the result of an error somewhere else, which results in this behavior. That is, if some my code (or third party) corrupts something that affects to allocation? But it seems that I do not do any hacks, any manipulations with pointers, etc.
>
> This is pretty heavily used code in druntime, my immediate thought is what is your code doing to cause this (I didn't see anything obvious)?

Yes, of course I understand perfectly well. And it seems to me that I am not doing anything "reprehensible".

Failure causes when code simply allocates by the "new" keyword, which internally calls GC's smallAlloc(). Fails on different points, depending on compilation options, compiled-in debug facilities, sanitizers, etc. And if class what allocation causes error manually moved into "heavy" by adding 64kB size field just another class allocation causes same error.

Maybe somewhere after destroy() I successfully write something into destroyed object field and this corrupts internal GC data? I'll try to remove everything destroy() calls right now

> Try using ldc's address sanitizer to see if that finds something.

Nothing, sanitizer only highlights point where is pointer to pool is broken:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==36969==ERROR: AddressSanitizer: SEGV on unknown address 0x000100000006 (pc 0x55c3ba20eac3 bp 0x523000005500 sp 0x7ffd7cb0fcf0 T0)
==36969==The signal is caused by a READ memory access.
    #0 0x55c3ba20eac3 in _D4core8internal2gc4impl12conservativeQw3Gcx10smallAllocMFNbmKmkxC8TypeInfoZPv (/home/denizzz/Dev/pukan3D/pukan+0x33fac3) (BuildId: 25246214f82ed318a32cc136c8e965179f4dcad3)

0x000100000006 is garbage value, placed by wrong allocation at swapchain.d:92:
s = new SyncFramesInFlight(device, commandBuffers[i]);
(if used debug configuration described in origin message):

3 days ago
On 13/05/2025 6:22 AM, Denis Feklushkin wrote:
>     Try using ldc's address sanitizer to see if that finds something.
> 
> Nothing, sanitizer only highlights point where is pointer to pool is broken:
> 
> 
>   AddressSanitizer:DEADLYSIGNAL
> 
> ==36969==ERROR: AddressSanitizer: SEGV on unknown address 0x000100000006 (pc 0x55c3ba20eac3 bp 0x523000005500 sp 0x7ffd7cb0fcf0 T0) ==36969==The signal is caused by a READ memory access. #0 0x55c3ba20eac3 in _D4core8internal2gc4impl12conservativeQw3Gcx10smallAllocMFNbmKmkxC8TypeInfoZPv (/home/denizzz/Dev/pukan3D/pukan+0x33fac3) (BuildId: 25246214f82ed318a32cc136c8e965179f4dcad3)
> 
> 0x000100000006 is garbage value, placed by wrong allocation at swapchain.d:92: s = new SyncFramesInFlight(device, commandBuffers[i]); (if used debug configuration described in origin message):

This is useful information, now you can minify your code to what causes it.

I suggest throwing dustmite at it, and looking for that segfault.

https://github.com/CyberShadow/DustMite

3 days ago
On Monday, 12 May 2025 at 18:22:16 UTC, Denis Feklushkin wrote:

> Maybe somewhere after destroy() I successfully write something into destroyed object field and this corrupts internal GC data? I'll try to remove everything destroy() calls right now

Removed all destroy() calls - nothing changed

3 days ago

On Monday, 12 May 2025 at 18:22:16 UTC, Denis Feklushkin wrote:

>

On Monday, 12 May 2025 at 15:40:01 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

On 13/05/2025 3:31 AM, Denis Feklushkin wrote:

>

Perhaps all this is the result of an error somewhere else, which results in this behavior. That is, if some my code (or third party) corrupts something that affects to allocation? But it seems that I do not do any hacks, any manipulations with pointers, etc.

This is pretty heavily used code in druntime, my immediate thought is what is your code doing to cause this (I didn't see anything obvious)?

Yes, of course I understand perfectly well. And it seems to me that I am not doing anything "reprehensible".

The "reprehensible" thing that almost always causes GC issues is use after free because you are interacting with C memory. I have not diagnosed the specific issue, but you are very much using some C libs to do complicated things.

I literally just fixed a bug at work that existed for 3 years because a GC object was being freed slightly early. Issue was -- we were using C memory that was owned by a GC object that was no longer referenced. GC runs -- destructor frees memory -- use after free.

Not saying this isn't some latent GC bug that has existed for a while. But the good news is that it's repeatable, so it should be possible to track down.

>

Failure causes when code simply allocates by the "new" keyword, which internally calls GC's smallAlloc(). Fails on different points, depending on compilation options, compiled-in debug facilities, sanitizers, etc. And if class what allocation causes error manually moved into "heavy" by adding 64kB size field just another class allocation causes same error.

Having errors very much points at the problem happening before new is called. If it's not always failing in the same spot, that sounds a lot like memory corruption. And very often the corruption happens long before the explosion.

>

Maybe somewhere after destroy() I successfully write something into destroyed object field and this corrupts internal GC data? I'll try to remove everything destroy() calls right now

First thing I would rule out is C memory being used to refer to GC objects. Focus on places where C memory is allocated, especially with things like callbacks + data pointer.

Another cause, as I mentioned above, is using a GC object to manage C memory, and then forgetting the GC object but remembering the C memory.

-Steve

3 days ago

First of all, I want to thank everyone for their help. And, yes - I forgot to check obvious things before I was deep into GC

On Monday, 12 May 2025 at 21:29:10 UTC, Steven Schveighoffer wrote:

>

Having errors very much points at the problem happening before new is called. If it's not always failing in the same spot, that sounds a lot like memory corruption. And very often the corruption happens long before the explosion.

>

First thing I would rule out is C memory being used to refer to GC objects. Focus on places where C memory is allocated, especially with things like callbacks + data pointer.

Another cause, as I mentioned above, is using a GC object to manage C memory, and then forgetting the GC object but remembering the C memory.

-Steve

So far I have done two things in this direction:

  1. I called GC.disable() at start of main()
  2. destroy() was removed from the code

It seems like this should eliminate probability of use after freeing and referring from C to D objects? Nothing has changed, issue is still here

dustmite needs a lot of time - I launched it but I'm still waiting

3 days ago

On Tuesday, 13 May 2025 at 10:08:02 UTC, Denis Feklushkin wrote:

>

It seems like this should eliminate probability of use after freeing and referring from C to D objects?

More precisely, probability of damage of internal GC structures

3 days ago
On 13/05/2025 10:08 PM, Denis Feklushkin wrote:
> dustmite needs a lot of time - I launched it but I'm still waiting

What I normally do is help dustmite.

Run it for a little bit, dup the file system, remove some cycles or dependent usage of a variable, run it again.

It can't always break chains, so it needs a bit of help.

3 days ago
On Tuesday, 13 May 2025 at 10:14:17 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 13/05/2025 10:08 PM, Denis Feklushkin wrote:
>> dustmite needs a lot of time - I launched it but I'm still waiting
>
> What I normally do is help dustmite.
>
> Run it for a little bit, dup the file system, remove some cycles or dependent usage of a variable, run it again.
>
> It can't always break chains, so it needs a bit of help.

Yes, I already did some for this.

At first (for the whole night) I decided to launch "dub dustmite", which (it seems) works only over whole sources with dependencies, and it was too big amount of work
« First   ‹ Prev
1 2