Thread overview
GC collecting "too much"..
Mar 25, 2012
simendsjo
Mar 25, 2012
bearophile
Mar 26, 2012
simendsjo
Mar 26, 2012
simendsjo
Mar 26, 2012
Timon Gehr
Mar 26, 2012
simendsjo
Mar 29, 2012
simendsjo
Mar 29, 2012
simendsjo
March 25, 2012
I'm doing some coding against a c library, and Ds GC keeps collecting c owned objects (I think - disabling the GC makes everything work)
But how can I figure out what the GC is (read: I am) fucking up?
I have some to!string(c_struct_field) and format("%s", c_struct_field) and field = c_fields[i] etc etc
March 25, 2012
On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
> I'm doing some coding against a c library, and Ds GC keeps collecting c owned objects (I think - disabling the GC makes everything work)

Three alternative solutions:
- Allocate from the C heap the memory that C will need to use, and free it manually or in a struct destructor (RAII) or with scope(exit).
- Keep a pointer to the D-GC memory in the D code too.
- In core.memory there are ways to disable scanning of a memory zone. Maybe it's usable for your purposes too.

Bye,
bearophile
March 26, 2012
On Sun, 25 Mar 2012 22:18:02 +0200, bearophile <bearophileHUGS@lycos.com> wrote:

> On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
>> I'm doing some coding against a c library, and Ds GC keeps collecting c owned objects (I think - disabling the GC makes everything work)
>
> Three alternative solutions:
> - Allocate from the C heap the memory that C will need to use, and free it manually or in a struct destructor (RAII) or with scope(exit).
> - Keep a pointer to the D-GC memory in the D code too.
> - In core.memory there are ways to disable scanning of a memory zone. Maybe it's usable for your purposes too.
>
> Bye,
> bearophile


I've been able to find where the code fails, but now I don't understand what's happening at all.
Is calling GC.collect() from an extern(C) function undefined?

The following code just starts a mongoose web server and tries to run GC.collect in the handler.

extern(C) void* cb(mg_event event, mg_connection* conn, mg_request_info* request_info)
{
    GC.collect(); // segfault
    return null;
}

void main()
{
    auto opts = ["listening_ports", "6969"].map!(toUTFz!(char*))().array();
    mg_start(&cb, null, cast(const(char**))opts);
    GC.collect(); // no problem
    readln();
}

If I collect memory from main(), it works as expected (removing the collect from cb of course)
while(readln().chomp() != "q") { GC.collect(); }

The documentation in core.memory is a bit sparse on how the GC works. Are there any articles on the D GC?
March 26, 2012
On Mon, 26 Mar 2012 10:13:35 +0200, simendsjo <simendsjo@gmail.com> wrote:

> On Sun, 25 Mar 2012 22:18:02 +0200, bearophile <bearophileHUGS@lycos.com> wrote:
>
>> On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
>>> I'm doing some coding against a c library, and Ds GC keeps collecting c owned objects (I think - disabling the GC makes everything work)
>>
>> Three alternative solutions:
>> - Allocate from the C heap the memory that C will need to use, and free it manually or in a struct destructor (RAII) or with scope(exit).
>> - Keep a pointer to the D-GC memory in the D code too.
>> - In core.memory there are ways to disable scanning of a memory zone. Maybe it's usable for your purposes too.
>>
>> Bye,
>> bearophile
>
>
> I've been able to find where the code fails, but now I don't understand what's happening at all.
> Is calling GC.collect() from an extern(C) function undefined?
>
> The following code just starts a mongoose web server and tries to run GC.collect in the handler.
>
> extern(C) void* cb(mg_event event, mg_connection* conn, mg_request_info* request_info)
> {
>      GC.collect(); // segfault
>      return null;
> }
>
> void main()
> {
>      auto opts = ["listening_ports", "6969"].map!(toUTFz!(char*))().array();
>      mg_start(&cb, null, cast(const(char**))opts);
>      GC.collect(); // no problem
>      readln();
> }
>
> If I collect memory from main(), it works as expected (removing the collect from cb of course)
> while(readln().chomp() != "q") { GC.collect(); }
>
> The documentation in core.memory is a bit sparse on how the GC works. Are there any articles on the D GC?

It seems threads created in the c library is totally unknown to D. How can I make D aware of these threads when there is no library support for it?
March 26, 2012
On 03/26/2012 11:55 AM, simendsjo wrote:
> It seems threads created in the c library is totally unknown to D. How
> can I make D aware of these threads when there is no library support for
> it?

You may be looking for this:
http://dlang.org/phobos/core_thread.html#thread_attachThis
March 26, 2012
On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr@gmx.ch> wrote:

> On 03/26/2012 11:55 AM, simendsjo wrote:
>> It seems threads created in the c library is totally unknown to D. How
>> can I make D aware of these threads when there is no library support for
>> it?
>
> You may be looking for this:
> http://dlang.org/phobos/core_thread.html#thread_attachThis

Thanks, but I tried that too and couldn't get it to work.

I added the following:

extern(C) handler()
{
    synchronized // needed here to avoid the GC to collect while attaching thread?
    {
        if(!Thread.getThis()) // thread unknown to D
        {
            thread_attachThis();
            assert(Thread.getThis()); // now D knows about it
        }
    }
    GC.collect(); // still segfaults
}

Actually, using attachThis segfaults GC.collect() outside the thread handling code too.
March 29, 2012
On Mon, 26 Mar 2012 20:15:40 +0200, simendsjo <simendsjo@gmail.com> wrote:

> On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr@gmx.ch> wrote:
>
>> On 03/26/2012 11:55 AM, simendsjo wrote:
>>> It seems threads created in the c library is totally unknown to D. How
>>> can I make D aware of these threads when there is no library support for
>>> it?
>>
>> You may be looking for this:
>> http://dlang.org/phobos/core_thread.html#thread_attachThis
>
> Thanks, but I tried that too and couldn't get it to work.
>
> I added the following:
>
> extern(C) handler()
> {
>      synchronized // needed here to avoid the GC to collect while attaching thread?
>      {
>          if(!Thread.getThis()) // thread unknown to D
>          {
>              thread_attachThis();
>              assert(Thread.getThis()); // now D knows about it
>          }
>      }
>      GC.collect(); // still segfaults
> }
>
> Actually, using attachThis segfaults GC.collect() outside the thread handling code too.


Some info from gdb:

[New Thread 0xf7cfcb70 (LWP 18614)]
[New Thread 0xf74fbb70 (LWP 18615)]

Program received signal SIGUSR1, User defined signal 1.
0xf7fdd430 in __kernel_vsyscall ()

(gdb) info threads
  Id   Target Id         Frame
  3    Thread 0xf74fbb70 (LWP 18615) "main" 0xf7fdd430 in __kernel_vsyscall ()
  2    Thread 0xf7cfcb70 (LWP 18614) "main" 0xf7fdd430 in __kernel_vsyscall ()
* 1    Thread 0xf7dfd9c0 (LWP 18611) "main" 0xf7fdd430 in __kernel_vsyscall ()

(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf74fbb70 (LWP 18615)]
0x080855b4 in gc.gcx.Gcx.mark() ()
(gdb) info threads
  Id   Target Id         Frame
* 3    Thread 0xf74fbb70 (LWP 18615) "main" 0x080855b4 in gc.gcx.Gcx.mark() ()
  2    Thread 0xf7cfcb70 (LWP 18614) "main" 0xf7fdd430 in __kernel_vsyscall ()
  1    Thread 0xf7dfd9c0 (LWP 18611) "main" 0xf7fdd430 in __kernel_vsyscall ()

------------

And if I change the code to:
extern(C) handler()
{
    GC.collect();
}

[New Thread 0xf7cfcb70 (LWP 21454)]
[New Thread 0xf74fbb70 (LWP 21455)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf74fbb70 (LWP 21455)]
0x0808970e in core.thread.thread_suspendAll() ()
(gdb) info threads
  Id   Target Id         Frame
* 3    Thread 0xf74fbb70 (LWP 21455) "main" 0x0808970e in core.thread.thread_suspendAll() ()
  2    Thread 0xf7cfcb70 (LWP 21454) "main" 0xf7fdd430 in __kernel_vsyscall ()
  1    Thread 0xf7dfd9c0 (LWP 21451) "main" 0xf7fdd430 in __kernel_vsyscall ()


-----

Any idea where I should go from here?
March 29, 2012
On Thu, 29 Mar 2012 14:14:10 +0200, simendsjo <simendsjo@gmail.com> wrote:

> On Mon, 26 Mar 2012 20:15:40 +0200, simendsjo <simendsjo@gmail.com> wrote:
>
>> On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr@gmx.ch> wrote:
>>
>>> On 03/26/2012 11:55 AM, simendsjo wrote:
>>>> It seems threads created in the c library is totally unknown to D. How
>>>> can I make D aware of these threads when there is no library support for
>>>> it?
>>>
>>> You may be looking for this:
>>> http://dlang.org/phobos/core_thread.html#thread_attachThis
>>
>> Thanks, but I tried that too and couldn't get it to work.
>>
>> I added the following:
>>
>> extern(C) handler()
>> {
>>      synchronized // needed here to avoid the GC to collect while attaching thread?
>>      {
>>          if(!Thread.getThis()) // thread unknown to D
>>          {
>>              thread_attachThis();
>>              assert(Thread.getThis()); // now D knows about it
>>          }
>>      }
>>      GC.collect(); // still segfaults
>> }
>>
>> Actually, using attachThis segfaults GC.collect() outside the thread handling code too.

I compiled with a debug build of druntime, and it fails on dereferencing a pointer, pbot in gcx.d:2511:

void mark(void *pbot, void *ptop, int nRecurse)
        void **p1 = cast(void **)pbot;
(...)
        for (; p1 < p2; p1++)
        {
            auto p = cast(byte *)(*p1); // segfault!
(...)

gdb) print p1
$10 = (void **) 0xf74fc000
(gdb) print *p1
$11 = (void *) 0x0

Some more dbg info:

(gdb) bt
#0  0x08098753 in gc.gcx.Gcx.mark() (this=0x80df030, nRecurse=64, ptop=0xffffd190, pbot=0xf74f4df4) at src/gc/gcx.d:2511
#1  0x080986d6 in gc.gcx.Gcx.mark() (this=0x80df030, ptop=0xffffd190, pbot=0xf74f4df4) at src/gc/gcx.d:2494
#2  0x0809ef95 in core.thread.thread_scanAll() (this=0xf74f4cf8, p2=0xffffd190, p1=0xf74f4df4, type=<incomplete type>)
    at src/core/thread.d:2743
#3  0x080a6488 in thread_scanAllType (scan=..., curStackTop=0xf74f4df4) at src/core/thread.d:2698
#4  0x0809ef68 in thread_scanAll (scan=..., curStackTop=0xf74f4df4) at src/core/thread.d:2746
#5  0x08098c34 in gc.gcx.Gcx.fullcollect() (this=0x80df030, stackTop=0xf74f4df4) at src/gc/gcx.d:2776
#6  0x080989f9 in gc.gcx.Gcx.fullcollectshell() (this=0x80df030) at src/gc/gcx.d:2662
#7  0x08096da1 in gc.gcx.GC.fullCollect() (this=0x80df018) at src/gc/gcx.d:1372
#8  0x08086da6 in gc_collect () at src/gc/gc.d:154
#9  0x08085380 in core.memory.GC.collect() () at src/core/memory.d:101
#10 0x08080e2c in c_callback (event=<incomplete type>, conn=0x80ebec0, request_info=0x80ebec0) at main.d:54
#11 0xf7fd28c1 in handle_request(mg_connection*) () from ./libmongoose.so
#12 0xf7fd4b7d in worker_thread(mg_context*) () from ./libmongoose.so
#13 0xf7f8dd1a in start_thread (arg=0xf74fbb70) at pthread_create.c:304
#14 0xf7ed5e4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb) l 2511
2506            uint changes = 0;
2507
2508            //printf("marking range: %p -> %p\n", pbot, ptop);
2509            for (; p1 < p2; p1++)
2510            {
2511                auto p = cast(byte *)(*p1);
2512
2513                //if (log) debug(PRINTF) printf("\tmark %p\n", p);
2514                if (p >= minAddr && p < maxAddr)
2515                {



> And if I change the code to:
> extern(C) handler()
> {
>      GC.collect();
> }
(snip)
Checking with debug build of druntime, and I see it tries to suspend(thread_getThis()). This gives a null reference error as thread_getThis() returns null when it's not attached.