View mode: basic / threaded / horizontal-split · Log in · Help
March 25, 2012
GC collecting "too much"..
I'm doing some coding against a c library, and Ds GC keeps collecting c  
owned objects (I think - disabling the GC makes everything work)
But how can I figure out what the GC is (read: I am) fucking up?
I have some to!string(c_struct_field) and format("%s", c_struct_field) and  
field = c_fields[i] etc etc
March 25, 2012
Re: GC collecting "too much"..
On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
> I'm doing some coding against a c library, and Ds GC keeps 
> collecting c owned objects (I think - disabling the GC makes 
> everything work)

Three alternative solutions:
- Allocate from the C heap the memory that C will need to use, 
and free it manually or in a struct destructor (RAII) or with 
scope(exit).
- Keep a pointer to the D-GC memory in the D code too.
- In core.memory there are ways to disable scanning of a memory 
zone. Maybe it's usable for your purposes too.

Bye,
bearophile
March 26, 2012
Re: GC collecting "too much"..
On Sun, 25 Mar 2012 22:18:02 +0200, bearophile <bearophileHUGS@lycos.com>  
wrote:

> On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
>> I'm doing some coding against a c library, and Ds GC keeps collecting c  
>> owned objects (I think - disabling the GC makes everything work)
>
> Three alternative solutions:
> - Allocate from the C heap the memory that C will need to use, and free  
> it manually or in a struct destructor (RAII) or with scope(exit).
> - Keep a pointer to the D-GC memory in the D code too.
> - In core.memory there are ways to disable scanning of a memory zone.  
> Maybe it's usable for your purposes too.
>
> Bye,
> bearophile


I've been able to find where the code fails, but now I don't understand  
what's happening at all.
Is calling GC.collect() from an extern(C) function undefined?

The following code just starts a mongoose web server and tries to run  
GC.collect in the handler.

extern(C) void* cb(mg_event event, mg_connection* conn, mg_request_info*  
request_info)
{
    GC.collect(); // segfault
    return null;
}

void main()
{
    auto opts = ["listening_ports", "6969"].map!(toUTFz!(char*))().array();
    mg_start(&cb, null, cast(const(char**))opts);
    GC.collect(); // no problem
    readln();
}

If I collect memory from main(), it works as expected (removing the  
collect from cb of course)
while(readln().chomp() != "q") { GC.collect(); }

The documentation in core.memory is a bit sparse on how the GC works. Are  
there any articles on the D GC?
March 26, 2012
Re: GC collecting "too much"..
On Mon, 26 Mar 2012 10:13:35 +0200, simendsjo <simendsjo@gmail.com> wrote:

> On Sun, 25 Mar 2012 22:18:02 +0200, bearophile  
> <bearophileHUGS@lycos.com> wrote:
>
>> On Sunday, 25 March 2012 at 19:15:05 UTC, simendsjo wrote:
>>> I'm doing some coding against a c library, and Ds GC keeps collecting  
>>> c owned objects (I think - disabling the GC makes everything work)
>>
>> Three alternative solutions:
>> - Allocate from the C heap the memory that C will need to use, and free  
>> it manually or in a struct destructor (RAII) or with scope(exit).
>> - Keep a pointer to the D-GC memory in the D code too.
>> - In core.memory there are ways to disable scanning of a memory zone.  
>> Maybe it's usable for your purposes too.
>>
>> Bye,
>> bearophile
>
>
> I've been able to find where the code fails, but now I don't understand  
> what's happening at all.
> Is calling GC.collect() from an extern(C) function undefined?
>
> The following code just starts a mongoose web server and tries to run  
> GC.collect in the handler.
>
> extern(C) void* cb(mg_event event, mg_connection* conn, mg_request_info*  
> request_info)
> {
>      GC.collect(); // segfault
>      return null;
> }
>
> void main()
> {
>      auto opts = ["listening_ports",  
> "6969"].map!(toUTFz!(char*))().array();
>      mg_start(&cb, null, cast(const(char**))opts);
>      GC.collect(); // no problem
>      readln();
> }
>
> If I collect memory from main(), it works as expected (removing the  
> collect from cb of course)
> while(readln().chomp() != "q") { GC.collect(); }
>
> The documentation in core.memory is a bit sparse on how the GC works.  
> Are there any articles on the D GC?

It seems threads created in the c library is totally unknown to D. How can  
I make D aware of these threads when there is no library support for it?
March 26, 2012
Re: GC collecting "too much"..
On 03/26/2012 11:55 AM, simendsjo wrote:
> It seems threads created in the c library is totally unknown to D. How
> can I make D aware of these threads when there is no library support for
> it?

You may be looking for this:
http://dlang.org/phobos/core_thread.html#thread_attachThis
March 26, 2012
Re: GC collecting "too much"..
On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr@gmx.ch> wrote:

> On 03/26/2012 11:55 AM, simendsjo wrote:
>> It seems threads created in the c library is totally unknown to D. How
>> can I make D aware of these threads when there is no library support for
>> it?
>
> You may be looking for this:
> http://dlang.org/phobos/core_thread.html#thread_attachThis

Thanks, but I tried that too and couldn't get it to work.

I added the following:

extern(C) handler()
{
    synchronized // needed here to avoid the GC to collect while attaching  
thread?
    {
        if(!Thread.getThis()) // thread unknown to D
        {
            thread_attachThis();
            assert(Thread.getThis()); // now D knows about it
        }
    }
    GC.collect(); // still segfaults
}

Actually, using attachThis segfaults GC.collect() outside the thread  
handling code too.
March 29, 2012
Re: GC collecting "too much"..
On Mon, 26 Mar 2012 20:15:40 +0200, simendsjo <simendsjo@gmail.com> wrote:

> On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr@gmx.ch> wrote:
>
>> On 03/26/2012 11:55 AM, simendsjo wrote:
>>> It seems threads created in the c library is totally unknown to D. How
>>> can I make D aware of these threads when there is no library support  
>>> for
>>> it?
>>
>> You may be looking for this:
>> http://dlang.org/phobos/core_thread.html#thread_attachThis
>
> Thanks, but I tried that too and couldn't get it to work.
>
> I added the following:
>
> extern(C) handler()
> {
>      synchronized // needed here to avoid the GC to collect while  
> attaching thread?
>      {
>          if(!Thread.getThis()) // thread unknown to D
>          {
>              thread_attachThis();
>              assert(Thread.getThis()); // now D knows about it
>          }
>      }
>      GC.collect(); // still segfaults
> }
>
> Actually, using attachThis segfaults GC.collect() outside the thread  
> handling code too.


Some info from gdb:

[New Thread 0xf7cfcb70 (LWP 18614)]
[New Thread 0xf74fbb70 (LWP 18615)]

Program received signal SIGUSR1, User defined signal 1.
0xf7fdd430 in __kernel_vsyscall ()

(gdb) info threads
  Id   Target Id         Frame
  3    Thread 0xf74fbb70 (LWP 18615) "main" 0xf7fdd430 in  
__kernel_vsyscall ()
  2    Thread 0xf7cfcb70 (LWP 18614) "main" 0xf7fdd430 in  
__kernel_vsyscall ()
* 1    Thread 0xf7dfd9c0 (LWP 18611) "main" 0xf7fdd430 in  
__kernel_vsyscall ()

(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf74fbb70 (LWP 18615)]
0x080855b4 in gc.gcx.Gcx.mark() ()
(gdb) info threads
  Id   Target Id         Frame
* 3    Thread 0xf74fbb70 (LWP 18615) "main" 0x080855b4 in  
gc.gcx.Gcx.mark() ()
  2    Thread 0xf7cfcb70 (LWP 18614) "main" 0xf7fdd430 in  
__kernel_vsyscall ()
  1    Thread 0xf7dfd9c0 (LWP 18611) "main" 0xf7fdd430 in  
__kernel_vsyscall ()

------------

And if I change the code to:
extern(C) handler()
{
    GC.collect();
}

[New Thread 0xf7cfcb70 (LWP 21454)]
[New Thread 0xf74fbb70 (LWP 21455)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf74fbb70 (LWP 21455)]
0x0808970e in core.thread.thread_suspendAll() ()
(gdb) info threads
  Id   Target Id         Frame
* 3    Thread 0xf74fbb70 (LWP 21455) "main" 0x0808970e in  
core.thread.thread_suspendAll() ()
  2    Thread 0xf7cfcb70 (LWP 21454) "main" 0xf7fdd430 in  
__kernel_vsyscall ()
  1    Thread 0xf7dfd9c0 (LWP 21451) "main" 0xf7fdd430 in  
__kernel_vsyscall ()


-----

Any idea where I should go from here?
March 29, 2012
Re: GC collecting "too much"..
On Thu, 29 Mar 2012 14:14:10 +0200, simendsjo <simendsjo@gmail.com> wrote:

> On Mon, 26 Mar 2012 20:15:40 +0200, simendsjo <simendsjo@gmail.com>  
> wrote:
>
>> On Mon, 26 Mar 2012 17:10:34 +0200, Timon Gehr <timon.gehr@gmx.ch>  
>> wrote:
>>
>>> On 03/26/2012 11:55 AM, simendsjo wrote:
>>>> It seems threads created in the c library is totally unknown to D. How
>>>> can I make D aware of these threads when there is no library support  
>>>> for
>>>> it?
>>>
>>> You may be looking for this:
>>> http://dlang.org/phobos/core_thread.html#thread_attachThis
>>
>> Thanks, but I tried that too and couldn't get it to work.
>>
>> I added the following:
>>
>> extern(C) handler()
>> {
>>      synchronized // needed here to avoid the GC to collect while  
>> attaching thread?
>>      {
>>          if(!Thread.getThis()) // thread unknown to D
>>          {
>>              thread_attachThis();
>>              assert(Thread.getThis()); // now D knows about it
>>          }
>>      }
>>      GC.collect(); // still segfaults
>> }
>>
>> Actually, using attachThis segfaults GC.collect() outside the thread  
>> handling code too.

I compiled with a debug build of druntime, and it fails on dereferencing a  
pointer, pbot in gcx.d:2511:

void mark(void *pbot, void *ptop, int nRecurse)
        void **p1 = cast(void **)pbot;
(...)
        for (; p1 < p2; p1++)
        {
            auto p = cast(byte *)(*p1); // segfault!
(...)

gdb) print p1
$10 = (void **) 0xf74fc000
(gdb) print *p1
$11 = (void *) 0x0

Some more dbg info:

(gdb) bt
#0  0x08098753 in gc.gcx.Gcx.mark() (this=0x80df030, nRecurse=64,  
ptop=0xffffd190, pbot=0xf74f4df4) at src/gc/gcx.d:2511
#1  0x080986d6 in gc.gcx.Gcx.mark() (this=0x80df030, ptop=0xffffd190,  
pbot=0xf74f4df4) at src/gc/gcx.d:2494
#2  0x0809ef95 in core.thread.thread_scanAll() (this=0xf74f4cf8,  
p2=0xffffd190, p1=0xf74f4df4, type=<incomplete type>)
    at src/core/thread.d:2743
#3  0x080a6488 in thread_scanAllType (scan=..., curStackTop=0xf74f4df4) at  
src/core/thread.d:2698
#4  0x0809ef68 in thread_scanAll (scan=..., curStackTop=0xf74f4df4) at  
src/core/thread.d:2746
#5  0x08098c34 in gc.gcx.Gcx.fullcollect() (this=0x80df030,  
stackTop=0xf74f4df4) at src/gc/gcx.d:2776
#6  0x080989f9 in gc.gcx.Gcx.fullcollectshell() (this=0x80df030) at  
src/gc/gcx.d:2662
#7  0x08096da1 in gc.gcx.GC.fullCollect() (this=0x80df018) at  
src/gc/gcx.d:1372
#8  0x08086da6 in gc_collect () at src/gc/gc.d:154
#9  0x08085380 in core.memory.GC.collect() () at src/core/memory.d:101
#10 0x08080e2c in c_callback (event=<incomplete type>, conn=0x80ebec0,  
request_info=0x80ebec0) at main.d:54
#11 0xf7fd28c1 in handle_request(mg_connection*) () from ./libmongoose.so
#12 0xf7fd4b7d in worker_thread(mg_context*) () from ./libmongoose.so
#13 0xf7f8dd1a in start_thread (arg=0xf74fbb70) at pthread_create.c:304
#14 0xf7ed5e4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind  
further
(gdb) l 2511
2506            uint changes = 0;
2507
2508            //printf("marking range: %p -> %p\n", pbot, ptop);
2509            for (; p1 < p2; p1++)
2510            {
2511                auto p = cast(byte *)(*p1);
2512
2513                //if (log) debug(PRINTF) printf("\tmark %p\n", p);
2514                if (p >= minAddr && p < maxAddr)
2515                {



> And if I change the code to:
> extern(C) handler()
> {
>      GC.collect();
> }
(snip)
Checking with debug build of druntime, and I see it tries to  
suspend(thread_getThis()). This gives a null reference error as  
thread_getThis() returns null when it's not attached.
Top | Discussion index | About this forum | D home