December 05, 2014 Re: threading issues with D -> C -> Python | ||||
---|---|---|---|---|
| ||||
Posted in reply to Michael | On 12/04/2014 02:11 PM, Michael wrote: > On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote: >> >> dustmite? > > Not sure what went wrong with dustmite, but every time I tried it it > just started deleting all the files in the directory and setup.py would > give errors. I manually deleted a reasonable chunk of the code and I'm > left with these files which still seem to cause segfaults: > > Main code: http://pastebin.com/zqgNTk9w > PyD definitions: http://pastebin.com/6mRH3KZZ > setup.py: http://pastebin.com/i9Ph78UC > test code that causes segfaults: http://pastebin.com/1ukzShVh > > Cheers, > Michael. hmm.. looks like here it originates in python when it tries to acquire the GIL. specifically, pthread_cond_timedwait is segfaulting. in your code, execution inside a python thread makes it to receiveTimeout in get_image. it made it past receiveTimeout in acquire. then I guess there is a context switch. the main python thread throws an exception, but a number of things trigger the segfault. I think it's just the interpreter loop calling RestoreThread. backtrace looks like #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000003799d07bb3 in PyCOND_TIMEDWAIT (cond=0x379a063220 <gil_cond>, mut=0x379a0631e0 <gil_mutex>, us=5000) at /usr/src/debug/Python-3.3.2/Python/condvar.h:103 #2 take_gil (tstate=tstate@entry=0x604410) at /usr/src/debug/Python-3.3.2/Python/ceval_gil.h:224 #3 0x0000003799d081fb in PyEval_RestoreThread (tstate=tstate@entry=0x604410) ... It looks like this is the python main thread. I see two other threads. (i took out one of your python spawns) #2 looks to be your listener thread. std.concurrency.send seems to have gotten it into a gc_malloc, but it looks like it's just waiting. it's currently in sem_wait. this one would have been spawned in D code by #3 #3 is your other python thread. it is also in pthread_cond_timedwait. by its stack trace, receiveTimeout is just waiting. I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads. not exactly my area of expertise, this. |
December 06, 2014 Re: threading issues with D -> C -> Python | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ellery Newcomer | On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
> I guess tomorrow I can try messing around with thread_attachThis, as the
> fullcollect happening in #2 might be screwing with python data. But you
> aren't really passing anything from python to d or vice versa, so I'm
> not sure why the gc would need to know about the python threads.
by gum, thread_attachThis and thread_detachThis fix the issue!
now to figure out how to use them in the general case.
|
December 07, 2014 Re: threading issues with D -> C -> Python | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ellery Newcomer | On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
> On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
>
>> I guess tomorrow I can try messing around with thread_attachThis, as the
>> fullcollect happening in #2 might be screwing with python data. But you
>> aren't really passing anything from python to d or vice versa, so I'm
>> not sure why the gc would need to know about the python threads.
>
> by gum, thread_attachThis and thread_detachThis fix the issue!
>
> now to figure out how to use them in the general case.
This is great.. Thank you. I'm looking forward to being able to try the finished result.
|
December 08, 2014 Re: threading issues with D -> C -> Python | ||||
---|---|---|---|---|
| ||||
Posted in reply to Michael | On 12/07/2014 03:12 PM, Michael wrote:
> On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
>> On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
>>
>>> I guess tomorrow I can try messing around with thread_attachThis, as the
>>> fullcollect happening in #2 might be screwing with python data. But you
>>> aren't really passing anything from python to d or vice versa, so I'm
>>> not sure why the gc would need to know about the python threads.
>>
>> by gum, thread_attachThis and thread_detachThis fix the issue!
>>
>> now to figure out how to use them in the general case.
>
> This is great.. Thank you. I'm looking forward to being able to try the
> finished result.
It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered.
Actually, I suppose with a thread local flag the latter wouldn't be too bad.
Mind if I incorporate your example into pyd's test suite?
|
December 09, 2014 Re: threading issues with D -> C -> Python | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ellery Newcomer | On Monday, 8 December 2014 at 01:17:16 UTC, Ellery Newcomer wrote:
> On 12/07/2014 03:12 PM, Michael wrote:
>> On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
>>> On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
>>>
>>>> I guess tomorrow I can try messing around with thread_attachThis, as the
>>>> fullcollect happening in #2 might be screwing with python data. But you
>>>> aren't really passing anything from python to d or vice versa, so I'm
>>>> not sure why the gc would need to know about the python threads.
>>>
>>> by gum, thread_attachThis and thread_detachThis fix the issue!
>>>
>>> now to figure out how to use them in the general case.
>>
>> This is great.. Thank you. I'm looking forward to being able to try the
>> finished result.
>
> It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered.
>
> Actually, I suppose with a thread local flag the latter wouldn't be too bad.
>
> Mind if I incorporate your example into pyd's test suite?
Not at all. Go for it.
|
December 18, 2014 Re: threading issues with D -> C -> Python | ||||
---|---|---|---|---|
| ||||
Posted in reply to Michael | On 12/07/2014 03:12 PM, Michael wrote:
>> now to figure out how to use them in the general case.
>
> This is great.. Thank you. I'm looking forward to being able to try the
> finished result.
My build servers are broken at the moment, but I think I have this fixed, on linux at least.
|
Copyright © 1999-2021 by the D Language Foundation