On 12/04/2014 02:11 PM, Michael wrote:
> On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:
>>
>> dustmite?
>
> Not sure what went wrong with dustmite, but every time I tried it it
> just started deleting all the files in the directory and setup.py would
> give errors. I manually deleted a reasonable chunk of the code and I'm
> left with these files which still seem to cause segfaults:
>
> Main code: http://pastebin.com/zqgNTk9w
> PyD definitions: http://pastebin.com/6mRH3KZZ
> setup.py: http://pastebin.com/i9Ph78UC
> test code that causes segfaults: http://pastebin.com/1ukzShVh
>
> Cheers,
> Michael.

hmm.. looks like here it originates in python when it tries to acquire the GIL. specifically, pthread_cond_timedwait is segfaulting.

in your code, execution inside a python thread makes it to receiveTimeout in get_image. it made it past receiveTimeout in acquire.

then I guess there is a context switch. the main python thread throws an exception, but a number of things trigger the segfault. I think it's just the interpreter loop calling RestoreThread. backtrace looks like

#0  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000003799d07bb3 in PyCOND_TIMEDWAIT (cond=0x379a063220 <gil_cond>,
    mut=0x379a0631e0 <gil_mutex>, us=5000)
    at /usr/src/debug/Python-3.3.2/Python/condvar.h:103
#2  take_gil (tstate=tstate@entry=0x604410)
    at /usr/src/debug/Python-3.3.2/Python/ceval_gil.h:224
#3  0x0000003799d081fb in PyEval_RestoreThread (tstate=tstate@entry=0x604410)
...

It looks like this is the python main thread.

I see two other threads. (i took out one of your python spawns) #2 looks to be your listener thread. std.concurrency.send seems to have gotten it into a gc_malloc, but it looks like it's just waiting. it's currently in sem_wait. this one would have been spawned in D code by #3

#3 is your other python thread. it is also in pthread_cond_timedwait. by its stack trace, receiveTimeout is just waiting.


I guess tomorrow I can try messing around with thread_attachThis, as the fullcollect happening in #2 might be screwing with python data. But you aren't really passing anything from python to d or vice versa, so I'm not sure why the gc would need to know about the python threads.

not exactly my area of expertise, this.

On 12/04/2014 10:55 PM, Ellery Newcomer wrote:

> I guess tomorrow I can try messing around with thread_attachThis, as the
> fullcollect happening in #2 might be screwing with python data. But you
> aren't really passing anything from python to d or vice versa, so I'm
> not sure why the gc would need to know about the python threads.

by gum, thread_attachThis and thread_detachThis fix the issue!

now to figure out how to use them in the general case.

On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
> On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
>
>> I guess tomorrow I can try messing around with thread_attachThis, as the
>> fullcollect happening in #2 might be screwing with python data. But you
>> aren't really passing anything from python to d or vice versa, so I'm
>> not sure why the gc would need to know about the python threads.
>
> by gum, thread_attachThis and thread_detachThis fix the issue!
>
> now to figure out how to use them in the general case.

This is great.. Thank you. I'm looking forward to being able to try the finished result.

On 12/07/2014 03:12 PM, Michael wrote:
> On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
>> On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
>>
>>> I guess tomorrow I can try messing around with thread_attachThis, as the
>>> fullcollect happening in #2 might be screwing with python data. But you
>>> aren't really passing anything from python to d or vice versa, so I'm
>>> not sure why the gc would need to know about the python threads.
>>
>> by gum, thread_attachThis and thread_detachThis fix the issue!
>>
>> now to figure out how to use them in the general case.
>
> This is great.. Thank you. I'm looking forward to being able to try the
> finished result.

It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered.

Actually, I suppose with a thread local flag the latter wouldn't be too bad.

Mind if I incorporate your example into pyd's test suite?

On Monday, 8 December 2014 at 01:17:16 UTC, Ellery Newcomer wrote:
> On 12/07/2014 03:12 PM, Michael wrote:
>> On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:
>>> On 12/04/2014 10:55 PM, Ellery Newcomer wrote:
>>>
>>>> I guess tomorrow I can try messing around with thread_attachThis, as the
>>>> fullcollect happening in #2 might be screwing with python data. But you
>>>> aren't really passing anything from python to d or vice versa, so I'm
>>>> not sure why the gc would need to know about the python threads.
>>>
>>> by gum, thread_attachThis and thread_detachThis fix the issue!
>>>
>>> now to figure out how to use them in the general case.
>>
>> This is great.. Thank you. I'm looking forward to being able to try the
>> finished result.
>
> It would be great if there were some convenient hook in python to stick these calls. Near as I can tell, there isn't. That leaves you with either explicitly calling attach and detach with an exposed api, or pyd obsessively checking whether the current thread is registered.
>
> Actually, I suppose with a thread local flag the latter wouldn't be too bad.
>
> Mind if I incorporate your example into pyd's test suite?

Not at all. Go for it.

On 12/07/2014 03:12 PM, Michael wrote:
>> now to figure out how to use them in the general case.
>
> This is great.. Thank you. I'm looking forward to being able to try the
> finished result.

My build servers are broken at the moment, but I think I have this fixed, on linux at least.