On Wed, 2014-12-03 at 01:07 +0000, Michael via Digitalmars-d-learn wrote:
> Hi. I'm new here and this is my first post. I'm not sure this is the right subforum for it, but wasn't sure where else to put it either.
> 
> I've written a library to talk to some external hardware using a socket. It uses the std.concurrency threads to send messages between the main D-object for the hardware and the D-object for the sockets. I then wanted to be able to call these functions from Python. PyD appeared to be out of date, so I've been using a

As far as I can tell PyD is still active, but in a non-funded FOSS way, i.e. work happens as and when volunteers put time and effort in. I haven't tried PyD recently but it worked fine last time I did. If can set out what you tried and what didn't work, maybe there is a PyD solution, or a fix to PyD to give a solution?

> D -> C interface, and a C -> Python interface. The python code will often run from different python threads, so I then added yet another message-passing layer between the D->C interface and the D->hardware interface.

D's "big problem" is shared objects/dynamic link libraries. Without them you cannot interwork with Python at all. I have tried experiments on Linux creating shared libraries from D code with C linkage entry points to create classic Python extensions, and it appears to work fine. Except for having to start up the D heap and thread  management, should they be needed. But that is what PyD is there for. If I took my experiments any further I would end up recreating PyD or something like it.

It sounds like you are in a similar situation except that you appear to have an extra layer of C code. I am not sure a layer of C is needed between Python and D, it would be good to know more about why you seem to need it.

Your use case is interesting as I have more or less given up on using D in a Python context. CPython naturally requires C linkage shared objects for non-Python code so C, C++ or D would be fine except that everyone does it in C or C++, very few people in the arena have even heard of D. PyPy brings it's own issues with non-Python code but the now have a C capability. With Cython, Pythran, and more recently Numba there is increasing less and less need for any user written non-Python code. Hardware control would be done in C and PyPy now has a C linkage mechanism.

Of course for Python networking there is Twisted, Asyncio and Tornado so no Python folk are using non-Python code for handling networking.

> My problem is that this code routinely causes segmentation faults. I've spent a long time going through trying to figure out exactly what the causes are. I think there have been some related to D-exceptions not being handled gracefully by the C/Python code. Some more by stdout writing from multiple threads (which surprised me).
> 
> I'm fairly sure I have tackled both of these issues, but it still seems like Python threads and D threads don't mix well. When running the same functions from D, I am able to get no errors, but when run from Python/C it causes segfaults reliably.

Without seeing your code, it is difficult to say what may or may not be the problem, but I would guess it is about D infrastructure start up. I recollect being able to reliably get segfaults this way.

Are you using ctypes or CFFI on the Python side?

> Sorry for the large exposition. I am currently at the point of suspecting bugs in Phobos, but I am unskilled enough to tell for sure, and would appreciate any help.
> 
> The latest core dump gives a backtrace of almost entirely phobos commands:
> 
> #0  0x00007fe789ad3b97 in gc.gc.Gcx.fullcollect() () from
> /lib/libphobos2.so.0.66
> #1  0x00007fe789ad3294 in gc.gc.Gcx.bigAlloc() () from
> /lib/libphobos2.so.0.66
> #2  0x00007fe789ad0df1 in gc.gc.GC.mallocNoSync() () from
> /lib/libphobos2.so.0.66
> #3  0x00007fe789ad0c15 in gc.gc.GC.malloc() () from
> /lib/libphobos2.so.0.66
> #4  0x00007fe789ad6470 in gc_malloc () from
> /lib/libphobos2.so.0.66
> #5  0x00007fe789ae6d36 in _d_newitemT () from
> /lib/libphobos2.so.0.66
> #6  0x00007fe789e57112 in
> std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from
> /usr/lib/libv5camera.so
> #7  0x00007fe789e570b5 in
> std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from
> /usr/lib/libv5camera.so
> #8  0x00007fe789e562dc in
> std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from
> /usr/lib/libv5camera.so
> #9  0x00007fe789e561ea in
> std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from
> /usr/lib/libv5camera.so
> #10 0x00007fe789e5617d in
> std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar()
> () from /usr/lib/libv5camera.so
> #11 0x00007fe789e56132 in
> std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar()
> () from /usr/lib/libv5camera.so
> #12 0x00007fe789e61f09 in
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()
> () from /usr/lib/libv5camera.so
> #13 0x00007fe789e5b4ac in
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get()
> () from /usr/lib/libv5camera.so
> #14 0x00007fe789e57e8d in
> std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.__ctor()
> ()
>     from /usr/lib/libv5camera.so
> #15 0x00007fe789e581f1 in
> std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.handler()
> ()
>     from /usr/lib/libv5camera.so
> #16 0x00007fe789e57d0f in
> std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24__T5TupleTAyaTiTG65536kZ5TupleZ.opEquals()
> ()
>     from /usr/lib/libv5camera.so
> #17 0x00007fe789e57ba8 in
> std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from
> /usr/lib/libv5camera.so
> #18 0x00007fe789e62087 in
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()
> () from /usr/lib/libv5camera.so
> #19 0x00007fe789e621a3 in
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()
> () from /usr/lib/libv5camera.so
> #20 0x00007fe789e5b7f6 in
> std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get()
> () from /usr/lib/libv5camera.so
> #21 0x00007fe789ac7d51 in core.thread.Thread.run() () from
> /lib/libphobos2.so.0.66
> #22 0x00007fe789ac6f95 in thread_entryPoint () from
> /lib/libphobos2.so.0.66
> #23 0x00007fe79cee5182 in start_thread (arg=0x7fe77aca5700) at
> pthread_create.c:312
> #24 0x00007fe79cc11fbd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

My guess would be not properly initializing the D infrastructure from the incoming Python thread.

I would suggest that you want to avoid threads crossing the boundaries and just pass data via a shared channel. Unix pipes seem to work well in this context since they provide a language independent data channel.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer wrote:
> are you looking at this pyd: https://bitbucket.org/ariovistus/pyd

I'm looking at this one, which is what came up when googling "python to D"
http://pyd.dsource.org/

On Wednesday, 3 December 2014 at 06:30:07 UTC, Russel Winder via Digitalmars-d-learn wrote:
>
> As far as I can tell PyD is still active, but in a non-funded FOSS way,
> i.e. work happens as and when volunteers put time and effort in. I
> haven't tried PyD recently but it worked fine last time I did. If can
> set out what you tried and what didn't work, maybe there is a PyD
> solution, or a fix to PyD to give a solution?

Yeah apparently I might have used the wrong PyD. This might end up being all I need to do to fix my problem.


> D's "big problem" is shared objects/dynamic link libraries. Without them
> you cannot interwork with Python at all. I have tried experiments on
> Linux creating shared libraries from D code with C linkage entry points
> to create classic Python extensions, and it appears to work fine. Except
> for having to start up the D heap and thread  management, should they be
> needed. But that is what PyD is there for. If I took my experiments any
> further I would end up recreating PyD or something like it.

This is what I'm doing. I'm using the rt_init() function to setup the heap/thread management. Am I missing anything else here? It seems the issue is definitely more complicated than that.

I'll also point out that: http://dlang.org/interfaceToC had no information on calling rt_init first, although it seems like it should.


> It sounds like you are in a similar situation except that you appear to
> have an extra layer of C code. I am not sure a layer of C is needed
> between Python and D, it would be good to know more about why you seem
> to need it.

Well I wanted to compile D code and directly call it in Python, and since PyD didn't work for me, I instead tried the python-> C interface I already knew (https://docs.python.org/2/c-api/index.html) and worked my way through a C->D interface, which I understood to be relatively simple.

> My guess would be not properly initializing the D infrastructure from
> the incoming Python thread.
>
> I would suggest that you want to avoid threads crossing the boundaries
> and just pass data via a shared channel. Unix pipes seem to work well in
> this context since they provide a language independent data channel.

Yeah I'm leaning in that direction myself, although I might try the other PyD library first. I wanted to be able to use the D message-passing libraries to do the thread-safety stuff at first, because it was much easier than the alternative, but I'm not sure that's true anymore.

On Wed, 03 Dec 2014 20:41:46 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn@puremagic.com>
wrote:

> On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer wrote:
> > are you looking at this pyd: https://bitbucket.org/ariovistus/pyd
> 
> I'm looking at this one, which is what came up when googling
> "python to D"
> http://pyd.dsource.org/
ah, dsource strikes back! that vile site keep biting us again and again. let's hope that new admins will kill it for good.

On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via Digitalmars-d-learn wrote:
> ah, dsource strikes back! that vile site keep biting us again and
> again. let's hope that new admins will kill it for good.

Yeah. I've got the new PyD and it compiles and does everything I want much nicer, but it appears to have the exact same problems. When calling a python thread to my code, it can cause segfaults and hanging issues.

Cheers,
Michael.

On 12/03/2014 04:43 PM, Michael wrote:
> On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via
> Digitalmars-d-learn wrote:
>> ah, dsource strikes back! that vile site keep biting us again and
>> again. let's hope that new admins will kill it for good.
>
> Yeah. I've got the new PyD and it compiles and does everything I want
> much nicer, but it appears to have the exact same problems. When calling
> a python thread to my code, it can cause segfaults and hanging issues.
>
> Cheers,
> Michael.

okay. that's not too surprising.

If you can get me a minimal example, I'd be happy to have a look since pyd should probably support this case.

On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer wrote:
>
> okay. that's not too surprising.
>
> If you can get me a minimal example, I'd be happy to have a look since pyd should probably support this case.

Cool. Unfortunately most of the times I've attempted to reduce this down it always seems to work, but I think that's because I often did the example code in D. I'll play around with it and try to send you an example.

Cheers,
Michael.

On 12/03/2014 06:56 PM, Michael wrote:
> On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer wrote:
>>
>> okay. that's not too surprising.
>>
>> If you can get me a minimal example, I'd be happy to have a look since
>> pyd should probably support this case.
>
> Cool. Unfortunately most of the times I've attempted to reduce this down
> it always seems to work, but I think that's because I often did the
> example code in D. I'll play around with it and try to send you an example.
>
> Cheers,
> Michael.

dustmite?

On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:
>
> dustmite?

Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors. I manually deleted a reasonable chunk of the code and I'm left with these files which still seem to cause segfaults:

Main code: http://pastebin.com/zqgNTk9w
PyD definitions: http://pastebin.com/6mRH3KZZ
setup.py: http://pastebin.com/i9Ph78UC
test code that causes segfaults: http://pastebin.com/1ukzShVh

Cheers,
Michael.

On Thu, Dec 04, 2014 at 10:11:53PM +0000, Michael via Digitalmars-d-learn wrote:
> On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:
> >
> >dustmite?
> 
> Not sure what went wrong with dustmite, but every time I tried it it just started deleting all the files in the directory and setup.py would give errors.
[...]

Nothing is wrong with dustmite; one of its standard techniques to reduce code is to delete source files and see if the problem still happens (i.e., it's independent of that file). What you need to do is to craft a test script such that only the specific error you're looking for will return a success status, and everything else won't. Then when dustmite deletes an essential file, it will know that that was a wrong step and backtrack.


T

-- 
Food and laptops don't mix.