Kirk McDonald wrote:
> Walter Bright wrote:
> 
>> Kirk McDonald wrote:
>>
>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>
>>
>>
>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>
>> Will the shared libraries work with GDC?
> 
> 
> Ha! Well, at least this simple case does:
> 
[proof]
> 
> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
> 


Doesn't said boilerplate consist of linking the GC of the calling program in to the GC of the called so? why not do it the other way around? place the GC in its own so and have everything else link in to it? Not too clean for small projects but once you are using so's anyway it would be cleaner than putting gc hookup code all over the place.

BCS wrote:
> Kirk McDonald wrote:
> 
>> Walter Bright wrote:
>>
>>> Kirk McDonald wrote:
>>>
>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>
>>>
>>>
>>>
>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>
>>> Will the shared libraries work with GDC?
>>
>>
>>
>> Ha! Well, at least this simple case does:
>>
> [proof]
> 
>>
>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
>>
> 
> 
> Doesn't said boilerplate consist of linking the GC of the calling program in to the GC of the called so? why not do it the other way around? place the GC in its own so and have everything else link in to it? Not too clean for small projects but once you are using so's anyway it would be cleaner than putting gc hookup code all over the place.

If I am (for instance) writing an .so that will be loaded by the Python interpreter, it will not have a D GC to hook into.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki

Kirk McDonald wrote:
> Walter Bright wrote:
>> Kirk McDonald wrote:
>>
>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>
>>
>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>
>> Will the shared libraries work with GDC?
> 
> Ha! Well, at least this simple case does:
> 
> [myso2.d]
> import std.stdio;
> 
> export extern(C)
> void mysoprint() { writefln("Hello 'so' world!"); }
> 
> [myso.d]
> export extern(C) void mysoprint();
> 
> [test.d]
> import myso;
> 
> void main() {
>     mysoprint();
> }
> 
> $ gdc -shared -Wl,-soname,libmyso.so -o libmyso.so myso2.o -lc
> /usr/bin/ld: warning: creating a DT_TEXTREL in object.
> 
> (Not sure what that means...)
> 
> $ sudo cp libmyso.so /usr/lib
> $ gdc -c test.d
> $ gdc test.o -Wl,-lmyso -o test
> $ ./test
> Hello 'so' world!
> 
> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?

I think it is.  Perhaps the best approach would be to expose two extern (C) functions: one for startup and one for shutdown.  By default, these would be called automatically by internal/dmain2, but the user could opt to call them in DllInit or whatever if he knows that function will not be the entry point for his program.  In Ares, I'd probably call these "cr_init" and "cr_term" where "cr" means "compiler runtime."


Sean

BCS wrote:
> Kirk McDonald wrote:
>> Walter Bright wrote:
>>
>>> Kirk McDonald wrote:
>>>
>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>
>>>
>>>
>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>
>>> Will the shared libraries work with GDC?
>>
>>
>> Ha! Well, at least this simple case does:
>>
> [proof]
>>
>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
>>
> 
> 
> Doesn't said boilerplate consist of linking the GC of the calling program in to the GC of the called so? why not do it the other way around? place the GC in its own so and have everything else link in to it? Not too clean for small projects but once you are using so's anyway it would be cleaner than putting gc hookup code all over the place.

The issue is somewhat messy.  If Phobos is statically linked and the user DLL is loaded by a D app then there would be two GCs and two thread lists that need to cooperate.  It would be far preferable to put Phobos in a DLL of its own so only a single copy of this code is in use.  But now assume the DLL is loaded by a C app.  The user DLL would have to load/attach to the Phobos DLL.  I think the "cr_init" and "cr_term" functions I mentioned in my other post might have to be reentrant for everything to work properly?  I'll admit to having given the DLL issue basically no thought so far.


Sean

Sean Kelly wrote:
> Kirk McDonald wrote:
>> Walter Bright wrote:
>>> Kirk McDonald wrote:
>>>
>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>
>>>
>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>
>>> Will the shared libraries work with GDC?
>>
>> Ha! Well, at least this simple case does:
>>
>> [myso2.d]
>> import std.stdio;
>>
>> export extern(C)
>> void mysoprint() { writefln("Hello 'so' world!"); }
>>
>> [myso.d]
>> export extern(C) void mysoprint();
>>
>> [test.d]
>> import myso;
>>
>> void main() {
>>     mysoprint();
>> }
>>
>> $ gdc -shared -Wl,-soname,libmyso.so -o libmyso.so myso2.o -lc
>> /usr/bin/ld: warning: creating a DT_TEXTREL in object.
>>
>> (Not sure what that means...)
>>
>> $ sudo cp libmyso.so /usr/lib
>> $ gdc -c test.d
>> $ gdc test.o -Wl,-lmyso -o test
>> $ ./test
>> Hello 'so' world!
>>
>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
> 
> I think it is.  Perhaps the best approach would be to expose two extern (C) functions: one for startup and one for shutdown.  By default, these would be called automatically by internal/dmain2, but the user could opt to call them in DllInit or whatever if he knows that function will not be the entry point for his program.  In Ares, I'd probably call these "cr_init" and "cr_term" where "cr" means "compiler runtime."
> 

Could these be exposed to be the same on both platforms? If so that would be great.

Thanks,

- Dave

> 
> Sean

Sean Kelly wrote:
> BCS wrote:
> 
>> Kirk McDonald wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> Kirk McDonald wrote:
>>>>
>>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>>
>>>>
>>>>
>>>>
>>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>>
>>>> Will the shared libraries work with GDC?
>>>
>>>
>>>
>>> Ha! Well, at least this simple case does:
>>>
>> [proof]
>>
>>>
>>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
>>>
>>
>>
>> Doesn't said boilerplate consist of linking the GC of the calling program in to the GC of the called so? why not do it the other way around? place the GC in its own so and have everything else link in to it? Not too clean for small projects but once you are using so's anyway it would be cleaner than putting gc hookup code all over the place.
> 
> 
> The issue is somewhat messy.  If Phobos is statically linked and the user DLL is loaded by a D app then there would be two GCs and two thread lists that need to cooperate.  It would be far preferable to put Phobos in a DLL of its own so only a single copy of this code is in use.  But now assume the DLL is loaded by a C app.  The user DLL would have to load/attach to the Phobos DLL.  I think the "cr_init" and "cr_term" functions I mentioned in my other post might have to be reentrant for everything to work properly?  I'll admit to having given the DLL issue basically no thought so far.
> 
> 
> Sean

I just did some testing, and this is exactly right. I modified the old myso2.d to create a class and instantiate it:

[myso2.d]
module myso;

import std.stdio;

class A {
    ~this() { writefln("A.~this()"); }
    void foo() { writefln("A.foo()"); }
}

export extern(C)
void mysoprint() {
    A a = new A;
    a.foo();
}

When loaded by a D module, it works just fine. However, the following promptly causes Python to segfault when imported:

[testpy.d]
import python;
import std.stdio;

PyMethodDef testpy_methods[] = [
    { null, null, 0, null }
];

class A {
    ~this() { writefln("A.~this()"); }
    void foo() { writefln("A.foo()"); }
}

extern(C)
export void inittestpy() {
    Py_InitModule("testpy", testpy_methods);
    A a = new A;
    a.foo();
}

>>> import testpy
Segmentation fault
$

This, however, runs fine:

[testpy2.d]
import python;
import std.stdio;

PyMethodDef testpy_methods[] = [
    { null, null, 0, null }
];

extern(C)
export void inittestpy() {
    Py_InitModule("testpy", testpy_methods);
    writefln("Hello, world!");
}

>>> import testpy2
Hello, world!
>>>

It seems clear to me that the GC is causing the trouble.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki

Dave wrote:
> Sean Kelly wrote:
>> Kirk McDonald wrote:
>>> Walter Bright wrote:
>>>> Kirk McDonald wrote:
>>>>
>>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>>
>>>>
>>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>>
>>>> Will the shared libraries work with GDC?
>>>
>>> Ha! Well, at least this simple case does:
>>>
>>> [myso2.d]
>>> import std.stdio;
>>>
>>> export extern(C)
>>> void mysoprint() { writefln("Hello 'so' world!"); }
>>>
>>> [myso.d]
>>> export extern(C) void mysoprint();
>>>
>>> [test.d]
>>> import myso;
>>>
>>> void main() {
>>>     mysoprint();
>>> }
>>>
>>> $ gdc -shared -Wl,-soname,libmyso.so -o libmyso.so myso2.o -lc
>>> /usr/bin/ld: warning: creating a DT_TEXTREL in object.
>>>
>>> (Not sure what that means...)
>>>
>>> $ sudo cp libmyso.so /usr/lib
>>> $ gdc -c test.d
>>> $ gdc test.o -Wl,-lmyso -o test
>>> $ ./test
>>> Hello 'so' world!
>>>
>>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
>>
>> I think it is.  Perhaps the best approach would be to expose two extern (C) functions: one for startup and one for shutdown.  By default, these would be called automatically by internal/dmain2, but the user could opt to call them in DllInit or whatever if he knows that function will not be the entry point for his program.  In Ares, I'd probably call these "cr_init" and "cr_term" where "cr" means "compiler runtime."
>>
> 
> Could these be exposed to be the same on both platforms? If so that would be great.

Yup.  It would be a trivial change to internal/dmain2.  I'll probably do it in Ares in the next few days, just to have it as an option.


Sean

Sean Kelly wrote:
> BCS wrote:
> 
>> Kirk McDonald wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> Kirk McDonald wrote:
>>>>
>>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>>
>>>>
>>>>
>>>>
>>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>>
>>>> Will the shared libraries work with GDC?
>>>
>>>
>>>
>>> Ha! Well, at least this simple case does:
>>>
>> [proof]
>>
>>>
>>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
>>>
>>
>>
>> Doesn't said boilerplate consist of linking the GC of the calling program in to the GC of the called so? why not do it the other way around? place the GC in its own so and have everything else link in to it? Not too clean for small projects but once you are using so's anyway it would be cleaner than putting gc hookup code all over the place.
> 
> 
> The issue is somewhat messy.  If Phobos is statically linked and the user DLL is loaded by a D app then there would be two GCs and two thread lists that need to cooperate.  It would be far preferable to put Phobos in a DLL of its own so only a single copy of this code is in use.  But now assume the DLL is loaded by a C app.  The user DLL would have to load/attach to the Phobos DLL.  I think the "cr_init" and "cr_term" functions I mentioned in my other post might have to be reentrant for everything to work properly?  I'll admit to having given the DLL issue basically no thought so far.
> 
> 
> Sean

You would need a special Phobos to do it. For that matter, you would also want a hacked startup section to automatically hook into the gc.so, same for the so's themselves. (They have some start up code don't they?)

I guess what you'd have is a complete duplication of the D runtime. One set (what is there now) for static linking and one set (with the above hacks, and more I assume) for dynamic linking.

BCS wrote:
> Sean Kelly wrote:
>> BCS wrote:
>>
>>> Kirk McDonald wrote:
>>>
>>>> Walter Bright wrote:
>>>>
>>>>> Kirk McDonald wrote:
>>>>>
>>>>>> Here's something that has been annoying me, and this week-old thread is as good a place as any to bring it up: Shared library support on Linux. I could not take D seriously if it did a "1.0" release without this. I do hate to cram more on your plate, Walter, but I consider this a more serious issue than even this import thing that has gripped the newsgroup for the past week.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I know about the shared library issue on Linux. And to tell the truth, I've been procrastinating on it. The big job, -fPIC, is done. I don't know how much beyond that needs to be done.
>>>>>
>>>>> Will the shared libraries work with GDC?
>>>>
>>>>
>>>>
>>>> Ha! Well, at least this simple case does:
>>>>
>>> [proof]
>>>
>>>>
>>>> Sweet. However, I am a little concerned. When making DLLs on Windows, there is some boilerplate code needed to initialize and shut down the GC and do some other routine things. Is something like that needed here?
>>>>
>>>
>>>
>>> Doesn't said boilerplate consist of linking the GC of the calling program in to the GC of the called so? why not do it the other way around? place the GC in its own so and have everything else link in to it? Not too clean for small projects but once you are using so's anyway it would be cleaner than putting gc hookup code all over the place.
>>
>>
>> The issue is somewhat messy.  If Phobos is statically linked and the user DLL is loaded by a D app then there would be two GCs and two thread lists that need to cooperate.  It would be far preferable to put Phobos in a DLL of its own so only a single copy of this code is in use.  But now assume the DLL is loaded by a C app.  The user DLL would have to load/attach to the Phobos DLL.  I think the "cr_init" and "cr_term" functions I mentioned in my other post might have to be reentrant for everything to work properly?  I'll admit to having given the DLL issue basically no thought so far.
> 
> You would need a special Phobos to do it.

Well, that's basically what Ares is :-)  Though it would be easy enough for Walter to add to Phobos if he thought it was a good idea.

> For that matter, you would
> also want a hacked startup section to automatically hook into the gc.so, same for the so's themselves. (They have some start up code don't they?)

cr_init would initialize the GC, run module ctors, and any other special things that need doing.  cr_term would shutdown the GC, run module dtors, etc.  These functions would simply be called in internal/dmain2 instead of the code being inline.

> I guess what you'd have is a complete duplication of the D runtime. One set (what is there now) for static linking and one set (with the above hacks, and more I assume) for dynamic linking.

Dynamic linking is a pain, which is why I've avoided the issue thus far.  DDL is another option that I think is far preferable in instances where it can be used.  I'll admit I'm not at all keen on trying to turn the compiler runtime code into a DLL, what will all the exports likely required and such.


Sean

Kirk McDonald wrote:
> When loaded by a D module, it works just fine. However, the following promptly causes Python to segfault when imported:
> 

With these modifications, it works:

> [testpy.d]
> import python;
> import std.stdio;
> 
> PyMethodDef testpy_methods[] = [
>     { null, null, 0, null }
> ];
> 
> class A {
>     ~this() { writefln("A.~this()"); }
>     void foo() { writefln("A.foo()"); }
> }
> 

extern(C) {
    void gc_init();
    void gc_term();
}

> extern(C)
> export void inittestpy() {
    gc_init();
>     Py_InitModule("testpy", testpy_methods);
>     A a = new A;
>     a.foo();
> }

// The standard Linux dynamic library finalization function
extern(C)
void _fini() {
    gc_term();
}

It must be compiled with the -nostartfiles switch. When imported:

>>> import testpy
A.foo()
>>> ^D
A.~this()
$

However, I saw some notes about _fini being obsolete and possibly dangerous. Anyone more experienced with Linux coding want to comment?

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki