View mode: basic / threaded / horizontal-split · Log in · Help
September 27, 2012
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg 
wrote:
> On 2012-09-27 10:55, Maxim Fomin wrote:
>> On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg 
>> wrote:
>>> 1. Does this actually run?
>>>
>>
>> If it were non-runnable, I wouldn't posted it.
>>
>>> 2. This is statically linked with druntime and Phobos. What 
>>> happens
>>> when you create an executable that links with the D dynamic 
>>> library?
>>
>> Solution depends on a problem. I understood Andrei's post that 
>> he wanted
>> a .so file or DLL. I told originally that it is possible to 
>> make shared
>> libraries on linux. Now I see there is some misunderstanding. 
>> Is the
>> problem in diving D application on executables and shared 
>> libraries or
>> making druntime+phobos a shared library or loading a library 
>> at runtime?
>> A was speaking about the first.
>
> Actually, I seriously doubt everything is working as expected. 
> For example, what happens when an application loads (via dlopen 
> or similar) a D dynamic library:
>
> * Are exception handlers registered
> * Are module infos properly registered
> * What happens if I call Object.factory, will that find a class 
> in the dynamic library
> * Are GC sections registered
> * What happens when the library is unloaded, will all of the 
> above be unregistered and cleaned up
>
> A quick look in rt.minfo in druntime shows the it uses a single 
> array or linked list for all module infos. It expect the module 
> infos to be in just one place, i.e. in the application. There's 
> no handling when a dynamic library is loaded.
>
> Have a look at this:
>
> https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184
>
> That symbol points to the start of the linked list containing 
> module infos. That's only a single linked list, no handling of 
> module infos from multiple places.
>
> rt.minfo needs to be changed to use an associative array with 
> the keys pointing to images (executables and loaded dynamic 
> libraries) and the values mapped to module infos of the given 
> image. It also needs to intercept when a library is dynamically 
> loaded, extract the module infos and register it with the 
> runtime.
>
> I would think that the same is true for exception handling 
> tables, TLS and GC sections.

Posted code doesn't load libraries at runtime, it is just linked 
to shared libraries.
September 27, 2012
Re: dynamic library building and loading
On 9/27/12 3:37 PM, Maxim Fomin wrote:
> Posted code doesn't load libraries at runtime, it is just linked to
> shared libraries.

Exactly! (I can't believe I'm starting to get the hang of this...) But 
what we ultimately need is true dynamic loading of never-seen modules.

After the initial test I tried to load symbols with dlopen(). Indeed 
this works:

auto p = dlopen("liblib.so", RTLD_LAZY);

And it does work, all fine. To my dismay, as soon as I removed the flag 
"-llib" from the linker command line in the makefile, dlopen() didn't 
want to work anymore.

So I think in order to enable "true" dynamic loading, I'll need to 
generate PIC for druntime and phobos, and then link liblib.so like this:

dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2

Is that correct?

At that point, of course, a variety of issues will need to be resolved 
as people pointed out.


Andrei
September 27, 2012
Re: dynamic library building and loading
For me to get C or C++ to run a D function, I had to do the 
following:

// ====================
// C/C++ library source
// ====================

// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization & shutdown
void init();
void done();

void bar()
{
    foo(6, 7, 8);
}

int main(int argc, char **argv)
{
init();
	bar();
	done();
	return 0;
}


// ================
// D library source
// ================
import std.stdio;
static import core.runtime;

// sample D function for test
extern (C++) //	int foo(int i, int j, int k)
	void foo(int i, int j, int k)
	{
		writefln("i = %s", i);
		writefln("j = %s", j);
		writefln("k = %s", k);
		int t = i + j + k;
		writefln("Total = %s", t);
	}

// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after 
loading shared lib
    core.runtime.Runtime.initialize();
}

extern (C++) export void done() { // to be called before 
unloading shared lib
    core.runtime.Runtime.terminate();
}


// I had to include main even though this is a library.
int main()
{
   return 0;
}
September 28, 2012
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 07:54:29 UTC, Johannes Pfau 
wrote:
> Am Thu, 27 Sep 2012 08:26:36 +0200
> schrieb "Daniel Kozak" <kozzi11@gmail.com>:
>
>> Now I try it, and it is not required to build shared variant 
>> of druntime and phobos, only rebuild it with -fPIC
>> 
>
> In the end you'll probably need a shared druntime & phobos: 
> Let's say
> your main app doesn't use std.datetime and you statically link
> libphobos. Then the linker might strip std.datetime from your
> executable. If your shared library now needs std.datetime it 
> won't
> work.

I'm using C++ main app that dynamically loads shared libs at 
runtime, but I cannot load D shared libs because they cannot be 
linked to the non-fPIC built phobos2/druntime lib. It seems that 
the only way to get things working is to re-build the C++ code so 
that it is linked with non-fPIC phobos2, but this is not 
something I want to be doing.

So if I read this right, I can build phobos/druntime with -fPIC 
from the source code and it will work OK? If so, then why was 
this not done with the latest distrubution package, or is this 
only possible after a certain version number?

--rt
September 28, 2012
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 19:57:13 UTC, Andrei 
Alexandrescu wrote:
> So I think in order to enable "true" dynamic loading, I'll need 
> to generate PIC for druntime and phobos, and then link 
> liblib.so like this:
>
> dmd -fPIC -c lib.d
> gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2
>
> Is that correct?

Yes, that is correct.

The other thing missing is a phobos2.so for dynamic runtime 
linking.

-rt
September 28, 2012
Re: dynamic library building and loading
On 2012-09-28 02:04, Rob T wrote:

> I'm using C++ main app that dynamically loads shared libs at runtime,
> but I cannot load D shared libs because they cannot be linked to the
> non-fPIC built phobos2/druntime lib. It seems that the only way to get
> things working is to re-build the C++ code so that it is linked with
> non-fPIC phobos2, but this is not something I want to be doing.
>
> So if I read this right, I can build phobos/druntime with -fPIC from the
> source code and it will work OK? If so, then why was this not done with
> the latest distrubution package, or is this only possible after a
> certain version number?

No, it will not work ok. Depending on what you do in the D code it might 
work, but everything will not work. Perhaps if you limit yourself to a C 
subset. See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg
September 28, 2012
Re: dynamic library building and loading
On 2012-09-28 01:21, Rob T wrote:
> For me to get C or C++ to run a D function, I had to do the following:
>
> // ====================
> // C/C++ library source
> // ====================
>
> // sample D function from library
> void foo(int i, int j, int k);
> // D runtime initialization & shutdown
> void init();
> void done();
>
> void bar()
> {
>      foo(6, 7, 8);
> }
>
> int main(int argc, char **argv)
> {
> init();
>      bar();
>      done();
>      return 0;
> }
>
>
> // ================
> // D library source
> // ================
> import std.stdio;
> static import core.runtime;
>
> // sample D function for test
> extern (C++) //    int foo(int i, int j, int k)
>      void foo(int i, int j, int k)
>      {
>          writefln("i = %s", i);
>          writefln("j = %s", j);
>          writefln("k = %s", k);
>          int t = i + j + k;
>          writefln("Total = %s", t);
>      }
>
> // Had to initialize and shutdown D system from C/C++.
> extern (C++) export void init() { // to be called once after loading
> shared lib
>      core.runtime.Runtime.initialize();
> }
>
> extern (C++) export void done() { // to be called before unloading
> shared lib
>      core.runtime.Runtime.terminate();
> }
>
>
> // I had to include main even though this is a library.
> int main()
> {
>     return 0;
> }

That is not sufficient to have everything work. It might be ok if you 
limit yourself to a C subset of D. On some platforms, calling 
Runtime.initialize, will miss initializing some parts.

The implementation of the runtime initialization contains a lot of code 
duplication and some functions/branches are missing functionality. Example:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L346

The C main function handles the runtime initialization for a regular 
statically linked executable. If you initialize the runtime via 
"rt_init", which is called by "Runtime.initialize", it will miss to 
initialize some parts on some platforms. The C main function should 
really call "rt_init" to remove the code duplication.

Also see this post why everything will not properly work:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com


-- 
/Jacob Carlborg
September 28, 2012
Re: dynamic library building and loading
Am Fri, 28 Sep 2012 02:04:11 +0200
schrieb "Rob T" <rob@ucora.com>:

> 
> So if I read this right, I can build phobos/druntime with -fPIC 
> from the source code and it will work OK? If so, then why was 
> this not done with the latest distrubution package, or is this 
> only possible after a certain version number?
> 
> --rt

It shouldn't compile on x86 (32bit) with -fPIC because there's some
incompatible asm code in druntime and phobos. If dmd doesn't warn in
these cases, this code will fail as soon as it's called.

GDC used to warn about these errors, but as we dropped the D inline
assembler that code doesn't affect us anymore and there are no more
warnings.

The fix for this issue is simple: Do not modify the EBX register, or at
least save & restore it.

Old patch, which shows what needs to be done:
https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Example asm code which probably doesn't work:
https://github.com/D-Programming-Language/druntime/blob/master/src/core/cpuid.d#L432


I just tried building a shared druntime & phobos with gdc:
Druntime seems to be working, Phobos doesn't link here because
something's wrong with the fstat64 symbol. I haven't done any real
testing and we probably need some additional support in the runtime, but
the compiler part seems to be OK with gdc.
http://gdcproject.org/wiki/SharedRuntime
https://github.com/D-Programming-GDC/gdc/pull/32
September 28, 2012
Re: dynamic library building and loading
On 28 September 2012 12:14, Johannes Pfau <nospam@example.com> wrote:
> Am Fri, 28 Sep 2012 02:04:11 +0200
> schrieb "Rob T" <rob@ucora.com>:
>
>>
>> So if I read this right, I can build phobos/druntime with -fPIC
>> from the source code and it will work OK? If so, then why was
>> this not done with the latest distrubution package, or is this
>> only possible after a certain version number?
>>
>> --rt
>
> It shouldn't compile on x86 (32bit) with -fPIC because there's some
> incompatible asm code in druntime and phobos. If dmd doesn't warn in
> these cases, this code will fail as soon as it's called.
>
> GDC used to warn about these errors, but as we dropped the D inline
> assembler that code doesn't affect us anymore and there are no more
> warnings.
>
> The fix for this issue is simple: Do not modify the EBX register, or at
> least save & restore it.
>

In GDC D1 phobos, one hack around it was to use db to manually write
out the bytecode instruction (thus bypassing GCC 'ebx' clobbered
checks).  Which was a fiendish workaround to say the least. :~)


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
September 28, 2012
Re: dynamic library building and loading
I am trying out gdc 4.7 branch with -fPIC, but the info I'm 
getting is that even with a successulf PIC build I will still not 
be able to reliably execute D functions directly from C/C++ code.

> Old patch, which shows what needs to be done:
> https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Does anyone know why patches like this are taking so [bleeping] 
long to get recognized and implemented?

-rt
1 2 3 4 5 6 7
Top | Discussion index | About this forum | D home