View mode: basic / threaded / horizontal-split · Log in · Help
August 20, 2012
Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
(By dynamic loading I mean using something like the C library function
dlopen to explicitly load a shared object at runtime. I do not mean dynamic
linking in the usual passive sense.)

I just successfully got working a toy example of dynamically loading (from
a D program) a D shared object and then finding and calling functions
defined in the shared object, and not just with C linkage.

The main program, main.d, was compiled and linked somewhat normally, except
for passing the linker via gcc the flags necessary to ensure that the whole
of libphobos2.a is present and that all symbols in the resulting executable
are exposed dynamically.

The shared object source, dload.d was compiled to an object file containing
position independent code by dmd. Then I invoked the linker explicitly and
had it make a shared object without the D runtime system or Phobos. This is
the novel step, and it enables the shared object to resolve its linkage to
the D runtime system and Phobos at the point of being loaded, via callbacks
to the main program. Thus the troubles of D in shared objects are largely
circumvented. There is only one instance of phobos and D-runtime, in the
main program. (Once phobos and druntime are shared objects in the future
somewhere this will work with no code bloat.)

The static initialization code in dload.d is automatically executed when
the shared object libdload.so is loaded by the main program, because the
linker is also passed a flag indicating the static initialization block's
mangled name, dynamically determined from dload.o before linkage to
libdload.so occurs.

Finally, the mangled names of the functions to load are determined by a
call of a function with C linkage in dload.d from main.d that looks up
those names in an associative array initialized in the static
initialization block of dload.d where those mangled names are directly
available, so that full D linkage can be emulated, at least for functions.

One thing: the garbage collector needs to be aware of static and 'global' D
variables in the shared object. Can a technical expert verify that I've
done the right thing to achieve that happy state of affairs in this unusual
context?

So, what's overlooked here? I know that the static initialization code
cannot successfully throw an exception. Yet if a function in the shared
object is called from the main program and throws an exception, all is
well. (Try these.) See my comments in dload.d about this. What is it about
the implementation of exceptions that's problematic here?

All files attached, including a Makefile with the exact options passed to
dmd, gcc and ld.
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On 2012-08-20 05:40, Carl Sturtivant wrote:
> (By dynamic loading I mean using something like the C library function
> dlopen to explicitly load a shared object at runtime. I do not mean
> dynamic linking in the usual passive sense.)
>
> I just successfully got working a toy example of dynamically loading
> (from a D program) a D shared object and then finding and calling
> functions defined in the shared object, and not just with C linkage.
>
> The main program, main.d, was compiled and linked somewhat normally,
> except for passing the linker via gcc the flags necessary to ensure that
> the whole of libphobos2.a is present and that all symbols in the
> resulting executable are exposed dynamically.
>
> The shared object source, dload.d was compiled to an object file
> containing position independent code by dmd. Then I invoked the linker
> explicitly and had it make a shared object without the D runtime system
> or Phobos. This is the novel step, and it enables the shared object to
> resolve its linkage to the D runtime system and Phobos at the point of
> being loaded, via callbacks to the main program. Thus the troubles of D
> in shared objects are largely circumvented. There is only one instance
> of phobos and D-runtime, in the main program. (Once phobos and druntime
> are shared objects in the future somewhere this will work with no code
> bloat.)
>
> The static initialization code in dload.d is automatically executed when
> the shared object libdload.so is loaded by the main program, because the
> linker is also passed a flag indicating the static initialization
> block's mangled name, dynamically determined from dload.o before linkage
> to libdload.so occurs.
>
> Finally, the mangled names of the functions to load are determined by a
> call of a function with C linkage in dload.d from main.d that looks up
> those names in an associative array initialized in the static
> initialization block of dload.d where those mangled names are directly
> available, so that full D linkage can be emulated, at least for functions.
>
> One thing: the garbage collector needs to be aware of static and
> 'global' D variables in the shared object. Can a technical expert verify
> that I've done the right thing to achieve that happy state of affairs in
> this unusual context?
>
> So, what's overlooked here? I know that the static initialization code
> cannot successfully throw an exception. Yet if a function in the shared
> object is called from the main program and throws an exception, all is
> well. (Try these.) See my comments in dload.d about this. What is it
> about the implementation of exceptions that's problematic here?
>
> All files attached, including a Makefile with the exact options passed
> to dmd, gcc and ld.

I'm not sure I'm following what you exactly have done here but in 
general this is what needs to be done to make dynamic libraries properly 
work in D :

* Initialize module infos (module constructors and similar)
* Add TLS variables
* Add exception handling tables
* Add GC roots

The above four things need to be extracted from the loaded dynamic 
library and it gets loaded and preferably remove them as well when the 
dynamic library gets unloaded. Currently this is only extracted from the 
running executable. This is platform dependent but usually it's 
extracted using bracketed sections via extern C variables.

-- 
/Jacob Carlborg
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On Monday, 20 August 2012 at 07:26:40 UTC, Jacob Carlborg wrote:
> On 2012-08-20 05:40, Carl Sturtivant wrote:
>> (By dynamic loading I mean using something like the C library 
>> function
>> dlopen to explicitly load a shared object at runtime. I do not 
>> mean
>> dynamic linking in the usual passive sense.)
>>
>> I just successfully got working a toy example of dynamically 
>> loading
>> (from a D program) a D shared object and then finding and 
>> calling
>> functions defined in the shared object, and not just with C 
>> linkage.
>>
>> The main program, main.d, was compiled and linked somewhat 
>> normally,
>> except for passing the linker via gcc the flags necessary to 
>> ensure that
>> the whole of libphobos2.a is present and that all symbols in 
>> the
>> resulting executable are exposed dynamically.
>>
>> The shared object source, dload.d was compiled to an object 
>> file
>> containing position independent code by dmd. Then I invoked 
>> the linker
>> explicitly and had it make a shared object without the D 
>> runtime system
>> or Phobos. This is the novel step, and it enables the shared 
>> object to
>> resolve its linkage to the D runtime system and Phobos at the 
>> point of
>> being loaded, via callbacks to the main program. Thus the 
>> troubles of D
>> in shared objects are largely circumvented. There is only one 
>> instance
>> of phobos and D-runtime, in the main program. (Once phobos and 
>> druntime
>> are shared objects in the future somewhere this will work with 
>> no code
>> bloat.)
>>
>> The static initialization code in dload.d is automatically 
>> executed when
>> the shared object libdload.so is loaded by the main program, 
>> because the
>> linker is also passed a flag indicating the static 
>> initialization
>> block's mangled name, dynamically determined from dload.o 
>> before linkage
>> to libdload.so occurs.
>>
>> Finally, the mangled names of the functions to load are 
>> determined by a
>> call of a function with C linkage in dload.d from main.d that 
>> looks up
>> those names in an associative array initialized in the static
>> initialization block of dload.d where those mangled names are 
>> directly
>> available, so that full D linkage can be emulated, at least 
>> for functions.
>>
>> One thing: the garbage collector needs to be aware of static 
>> and
>> 'global' D variables in the shared object. Can a technical 
>> expert verify
>> that I've done the right thing to achieve that happy state of 
>> affairs in
>> this unusual context?
>>
>> So, what's overlooked here? I know that the static 
>> initialization code
>> cannot successfully throw an exception. Yet if a function in 
>> the shared
>> object is called from the main program and throws an 
>> exception, all is
>> well. (Try these.) See my comments in dload.d about this. What 
>> is it
>> about the implementation of exceptions that's problematic here?
>>
>> All files attached, including a Makefile with the exact 
>> options passed
>> to dmd, gcc and ld.
>
> I'm not sure I'm following what you exactly have done here but 
> in general this is what needs to be done to make dynamic 
> libraries properly work in D :
>
> * Initialize module infos (module constructors and similar)
> * Add TLS variables
> * Add exception handling tables
> * Add GC roots
>
> The above four things need to be extracted from the loaded 
> dynamic library and it gets loaded and preferably remove them 
> as well when the dynamic library gets unloaded. Currently this 
> is only extracted from the running executable. This is platform 
> dependent but usually it's extracted using bracketed sections 
> via extern C variables.

Should this be made automatically by the compiler?

This would be my expectation based on my experience with dynamic 
libraries in Turbo Pascal/Delphi.

--
Paulo
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On 2012-08-20 13:38, Paulo Pinto wrote:

> Should this be made automatically by the compiler?
>
> This would be my expectation based on my experience with dynamic
> libraries in Turbo Pascal/Delphi.

This should be handled automatically by the runtime.

-- 
/Jacob Carlborg
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On Monday, 20 August 2012 at 12:15:28 UTC, Jacob Carlborg wrote:
> On 2012-08-20 13:38, Paulo Pinto wrote:
>
>> Should this be made automatically by the compiler?
>>
>> This would be my expectation based on my experience with 
>> dynamic
>> libraries in Turbo Pascal/Delphi.
>
> This should be handled automatically by the runtime.

Ah ok, from your explanation I understood as something we are 
required to do manually.
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On 2012-08-20 16:22, Paulo Pinto wrote:

> Ah ok, from your explanation I understood as something we are required
> to do manually.

I just explained what changes need to be done in order for it to work. 
So we need to modify druntime to do what I listed above.

-- 
/Jacob Carlborg
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
>
> I'm not sure I'm following what you exactly have done here but 
> in general this is what needs to be done to make dynamic 
> libraries properly work in D :
>
> * Initialize module infos (module constructors and similar)
> * Add TLS variables
> * Add exception handling tables
> * Add GC roots
>
> The above four things need to be extracted from the loaded 
> dynamic library and it gets loaded and preferably remove them 
> as well when the dynamic library gets unloaded. Currently this 
> is only extracted from the running executable. This is platform 
> dependent but usually it's extracted using bracketed sections 
> via extern C variables.

OK, good to know. Any further hints about these, or where I can 
look?

What I've done is use the C dynamic loading library (header 
dlfcn.h) to manually load a shared object written in D, dload.d, 
from a D program (main.d), which then successfully calls 
functions in dload.d that are not defined extern(C).

I am attempting a do-it-yourself dynamic loading in D, where I 
explicitly do all the administration manually to make it work, 
rather than rely upon D to do it automatically. Hence my use of 
the C dynamic loading library, which knows nothing of additional 
work D must do.

Reason for this approach: the newsgroups indicate that dynamic 
loading in D does not work as yet.

What's novel is that I explicitly excluded from the shared object 
anything but code generated directly from dload.d:

dmd -c dload.d -m64 -fPIC
ld dload.o -shared -o libdload.so -m elf_x86_64 -E

And I included the whole of libphobos.a in the build of the main 
program:

dmd -c main.d -m64
gcc main.o -o main -m64 -Wl,-E -ldl -Wl,--whole-archive -lphobos2 
\
-Wl,--no-whole-archive -lcurl -lpthread -lm -lrt

This tactic is in the hope that all parts of D used in the shared 
library will find their linkage in the main program when it's 
dynamically loaded. (And the -E option passed to the linker in 
both cases is to expose all symbols for exactly this purpose.)

Reason for this approach: the newsgroups indicate that D 
runtime/phobos (all currently in libphobos.a it seems) does not 
initialize properly in a shared library, so I ensure that it's 
not present at all, and endeavor to have the shared object 
implicitly use the properly working D runtime/phobos in the main 
executable.

[And besides, at 64 bits all code in shared libraries apparently 
must be position independent, so even if I wanted to link parts 
of libphobos.a into the shared library, I couldn't without 
recompiling libphobos.a with the -fPIC option!]

This bare-bones-in-the-shared-library approach has worked well in 
my toy example. The only thing that apparently doesn't work is if 
an exception is thrown from the static initializer 'static 
this()' in the shared object. If an exception results from a call 
chain initiated by the main program even if thrown from a 
function in the shared object, all is well it seems.

Incidentally, I fibbed slightly about the ld command used to link 
the shared object. It also contains a trailing

  -init=$(shell staticCtor dload.o)

which enables the linker to bind in execution of the static 
initializer 'static this()' so that it runs automatically when 
the shared library is loaded. staticCtor is a script that 
analyzes dload.o to find its name, which is mangled of course.

If anyone can give me any more specific information about what 
else I can make happen manually to complete effective linkage at 
the D level I'd be grateful. I'm not stopping the investigation 
here!
August 20, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On 2012-08-20 17:16, Carl Sturtivant wrote:
>
>>
>> I'm not sure I'm following what you exactly have done here but in
>> general this is what needs to be done to make dynamic libraries
>> properly work in D :
>>
>> * Initialize module infos (module constructors and similar)
>> * Add TLS variables
>> * Add exception handling tables
>> * Add GC roots
>>
>> The above four things need to be extracted from the loaded dynamic
>> library and it gets loaded and preferably remove them as well when the
>> dynamic library gets unloaded. Currently this is only extracted from
>> the running executable. This is platform dependent but usually it's
>> extracted using bracketed sections via extern C variables.
>
> OK, good to know. Any further hints about these, or where I can look?

From druntime:

* rt.minfo - module infos. The "_moduleinfo_array" variable contains the 
module infos.

* rt.deh2 - exception handling. On Mac OS X the "_deh_eh_array" contains 
the exception handling tables. On the other Posix systems the sections 
bracketed by "_deh_beg" and "_deh_end" contains the tables.

* rt.thread - threading, TLS. On Mac OS X the TLS variables are stored 
in "_tls_data_array". On the other Posix systems they're are again 
stored in bracketed sections: "_tlsstart" and "_tlsend".

* rt.memory - GC roots. I think this module contains code related to 
adding GC roots. rt.memory_osx for Mac OS X.

You can also take a look at work by Martin Nowak:

https://github.com/dawgfoto/druntime/commits/SharedRuntime

> What I've done is use the C dynamic loading library (header dlfcn.h) to
> manually load a shared object written in D, dload.d, from a D program
> (main.d), which then successfully calls functions in dload.d that are
> not defined extern(C).
>
> I am attempting a do-it-yourself dynamic loading in D, where I
> explicitly do all the administration manually to make it work, rather
> than rely upon D to do it automatically. Hence my use of the C dynamic
> loading library, which knows nothing of additional work D must do.

The functions from dlfcn.h is what D would use as well. Possibly wrapped 
in some D function that does some additional work.

-- 
/Jacob Carlborg
August 22, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
>snip<

If we had dynamic loading, would we be able to do dependency
injection in D?
August 22, 2012
Re: Dynamic loading, D all the way (dmd 64bit 2.060/Ubuntu 64bit 12.04/x86_64)
On Wednesday, 22 August 2012 at 15:51:05 UTC, Philip Daniels 
wrote:
>>snip<
>
> If we had dynamic loading, would we be able to do dependency
> injection in D?

Dependency injection does not require dynamic loading per se.

It is all about using interfaces instead of classes, and 
initializing the corresponding instance members.

You just need some piece of code that takes the responsibility of 
locating such interfaces, by registering the classes somehow, or 
by compile time reflection, which gets called on program 
initialization.

You can do this in D today.

--
Paulo
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home