I think that I'll defer the support for runtime loading of shared library (plugins)
in favor of getting linked shared library support done now.
There are several issues that require more thoughts.

 - Per-thread initialization of modules is somewhat tricky.
   Doing it in Runtime.loadLibrary requires knowledge of shared library dependencies
   because different threads might share dependencies but this is not provided by libc/libdl.

 - Libraries might not be unloaded as long as GC collected class instances still exist because
   finalization fails otherwise.

 - Getting symbols through mangled names is difficult/unstable.

 - D libraries used by a C library should provide proper runtime initialization
   even if the C library is used by a D application.

Any ideas or use-cases for plugins are welcome.

martin

On 2012-01-02 20:20, Martin Nowak wrote:
> I think that I'll defer the support for runtime loading of shared
> library (plugins)
> in favor of getting linked shared library support done now.
> There are several issues that require more thoughts.
>
> - Per-thread initialization of modules is somewhat tricky.
> Doing it in Runtime.loadLibrary requires knowledge of shared library
> dependencies
> because different threads might share dependencies but this is not
> provided by libc/libdl.
>
> - Libraries might not be unloaded as long as GC collected class
> instances still exist because
> finalization fails otherwise.
>
> - Getting symbols through mangled names is difficult/unstable.
>
> - D libraries used by a C library should provide proper runtime
> initialization
> even if the C library is used by a D application.
>
> Any ideas or use-cases for plugins are welcome.
>
> martin


- Initializing module infos
- Initializing exception handling tables
- Running module constructors
- Initializing TLS

Then also unload all this when the library is unloaded.

On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it will work, hopefully, transparently for the user. D libraries used by C wouldn't need any different handling. Because they will be linked with druntime it can initializing everything with the help of "_dyld_register_func_for_add_image".

-- 
/Jacob Carlborg

On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote:

> On 2012-01-02 20:20, Martin Nowak wrote:
>> I think that I'll defer the support for runtime loading of shared
>> library (plugins)
>> in favor of getting linked shared library support done now.
>> There are several issues that require more thoughts.
>>
>> - Per-thread initialization of modules is somewhat tricky.
>> Doing it in Runtime.loadLibrary requires knowledge of shared library
>> dependencies
>> because different threads might share dependencies but this is not
>> provided by libc/libdl.
>>
>> - Libraries might not be unloaded as long as GC collected class
>> instances still exist because
>> finalization fails otherwise.
>>
>> - Getting symbols through mangled names is difficult/unstable.
>>
>> - D libraries used by a C library should provide proper runtime
>> initialization
>> even if the C library is used by a D application.
>>
>> Any ideas or use-cases for plugins are welcome.
>>
>> martin
>
>
> - Initializing module infos
> - Initializing exception handling tables
> - Running module constructors
> - Initializing TLS
>
> Then also unload all this when the library is unloaded.
>
It seems that libraries can't be unloaded deterministically,
because GC finalization still references them.

> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it will work, hopefully, transparently for the user. D libraries used by C wouldn't need any different handling. Because they will be linked with druntime it can initializing everything with the help of "_dyld_register_func_for_add_image".
>

That was the approach I took and it is partly a dead-end.

I have a mechanism similar to _dyld_register_func_for_add_image
but runtime loaders have no notion of per-thread initialization,
i.e. when two threads load the same library only the first one will
actually cause the image to be loaded.
This implies that the second thread would need to check all
dependencies of the loaded library to do the initialization.
I've written something along this line but it requires to
exploit/rewrite part of the runtime linker.
Using dlmopen on linux would be a terrible inefficient hack
around this issue, it allows to load libraries multiple times.

On 2012-01-02 21:57, Martin Nowak wrote:
> On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote:
>
>> On 2012-01-02 20:20, Martin Nowak wrote:
>>> I think that I'll defer the support for runtime loading of shared
>>> library (plugins)
>>> in favor of getting linked shared library support done now.
>>> There are several issues that require more thoughts.
>>>
>>> - Per-thread initialization of modules is somewhat tricky.
>>> Doing it in Runtime.loadLibrary requires knowledge of shared library
>>> dependencies
>>> because different threads might share dependencies but this is not
>>> provided by libc/libdl.
>>>
>>> - Libraries might not be unloaded as long as GC collected class
>>> instances still exist because
>>> finalization fails otherwise.
>>>
>>> - Getting symbols through mangled names is difficult/unstable.
>>>
>>> - D libraries used by a C library should provide proper runtime
>>> initialization
>>> even if the C library is used by a D application.
>>>
>>> Any ideas or use-cases for plugins are welcome.
>>>
>>> martin
>>
>>
>> - Initializing module infos
>> - Initializing exception handling tables
>> - Running module constructors
>> - Initializing TLS
>>
>> Then also unload all this when the library is unloaded.
>>
> It seems that libraries can't be unloaded deterministically,
> because GC finalization still references them.
>
>> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
>> it will work, hopefully, transparently for the user. D libraries used
>> by C wouldn't need any different handling. Because they will be linked
>> with druntime it can initializing everything with the help of
>> "_dyld_register_func_for_add_image".
>>
>
> That was the approach I took and it is partly a dead-end.
>
> I have a mechanism similar to _dyld_register_func_for_add_image
> but runtime loaders have no notion of per-thread initialization,
> i.e. when two threads load the same library only the first one will
> actually cause the image to be loaded.
> This implies that the second thread would need to check all
> dependencies of the loaded library to do the initialization.
> I've written something along this line but it requires to
> exploit/rewrite part of the runtime linker.
> Using dlmopen on linux would be a terrible inefficient hack
> around this issue, it allows to load libraries multiple times.

I'm not quite sure I understand. Most of the things that should be done, initializing module infos and so on, should only be done once.

-- 
/Jacob Carlborg

On 2012-01-02 21:57, Martin Nowak wrote:
>> Then also unload all this when the library is unloaded.
>>
> It seems that libraries can't be unloaded deterministically,
> because GC finalization still references them.

Could you elaborate this, I guess I'm not really familiar with the GC finalization.

-- 
/Jacob Carlborg

On Tue, 03 Jan 2012 08:22:47 +0100, Jacob Carlborg <doob@me.com> wrote:

> On 2012-01-02 21:57, Martin Nowak wrote:
>>> Then also unload all this when the library is unloaded.
>>>
>> It seems that libraries can't be unloaded deterministically,
>> because GC finalization still references them.
>
> Could you elaborate this, I guess I'm not really familiar with the GC finalization.
>
I stumbled over this issue while doing extended testing.
Consider unloading of a library, this requires you to not
use any data/functions from that library afterward.
If you have instantiated a class from that library the GC
will access it's classinfo when collecting it, but the
classinfo resides in the unloaded library as do finalizers.

Not collection all classes from a shared library before
unloading that library is currently impracticable.

We could allocate classes from a specific allocator associated
with the library, somewhat similar to how TLS is allocated for
each library.

On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob@me.com> wrote:

> On 2012-01-02 21:57, Martin Nowak wrote:
>> On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote:
>>
>>> On 2012-01-02 20:20, Martin Nowak wrote:
>>>> I think that I'll defer the support for runtime loading of shared
>>>> library (plugins)
>>>> in favor of getting linked shared library support done now.
>>>> There are several issues that require more thoughts.
>>>>
>>>> - Per-thread initialization of modules is somewhat tricky.
>>>> Doing it in Runtime.loadLibrary requires knowledge of shared library
>>>> dependencies
>>>> because different threads might share dependencies but this is not
>>>> provided by libc/libdl.
>>>>
>>>> - Libraries might not be unloaded as long as GC collected class
>>>> instances still exist because
>>>> finalization fails otherwise.
>>>>
>>>> - Getting symbols through mangled names is difficult/unstable.
>>>>
>>>> - D libraries used by a C library should provide proper runtime
>>>> initialization
>>>> even if the C library is used by a D application.
>>>>
>>>> Any ideas or use-cases for plugins are welcome.
>>>>
>>>> martin
>>>
>>>
>>> - Initializing module infos
>>> - Initializing exception handling tables
>>> - Running module constructors
>>> - Initializing TLS
>>>
>>> Then also unload all this when the library is unloaded.
>>>
>> It seems that libraries can't be unloaded deterministically,
>> because GC finalization still references them.
>>
>>> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
>>> it will work, hopefully, transparently for the user. D libraries used
>>> by C wouldn't need any different handling. Because they will be linked
>>> with druntime it can initializing everything with the help of
>>> "_dyld_register_func_for_add_image".
>>>
>>
>> That was the approach I took and it is partly a dead-end.
>>
>> I have a mechanism similar to _dyld_register_func_for_add_image
>> but runtime loaders have no notion of per-thread initialization,
>> i.e. when two threads load the same library only the first one will
>> actually cause the image to be loaded.
>> This implies that the second thread would need to check all
>> dependencies of the loaded library to do the initialization.
>> I've written something along this line but it requires to
>> exploit/rewrite part of the runtime linker.
>> Using dlmopen on linux would be a terrible inefficient hack
>> around this issue, it allows to load libraries multiple times.
>
> I'm not quite sure I understand. Most of the things that should be done, initializing module infos and so on, should only be done once.
>
Yes most, but not all.
The core issue here is that C++'s __thread doesn't allow dynamic initializers,
thus there is no infrastructure to do such things. And really a clean approach
would be to extend libc/ld.so.

The trick seems to be mapping in TLS (on OSX anyway) and running static crore at the right time. Are there other issues as well?

Sent from my iPhone

On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg@dawgfoto.de> wrote:

> On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob@me.com> wrote:
> 
>> On 2012-01-02 21:57, Martin Nowak wrote:
>>> On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote:
>>> 
>>>> On 2012-01-02 20:20, Martin Nowak wrote:
>>>>> I think that I'll defer the support for runtime loading of shared
>>>>> library (plugins)
>>>>> in favor of getting linked shared library support done now.
>>>>> There are several issues that require more thoughts.
>>>>> 
>>>>> - Per-thread initialization of modules is somewhat tricky.
>>>>> Doing it in Runtime.loadLibrary requires knowledge of shared library
>>>>> dependencies
>>>>> because different threads might share dependencies but this is not
>>>>> provided by libc/libdl.
>>>>> 
>>>>> - Libraries might not be unloaded as long as GC collected class
>>>>> instances still exist because
>>>>> finalization fails otherwise.
>>>>> 
>>>>> - Getting symbols through mangled names is difficult/unstable.
>>>>> 
>>>>> - D libraries used by a C library should provide proper runtime
>>>>> initialization
>>>>> even if the C library is used by a D application.
>>>>> 
>>>>> Any ideas or use-cases for plugins are welcome.
>>>>> 
>>>>> martin
>>>> 
>>>> 
>>>> - Initializing module infos
>>>> - Initializing exception handling tables
>>>> - Running module constructors
>>>> - Initializing TLS
>>>> 
>>>> Then also unload all this when the library is unloaded.
>>>> 
>>> It seems that libraries can't be unloaded deterministically, because GC finalization still references them.
>>> 
>>>> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it will work, hopefully, transparently for the user. D libraries used by C wouldn't need any different handling. Because they will be linked with druntime it can initializing everything with the help of "_dyld_register_func_for_add_image".
>>>> 
>>> 
>>> That was the approach I took and it is partly a dead-end.
>>> 
>>> I have a mechanism similar to _dyld_register_func_for_add_image
>>> but runtime loaders have no notion of per-thread initialization,
>>> i.e. when two threads load the same library only the first one will
>>> actually cause the image to be loaded.
>>> This implies that the second thread would need to check all
>>> dependencies of the loaded library to do the initialization.
>>> I've written something along this line but it requires to
>>> exploit/rewrite part of the runtime linker.
>>> Using dlmopen on linux would be a terrible inefficient hack
>>> around this issue, it allows to load libraries multiple times.
>> 
>> I'm not quite sure I understand. Most of the things that should be done, initializing module infos and so on, should only be done once.
>> 
> Yes most, but not all.
> The core issue here is that C++'s __thread doesn't allow dynamic initializers,
> thus there is no infrastructure to do such things. And really a clean approach
> would be to extend libc/ld.so.

On 2012-01-03 17:49, Martin Nowak wrote:
> On Tue, 03 Jan 2012 08:22:47 +0100, Jacob Carlborg <doob@me.com> wrote:
>
>> On 2012-01-02 21:57, Martin Nowak wrote:
>>>> Then also unload all this when the library is unloaded.
>>>>
>>> It seems that libraries can't be unloaded deterministically,
>>> because GC finalization still references them.
>>
>> Could you elaborate this, I guess I'm not really familiar with the GC
>> finalization.
>>
> I stumbled over this issue while doing extended testing.
> Consider unloading of a library, this requires you to not
> use any data/functions from that library afterward.
> If you have instantiated a class from that library the GC
> will access it's classinfo when collecting it, but the
> classinfo resides in the unloaded library as do finalizers.

Do they? Isn't classinfo part of the module info and the module info needs to be collected in one place otherwise ClassInfo.find and similar methods won't work correctly?

Or can those functions be implemented to iterate all loaded libraries and then look through each module info?

> Not collection all classes from a shared library before
> unloading that library is currently impracticable.

Can't the runtime just run all finalizers for a given library when it's unloaded?

-- 
/Jacob Carlborg

On 2012-01-03 17:53, Martin Nowak wrote:
> Yes most, but not all.
> The core issue here is that C++'s __thread doesn't allow dynamic
> initializers,
> thus there is no infrastructure to do such things. And really a clean
> approach
> would be to extend libc/ld.so.

First, __thread isn't supported in Mac OS X (if we're talking about that). For all the operating systems that do support TLS I'm pretty sure that TLS and dynamic libraries work.

This documentation:

http://www.akkadia.org/drepper/tls.pdf

mentions several different TLS modes, some used for dynamic libraries and some used for other cases.

-- 
/Jacob Carlborg