January 02, 2012 Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
I think that I'll defer the support for runtime loading of shared library (plugins) in favor of getting linked shared library support done now. There are several issues that require more thoughts. - Per-thread initialization of modules is somewhat tricky. Doing it in Runtime.loadLibrary requires knowledge of shared library dependencies because different threads might share dependencies but this is not provided by libc/libdl. - Libraries might not be unloaded as long as GC collected class instances still exist because finalization fails otherwise. - Getting symbols through mangled names is difficult/unstable. - D libraries used by a C library should provide proper runtime initialization even if the C library is used by a D application. Any ideas or use-cases for plugins are welcome. martin |
January 02, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Martin Nowak | On 2012-01-02 20:20, Martin Nowak wrote: > I think that I'll defer the support for runtime loading of shared > library (plugins) > in favor of getting linked shared library support done now. > There are several issues that require more thoughts. > > - Per-thread initialization of modules is somewhat tricky. > Doing it in Runtime.loadLibrary requires knowledge of shared library > dependencies > because different threads might share dependencies but this is not > provided by libc/libdl. > > - Libraries might not be unloaded as long as GC collected class > instances still exist because > finalization fails otherwise. > > - Getting symbols through mangled names is difficult/unstable. > > - D libraries used by a C library should provide proper runtime > initialization > even if the C library is used by a D application. > > Any ideas or use-cases for plugins are welcome. > > martin - Initializing module infos - Initializing exception handling tables - Running module constructors - Initializing TLS Then also unload all this when the library is unloaded. On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it will work, hopefully, transparently for the user. D libraries used by C wouldn't need any different handling. Because they will be linked with druntime it can initializing everything with the help of "_dyld_register_func_for_add_image". -- /Jacob Carlborg |
January 02, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jacob Carlborg | On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote: > On 2012-01-02 20:20, Martin Nowak wrote: >> I think that I'll defer the support for runtime loading of shared >> library (plugins) >> in favor of getting linked shared library support done now. >> There are several issues that require more thoughts. >> >> - Per-thread initialization of modules is somewhat tricky. >> Doing it in Runtime.loadLibrary requires knowledge of shared library >> dependencies >> because different threads might share dependencies but this is not >> provided by libc/libdl. >> >> - Libraries might not be unloaded as long as GC collected class >> instances still exist because >> finalization fails otherwise. >> >> - Getting symbols through mangled names is difficult/unstable. >> >> - D libraries used by a C library should provide proper runtime >> initialization >> even if the C library is used by a D application. >> >> Any ideas or use-cases for plugins are welcome. >> >> martin > > > - Initializing module infos > - Initializing exception handling tables > - Running module constructors > - Initializing TLS > > Then also unload all this when the library is unloaded. > It seems that libraries can't be unloaded deterministically, because GC finalization still references them. > On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it will work, hopefully, transparently for the user. D libraries used by C wouldn't need any different handling. Because they will be linked with druntime it can initializing everything with the help of "_dyld_register_func_for_add_image". > That was the approach I took and it is partly a dead-end. I have a mechanism similar to _dyld_register_func_for_add_image but runtime loaders have no notion of per-thread initialization, i.e. when two threads load the same library only the first one will actually cause the image to be loaded. This implies that the second thread would need to check all dependencies of the loaded library to do the initialization. I've written something along this line but it requires to exploit/rewrite part of the runtime linker. Using dlmopen on linux would be a terrible inefficient hack around this issue, it allows to load libraries multiple times. |
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Martin Nowak | On 2012-01-02 21:57, Martin Nowak wrote: > On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote: > >> On 2012-01-02 20:20, Martin Nowak wrote: >>> I think that I'll defer the support for runtime loading of shared >>> library (plugins) >>> in favor of getting linked shared library support done now. >>> There are several issues that require more thoughts. >>> >>> - Per-thread initialization of modules is somewhat tricky. >>> Doing it in Runtime.loadLibrary requires knowledge of shared library >>> dependencies >>> because different threads might share dependencies but this is not >>> provided by libc/libdl. >>> >>> - Libraries might not be unloaded as long as GC collected class >>> instances still exist because >>> finalization fails otherwise. >>> >>> - Getting symbols through mangled names is difficult/unstable. >>> >>> - D libraries used by a C library should provide proper runtime >>> initialization >>> even if the C library is used by a D application. >>> >>> Any ideas or use-cases for plugins are welcome. >>> >>> martin >> >> >> - Initializing module infos >> - Initializing exception handling tables >> - Running module constructors >> - Initializing TLS >> >> Then also unload all this when the library is unloaded. >> > It seems that libraries can't be unloaded deterministically, > because GC finalization still references them. > >> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then >> it will work, hopefully, transparently for the user. D libraries used >> by C wouldn't need any different handling. Because they will be linked >> with druntime it can initializing everything with the help of >> "_dyld_register_func_for_add_image". >> > > That was the approach I took and it is partly a dead-end. > > I have a mechanism similar to _dyld_register_func_for_add_image > but runtime loaders have no notion of per-thread initialization, > i.e. when two threads load the same library only the first one will > actually cause the image to be loaded. > This implies that the second thread would need to check all > dependencies of the loaded library to do the initialization. > I've written something along this line but it requires to > exploit/rewrite part of the runtime linker. > Using dlmopen on linux would be a terrible inefficient hack > around this issue, it allows to load libraries multiple times. I'm not quite sure I understand. Most of the things that should be done, initializing module infos and so on, should only be done once. -- /Jacob Carlborg |
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Martin Nowak | On 2012-01-02 21:57, Martin Nowak wrote: >> Then also unload all this when the library is unloaded. >> > It seems that libraries can't be unloaded deterministically, > because GC finalization still references them. Could you elaborate this, I guess I'm not really familiar with the GC finalization. -- /Jacob Carlborg |
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jacob Carlborg | On Tue, 03 Jan 2012 08:22:47 +0100, Jacob Carlborg <doob@me.com> wrote:
> On 2012-01-02 21:57, Martin Nowak wrote:
>>> Then also unload all this when the library is unloaded.
>>>
>> It seems that libraries can't be unloaded deterministically,
>> because GC finalization still references them.
>
> Could you elaborate this, I guess I'm not really familiar with the GC finalization.
>
I stumbled over this issue while doing extended testing.
Consider unloading of a library, this requires you to not
use any data/functions from that library afterward.
If you have instantiated a class from that library the GC
will access it's classinfo when collecting it, but the
classinfo resides in the unloaded library as do finalizers.
Not collection all classes from a shared library before
unloading that library is currently impracticable.
We could allocate classes from a specific allocator associated
with the library, somewhat similar to how TLS is allocated for
each library.
|
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jacob Carlborg | On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob@me.com> wrote:
> On 2012-01-02 21:57, Martin Nowak wrote:
>> On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote:
>>
>>> On 2012-01-02 20:20, Martin Nowak wrote:
>>>> I think that I'll defer the support for runtime loading of shared
>>>> library (plugins)
>>>> in favor of getting linked shared library support done now.
>>>> There are several issues that require more thoughts.
>>>>
>>>> - Per-thread initialization of modules is somewhat tricky.
>>>> Doing it in Runtime.loadLibrary requires knowledge of shared library
>>>> dependencies
>>>> because different threads might share dependencies but this is not
>>>> provided by libc/libdl.
>>>>
>>>> - Libraries might not be unloaded as long as GC collected class
>>>> instances still exist because
>>>> finalization fails otherwise.
>>>>
>>>> - Getting symbols through mangled names is difficult/unstable.
>>>>
>>>> - D libraries used by a C library should provide proper runtime
>>>> initialization
>>>> even if the C library is used by a D application.
>>>>
>>>> Any ideas or use-cases for plugins are welcome.
>>>>
>>>> martin
>>>
>>>
>>> - Initializing module infos
>>> - Initializing exception handling tables
>>> - Running module constructors
>>> - Initializing TLS
>>>
>>> Then also unload all this when the library is unloaded.
>>>
>> It seems that libraries can't be unloaded deterministically,
>> because GC finalization still references them.
>>
>>> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then
>>> it will work, hopefully, transparently for the user. D libraries used
>>> by C wouldn't need any different handling. Because they will be linked
>>> with druntime it can initializing everything with the help of
>>> "_dyld_register_func_for_add_image".
>>>
>>
>> That was the approach I took and it is partly a dead-end.
>>
>> I have a mechanism similar to _dyld_register_func_for_add_image
>> but runtime loaders have no notion of per-thread initialization,
>> i.e. when two threads load the same library only the first one will
>> actually cause the image to be loaded.
>> This implies that the second thread would need to check all
>> dependencies of the loaded library to do the initialization.
>> I've written something along this line but it requires to
>> exploit/rewrite part of the runtime linker.
>> Using dlmopen on linux would be a terrible inefficient hack
>> around this issue, it allows to load libraries multiple times.
>
> I'm not quite sure I understand. Most of the things that should be done, initializing module infos and so on, should only be done once.
>
Yes most, but not all.
The core issue here is that C++'s __thread doesn't allow dynamic initializers,
thus there is no infrastructure to do such things. And really a clean approach
would be to extend libc/ld.so.
|
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
The trick seems to be mapping in TLS (on OSX anyway) and running static crore at the right time. Are there other issues as well?
Sent from my iPhone
On Jan 3, 2012, at 8:53 AM, "Martin Nowak" <dawg@dawgfoto.de> wrote:
> On Tue, 03 Jan 2012 08:20:38 +0100, Jacob Carlborg <doob@me.com> wrote:
>
>> On 2012-01-02 21:57, Martin Nowak wrote:
>>> On Mon, 02 Jan 2012 20:38:50 +0100, Jacob Carlborg <doob@me.com> wrote:
>>>
>>>> On 2012-01-02 20:20, Martin Nowak wrote:
>>>>> I think that I'll defer the support for runtime loading of shared
>>>>> library (plugins)
>>>>> in favor of getting linked shared library support done now.
>>>>> There are several issues that require more thoughts.
>>>>>
>>>>> - Per-thread initialization of modules is somewhat tricky.
>>>>> Doing it in Runtime.loadLibrary requires knowledge of shared library
>>>>> dependencies
>>>>> because different threads might share dependencies but this is not
>>>>> provided by libc/libdl.
>>>>>
>>>>> - Libraries might not be unloaded as long as GC collected class
>>>>> instances still exist because
>>>>> finalization fails otherwise.
>>>>>
>>>>> - Getting symbols through mangled names is difficult/unstable.
>>>>>
>>>>> - D libraries used by a C library should provide proper runtime
>>>>> initialization
>>>>> even if the C library is used by a D application.
>>>>>
>>>>> Any ideas or use-cases for plugins are welcome.
>>>>>
>>>>> martin
>>>>
>>>>
>>>> - Initializing module infos
>>>> - Initializing exception handling tables
>>>> - Running module constructors
>>>> - Initializing TLS
>>>>
>>>> Then also unload all this when the library is unloaded.
>>>>
>>> It seems that libraries can't be unloaded deterministically, because GC finalization still references them.
>>>
>>>> On Mac OS X, can't "_dyld_register_func_for_add_image" be used? Then it will work, hopefully, transparently for the user. D libraries used by C wouldn't need any different handling. Because they will be linked with druntime it can initializing everything with the help of "_dyld_register_func_for_add_image".
>>>>
>>>
>>> That was the approach I took and it is partly a dead-end.
>>>
>>> I have a mechanism similar to _dyld_register_func_for_add_image
>>> but runtime loaders have no notion of per-thread initialization,
>>> i.e. when two threads load the same library only the first one will
>>> actually cause the image to be loaded.
>>> This implies that the second thread would need to check all
>>> dependencies of the loaded library to do the initialization.
>>> I've written something along this line but it requires to
>>> exploit/rewrite part of the runtime linker.
>>> Using dlmopen on linux would be a terrible inefficient hack
>>> around this issue, it allows to load libraries multiple times.
>>
>> I'm not quite sure I understand. Most of the things that should be done, initializing module infos and so on, should only be done once.
>>
> Yes most, but not all.
> The core issue here is that C++'s __thread doesn't allow dynamic initializers,
> thus there is no infrastructure to do such things. And really a clean approach
> would be to extend libc/ld.so.
|
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Martin Nowak | On 2012-01-03 17:49, Martin Nowak wrote: > On Tue, 03 Jan 2012 08:22:47 +0100, Jacob Carlborg <doob@me.com> wrote: > >> On 2012-01-02 21:57, Martin Nowak wrote: >>>> Then also unload all this when the library is unloaded. >>>> >>> It seems that libraries can't be unloaded deterministically, >>> because GC finalization still references them. >> >> Could you elaborate this, I guess I'm not really familiar with the GC >> finalization. >> > I stumbled over this issue while doing extended testing. > Consider unloading of a library, this requires you to not > use any data/functions from that library afterward. > If you have instantiated a class from that library the GC > will access it's classinfo when collecting it, but the > classinfo resides in the unloaded library as do finalizers. Do they? Isn't classinfo part of the module info and the module info needs to be collected in one place otherwise ClassInfo.find and similar methods won't work correctly? Or can those functions be implemented to iterate all loaded libraries and then look through each module info? > Not collection all classes from a shared library before > unloading that library is currently impracticable. Can't the runtime just run all finalizers for a given library when it's unloaded? -- /Jacob Carlborg |
January 03, 2012 Re: Ideas for runtime loading of shared libraries. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Martin Nowak | On 2012-01-03 17:53, Martin Nowak wrote: > Yes most, but not all. > The core issue here is that C++'s __thread doesn't allow dynamic > initializers, > thus there is no infrastructure to do such things. And really a clean > approach > would be to extend libc/ld.so. First, __thread isn't supported in Mac OS X (if we're talking about that). For all the operating systems that do support TLS I'm pretty sure that TLS and dynamic libraries work. This documentation: http://www.akkadia.org/drepper/tls.pdf mentions several different TLS modes, some used for dynamic libraries and some used for other cases. -- /Jacob Carlborg |
Copyright © 1999-2021 by the D Language Foundation