I've been doing some work on shared libraries for OSX, and have come across a potential problem, which I'm not sure what to do with.

Currently, when a thread is spawned, that thread calls all the TLS ctors, then runs the thread entry point function, and finally calls the TLS dtors before the thread terminates.

Example, for windows:

https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236

So, the question is, how do dynamic libraries interact here?

Example: A dynamic library contains D modules with TLS ctors. If I start a few threads, and then load a dynamic library, shouldn't the TLS ctors in the dynamic library be called for each running thread? If my assumption is correct, the next question is, how do you do this?

I don't think you can hijack each thread and have it run the TLS ctors, and you can run them all from the thread loading the shared library because of synchronization issues. So what's the solution? Should TLS ctors in dynamic libraries simply be specified not to run, or could they somehow be run lazily at the first TLS access in a dynamic library?

Any thoughts on this would be appreciated.

Thanks,
   Bit

Am Thu, 17 Mar 2016 20:27:39 +0000
schrieb bitwise <bitwise.pvt@gmail.com>:

> I've been doing some work on shared libraries for OSX, and have come across a potential problem, which I'm not sure what to do with.
> 
> Currently, when a thread is spawned, that thread calls all the TLS ctors, then runs the thread entry point function, and finally calls the TLS dtors before the thread terminates.
> 
> Example, for windows:
> 
> https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236
> 
> So, the question is, how do dynamic libraries interact here?
> 
> Example: A dynamic library contains D modules with TLS ctors. If I start a few threads, and then load a dynamic library, shouldn't the TLS ctors in the dynamic library be called for each running thread? If my assumption is correct, the next question is, how do you do this?
> 
> I don't think you can hijack each thread and have it run the TLS ctors, and you can run them all from the thread loading the shared library because of synchronization issues. So what's the solution? Should TLS ctors in dynamic libraries simply be specified not to run, or could they somehow be run lazily at the first TLS access in a dynamic library?
> 
> Any thoughts on this would be appreciated.
> 
> Thanks,
>     Bit
> 

It's been some time since I looked at that code, but IIRC TLS ctors for newly loaded libraries are not run in old threads. There's nothing we can do about this without help from the C runtime.


As TLS ctors are meant to initialize TLS storage the correct time to run these is after the per-thread TLS storage allocation. (The C runtime can actually allocate TLS memory lazily, on the first access to a TLS variable in a module). The C runtimes of course know when TLS storage is allocated, but they do not provide hooks we could use.

On Thursday, 17 March 2016 at 20:54:57 UTC, Johannes Pfau wrote:
> It's been some time since I looked at that code, but IIRC TLS ctors for newly loaded libraries are not run in old threads. There's nothing we can do about this without help from the C runtime.

Hmm... are all module _destructors_ called when those threads exit? That would mean that some destructors may be called without the corresponding constructors having run...

Am Fri, 18 Mar 2016 14:35:41 +0000
schrieb Marc Schütz <schuetzm@gmx.net>:

> On Thursday, 17 March 2016 at 20:54:57 UTC, Johannes Pfau wrote:
> > It's been some time since I looked at that code, but IIRC TLS ctors for newly loaded libraries are not run in old threads. There's nothing we can do about this without help from the C runtime.
> 
> Hmm... are all module _destructors_ called when those threads exit? That would mean that some destructors may be called without the corresponding constructors having run...

No. Each thread keeps a thread-local list of shared libraries it knows about. This list is inherited from the parent thread on thread creation. It also gets updated if a new library is loaded at runtime (but only for the loading thread!). The runtime only runs constructors and destructors for libraries in the list. Additionally the GC only scans TLS memory of libraries in the list.


As an example:
* Create 2 Threads, TA & TB
* TB creates thread TB1
* TB loads library libA
* TB creates new thread TB2
* TA creates new thread TA1

Threads TB and TB2 will run TLS constructors and destructors for libA. modules. These threads will also scan TLS memory from modules in libA.

Threads TA, TA1 and TB1 will not run TLS constructors or destructors for libA. Additionally, the GC will not scan TLS memory for libA for these threads.

So you can only safely use modules from libA in thread TB and TB2.

Relevant links: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_elf_shared.d#L198 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_elf_shared.d#L281

https://github.com/D-Programming-Language/druntime/blob/master/src/core/thread.d#L385 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L312 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_elf_shared.d#L49 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_elf_shared.d#L76

On 17/03/16 21:27, bitwise wrote:
> I've been doing some work on shared libraries for OSX, and have come
> across a potential problem, which I'm not sure what to do with.
>
> Currently, when a thread is spawned, that thread calls all the TLS
> ctors, then runs the thread entry point function, and finally calls the
> TLS dtors before the thread terminates.
>
> Example, for windows:
>
> https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236
>
>
> So, the question is, how do dynamic libraries interact here?
>
> Example: A dynamic library contains D modules with TLS ctors. If I start
> a few threads, and then load a dynamic library, shouldn't the TLS ctors
> in the dynamic library be called for each running thread? If my
> assumption is correct, the next question is, how do you do this?
>
> I don't think you can hijack each thread and have it run the TLS ctors,
> and you can run them all from the thread loading the shared library
> because of synchronization issues. So what's the solution? Should TLS
> ctors in dynamic libraries simply be specified not to run, or could they
> somehow be run lazily at the first TLS access in a dynamic library?
>
> Any thoughts on this would be appreciated.

How does it behave on Linux? It already supports dynamic libraries.

-- 
/Jacob Carlborg

On Friday, 18 March 2016 at 16:23:00 UTC, Johannes Pfau wrote:
> Am Fri, 18 Mar 2016 14:35:41 +0000
> schrieb Marc Schütz <schuetzm@gmx.net>:
>
> [...]

Thanks for the explanation!

When I first looked at the elf implementation, I was confused by what was goign on with pinLoadedLibraries()/inheritLoadedLibraries(). With this problem in mind though, it all makes sense now.

    Bit

On Friday, 18 March 2016 at 18:52:25 UTC, Jacob Carlborg wrote:
> On 17/03/16 21:27, bitwise wrote:
>> I've been doing some work on shared libraries for OSX, and have come
>> across a potential problem, which I'm not sure what to do with.
>>
>> Currently, when a thread is spawned, that thread calls all the TLS
>> ctors, then runs the thread entry point function, and finally calls the
>> TLS dtors before the thread terminates.
>>
>> Example, for windows:
>>
>> https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236
>>
>>
>> So, the question is, how do dynamic libraries interact here?
>>
>> Example: A dynamic library contains D modules with TLS ctors. If I start
>> a few threads, and then load a dynamic library, shouldn't the TLS ctors
>> in the dynamic library be called for each running thread? If my
>> assumption is correct, the next question is, how do you do this?
>>
>> I don't think you can hijack each thread and have it run the TLS ctors,
>> and you can run them all from the thread loading the shared library
>> because of synchronization issues. So what's the solution? Should TLS
>> ctors in dynamic libraries simply be specified not to run, or could they
>> somehow be run lazily at the first TLS access in a dynamic library?
>>
>> Any thoughts on this would be appreciated.
>
> How does it behave on Linux? It already supports dynamic libraries.

After looking at the elf implementation, Johannes's explanation seems to be correct. I don't think it's the best solution though.

I'm wondering if the _exact_ behavior of the current solution was intentional. It seems like everything will work fine as long as you only use statically linked shared libs, so I'm wondering if dynamic linking was fully considered at the time the current solution was written. As far as dynamic linking goes, I don't like how the current solution works. It can lead to very confusing problems, and I think it would be better to do an all-or-nothing approach:

-statically linked shared libs always have shared static ctors called
-statically linked shared libs always have TLS static ctors called for all new threads
-dynamically linked shared libs always have shared static ctors called
-dynamically linked shared libs NEVER have TLS static ctors called for ANY thread

I can see how one could argue for benefits of the current approach, but I don't think it's worth exposing people to that kind of confusion, to have partially working TLS static ctors.

Thoughts?

    Bit

On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:
> Thoughts?
>
>     Bit

Ok..How about "Thought"? One thought will do. I'll take what I can get ;)

     Bit

On Friday, 25 March 2016 at 16:33:26 UTC, bitwise wrote:
> On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:
>> Thoughts?
>>
>>     Bit
>
> Ok..How about "Thought"? One thought will do. I'll take what I can get ;)
>
>      Bit

Martin is the guy to talk to, he wrote all that code.  David might have an opinion, as he adapted it for ldc.  Most everybody else doesn't go into those weeds.

On Saturday, 26 March 2016 at 03:08:13 UTC, Joakim wrote:
> On Friday, 25 March 2016 at 16:33:26 UTC, bitwise wrote:
>> On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:
>>> Thoughts?
>>>
>>>     Bit
>>
>> Ok..How about "Thought"? One thought will do. I'll take what I can get ;)
>>
>>      Bit
>
> Martin is the guy to talk to, he wrote all that code.  David might have an opinion, as he adapted it for ldc.  Most everybody else doesn't go into those weeds.

I was hoping to have some more people weigh in with their experience with plugins, whether or not they're expected to be multi-threaded, thread-safe, etc.. I think the current design is fragile, and given the limited usage of D shared libraries atm, I think it's a good time to come up with something a bit more solid/predictable.

Anyways, I guess I'll have to email Martin and David and see what they say.

    Bit