Thread overview
Building several dynamic libraries with one shared GC
Sep 12
NonNull
Sep 12
frame
Sep 12
NonNull
Sep 12
NonNull
Sep 12
NonNull
September 12

I am making a plug-in development system for a high performance Linux application that already exists and is written in C and will not be modified for this purpose. It is already has an API for plugins written in C. A plug-in simply makes new functions (e.g. manipulating text and numbers) available to be used inside the application. The D GC will intentionally be available inside a plugin.

I have written a D source file containing templates that do the work when a plugin is written by a third party in D. A plugin will expose a C API for the functions made available to the application, and any data in D made available to the application will be copied to the application's managed storage and vice versa, so there are no GC issues.

All of this is to make it as simple as possible for a third party to write such a plugin without worrying about any of these issues, and use D in a simplistic scripting language-like way. This is for non-experts. They just use the templates and build a dynamic library and all of the above is done for them.

If several plugins are built by different third parties, each dynamic library will have its own GC and copy of druntime right now. How can I organize that there is one separate dynamic library to share these among all plugins?

September 12

On Sunday, 12 September 2021 at 14:31:04 UTC, NonNull wrote:

>

If several plugins are built by different third parties, each dynamic library will have its own GC and copy of druntime right now. How can I organize that there is one separate dynamic library to share these among all plugins?

Shouldn't the runtime not already be shared on Linux? The Runtime.loadLibrary specs say
If the library contains a D runtime it will be integrated with the current runtime.

This should be true for the GC too. At least the memory is shared because as I remember I could access __gshared variables,

whereas on Windows nothing like this works and any DLL will spawn a new thread (for each thread you use too).

September 12

On Sunday, 12 September 2021 at 16:23:13 UTC, frame wrote:

>

Shouldn't the runtime not already be shared on Linux? The Runtime.loadLibrary specs say
If the library contains a D runtime it will be integrated with the current runtime.

This should be true for the GC too. At least the memory is shared because as I remember I could access __gshared variables,

whereas on Windows nothing like this works and any DLL will spawn a new thread (for each thread you use too).

The plugin libraries expose a C API and are dynamically loaded by the application which is written in C, so presumably using dlopen. No D runtime there. Still your reply does suggest a way to proceed...

September 12
On 9/12/21 7:31 AM, NonNull wrote:
> I am making a plug-in development system for a high performance Linux
> application that already exists and is written in C and will not be
> modified for this purpose.

I've done something similar but in my case the main application is in D and the plugins are code-generated D (hand edited by non-D developers).

However, as is common in software, the requirements changed :) and we wanted to use it as a library as well; so, plugins became part of a main D library (which exposed C functions).

All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called.

(I tried dlopen after guessing intelligently the name of the 'shared static this' function (not obvious); it was not satisfactory and I don't remember exactly why not.)

Later I learned, this could be because merely loading a plugin might not be enough, and perhaps the main library might have to use a feature of the library as well:

  https://forum.dlang.org/post/sdb5jb$2rk3$1@digitalmars.com

If that link indeed explains the issue, I did not know about it when we worked around the initialization problem by running a D daemon per thread behind this main library.

Each thread of the library passes information to its daemon through shared memory and the daeman does it's thing and returns the result back. (The difference is, the plugins are linked to the daemon; avoiding the problem initialization scenario; 'shared static this' are called from a D environment.)

> If several plugins are built by different third parties, each dynamic
> library will have its own GC and copy of druntime right now.

Like the user 'frame', I don't think that's the case.

Going off topic, I started thinking about this "one daemon per library thread" idea: becaues I am under the impression that running multiple daemons does not put much stress on the system under Linux, this idea can automatically translate to "independent GCs for each thread". For example, the main library has 10 threads using 10 separate daemons on their backgrounds; each daemon handles its own GC needs without even knowing about the other daemons that are effectively working for the same main library. I haven't experimented with this yet.

Ali

September 12
On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote:
> All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called.
>
> (I tried dlopen after guessing intelligently the name of the 'shared static this' function (not obvious); it was not satisfactory and I don't remember exactly why not.)
>
> Later I learned, this could be because merely loading a plugin might not be enough, and perhaps the main library might have to use a feature of the library as well:
>
>   https://forum.dlang.org/post/sdb5jb$2rk3$1@digitalmars.com
>
[...]
>
> > If several plugins are built by different third parties, each
> dynamic
> > library will have its own GC and copy of druntime right now.
>
> Like the user 'frame', I don't think that's the case.


I hope you're right about this last. Not sure how static versus dynamic linking affects it. ldc2 seems to default to dynamic linking for phobos and have druntime in a dynamic library too. [Not clear what DMD does with druntime when it dynamically links to phobos.] In that ideal situation you seem to be right. Not sure how a plugin can be distributed as an executable only (to those without a D compiler installed) without static linking and then what? I have a mess to sort out.
Any info or suggestions?

September 12
On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote:
> All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called.

So here, your main dynamic library in turn dynamically loads plugins. Did you try simply calling a function exported by a plugin from the static constructor in the main library after it had made the call to initialize druntime to see if that stimulated running the plugin's static constructors first? The problem you linked to suggests that might do the job. I haven't run into this problem yet myself. But I'm interested.
September 12
On 9/12/21 1:25 PM, NonNull wrote:
> On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote:
>> All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called.
> 
> So here, your main dynamic library in turn dynamically loads plugins. Did you try simply calling a function exported by a plugin from the static constructor in the main library after it had made the call to initialize druntime to see if that stimulated running the plugin's static constructors first? The problem you linked to suggests that might do the job. I haven't run into this problem yet myself. But I'm interested.

I did not try it because I learned about that potential cause after we worked around the issue with the daemons.

Ali