Thread overview
Shared library support: Where we are at in 2023
Apr 22, 2023
Walter Bright
Apr 23, 2023
ryuukk_
April 21, 2023

Part 2 of: https://forum.dlang.org/post/ghsigisnciolxsiyaehu@forum.dlang.org

Rainer has been working on dmd support of DLL's, if completed this should fix export on Windows minus the ability to annotate DllImport/Internal in: https://github.com/dlang/dmd/pull/14849

So I've got a list of bugs and enhancements that'll get us further down the road of shared library support.

All of these feature are in my WIP export DIP, however as pretty much everything there could probably be implemented without the DIP lets start with the things I think have the greatest improvements in both mine and other peoples lives.

As an update to my own stuff, -betterC DLL with full D executable (unit testing) works even with dmd on Windows. Implementing the list of issues brought up here should allow anyone to repeat what I've done with minimal understanding of how linkers work as long as the appropriete symbols are annotated with export.

Telling the compiler about symbol location

The compiler needs to know if a module is going inside of the binary or not (i.e. is in a dependency shared library).

  1. It allows for the compiler to error if a templated symbol when instantiated references a non-exported symbol when the non-exported symbol is in another binary.
  2. Allows automatic differentiation between DllImport and internal of symbols marked export.
  3. Prevent unittests triggering the need for ModuleInfo when the unittest is in a module that is in an external binary.

This could be done in one of two ways, however one is the clear winner.

a. Use a pattern to match the full module name. This would not be automatic at the package/build manager level without slowing things down and would not be very cheap to process.
b. Use a second -I switch. This would be automatic at the package/build manager level and over all should be very cheap.

Fixes:

https://issues.dlang.org/show_bug.cgi?id=22367

Completes:

https://issues.dlang.org/show_bug.cgi?id=23850

Removal of workaround:

ModuleInfo stubs.

https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/moduleinfostubs.d

Prevents linker warnings:

https://learn.microsoft.com/en-us/cpp/error-messages/tool-errors/linker-tools-warning-lnk4217?view=msvc-170

https://learn.microsoft.com/en-us/cpp/error-messages/tool-errors/linker-tools-warning-lnk4286?view=msvc-170

This is quite an exciting possibility. Not having to mark symbols as DllImport/Internal will allow (in majority of cases) for these warnings to not occur. We will still need a way to annotate this information (easily solved design wise, but will leave it for later).

Just exporting a symbol will be enough even on Windows as long as the package/build manager helps out a little bit.

Generated symbols

When the compiler generates a symbol it should always be exported if it has other symbols in the encapsulation unit that are exported or is itself exported.

Encapsulation unit in this case includes: modules, classes, structs, unions, template blocks.

A non-exhaustive list of symbols that should be exported:

  1. TypeInfo
  2. ModuleInfo
  3. __init
  4. __fieldDtor
  5. __invariant
  6. opCmp
  7. opEquals
  8. toHash

Once export is fixed will fix:

https://issues.dlang.org/show_bug.cgi?id=6019

https://issues.dlang.org/show_bug.cgi?id=23177

Removal of workaround:

Allows to get rid of majority of manual exports in one of my DLL's.

https://github.com/Project-Sidero/basic_memory/blob/main/msvc_exports.def

There are some symbols that are from druntime (like atomics) that should be fixed with annotating them with export.

_D4core6atomic__T8atomicOpVAyaa2_2d3dTlTiZQzFNaNbNiNfKOliZl

_D4core8internal11destruction__T11__ArrayDtorTS6sidero4base10containers3map17concurrenthashmap__T21ConcurrentHashMapNodeTSQCxQCt4text7unicode13readonly_utf811String_UTF8TSQEuQEq8datetime4timeQHc4iana6TZFileZQEg6BucketZQHeFNbNiNfMAQHcZv

RTInfo should be emitted when used not on declaration

RTInfo should be emitted when used, this prevents unresolved external symbols with -betterC dependencies.

Fixes:

https://issues.dlang.org/show_bug.cgi?id=23820

Thread crt constructor

This may only apply to Windows, but have a thread C runtime constructor/destructor will allow us to kill off SimpleDllMain.

DllMain is a wholly optional concept in C/C++ land. There is no reason for it to exist in D either.

crt_constructor/destructor already serves half of its purpose, only left is the per-thread behavior.

There will be some other behaviors that the compiler will need to switch from DllMain detection to performing when -shared is passed. To complete this transition.

Completes:

https://issues.dlang.org/show_bug.cgi?id=23756

Set object files file name to be the full module name with package

Linkers like to de-duplicate object files. To do this, the cheapest way is to compare file names.

Problem is dmd is using only the base name and not the full package + module name.

LDC supports this as opt-in, but I argue that this should be the default.

It also means if you use packages if you have two modules with the same base name they won't conflict.

Completes:

https://issues.dlang.org/show_bug.cgi?id=3541

April 22, 2023
Thank you for taking the time to work all this stuff out. Nice!
April 23, 2023

If you are interested i can send your my project, i hotreload game scenes using DLL

With dmd: it freeze
With LDC: everyone works

Or, if you don't mind, could you send me a compiled dmd.exe so i could try your changes?

My project is currently on hold because of this issue and the TLS one

April 24, 2023
On 24/04/2023 4:29 AM, ryuukk_ wrote:
> If you are interested i can send your my project, i hotreload game scenes using DLL
> 
> With dmd: it freeze
> With LDC: everyone works

I can certainly take a look, Discord or email (firstname @ lastname . co.nz).

> Or, if you don't mind, could you send me a compiled dmd.exe so i could try your changes?

I don't have any changes to dmd, I'm working more at the requirements level to figure out what needs to happen to get things usable. Discovering problems and proposing possible solutions is hard enough ;)