May 08
On 08/05/2024 7:15 PM, Walter Bright wrote:
> On 5/7/2024 8:45 PM, Richard (Rikki) Andrew Cattermole wrote:
>>> They don't have much in common with shared libraries on OSX and Posix.
>>
>> They do have plenty in common, this is a misconception I really want to get you off of. There is a dedicated heading for this ``Is a Dynamic Link Library a Shared Library?``.
> 
> Isn't it true that DLLs on Windows share their global data segment with all users of the DLL? While Linux shared libraries have a separate data segment for each process?
> 
> This is a very major difference.

So I was going to write out that I have no idea how you came to this conclusion and I'd love to hear the story that makes you think that this is true for a OS that is used by financial, governmental and military organizations.

And then I found MSVC link's ``/SECTION`` flag.

https://learn.microsoft.com/en-us/cpp/build/reference/section-specify-section-attributes?view=msvc-170

See the flag ``S``.

https://www.codeproject.com/Articles/240/How-to-share-a-data-segment-in-a-DLL

I haven't been able to find an equivalent linux ld flag.

Either way, this isn't the default and in no way would I ever consider that it should be used! Sounds like a big ball of insanity.

May 08

On Wednesday, 8 May 2024 at 07:15:20 UTC, Walter Bright wrote:

>

On 5/7/2024 8:45 PM, Richard (Rikki) Andrew Cattermole wrote:

> >

They don't have much in common with shared libraries on OSX and Posix.

They do have plenty in common, this is a misconception I really want to get you off of. There is a dedicated heading for this Is a Dynamic Link Library a Shared Library?.

Isn't it true that DLLs on Windows share their global data segment with all users of the DLL? While Linux shared libraries have a separate data segment for each process?

I've been writing DLL's since Windows 2000, and this has not been true for the entire time I've been doing this. As far as I know, if this was ever true, then it hasn't been true since Windows 3.1. The global data segment is shared at the per-process level, no further. And I for one do not want to consider the absolute chaos that would ensue if it were any other way... <runs away in terror>.

May 09
On Wednesday, 8 May 2024 at 05:13:55 UTC, Gregor Mückl wrote:
> On Wednesday, 8 May 2024 at 03:08:15 UTC, Walter Bright wrote:
>> Thanks for writing this.
>>
>> Are you writing solely about DLLs on Windows? They don't have much in common with shared libraries on OSX and Posix.
>
> That is confusing me as well. DLLs share concepts with shared libraries on other platforms, but they have subtle differences. The ones that come to my mind:
>
> - Shared libraries export everything by default. DLLs export nothing by default. This relates to the non-standard declspec(dllexport) declaration supported by MSVC to mark exported symbols.
>
> - Unix system linkers take shared libraries as input files directly. Windows linkers require import libraries. These import libraries contain thunks that jump to the real code in the DLL. Those thunks can be avoided if the compiler knows a symbol comes from a DLL. This is why declspec(dllimport) exists in MSVC (as a performance optimization).
>
> - DllMain() is a Windows only construct. If it is present, it is invoked for a lot of different events (PROCESS_ATTACH, THREAD_ATTACH...). Some Unix/Posix OSes support callbacks for loading/unloading libraries at most. The mechanisms are not equivalent.
>
> - And then there are all the funny ways in which static initialization in C++ can break in combination with Unix shared libraries. There are some fun, really opaque pitfalls like static constructors getting executed multiple times (and at times when you probably woudldn't expect). I don't think the same is true on Windows.
>
> These differences result in a number of things that are different in one model and not the other. On Unix, it's legal to have name collisions between symbols exported from different libraries. Typically, the first encountered symbol wins. This allows mechanisms like LD_PRELOAD to work and and use a program with a replacement malloc() implementation, for example. There is no Windows equivalent for this. You'd have to provide a shim DLL in the search path that provides all symbols.


This is also not fully correct.

The Windows DLL model is also present on Aix, including having export files, and similar kind of linker features. While Aix adopted ELf later on, COFF is still quite prevalent, having evolved into XCOFF.

Although no longer relevant, Symbian also used the same DLL model.

Then we Amiga Libraries, BeOS, IBM i, IBM z and Unisys ClearPath MCP, all of which are kind of their own thing.

While they all might be irrelevant for D, there is a bit more to shared libraries as only POSIX vs Windows.



May 09
On 09/05/2024 6:39 PM, Paulo Pinto wrote:
> The Windows DLL model is also present on Aix, including having export files, and similar kind of linker features. While Aix adopted ELf later on, COFF is still quite prevalent, having evolved into XCOFF.
> 
> Although no longer relevant, Symbian also used the same DLL model.
> 
> Then we Amiga Libraries, BeOS, IBM i, IBM z and Unisys ClearPath MCP, all of which are kind of their own thing.
> 
> While they all might be irrelevant for D, there is a bit more to shared libraries as only POSIX vs Windows.

Indeed, pretty much all platforms will do tuning for shared libraries to make it fit their needs. Windows is an extreme example of it, and is in active use by the D community so is worthy of a lot of attention.

I tried to explain that there are tunings that are used but apparently I didn't do so well at it as I don't have experience with any of them.

Any additional write up you would be willing to do, I'd be happy to add it attributed.
May 09

On Monday, 6 May 2024 at 03:28:47 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

This post is meant to be a highly enlightening and entertaining explanation (or should I say it shouldn't cure anyones insomnia) of just how many things can go wrong with shared libraries if they are not worked with right regardless of platform.

[...]

There were a few things here I didn't understand; sometimes whole sentences. I don't know what "siblings shared libraries" are, and "intermediary static libraries" only made sense to me further down when I understood that it was about linking several static libraries into a dynamic one.

Some questions:

On everything druntime-related, how is it done in C++? C (unless freestanding) and C++ both have runtimes despite some people pretending otherwise.

"you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with export on a declaration (as opposed to a definition)? That is, my understanding is that export with no body means dllimport and export with a body means dllexport.

Where does the need for "private but export" come from again? Is there an equivalent in C++ (static dllexport?), or does this only happen due to something specific to D like T.init?

"By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.

May 10
On 10/05/2024 10:04 AM, Atila Neves wrote:
> On Monday, 6 May 2024 at 03:28:47 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> This post is meant to be a highly enlightening and entertaining explanation (or should I say it shouldn't cure anyones insomnia) of just how many things can go wrong with shared libraries if they are not worked with right regardless of platform.
>>
>> [...]
> 
> There were a few things here I didn't understand; sometimes whole sentences. I don't know what "siblings shared libraries" are, and "intermediary static libraries" only made sense to me further down when I understood that it was about linking several static libraries into a dynamic one.
> 
> Some questions:
> 
> On everything druntime-related, how is it done in C++? C (unless freestanding) and C++ both have runtimes despite some people pretending otherwise.

You use the shared library build of it by default.

You can opt into using a switch to use a static library instead, but you are on your own.

This reflects my recommendations.

Oh hey, similar recommendations I gave can be found here: https://learn.microsoft.com/en-us/cpp/c-runtime-library/crt-library-features?view=msvc-170#what-problems-exist-if-an-application-uses-more-than-one-crt-version

"When you build a release version of your project, one of the basic C runtime libraries (libcmt.lib, msvcmrt.lib, msvcrt.lib) is linked by default"

libcmt.lib	Statically links the native CRT startup into your code.

msvcmrt.lib	Static library for the mixed native and managed CRT startup for use with DLL UCRT and vcruntime.

msvcrt.lib	Static library for the native CRT startup for use with DLL UCRT and vcruntime.

> "you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with `export` on a declaration (as opposed to a definition)? That is, my understanding is that `export` with no body means `dllimport` and `export` with a body means `dllexport`.

Keep in mind that nobody that I know of is using it to mean this today.

Also a D module may be in binary, but it may have declarations that point to something that is not.

See the bindings in druntime where it has D symbols.

https://github.com/dlang/dmd/blob/5fc02ba152ccaa71711f3ed84b6d44a2a940f206/druntime/src/core/stdc/stdio.d#L1191

This is why you don't pretend that one symbol in ``DllImport`` mode means the entire module is because it may be a binding to something else.

Note: there are plenty of examples of it which are not templated, like structs that have to be initialized using the D init array.

> Where does the need for "private but export" come from again? Is there an equivalent in C++ (`static dllexport`?), or does this only happen due to something specific to D like `T.init`?

It happens because D confuses exportation which is a linker concept, with a language visibility concept.

In C/C++ it uses a completely separate attribute to donate that it does not affect Member Access Control such as ``private``.

But you can see this with things like templates, you want to access an internal symbol but don't want somebody else to? Yeah no, can't do that today.

https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170

You have to be very explicit in c/c++ over this, we do not have that level of control (apart from saying do not export this symbol via ``@hidden``).

> "By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.

Yes two. They are referenced in the article.

Note: they are not duplicates.

Okay I lie there is a bunch more.

https://issues.dlang.org/show_bug.cgi?id=23850

https://issues.dlang.org/show_bug.cgi?id=23177

https://issues.dlang.org/show_bug.cgi?id=23974

https://issues.dlang.org/show_bug.cgi?id=6019

https://issues.dlang.org/show_bug.cgi?id=9816

Here is my workaround code to get around this problem: https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/moduleinfostubs.d

Not something I am proud of needing.
May 10
On Friday, 10 May 2024 at 02:05:18 UTC, Richard (Rikki) Andrew Cattermole wrote:
>
> On 10/05/2024 10:04 AM, Atila Neves wrote:
>> On Monday, 6 May 2024 at 03:28:47 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>> This post is meant to be a highly enlightening and entertaining explanation (or should I say it shouldn't cure anyones insomnia) of just how many things can go wrong with shared libraries if they are not worked with right regardless of platform.
>>>
>>> [...]
>> 

>> "you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with `export` on a declaration (as opposed to a definition)? That is, my understanding is that `export` with no body means `dllimport` and `export` with a body means `dllexport`.
>
> Keep in mind that nobody that I know of is using it to mean this today.

Maybe the recommendation should then be that they should? Doesn't the point still stand that "you cannot tell the compiler that a module is not in your binary" isn't actually true? I saw in one issue where there was a problem with variable declaration though, where dllimport/dllexport was determined by the presence or not of an initialiser, which is... yuck.

> Also a D module may be in binary, but it may have declarations that point to something that is not.
>
> See the bindings in druntime where it has D symbols.
>
> https://github.com/dlang/dmd/blob/5fc02ba152ccaa71711f3ed84b6d44a2a940f206/druntime/src/core/stdc/stdio.d#L1191

The relevant code is:

    private extern shared FILE[3] __sF;
    @property auto stdin()() { return &__sF[0]; }

`__sF` is declared extern, i.e., not in binary. I don't understand what the issue would be?

> This is why you don't pretend that one symbol in ``DllImport`` mode means the entire module is because it may be a binding to something else.

Why would the entire module be dllimport?

> Note: there are plenty of examples of it which are not templated, like structs that have to be initialized using the D init array.

Yes, but I don't know how this is related to the above.

>> Where does the need for "private but export" come from again? Is there an equivalent in C++ (`static dllexport`?), or does this only happen due to something specific to D like `T.init`?
>
> It happens because D confuses exportation which is a linker concept, with a language visibility concept.

My question is: when would I want to export a private symbol?

> In C/C++ it uses a completely separate attribute to donate that it does not affect Member Access Control such as ``private``.

On Windows, C/C++ compilers use a non-standard extension to do so, but yes. AFAIK (and I could well be wrong), one can't dllexport something that's static? You'd have to put it in a header, and it'd be compiled into the current translation unit anyway, so I also don't understand why you'd want to.

> But you can see this with things like templates, you want to access an internal symbol but don't want somebody else to? Yeah no, can't do that today.
>
> https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170

Assuming the link is supposed to elucidate the template comment, I don't understand the relevance. Otherwise, what does "you want to access an internal symbol but don't want somebody else to" mean?

> You have to be very explicit in c/c++ over this, we do not have that level of control (apart from saying do not export this symbol via ``@hidden``).

This could mean several things. We have the control over individual symbols that are actually in the source code with `export`. Is the comment above about things like T.init?

>> "By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.
>
> Yes two. They are referenced in the article.
>
> Note: they are not duplicates.
>
> Okay I lie there is a bunch more.

Thanks!

On a somewhat related note, we use dlls at work and seem to have fixed "everything" by using ldc and `-fvisibility=hidden -dllimport=defaultLibsOnly`, as well as `-link-defaultlib-shared`.
May 11
On 11/05/2024 1:27 AM, Atila Neves wrote:
>>> "you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with `export` on a declaration (as opposed to a definition)? That is, my understanding is that `export` with no body means `dllimport` and `export` with a body means `dllexport`.
>>
>> Keep in mind that nobody that I know of is using it to mean this today.
> 
> Maybe the recommendation should then be that they should? Doesn't the point still stand that "you cannot tell the compiler that a module is not in your binary" isn't actually true? I saw in one issue where there was a problem with variable declaration though, where dllimport/dllexport was determined by the presence or not of an initialiser, which is... yuck.

"you cannot tell the compiler that a module is not in your binary" is true, there is no syntax or cli flag to do this today. Note: this is for the entire module, its metadata (``ModuleInfo``) not just the user written symbols.

There probably should be syntax on the module to specify it being out of binary. However that would not be suitable for the majority of programmers and most definitely is not suitable for build managers as it would require it to modify/create the files. That could come later once we have more experience with reliable support.

But yes having an initializer or not, should not determine the symbol mode. I did my best to clean it up in a way that would keep things simple and not break the world.

This is why I've simplified things down to:

Use ``export`` + ``extern`` to go into ``DllImport`` mode.

Or rely on the external import path switch to set the ``extern`` for the majority of users. Which is ideal for things like the di generator or build managers ;)

Or use the dllimport override switch to set all symbols found from a module that is known to be out of binary as ``DllImport`` (helps with mixing some imports being in binary and some out).

>> Also a D module may be in binary, but it may have declarations that point to something that is not.
>>
>> See the bindings in druntime where it has D symbols.
>>
>> https://github.com/dlang/dmd/blob/5fc02ba152ccaa71711f3ed84b6d44a2a940f206/druntime/src/core/stdc/stdio.d#L1191
> 
> The relevant code is:
> 
>      private extern shared FILE[3] __sF;
>      @property auto stdin()() { return &__sF[0]; }
> 
> `__sF` is declared extern, i.e., not in binary. I don't understand what the issue would be?

Yeah that's not the best example, but keep in mind that symbol is not in ``DllImport`` mode, its ``Internal``.

And that right there is a problem.

If all those symbols were declared using positive annotation (like they should be), then anything that has D symbols like the structs such as:

https://github.com/dlang/dmd/blob/5fc02ba152ccaa71711f3ed84b6d44a2a940f206/druntime/src/core/stdc/stdio.d#L872

Would also be affected if the following was applied.

>> This is why you don't pretend that one symbol in ``DllImport`` mode means the entire module is because it may be a binding to something else.
> 
> Why would the entire module be dllimport?

Walter came up with this idea a while back, so I'm a tad defensive towards it.

https://github.com/dlang/dmd/pull/15298

Me and Martin had to really fight Walter on it, as it would mean breaking peoples builds in incredibly frustrating ways.

The problem is once you start making assumptions about if a module is in or out of your binary without direct instruction by the user, you can mess up your dependencies between modules at the minimum. So if you depend on another it may not initialize before you.

It needs to be explicit, or its going to ruin someones day pretty quickly (hence the external import path switch).

Here is a binding module that is compiled into your executable.

```d
module binding;

export extern void otherFunction();

shared static this() {
	import std.stdio;
	writeln("binding");
}
```

```d
module app;
import binding;

void main() {
	otherFunction();
}

shared static this() {
	import std.stdio;
	writeln("main");
}
```

``$ dmd main.d binding.d someImport.lib``

With Walter's idea the ``app`` module constructor could be called before ``binding``'s does, which absolutely should not happen.

>> Note: there are plenty of examples of it which are not templated, like structs that have to be initialized using the D init array.
> 
> Yes, but I don't know how this is related to the above.

If you were to use Walter's approach to determining if a module is out of binary simply because there is a symbol in ``DllImport`` mode, and then applied it to all declarations (as if the dllimport override switch was set to ``DllImport`` everything) it'll cause link errors.

Basically a module can have some symbols out of binary whilst the rest are in. So you can't make this sort of assumption.

This is an unfortunate side effect of analyzing Walter's idea to its natural conclusion. Me getting a bit unhappy towards any suggestion of inferring out of binary status, its pain in waiting.

>>> Where does the need for "private but export" come from again? Is there an equivalent in C++ (`static dllexport`?), or does this only happen due to something specific to D like `T.init`?
>>
>> It happens because D confuses exportation which is a linker concept, with a language visibility concept.
> 
> My question is: when would I want to export a private symbol?

Q1: Should a given symbol be private, yes?
Q2: Does any template use the previous symbol?

A: That is why you would export a private symbol.

Everything in here needs to be exported and should have package visibility so that it is not available for anyone else to access: https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/console/internal/rawwrite.d

See some usage here: https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/console/internal/writer.d#L70

I seem to be severely lacking in my ability to explain that having templates be limited to accessing only public things is not a good idea. This has been a bit of an annoyance for me, it goes in the face of all my interests in program security by introducing uncertainty that has no reason to exist.

In practice it means people can go and mess around with your internal state without any language protection. You are better off using negative annotation and never touching positive. Its safer. A lot safer.

Needless to say they are in a few places in my code base:

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/rawread.d#L126

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/prettyprint.d#L67

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/path/file.d#L1012

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/logger.d#L447

Loggers, console read/write, are where I have hit it.

But I'm sure you could come up with other examples of where you don't want others directly touching your _internal_ symbols, but still need your code which may be compiled into another binary to touch them.

>> In C/C++ it uses a completely separate attribute to donate that it does not affect Member Access Control such as ``private``.
> 
> On Windows, C/C++ compilers use a non-standard extension to do so, but yes. AFAIK (and I could well be wrong), one can't dllexport something that's static? You'd have to put it in a header, and it'd be compiled into the current translation unit anyway, so I also don't understand why you'd want to.

Yes the extension is https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170

For D apart from maybe metadata, only templates can go into another binary.

>> But you can see this with things like templates, you want to access an internal symbol but don't want somebody else to? Yeah no, can't do that today.
>>
>> https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170
> 
> Assuming the link is supposed to elucidate the template comment, I don't understand the relevance. Otherwise, what does "you want to access an internal symbol but don't want somebody else to" mean?

```d
module database_access;

@safe:

void doAThing(T)() {
	iCanKillYourDatabase(false, T.sizeof);
}


/*private:*/
export:

void iCanKillYourDatabase(bool doWrongThing = true, size_t sizeOfThing) {
	if (doWrongThing)
		database.corruptSilently();
}
```

A bit dramatic (due to unrealistic nature), but that should get the point across that ``iCanKillYourDatabase`` should be private but also exported.

The kernel might not stop you from doing a bad thing, but D should be making it a lot harder by making ``iCanKillYourDatabase`` private.

Note: you can of course still gain access to it via ``dlopen``, but that is not ``@safe`` code and you would need to gain access to the symbol name before you could do that.

>> You have to be very explicit in c/c++ over this, we do not have that level of control (apart from saying do not export this symbol via ``@hidden``).
> 
> This could mean several things. We have the control over individual symbols that are actually in the source code with `export`. Is the comment above about things like T.init?

For generated symbols like ``T.init``, ``opCmp``, ``ModuleInfo`` ext., we have zero control over these currently. Either you use a linker script or you use negative annotation. Everyone I know of uses negative annotation (although I support both).

For other symbols we can control not to export per symbol, and if we want to export and have it be public then we can use the export keyword.

In C/C++ land, you control if its ``DllExport`` or ``DllImport`` with an attribute directly. With my DIP you would use ``export`` and ``export`` with ``extern`` to denote each.

>>> "By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.
>>
>> Yes two. They are referenced in the article.
>>
>> Note: they are not duplicates.
>>
>> Okay I lie there is a bunch more.
> 
> Thanks!
> 
> On a somewhat related note, we use dlls at work and seem to have fixed "everything" by using ldc and `-fvisibility=hidden -dllimport=defaultLibsOnly`, as well as `-link-defaultlib-shared`.

``-link-defaultlib-shared`` sets the druntime to be a shared library (which lets face it should be the default).

``-dllimport=defaultLibsOnly`` all symbols for druntime and phobos are defaulting to ``DllImport`` but none others.

As for ``-fvisibility=hidden``, that would imply that you are using positive annotation in every code base. Which is curious considering Martin's prior work has been the exact opposite to this.

I don't know enough details of Symmetry's projects or how they are laid out to comment about them beyond the tidbits I get. Switching from negative to positive annotation would be a massive undertaking so the notion that you have switched to positive annotation is a statement I am having trouble coinciding it with what I know.

Perhaps the one you are referring to is a plugin with a known fixed public API? That being positive annotation and everything else being compiled in being hidden would make sense.
May 14
On Friday, 10 May 2024 at 15:21:15 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 11/05/2024 1:27 AM, Atila Neves wrote:
>>>> "you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with `export` on a declaration (as opposed to a definition)? That is, my understanding is that `export` with no body means `dllimport` and `export` with a body means `dllexport`.
>>>
>>> Keep in mind that nobody that I know of is using it to mean this today.
>> 
>> Maybe the recommendation should then be that they should? Doesn't the point still stand that "you cannot tell the compiler that a module is not in your binary" isn't actually true? I saw in one issue where there was a problem with variable declaration though, where dllimport/dllexport was determined by the presence or not of an initialiser, which is... yuck.
>
> "you cannot tell the compiler that a module is not in your binary" is true, there is no syntax or cli flag to do this today.

Correct, sorry, I thought we were talking about symbols and somehow missed "module". The question would then be: why would someone want to tell the compiler that an entire module is out of binary?

> But yes having an initializer or not, should not determine the symbol mode. I did my best to clean it up in a way that would keep things simple and not break the world.
>
> This is why I've simplified things down to:
>
> Use ``export`` + ``extern`` to go into ``DllImport`` mode.

Makes sense.

> Or rely on the external import path switch to set the ``extern`` for the majority of users. Which is ideal for things like the di generator or build managers ;)

Do you mean "build systems"?

> Or use the dllimport override switch to set all symbols found from a module that is known to be out of binary as ``DllImport`` (helps with mixing some imports being in binary and some out).
>
> Yeah that's not the best example, but keep in mind that symbol is not in ``DllImport`` mode, its ``Internal``.
>
> And that right there is a problem.

And wouldn't the solution be to add `export`?

>> Why would the entire module be dllimport?
>
> Walter came up with this idea a while back, so I'm a tad defensive towards it.
>
> https://github.com/dlang/dmd/pull/15298

I went through all of that and am still confused as to whether you want or don't want whole modules to be declared out of binary, or why one would want to do that.

> With Walter's idea the ``app`` module constructor could be called before ``binding``'s does, which absolutely should not happen.

I don't know why not, nor did I understand the relevance of the example.

> Basically a module can have some symbols out of binary whilst the rest are in. So you can't make this sort of assumption.

Ok.

>> My question is: when would I want to export a private symbol?
>
> Q1: Should a given symbol be private, yes?
> Q2: Does any template use the previous symbol?
>
> A: That is why you would export a private symbol.

Can't we do what C++ does and stick the private symbol where the template is being instantiated?

> ```d
> module database_access;
>
> @safe:
>
> void doAThing(T)() {
> 	iCanKillYourDatabase(false, T.sizeof);
> }
>
>
> /*private:*/
> export:
>
> void iCanKillYourDatabase(bool doWrongThing = true, size_t sizeOfThing) {
> 	if (doWrongThing)
> 		database.corruptSilently();
> }
> ```
>
> A bit dramatic (due to unrealistic nature), but that should get the point across that ``iCanKillYourDatabase`` should be private but also exported.

Inline it instead? The code is right there.

> For generated symbols like ``T.init``, ``opCmp``, ``ModuleInfo`` ext., we have zero control over these currently. Either you use a linker script or you use negative annotation. Everyone I know of uses negative annotation (although I support both).

What would the solution be?

>>>> "By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.
>>>
>>> Yes two. They are referenced in the article.
>>>
>>> Note: they are not duplicates.
>>>
>>> Okay I lie there is a bunch more.
>> 
>> Thanks!
>> 
>> On a somewhat related note, we use dlls at work and seem to have fixed "everything" by using ldc and `-fvisibility=hidden -dllimport=defaultLibsOnly`, as well as `-link-defaultlib-shared`.
>
> ``-link-defaultlib-shared`` sets the druntime to be a shared library (which lets face it should be the default).

Given how much people go on about how great Go is because it links statically (even though C/C++ have been able to do this for basically forever if you opt-in), I'm not sure of that.

> ``-dllimport=defaultLibsOnly`` all symbols for druntime and phobos are defaulting to ``DllImport`` but none others.
>
> As for ``-fvisibility=hidden``, that would imply that you are using positive annotation in every code base. Which is curious considering Martin's prior work has been the exact opposite to this.

I grepped for `export` and it appears in quite a few places as expected.

> Perhaps the one you are referring to is a plugin with a known fixed public API? That being positive annotation and everything else being compiled in being hidden would make sense.

I think that's what we're doing yes, and that the fact that we are is a good thing.


May 15
On 15/05/2024 6:19 AM, Atila Neves wrote:
> On Friday, 10 May 2024 at 15:21:15 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> On 11/05/2024 1:27 AM, Atila Neves wrote:
>>>>> "you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with `export` on a declaration (as opposed to a definition)? That is, my understanding is that `export` with no body means `dllimport` and `export` with a body means `dllexport`.
>>>>
>>>> Keep in mind that nobody that I know of is using it to mean this today.
>>>
>>> Maybe the recommendation should then be that they should? Doesn't the point still stand that "you cannot tell the compiler that a module is not in your binary" isn't actually true? I saw in one issue where there was a problem with variable declaration though, where dllimport/dllexport was determined by the presence or not of an initialiser, which is... yuck.
>>
>> "you cannot tell the compiler that a module is not in your binary" is true, there is no syntax or cli flag to do this today.
> 
> Correct, sorry, I thought we were talking about symbols and somehow missed "module". The question would then be: why would someone want to tell the compiler that an entire module is out of binary?

1. Metadata (ModuleInfo, TypeInfo, RTInfo)
2. It is almost certainly going to be correct and ``-I`` is almost certainly at play, therefore we can leverage this information to turn ``export`` for positive annotation into ``DllImport``.

Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, void*]``

Metadata quite literally is incapable of crossing the DLL boundary without knowing if its in or out of binary. It prevents linking.

If it does link, that pointer could very well be pointing at a jump instruction. Not exactly a fun day if you want to debug it.

>> But yes having an initializer or not, should not determine the symbol mode. I did my best to clean it up in a way that would keep things simple and not break the world.
>>
>> This is why I've simplified things down to:
>>
>> Use ``export`` + ``extern`` to go into ``DllImport`` mode.
> 
> Makes sense.

This also plays into the external import path switch, since we know that is almost certainly correct, adding the extern on make a very clean simple solution for positive annotation!

>> Or rely on the external import path switch to set the ``extern`` for the majority of users. Which is ideal for things like the di generator or build managers ;)
> 
> Do you mean "build systems"?

They are interchangeable at this level in my mind, but yes.

>> Or use the dllimport override switch to set all symbols found from a module that is known to be out of binary as ``DllImport`` (helps with mixing some imports being in binary and some out).
>>
>> Yeah that's not the best example, but keep in mind that symbol is not in ``DllImport`` mode, its ``Internal``.
>>
>> And that right there is a problem.
> 
> And wouldn't the solution be to add `export`?

Both druntime and PhobosV2 will remain in using negative annotation for the foreseeable future. The amount of work to convert that to positive is quite significant because its not just slapping export on things, you also have to test it.

Now do the rest of the ecosystem but with people who don't know what they are doing.

>>> Why would the entire module be dllimport?
>>
>> Walter came up with this idea a while back, so I'm a tad defensive towards it.
>>
>> https://github.com/dlang/dmd/pull/15298
> 
> I went through all of that and am still confused as to whether you want or don't want whole modules to be declared out of binary, or why one would want to do that.

Being out of binary is what that PR does, by deriving it based upon any symbol being in DllImport mode.

We need to know if it is out of binary (see at top of this comment especially here), and will give false positives if you use positive annotation due the deriviation.

>> With Walter's idea the ``app`` module constructor could be called before ``binding``'s does, which absolutely should not happen.
> 
> I don't know why not, nor did I understand the relevance of the example.

It shows that shared libraries weren't involved, yet because a module was derived as out of binary things didn't work correctly.

>>> My question is: when would I want to export a private symbol?
>>
>> Q1: Should a given symbol be private, yes?
>> Q2: Does any template use the previous symbol?
>>
>> A: That is why you would export a private symbol.
> 
> Can't we do what C++ does and stick the private symbol where the template is being instantiated?

So you want even more global state?

And why am I feeling like there is a shifting sands feeling going on for.

>> ```d
>> module database_access;
>>
>> @safe:
>>
>> void doAThing(T)() {
>>     iCanKillYourDatabase(false, T.sizeof);
>> }
>>
>>
>> /*private:*/
>> export:
>>
>> void iCanKillYourDatabase(bool doWrongThing = true, size_t sizeOfThing) {
>>     if (doWrongThing)
>>         database.corruptSilently();
>> }
>> ```
>>
>> A bit dramatic (due to unrealistic nature), but that should get the point across that ``iCanKillYourDatabase`` should be private but also exported.
> 
> Inline it instead? The code is right there.

What if its accessing global state (and perhaps giving you access to it via callback)?

What if it is global state instead of a function?

What makes you think that it can be inlined?

>> For generated symbols like ``T.init``, ``opCmp``, ``ModuleInfo`` ext., we have zero control over these currently. Either you use a linker script or you use negative annotation. Everyone I know of uses negative annotation (although I support both).
> 
> What would the solution be?

Some cannot be like ``ModuleInfo``, others have be exported based upon if other things in the encapsulation are exported.

Its either that or we export literally everything inside of the encapsulation unit (I don't like that at all).

Or we invent new syntax... Again not a fan.

>>>>> "By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.
>>>>
>>>> Yes two. They are referenced in the article.
>>>>
>>>> Note: they are not duplicates.
>>>>
>>>> Okay I lie there is a bunch more.
>>>
>>> Thanks!
>>>
>>> On a somewhat related note, we use dlls at work and seem to have fixed "everything" by using ldc and `-fvisibility=hidden -dllimport=defaultLibsOnly`, as well as `-link-defaultlib-shared`.
>>
>> ``-link-defaultlib-shared`` sets the druntime to be a shared library (which lets face it should be the default).
> 
> Given how much people go on about how great Go is because it links statically (even though C/C++ have been able to do this for basically forever if you opt-in), I'm not sure of that.

The way I view it is as thus:

- If you use D shared libraries with a static runtime, you're going to have your program have indeterminate behavior.
- On the other hand, if you don't use shared libraries with a shared runtime it works.

In the latter you will need to copy the druntime/phobos shared library, but hey the system loader will tell you if you forgot!

In the former there is no warning, it will happily do the wrong thing with no warnings and it might not even crash it could just corrupt data instead.

So from my perspective its better to be opt-in for static runtime/phobos builds if you know you don't need it, than the opposite.