 | Posted by Richard (Rikki) Andrew Cattermole
in reply to Atila Neves |  Permalink Reply |
|
Richard (Rikki) Andrew Cattermole 
Posted in reply to Atila Neves
| On 11/05/2024 1:27 AM, Atila Neves wrote:
>>> "you cannot tell the compiler that a module is not in your binary." - isn't this exactly what happens with `export` on a declaration (as opposed to a definition)? That is, my understanding is that `export` with no body means `dllimport` and `export` with a body means `dllexport`.
>>
>> Keep in mind that nobody that I know of is using it to mean this today.
>
> Maybe the recommendation should then be that they should? Doesn't the point still stand that "you cannot tell the compiler that a module is not in your binary" isn't actually true? I saw in one issue where there was a problem with variable declaration though, where dllimport/dllexport was determined by the presence or not of an initialiser, which is... yuck.
"you cannot tell the compiler that a module is not in your binary" is true, there is no syntax or cli flag to do this today. Note: this is for the entire module, its metadata (``ModuleInfo``) not just the user written symbols.
There probably should be syntax on the module to specify it being out of binary. However that would not be suitable for the majority of programmers and most definitely is not suitable for build managers as it would require it to modify/create the files. That could come later once we have more experience with reliable support.
But yes having an initializer or not, should not determine the symbol mode. I did my best to clean it up in a way that would keep things simple and not break the world.
This is why I've simplified things down to:
Use ``export`` + ``extern`` to go into ``DllImport`` mode.
Or rely on the external import path switch to set the ``extern`` for the majority of users. Which is ideal for things like the di generator or build managers ;)
Or use the dllimport override switch to set all symbols found from a module that is known to be out of binary as ``DllImport`` (helps with mixing some imports being in binary and some out).
>> Also a D module may be in binary, but it may have declarations that point to something that is not.
>>
>> See the bindings in druntime where it has D symbols.
>>
>> https://github.com/dlang/dmd/blob/5fc02ba152ccaa71711f3ed84b6d44a2a940f206/druntime/src/core/stdc/stdio.d#L1191
>
> The relevant code is:
>
> private extern shared FILE[3] __sF;
> @property auto stdin()() { return &__sF[0]; }
>
> `__sF` is declared extern, i.e., not in binary. I don't understand what the issue would be?
Yeah that's not the best example, but keep in mind that symbol is not in ``DllImport`` mode, its ``Internal``.
And that right there is a problem.
If all those symbols were declared using positive annotation (like they should be), then anything that has D symbols like the structs such as:
https://github.com/dlang/dmd/blob/5fc02ba152ccaa71711f3ed84b6d44a2a940f206/druntime/src/core/stdc/stdio.d#L872
Would also be affected if the following was applied.
>> This is why you don't pretend that one symbol in ``DllImport`` mode means the entire module is because it may be a binding to something else.
>
> Why would the entire module be dllimport?
Walter came up with this idea a while back, so I'm a tad defensive towards it.
https://github.com/dlang/dmd/pull/15298
Me and Martin had to really fight Walter on it, as it would mean breaking peoples builds in incredibly frustrating ways.
The problem is once you start making assumptions about if a module is in or out of your binary without direct instruction by the user, you can mess up your dependencies between modules at the minimum. So if you depend on another it may not initialize before you.
It needs to be explicit, or its going to ruin someones day pretty quickly (hence the external import path switch).
Here is a binding module that is compiled into your executable.
```d
module binding;
export extern void otherFunction();
shared static this() {
import std.stdio;
writeln("binding");
}
```
```d
module app;
import binding;
void main() {
otherFunction();
}
shared static this() {
import std.stdio;
writeln("main");
}
```
``$ dmd main.d binding.d someImport.lib``
With Walter's idea the ``app`` module constructor could be called before ``binding``'s does, which absolutely should not happen.
>> Note: there are plenty of examples of it which are not templated, like structs that have to be initialized using the D init array.
>
> Yes, but I don't know how this is related to the above.
If you were to use Walter's approach to determining if a module is out of binary simply because there is a symbol in ``DllImport`` mode, and then applied it to all declarations (as if the dllimport override switch was set to ``DllImport`` everything) it'll cause link errors.
Basically a module can have some symbols out of binary whilst the rest are in. So you can't make this sort of assumption.
This is an unfortunate side effect of analyzing Walter's idea to its natural conclusion. Me getting a bit unhappy towards any suggestion of inferring out of binary status, its pain in waiting.
>>> Where does the need for "private but export" come from again? Is there an equivalent in C++ (`static dllexport`?), or does this only happen due to something specific to D like `T.init`?
>>
>> It happens because D confuses exportation which is a linker concept, with a language visibility concept.
>
> My question is: when would I want to export a private symbol?
Q1: Should a given symbol be private, yes?
Q2: Does any template use the previous symbol?
A: That is why you would export a private symbol.
Everything in here needs to be exported and should have package visibility so that it is not available for anyone else to access: https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/console/internal/rawwrite.d
See some usage here: https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/console/internal/writer.d#L70
I seem to be severely lacking in my ability to explain that having templates be limited to accessing only public things is not a good idea. This has been a bit of an annoyance for me, it goes in the face of all my interests in program security by introducing uncertainty that has no reason to exist.
In practice it means people can go and mess around with your internal state without any language protection. You are better off using negative annotation and never touching positive. Its safer. A lot safer.
Needless to say they are in a few places in my code base:
https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/rawread.d#L126
https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/prettyprint.d#L67
https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/path/file.d#L1012
https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/logger.d#L447
Loggers, console read/write, are where I have hit it.
But I'm sure you could come up with other examples of where you don't want others directly touching your _internal_ symbols, but still need your code which may be compiled into another binary to touch them.
>> In C/C++ it uses a completely separate attribute to donate that it does not affect Member Access Control such as ``private``.
>
> On Windows, C/C++ compilers use a non-standard extension to do so, but yes. AFAIK (and I could well be wrong), one can't dllexport something that's static? You'd have to put it in a header, and it'd be compiled into the current translation unit anyway, so I also don't understand why you'd want to.
Yes the extension is https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170
For D apart from maybe metadata, only templates can go into another binary.
>> But you can see this with things like templates, you want to access an internal symbol but don't want somebody else to? Yeah no, can't do that today.
>>
>> https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170
>
> Assuming the link is supposed to elucidate the template comment, I don't understand the relevance. Otherwise, what does "you want to access an internal symbol but don't want somebody else to" mean?
```d
module database_access;
@safe:
void doAThing(T)() {
iCanKillYourDatabase(false, T.sizeof);
}
/*private:*/
export:
void iCanKillYourDatabase(bool doWrongThing = true, size_t sizeOfThing) {
if (doWrongThing)
database.corruptSilently();
}
```
A bit dramatic (due to unrealistic nature), but that should get the point across that ``iCanKillYourDatabase`` should be private but also exported.
The kernel might not stop you from doing a bad thing, but D should be making it a lot harder by making ``iCanKillYourDatabase`` private.
Note: you can of course still gain access to it via ``dlopen``, but that is not ``@safe`` code and you would need to gain access to the symbol name before you could do that.
>> You have to be very explicit in c/c++ over this, we do not have that level of control (apart from saying do not export this symbol via ``@hidden``).
>
> This could mean several things. We have the control over individual symbols that are actually in the source code with `export`. Is the comment above about things like T.init?
For generated symbols like ``T.init``, ``opCmp``, ``ModuleInfo`` ext., we have zero control over these currently. Either you use a linker script or you use negative annotation. Everyone I know of uses negative annotation (although I support both).
For other symbols we can control not to export per symbol, and if we want to export and have it be public then we can use the export keyword.
In C/C++ land, you control if its ``DllExport`` or ``DllImport`` with an attribute directly. With my DIP you would use ``export`` and ``export`` with ``extern`` to denote each.
>>> "By not exporting ModuleInfo and assuming it is available the compiler introduces a hidden dependency on a generated symbol that may not exist." - do we have an issue for that? I searched for ModuleInfo in the issues but none of them looked like a match to me.
>>
>> Yes two. They are referenced in the article.
>>
>> Note: they are not duplicates.
>>
>> Okay I lie there is a bunch more.
>
> Thanks!
>
> On a somewhat related note, we use dlls at work and seem to have fixed "everything" by using ldc and `-fvisibility=hidden -dllimport=defaultLibsOnly`, as well as `-link-defaultlib-shared`.
``-link-defaultlib-shared`` sets the druntime to be a shared library (which lets face it should be the default).
``-dllimport=defaultLibsOnly`` all symbols for druntime and phobos are defaulting to ``DllImport`` but none others.
As for ``-fvisibility=hidden``, that would imply that you are using positive annotation in every code base. Which is curious considering Martin's prior work has been the exact opposite to this.
I don't know enough details of Symmetry's projects or how they are laid out to comment about them beyond the tidbits I get. Switching from negative to positive annotation would be a massive undertaking so the notion that you have switched to positive annotation is a statement I am having trouble coinciding it with what I know.
Perhaps the one you are referring to is a plugin with a known fixed public API? That being positive annotation and everything else being compiled in being hidden would make sense.
|
