On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) Andrew Cattermole wrote:
>
> On 15/05/2024 6:19 AM, Atila Neves wrote:
>> [...]
>
> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)

These are whats, not whys.

> 2. It is almost certainly going to be correct

What is most certainly correct?

> and ``-I`` is almost certainly at play, therefore we can leverage this information to turn ``export`` for positive annotation into ``DllImport``.

We already have a mechanism to do this.

> Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, void*]``

No, but I don't understand the question.

> Metadata quite literally is incapable of crossing the DLL boundary without knowing if its in or out of binary. It prevents linking.

What prevents linking?

>> [...]
>
> This also plays into the external import path switch, since we know that is almost certainly correct, adding the extern on make a very clean simple solution for positive annotation!

I don't see how `extern` is related to a potential external import path switch.

>> [...]
>
> Both druntime and PhobosV2 will remain in using negative annotation for the foreseeable future. The amount of work to convert that to positive is quite significant because its not just slapping export on things, you also have to test it.

`export:`?

What needs to be tested?

> Now do the rest of the ecosystem but with people who don't know what they are doing.

I think it's unlikely that most projects would need to work in dlls.

>> [...]
>
> So you want even more global state?

?

>> [...]
>
> What if its accessing global state (and perhaps giving you access to it via callback)?
>
> What if it is global state instead of a function?
>
> What makes you think that it can be inlined?

C++.


>> [...]
>
> Some cannot be like ``ModuleInfo``, others have be exported based upon if other things in the encapsulation are exported.

What's an "encapsulation"?

> Its either that or we export literally everything inside of the encapsulation unit (I don't like that at all).

?

>> [...]
>
> The way I view it is as thus:
>
> - If you use D shared libraries with a static runtime, you're going to have your program have indeterminate behavior.

Which doesn't seem to be what happens most often.

> - On the other hand, if you don't use shared libraries with a shared runtime it works.

It doesn't seem this is what most people want.

> So from my perspective its better to be opt-in for static runtime/phobos builds if you know you don't need it, than the opposite.

Again, the raves from people new to Go would suggest otherwise. So would the lack of requests for this in D.

On 16/05/2024 5:30 AM, Atila Neves wrote:
> On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>
>> On 15/05/2024 6:19 AM, Atila Neves wrote:
>>> [...]
>>
>> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)
> 
> These are whats, not whys.

As always, this entire thread is all about why things do not link, load or run.

All three of those are examples of things that because they exist do not link, load or run at runtime correctly.

These files should not exist:

https://github.com/Project-Sidero/basic_memory/blob/main/msvc_exports.def

https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/dllmain.d

https://github.com/Project-Sidero/basic_memory/blob/main/source/sidero/base/moduleinfostubs.d

https://github.com/Project-Sidero/eventloop/blob/master/source/sidero/eventloop/rtinfoimplstub.d

https://github.com/Project-Sidero/image/blob/master/source/sidero/image/rtinfoimplstub.d

https://github.com/Project-Sidero/image/blob/master/source/sidero/image/dllmain.d

>> 2. It is almost certainly going to be correct
> 
> What is most certainly correct?

If a module is found via a path provided to the external import path switch to be out of binary.

>> and ``-I`` is almost certainly at play, therefore we can leverage this information to turn ``export`` for positive annotation into ``DllImport``.
> 
> We already have a mechanism to do this.

Ugh no? Not for positive annotation we do not.

>> Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, void*]``
> 
> No, but I don't understand the question.

ModuleInfo goes into an array during loading.

For ones in binary that is an array of them.

Because we can only bind to the jmp wrapper function, or the DllImport global pointer /if/ we know that it is in another binary we end up with an array that has both ModuleInfo's and pointers to ModuleInfo's there is no way to reliably distinguish between the two.

>> Metadata quite literally is incapable of crossing the DLL boundary without knowing if its in or out of binary. It prevents linking.
> 
> What prevents linking?

Only functions can work across the DLL boundary, ROM data like arrays or meta data like ModuleInfo cannot cross directly.

If you know it is out of binary then you can perform a load against ``ModuleInfo*``, but you cannot patch against ``ModuleInfo``.

Its not like how it is when its all in binary. This is not something any user of D should be needing to attempt a work around for and **I have working ones** puke.

>>> [...]
>>
>> This also plays into the external import path switch, since we know that is almost certainly correct, adding the extern on make a very clean simple solution for positive annotation!
> 
> I don't see how `extern` is related to a potential external import path switch.

Simple, it adds it.

By adding it you convert an ``export`` to ``export`` with ``extern`` which the DIP defines as ``DllImport``.

An ``export`` by itself would remain in ``internal`` mode.

This is desirable when you are doing multiple step builds, such as incremental compilation or ya know using static libraries like dub requires today.

>>> [...]
>>
>> Both druntime and PhobosV2 will remain in using negative annotation for the foreseeable future. The amount of work to convert that to positive is quite significant because its not just slapping export on things, you also have to test it.
> 
> `export:`?
> 
> What needs to be tested?

Anytime you convert a module from negative annotation to positive, everything has to be tested to verify that it still links, loads and runs correctly.

>> Now do the rest of the ecosystem but with people who don't know what they are doing.
> 
> I think it's unlikely that most projects would need to work in dlls.

It doesn't matter, it only takes one project that depends upon another for the dependency to have to care about it.

>>> [...]
>>
>> So you want even more global state?
> 
> ?

In the above you were promoting duplicating global state as a solution instead of letting the user mark something that is private as exported.

It's an insane idea.

Imagine there being multiple different mallocs inside each shared library!

Lots more ram usage, and plenty of free's not freeing because the state wasn't shared.

And that assume you even succeed at making that link at all.

>>> [...]
>>
>> What if its accessing global state (and perhaps giving you access to it via callback)?
>>
>> What if it is global state instead of a function?
>>
>> What makes you think that it can be inlined?
> 
> C++.

Well C++ isn't D.

They have things we don't have, different expectations.

Keep in mind that D's official build manager is based upon npm, not CMake. We use D very differently than C++ users use C++.

You have to understand how D is being used here, there is not substitute for experience supporting people with shared library support and seeing what they are trying to do.

>>> [...]
>>
>> Some cannot be like ``ModuleInfo``, others have be exported based upon if other things in the encapsulation are exported.
> 
> What's an "encapsulation"?

struct/class/union

Basically anything that contains other symbols.

>> Its either that or we export literally everything inside of the encapsulation unit (I don't like that at all).
> 
> ?

That is negative annotation. It is the default in dub and is the only reliable method of using shared libraries in D and even then its no where near as reliable as it should be as you cannot use (sub)packages.

>>> [...]
>>
>> The way I view it is as thus:
>>
>> - If you use D shared libraries with a static runtime, you're going to have your program have indeterminate behavior.
> 
> Which doesn't seem to be what happens most often.

Right, we don't default to that with dub when using shared libraries, because it can only ever introduce undesirable behavior.

>> - On the other hand, if you don't use shared libraries with a shared runtime it works.
> 
> It doesn't seem this is what most people want.

Defaults should never be foot-gun heavy. That is what you are promoting here by suggesting the status quo with static library builds of druntime/phobos are a good thing.

It's why C always defaults to shared libraries for its libc and with that the C++ runtime as well.

>> So from my perspective its better to be opt-in for static runtime/phobos builds if you know you don't need it, than the opposite.
> 
> Again, the raves from people new to Go would suggest otherwise. So would the lack of requests for this in D.

Go isn't D, its used in a very different way to D is.

You are very much mistaken about lack of requests, they are not needed because dub automates the change of defaults.

When I first introduced support for shared libraries into dub, you had to explicitly set that you wanted a shared library build of druntime/phobos as well as setting dllimport to all and visibility to public.

The reason nobody has talked about it is because its done for them automatically when they use dub. If they use something else, then they probably know how to figure out what they need to do on their own (I've had no support requests for anything that isn't dub).

On Thursday, 16 May 2024 at 02:15:33 UTC, Richard (Rikki) Andrew Cattermole wrote:
>
> On 16/05/2024 5:30 AM, Atila Neves wrote:
>> On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>
>>> On 15/05/2024 6:19 AM, Atila Neves wrote:
>>>> [...]
>>>
>>> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)
>> 
> All three of those are examples of things that because they exist do not link, load or run at runtime correctly.

I still don't understand what are the situations under which things don't work. Or why they require marking an entire module as out of binary.

> These files should not exist:

Oof. No, they should not. I understand you have workarounds here for problems with shared libraries in D, but I still don't understand why these workarounds are necessary.

>>> 2. It is almost certainly going to be correct
>> 
>> What is most certainly correct?
>
> If a module is found via a path provided to the external import path switch to be out of binary.

This "begs the assumption" that such a switch exists.

>>> and ``-I`` is almost certainly at play, therefore we can leverage this information to turn ``export`` for positive annotation into ``DllImport``.
>> 
>> We already have a mechanism to do this.
>
> Ugh no? Not for positive annotation we do not.

We do?

    export void foo();    // dllimport
    export void foo() { } // dllexport

>>> Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, void*]``
>> 
>> No, but I don't understand the question.
>
> ModuleInfo goes into an array during loading.
>
> For ones in binary that is an array of them.
> [...]

I get an inkling of what the problem is here with the description, but:

* I don't actually know
* I have a gut feeling there's a solution that isn't too complicated

> Its not like how it is when its all in binary. This is not something any user of D should be needing to attempt a work around for and **I have working ones** puke.

I agree.

>> I don't see how `extern` is related to a potential external import path switch.
>
> Simple, it adds it.

Right, but why does it need to?

    extern export int foo; // dllimport
    export int foo;        // dllexport

> This is desirable when you are doing multiple step builds, such as incremental compilation or ya know using static libraries like dub requires today.

We do this at work everyday; I don't get why the switch would be needed.

>> What needs to be tested?
>
> Anytime you convert a module from negative annotation to positive, everything has to be tested to verify that it still links, loads and runs correctly.

What's "everything"?

> In the above you were promoting duplicating global state as a solution instead of letting the user mark something that is private as exported.
>
> It's an insane idea.

It's what C++ does. There's no way to do it without putting it into a header and marking it `static`.

> Imagine there being multiple different mallocs inside each shared library!

Because of a global variable?

>>> What makes you think that it can be inlined?
>> 
>> C++.
>
> Well C++ isn't D.
>
> They have things we don't have, different expectations.

And we have things they don't have; but given that dll{import,export} is a Microsoft C/C++ thing, I think that it's a good idea to use them as inspiration for what we should do.

> Keep in mind that D's official build manager is based upon npm, not CMake. We use D very differently than C++ users use C++.

I don't see how this is relevant, especially since there are multiple (meta) build systems one can use for both.

> You have to understand how D is being used here

Ok, how is it being used in a way that's so different from how C++ is used?

> That is negative annotation. It is the default in dub

How is it the default in dub?

> Defaults should never be foot-gun heavy.

I agree.

> That is what you are promoting here by suggesting the status quo with static library builds of druntime/phobos are a good thing.

I don't see how; 0 bullets in my foot due to this so far in 10 years.

> It's why C always defaults to shared libraries for its libc and with that the C++ runtime as well.

I'm 99.9% sure that it was to save space and for every dependendent app to be updated automatically to the new version of the libraries.

>>> So from my perspective its better to be opt-in for static runtime/phobos builds if you know you don't need it, than the opposite.
>> 
>> Again, the raves from people new to Go would suggest otherwise. So would the lack of requests for this in D.
>
> Go isn't D, its used in a very different way to D is.

Yes/no. I'd rather statically link, personally.

> You are very much mistaken about lack of requests, they are not needed because dub automates the change of defaults.

I'm confused; are there requests or not?

> When I first introduced support for shared libraries into dub, you had to explicitly set that you wanted a shared library build of druntime/phobos as well as setting dllimport to all and visibility to public.

Ok. And now it automatically says "dynamically link"? If that's the case, I didn't know that, and now I'm wondering why we have an explicit flag for that in a dub.sdl.

On 17/05/2024 1:46 AM, Atila Neves wrote:
> On Thursday, 16 May 2024 at 02:15:33 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>
>> On 16/05/2024 5:30 AM, Atila Neves wrote:
>>> On Tuesday, 14 May 2024 at 22:13:19 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>>
>>>> On 15/05/2024 6:19 AM, Atila Neves wrote:
>>>>> [...]
>>>>
>>>> 1. Metadata (ModuleInfo, TypeInfo, RTInfo)
>>>
>> All three of those are examples of things that because they exist do not link, load or run at runtime correctly.
> 
> I still don't understand what are the situations under which things don't work. Or why they require marking an entire module as out of binary.

They are data not functions.

See Martin's post. https://forum.dlang.org/post/dobouzmhwabquswguunk@forum.dlang.org

They literally cannot cross the DLL boundary if you do not know that they are out of binary.

>> These files should not exist:
> 
> Oof. No, they should not. I understand you have workarounds here for problems with shared libraries in D, but I still don't understand why these workarounds are necessary.

Yeah I can tell, you're not quite getting the difference between data and functions here and unfortunately they matter if you are using positive annotation.

If I dropped positive annotation and went to negative I could get rid of them even for dmd (tested after Rainer added support).

>>>> and ``-I`` is almost certainly at play, therefore we can leverage this information to turn ``export`` for positive annotation into ``DllImport``.
>>>
>>> We already have a mechanism to do this.
>>
>> Ugh no? Not for positive annotation we do not.
> 
> We do?
> 
>      export void foo();    // dllimport
>      export void foo() { } // dllexport

This is no different than initializers on a global variable determining its symbol mode.

Think about the .di generator removing bodies, and suddenly you can no longer ship both static and shared libraries of your library.

One set of interface files to a library is better than two, one for each build.

>>>> Can you spot the problem with this: ``[void*]`` vs ``[void*, void**, void*]``
>>>
>>> No, but I don't understand the question.
>>
>> ModuleInfo goes into an array during loading.
>>
>> For ones in binary that is an array of them.
>> [...]
> 
> I get an inkling of what the problem is here with the description, but:
> 
> * I don't actually know
> * I have a gut feeling there's a solution that isn't too complicated

The only "easy" solution I know of is to nullify out if it can't load the symbol. That assumes the linker/loader supports it, and unfortunately as a solution it only solves ModuleInfo.

You still have to deal with other data like TypeInfo or .init.

Each time I had to deal with .init not crossing over it took me days to figure it out and here is my workaround to that:

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/specifier.d#L262

https://github.com/Project-Sidero/basic_memory/blob/65ee9d0c4b1d4bc666772b21d4afdec30835120c/source/sidero/base/text/format/prettyprint.d#L28

>>> I don't see how `extern` is related to a potential external import path switch.
>>
>> Simple, it adds it.
> 
> Right, but why does it need to?
> 
>      extern export int foo; // dllimport
>      export int foo;        // dllexport

What you have written is behavior my DIP promotes.

However the question is what happens when you have ``export int foo;`` in a file passed to ``-I``.

In my DIP that is in internal symbol mode, not ``DllImport``.

This allows for both static libraries and shared libraries to be used with one set of D files and in turn keeps the .di generator simpler.

>> This is desirable when you are doing multiple step builds, such as incremental compilation or ya know using static libraries like dub requires today.
> 
> We do this at work everyday; I don't get why the switch would be needed.

That's because you're building it as a plugin with a set exported interface.

This is a much more limited scenario that does not reflect projects like game engines or a standard library and should not be compared.

>>> What needs to be tested?
>>
>> Anytime you convert a module from negative annotation to positive, everything has to be tested to verify that it still links, loads and runs correctly.
> 
> What's "everything"?

All symbols that you change the symbol mode on.

>> In the above you were promoting duplicating global state as a solution instead of letting the user mark something that is private as exported.
>>
>> It's an insane idea.
> 
> It's what C++ does. There's no way to do it without putting it into a header and marking it `static`.
> 
>> Imagine there being multiple different mallocs inside each shared library!
> 
> Because of a global variable?

Bad example, but yes.

If you are requiring duplicated globals that means you can end up with a library with N state, rather than 1.

With template instantiations that is a very real problem.

However I am now wondering if we might be miscommunicating on this.

>>>> What makes you think that it can be inlined?
>>>
>>> C++.
>>
>> Well C++ isn't D.
>>
>> They have things we don't have, different expectations.
> 
> And we have things they don't have; but given that dll{import,export} is a Microsoft C/C++ thing, I think that it's a good idea to use them as inspiration for what we should do.

They have no QoL stuff at this level for us to take inspiration from.

You use the macro preprocessor to set when something is DllImport versus internal.

As for terminology yes, we absolutely must migrate over. Everyone has. Both LLVM and GCC use it, it's the standard model at the compiler level now.

>> Keep in mind that D's official build manager is based upon npm, not CMake. We use D very differently than C++ users use C++.
> 
> I don't see how this is relevant, especially since there are multiple (meta) build systems one can use for both.
> 
>> You have to understand how D is being used here
> 
> Ok, how is it being used in a way that's so different from how C++ is used?

Basically people expect to be able to import and for things to work.

Nobody should be seeing linker warnings when using dub just because they have a dependency and are building a shared library.

It is abnormal outside of plugins for people to be using positive annotation. They are exclusively negative annotation based upon the support I have given over a 2+ year period.

We have people in the community who do not wish to know that linkers exist and still manage (with help) to use shared libraries.

C++ on the other hand has for most of its life been exclusively positive annotation. Very different knowledge and willingness to deal with these details.

>> That is negative annotation. It is the default in dub
> 
> How is it the default in dub?

https://github.com/dlang/dub/blob/master/source/dub/compilers/ldc.d#L172

Its hard wired at the compiler personality level to change defaults.

Note: default on other platforms is negative annotation (which makes it highly inconsistent, which my DIP fixes).

>> It's why C always defaults to shared libraries for its libc and with that the C++ runtime as well.
> 
> I'm 99.9% sure that it was to save space and for every dependendent app to be updated automatically to the new version of the libraries.

That is a nice side benefit. It also save RAM for the read only segments (that could be large due to tables).

But it has global state for things like malloc, that you really want to be unified within the entire application.

Otherwise you get memory leaks.

>> You are very much mistaken about lack of requests, they are not needed because dub automates the change of defaults.
> 
> I'm confused; are there requests or not?

I got it wrong, it wasn't dub that was changing defaults. Its ldc.

```
--link-defaultlib-shared                     - Link with shared versions of default libraries. Defaults to true when generating a shared library (-shared)
```

So no requests, because it is already switched (for shared libraries but not executables)!

>> When I first introduced support for shared libraries into dub, you had to explicitly set that you wanted a shared library build of druntime/phobos as well as setting dllimport to all and visibility to public.
> 
> Ok. And now it automatically says "dynamically link"? If that's the case, I didn't know that, and now I'm wondering why we have an explicit flag for that in a dub.sdl.

I got it wrong that its ldc doing it, although it did do it once back in 2016.

I don't know why its there. Is that on an executable?