On Wednesday, 10 June 2020 at 01:48:20 UTC, Walter Bright wrote:
> On 6/9/2020 4:31 PM, Manu wrote:
>> Redundant work,
>
> It isn't redundant work to do it in the linker. It is redundant to do half of the linker's job in the compiler, throw away the other half of it, and re-do it in the linker.
>
> > undesired object bloat,
>
> Not a problem since we moved on from floppy disks.

People complain about Visual Studio's download size, most of the download is library files.

I imagine it is also a problem for high performance computing as well. So, yes still a problem.

> > and various potential link issues/errors.
>
> I suppose that fits in with the common notion that the linker is black magic full of trolls and dragons. It isn't, it's a boringly simple program. Although it is true that probably only 1 in 100 programmers can explain what a linker does.
>
> (Most of the complexity of linkers is not inherent, it is the result of file formats designed by hamsters and workarounds for compiler and loader bugs. You have to pity the linker developer - there's no glamor, nobody understands what they do, nobody praises them, they just dump on them with "what does 'undefined symbol mean' - must be a bug in the linker" questions.)

I think maybe new programmers think that, ones that come from languages like python or otherwise where they don't have to deal with the abomination known as a linker. That's the  thing, the way the workflow is setup the problem usually isn't with the linker which makes it really annoying to try and find where the problem actually is. Especially if it's a compiler bug :).

On Wednesday, 10 June 2020 at 01:48:20 UTC, Walter Bright wrote:
> On 6/9/2020 4:31 PM, Manu wrote:
>> [...]
>
> It isn't redundant work to do it in the linker. It is redundant to do half of the linker's job in the compiler, throw away the other half of it, and re-do it in the linker.
>
> [...]

we have a project that compiles with "-allinst -allinst -allinst".
To avoid such errors.
(the repetition comes from dub appending to the dflags)

On Wednesday, 10 June 2020 at 18:03:21 UTC, Avrina wrote:
> [snip]
>
> I imagine it is also a problem for high performance computing as well. So, yes still a problem.
> [snip]

For HPC, it would only be an issue during development (where you can probably do a lot of the work on your own machine), rather than during production.

On Wednesday, 10 June 2020 at 13:55:40 UTC, Steven Schveighoffer wrote:
> On 6/9/20 8:20 PM, Stefan Koch wrote:
>> Yes making the function a function template will do that.
>> And Manu is fully aware of it.
>> 
>> It's a nasty hack.
>> And in certain cases it screws with overload-resolution.
>
> It's not exactly the same. If the compiler thinks an imported module has instantiated it, it will not include it in the object file (unless you use -allinst).

Nope, -allinst doesn't make sure all templates are instantiated in each object file. I'll link again to https://github.com/ldc-developers/ldc/pull/3422#issuecomment-625945508.

On Wednesday, 10 June 2020 at 01:48:20 UTC, Walter Bright wrote:
> On 6/9/2020 4:31 PM, Manu wrote:
>> Redundant work,
>
> It isn't redundant work to do it in the linker. It is redundant to do half of the linker's job in the compiler, throw away the other half of it, and re-do it in the linker.

It is redundant to do the job in the compiler, throw *that* away and then shrug it off with "meh, the linker will take care of it". Which is precisely what compilers are doing. Which is precisely what you suggested to be doing in the issue report.
...And of course, the linker doesn't take care of it, which is why LTO came to be - to solve a problem that doesn't need to exist.

And DMD does not need a built-in linker.

On 6/10/20 2:54 PM, kinke wrote:
> On Wednesday, 10 June 2020 at 13:55:40 UTC, Steven Schveighoffer wrote:
>> On 6/9/20 8:20 PM, Stefan Koch wrote:
>>> Yes making the function a function template will do that.
>>> And Manu is fully aware of it.
>>>
>>> It's a nasty hack.
>>> And in certain cases it screws with overload-resolution.
>>
>> It's not exactly the same. If the compiler thinks an imported module has instantiated it, it will not include it in the object file (unless you use -allinst).
> 
> Nope, -allinst doesn't make sure all templates are instantiated in each object file. I'll link again to https://github.com/ldc-developers/ldc/pull/3422#issuecomment-625945508.

I can't tell from that discussion what you mean.

My understanding with -allinst is that the compiler is supposed to instantiate all templates even if it thinks a dependent library has already done it.

But I'm happy to defer to you as an actual developer of the compiler, I'm not. I have had experience as a user with the compiler making wrong decisions about whether it should emit code for a template instance or not.

In any case, my point is that just making it a template isn't the equivalent as has been suggested.

-Steve

On Wednesday, 10 June 2020 at 19:20:55 UTC, Steven Schveighoffer wrote:
> My understanding with -allinst is that the compiler is supposed to instantiate all templates even if it thinks a dependent library has already done it.
>
> But I'm happy to defer to you as an actual developer of the compiler, I'm not. I have had experience as a user with the compiler making wrong decisions about whether it should emit code for a template instance or not.
>
> In any case, my point is that just making it a template isn't the equivalent as has been suggested.

You're spot-on, except for the fact that a template instance is emitted into at most 1 object file per compiler command-line. So if you compile a static lib with 100 modules in one go, and each module instantiates some common template, it's emitted into exactly one object file when using -allinst (and possibly -unittest too, I don't remember exactly), and *might* be emitted into a (single) object file without -allinst (because as you said, if the compiler sees another non-compiled imported module instantiating it, the compiler elides its codegen and depends on that external definition later when linking).

TLDR: Templates, with the current emission scheme, can definitely not be used to emit a function into every referencing CU.

On 6/10/2020 11:03 AM, Avrina wrote:
> I think maybe new programmers think that,

I've seen near total linker puzzlement from people with long experience with C and C++. I don't know how this happens, just that it is commonplace.

> ones that come from languages like python or otherwise where they don't have to deal with the abomination known as a linker.

"Abomination" is a misunderstanding of what a linker is and does.

> That's the  thing, the way the workflow is setup the problem usually isn't with the linker which makes it really annoying to try and find where the problem actually is.

Understanding the simple process of linking is key to quickly resolving whether a problem is linker or compiler.

I wrote a "how linkers work" chapter for Kevlin Henney's book "97 Things Every Programmer Should Know":

https://www.amazon.com/Things-Every-Programmer-Should-Know/dp/0596809484/

No, I don't get any payments from that book. I was talking to Kevlin once grousing about nobody understanding linkers, and he invited me to write a chapter :-)

On 6/10/2020 12:04 PM, Stanislav Blinov wrote:
> It is redundant to do the job in the compiler, throw *that* away and then shrug it off with "meh, the linker will take care of it". Which is precisely what compilers are doing.

All I can say is you're misunderstanding what linkers do.

On Wednesday, 10 June 2020 at 22:29:59 UTC, Walter Bright wrote:
> ...
> I wrote a "how linkers work" chapter for Kevlin Henney's book "97 Things Every Programmer Should Know":
>
> https://www.amazon.com/Things-Every-Programmer-Should-Know/dp/0596809484/
>
> No, I don't get any payments from that book. I was talking to Kevlin once grousing about nobody understanding linkers, and he invited me to write a chapter :-)

Interesting: https://github.com/97-things/97-things-every-programmer-should-know/blob/master/en/thing_53/README.md

Matheus.