Thread overview
Can anyone provide an example of how D templates are overridable by global symbols?
Dec 09, 2021
Siarhei Siamashka
Dec 09, 2021
Siarhei Siamashka
Jan 27, 2022
Siarhei Siamashka
Jan 27, 2022
kinke
Jan 28, 2022
Siarhei Siamashka
Jan 28, 2022
H. S. Teoh
Jan 29, 2022
Siarhei Siamashka
Jan 29, 2022
H. S. Teoh
Feb 03, 2022
Siarhei Siamashka
Jan 29, 2022
Ali Çehreli
December 09, 2021

A quote of Iain Buclaw from https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102765 about GDC behaviour:

>

D semantics for template symbols is that they must be overridable - even by normal global symbols.

>

So in version 11.1, the default linkage for templates was switched over to weak, and with that, you can't safely inline them without violating ODR.

My (most likely wrong) interpretation of this is that the D language standard somehow makes it impossible to make template inlining decisions at the compilation stage and this job has to be delegated to the linker. And as a result, the use of LTO becomes required for generating fast binaries. Another implication is that fast incremental rebuilds of optimized binaries are likely highly problematic.

So I have two questions:

  1. What is the exact wording of the D language standard on this matter?

  2. How would one construct a simple example of a template symbol getting successfully overridden by a global symbol?

Thanks!

December 09, 2021

On Thursday, 9 December 2021 at 20:53:52 UTC, Siarhei Siamashka wrote:

>
  1. How would one construct a simple example of a template symbol getting successfully overridden by a global symbol?

Here's my unsuccessful attempt:

module template_f;

T f(T)(T a, T b) { return a + b; }
module nontemplate_f;

int f(int a, int b) { return a - b; }
import std.stdio;

import template_f;
import nontemplate_f;

void main()
{
  f(2, 1).writeln;
}

$ gdc-10.3.0 test.d
test.d:8:4: error: nontemplate_f.f at nontemplate_f.d:3:5 conflicts with template_f.f!int.f at template_f.d:3:3
    8 |   f(2, 1).writeln;
      |    ^

This is prohibited at the compilation stage and doesn't even reach the linker.

I guess, something a bit more elaborate needs to be done to simulate a global symbol from a rogue object file overriding a template from phobos in the resulting compiled binary. The question is how to achieve this.

January 27, 2022

On Thursday, 9 December 2021 at 21:06:54 UTC, Siarhei Siamashka wrote:

>

On Thursday, 9 December 2021 at 20:53:52 UTC, Siarhei Siamashka wrote:

>

How would one construct a simple example of a template symbol getting successfully overridden by a global symbol?

Forgot to mention that a template function can be overridden by another function with the same name. But only as long as all of this happens in the scope of a single module. Here are a few examples (all of them successfully compile and run):

import std.stdio;

T f(T)(T a, T b) { return a + b; }
int f(int a, int b) { return a - b; }

void main()
{
  f(2, 1).writeln; // prints "1"
}
import std.stdio;

int f(int a, int b) { return a - b; }
T f(T)(T a, T b) { return a + b; }

void main()
{
  f(2, 1).writeln; // prints "1"
}
import std.stdio;

import template_f;
int f(int a, int b) { return a - b; }

void main()
{
  f(2, 1).writeln; // prints "1"
}
import std.stdio;

import nontemplate_f;
T f(T)(T a, T b) { return a + b; }

void main()
{
  f(2, 1).writeln; // prints "3"
}

This mostly agrees with the following part of the D language specification: https://dlang.org/spec/module.html#name_lookup

Except that having a template function and a non-template function with the same name within the same module scope doesn't seem to be explicitly documented in the D specification. But such name clash appears to be resolved in favor of a non-template function. And this behavior shouldn't inhibit functions inlining.

January 27, 2022

An example:

a.d:

import core.stdc.stdio;

void foo()() {
    version (Oops)
        printf("  foo - oops\n");
    else
        printf("  foo\n");
}

void doA() {
    printf("doA:\n");
    foo!();
}

b.d:

import core.stdc.stdio;
import a;

void main() {
    printf("main:\n");
    foo!();
    doA();
}
$ dmd -c a.d -version=Oops
$ dmd -c b.d
$ dmd a.o b.o -of=ab
$ ./ab
main:
  foo - oops
doA:
  foo - oops
$ dmd b.o a.o -of=ba
$ ./ba
main:
  foo
doA:
  foo

Each object file contains a foo!() instantiation (in a.o, the Oops version). No inlining, so the linker takes one of the weak definitions, and we end up with a consistent behavior for both calls - but the picked version is determined by the order of the object files.

Now if the calls are inlined, the behavior might not be consistent anymore. So separate compilations with different compiler flags can cause observable differences.

January 28, 2022

On Thursday, 27 January 2022 at 21:50:12 UTC, kinke wrote:

>

An example:

[...]

Now if the calls are inlined, the behavior might not be consistent anymore. So separate compilations with different compiler flags can cause observable differences.

Thanks! This was very informative. Though I'm not convinced that having a single (but randomly chosen) function used across the whole program is much better than a random mix of multiple versions. Especially if (unlike your example) this function doesn't identify itself in a user visible way. Both cases are bad, one is just much worse than the other.

Internet seems to disagree about what happens when multiple weak symbols are encountered and various interpretations can be found: "Given multiple weak symbols, choose any of the weak symbols", "if there exists several weak symbols, GCC will choose one that have the largest size (memory occupation)", etc. And I'm not inclined to happily rely on either of these opinions.

January 28, 2022
On Fri, Jan 28, 2022 at 11:01:41PM +0000, Siarhei Siamashka via Digitalmars-d-learn wrote: [...]
> Internet seems to disagree about what happens when multiple weak symbols are encountered and various interpretations can be found: "Given multiple weak symbols, choose any of the weak symbols", "if there exists several weak symbols, GCC will choose one that have the largest size (memory occupation)", etc. And I'm not inclined to happily rely on either of these opinions.

You don't have to rely on any opinions. Try it out yourself and find out for sure. E.g., compile several versions of exactly the same function (e.g, each printing something different), make sure you mark them as weak functions and rename the object files into different names.  Link them all together with another object file that contains main() that calls the weak symbol. Running the program ought to tell you which version got linked.  Try linking in different orders (specify the object files in different orders in your compile/link command) to see what differences there might be.


T

-- 
Democracy: The triumph of popularity over principle. -- C.Bond
January 29, 2022

On Friday, 28 January 2022 at 23:43:00 UTC, H. S. Teoh wrote:

>

You don't have to rely on any opinions. Try it out yourself and find out for sure.

I guess, my problem and the source of all confusion is that I'm way too used to developing C++ code. And in the C++ ecosystem your recommendation is a recipe for disaster. It's absolutely necessary to have perfect understanding about what's going on and which guarantees are provided. Accidentally relying on undefined behavior will backfire, because Murphy's law is unfortunately very real.

January 28, 2022
On Sat, Jan 29, 2022 at 12:17:49AM +0000, Siarhei Siamashka via Digitalmars-d-learn wrote:
> On Friday, 28 January 2022 at 23:43:00 UTC, H. S. Teoh wrote:
> > You don't have to rely on any opinions. Try it out yourself and find out for sure.
> 
> I guess, my problem and the source of all confusion is that I'm way too used to developing C++ code. And in the C++ ecosystem your recommendation is a recipe for disaster. It's absolutely necessary to have perfect understanding about what's going on and which guarantees are provided. Accidentally relying on undefined behavior will backfire, because [Murphy's law](https://en.wikipedia.org/wiki/Murphy%27s_law) is unfortunately very real.

Trying out what I suggested on different OS's and toolchains will give you a good idea of what's actually out there.


T

-- 
If lightning were to ever strike an orchestra, it'd always hit the conductor first.
January 28, 2022
On 1/28/22 16:17, Siarhei Siamashka wrote:
> On Friday, 28 January 2022 at 23:43:00 UTC, H. S. Teoh wrote:
>> You don't have to rely on any opinions. Try it out yourself and find
>> out for sure.
>
> I guess, my problem and the source of all confusion is that I'm way too
> used to developing C++ code.

I am confused too. Weak symbols are a concept beyond D and C++ so it should be the same with C++. Testing, the following C++ program does compile foo<int> as a weak symbol as well:

template <class T>
void foo() {
}

int main() {
  foo<int>();
}

> And in the C++ ecosystem your
> recommendation is a recipe for disaster.

And it is.

> It's absolutely necessary to
> have perfect understanding about what's going on and which guarantees
> are provided.

Good luck with that. :) There aren't many people who know what linkers and loaders actually do.

> Accidentally relying on undefined behavior will backfire,
> because [Murphy's law](https://en.wikipedia.org/wiki/Murphy%27s_law) is
> unfortunately very real.

Yes.

What Johan said makes the most sense to me: The onus of ensuring ODR is on the user. Given the state of languages and linkers, I have to ensure that.

Ali

February 03, 2022

On Saturday, 29 January 2022 at 00:52:10 UTC, H. S. Teoh wrote:

>

Trying out what I suggested on different OS's and toolchains will give you a good idea of what's actually out there.

I will just reply with a quote from https://forum.dlang.org/post/mailman.400.1643853436.20251.digitalmars-d-learn@puremagic.com : "In any case, just because it worked by chance does not mean it's OK".