http://d.puremagic.com/issues/show_bug.cgi?id=11284

           Summary: add -allinst compiler switch
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody@puremagic.com
        ReportedBy: bugzilla@digitalmars.com


--- Comment #0 from Walter Bright <bugzilla@digitalmars.com> 2013-10-16 16:36:37 PDT ---
Given these two files:

  ----- a.d ------
  int foo(T)(T i) { return i;)
  enum x = foo(3);
  enum f = foo(2.3);
  ---- test.d ----
  import a;
  int y = foo(4);
  ----------------

Compiling with 2.063 or earlier with:

    dmd test -c

would generate an instance of foo!int and foo!double into test.obj. In fact, every file that imported a.d would have those instantiations inserted into its resulting object file. This didn't affect the executable, as the linker would remove duplicates (the magic of COMDAT segments).

However, template heavy imports can and did result in inordinately massive .obj files, for example, try importing std.stdio.

Starting with 2.064, the compiler notices that foo!double is instantiated only by import a, and so the code generated for foo!double would presumably be in the object file a.obj, and so the compiler does not have to insert it into test.obj. This can potentially result in large compiler speed increases, which are a big feature of D.

If the foo!double is not in a.obj, then the user sees "undefined symbol foo!double" coming from the linker in code that worked with 2.063, and files a bug report. This circumstance can appear when:

1. a.obj is never linked in, i.e. a.d is regarded as a "header only" file.

2. a.d was modified but a.obj was never recompiled with the newer a.d.

3. a.d's template instantiations are behind VersionDeclaration's and the versions in effect when a.d is compiled are different than the versions when test.d is compiled.

4. Other forms of conditional compilation analogous to case (3).

I regard these as user build system irregularities, not bugs. However, user confusion over this, and frustration over "it worked in 2.063 and now I get this error message, what's wrong" is very real. Hence, I propose the -allinst switch which will generate code for all instantiated templates, like 2.063 does.

This way, the user can throw the switch and continue working as before, or if he wants to track down the source of the undefined symbol at least the switch provides a starting point in diagnosing if it is one of cases 1..4 or something else.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284


jfanatiker@gmx.at changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jfanatiker@gmx.at


--- Comment #1 from jfanatiker@gmx.at 2013-10-18 01:35:45 PDT ---
It makes sense, but in this particular case: Why is foo!double put into a.obj in the first place, when it is only used in CTFE? From reading the NG, it seems there are a lot of unnecessary templates instantiated, for things like template constraints.

Best regards,
Robert

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284


Dicebot <public@dicebot.lv> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |public@dicebot.lv


--- Comment #2 from Dicebot <public@dicebot.lv> 2013-10-18 05:52:29 PDT ---
(In reply to comment #1)
> It makes sense, but in this particular case: Why is foo!double put into a.obj in the first place, when it is only used in CTFE? From reading the NG, it seems there are a lot of unnecessary templates instantiated, for things like template constraints.
> 
> Best regards,
> Robert

There are 2 reasons for this.

First, related part of DMD source code is damn complicated. I have wasted several months trying to figure this out and when Walter came out with his own partial solution I was hardly even able to understand reasoning behind those exact changes.

But probably I am just stupid. Second issue is much more important though:

Currently D spec does not allow any symbol visibility optimizations. The fact that symbol X is used only during CTFE during compilation of module A does not mean it won't be used in runtime from module B. Or even linked dynamically. Any really clever stuff can be done here only after specification is restricted.

I have had a proposal for internal linkage attribute ages ago which is tightly related but I am not skilled enough to implement it right now.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284



--- Comment #3 from Walter Bright <bugzilla@digitalmars.com> 2013-10-18 15:58:07 PDT ---
https://github.com/D-Programming-Language/dmd/pull/2680

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284


Max Samukha <samukha@voliacable.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |samukha@voliacable.com


--- Comment #4 from Max Samukha <samukha@voliacable.com> 2013-10-18 16:07:29 PDT ---
(In reply to comment #1)
> It makes sense, but in this particular case: Why is foo!double put into a.obj in the first place, when it is only used in CTFE? From reading the NG, it seems there are a lot of unnecessary templates instantiated, for things like template constraints.
> 
> Best regards,
> Robert

That is the problem. There should be a way to specify that a function is intended for use only in CTFE. I tend to wrap the body of such functions in "if (__ctfe) {...} else assert(false); but that still sucks.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284



--- Comment #6 from Dicebot <public@dicebot.lv> 2013-10-18 17:44:14 PDT ---
(In reply to comment #5)
> One way to improve the situation a little that I think doesn't require compiler changes is to add a "private":
> 
> private int foo(int x) { return x ^^ 2; }
> 
> If this "foo" is used only at compile-time inside the module, then perhaps it should not appear in the module object.

Unfortunately, it is not that simple.

First, it may break existing code as currently it has external linkage and can be used from C modules. Second, some private symbols may actually leak outside of the module via public aliases and function parameters / return values. This is partially explained here:

http://wiki.dlang.org/Access_specifiers_and_visibility http://wiki.dlang.org/DIP22.1

What is more important, though, is that templates and CTFE functions are rarely used in the same module they are declared. I have several ideas for more efficient design that can fix this but all contain breaking changes when it comes to separate compilation (in absence of whole program optimization at least).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284



--- Comment #7 from jfanatiker@gmx.at 2013-10-20 02:01:54 PDT ---
> Currently D spec does not allow any symbol visibility optimizations. The fact that symbol X is used only during CTFE during compilation of module A does not mean it won't be used in runtime from module B. Or even linked dynamically. Any really clever stuff can be done here only after specification is restricted.

I don't really get this. I am not talking about omitting regular functions which are only used during CTFE in a module, because obviously the function might be of use at runtime from within another module. But why do we put a template instantiation in a.obj that never really got instantiated? If b.d uses it, it should go to b.obj just as if a.d would not have used it at all.

I don't get why the use in CTFE triggers template instantiations.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284


Denis Shelomovskij <verylonglogin.reg@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |verylonglogin.reg@gmail.com


--- Comment #8 from Denis Shelomovskij <verylonglogin.reg@gmail.com> 2013-10-20 15:10:52 MSD ---
(In reply to comment #0)
> I regard these as user build system irregularities, not bugs. However, user confusion over this, and frustration over "it worked in 2.063 and now I get this error message, what's wrong" is very real. Hence, I propose the -allinst switch which will generate code for all instantiated templates, like 2.063 does.

Every release "accept-invalid" bugs are fixed and incorrect user code that used to work become uncompileable (showing user his mistakes and making user happy :) ).

If in this particular case invalid compilation actions are too common to be
broken immediately:
1. Links to user complains are needed to proof its huge count.
2. We have to find and resolve our mistake (e.g. lack of docs) that causes D
user to do such errors in building process.

Also IMHO points 2, 3, and 4 from the description are obviously user build process bugs that have to be pointed out by a tool-chain like it do now to prevent the worst case of buildable but incorrectly working application.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284



--- Comment #9 from Denis Shelomovskij <verylonglogin.reg@gmail.com> 2013-10-20 15:17:22 MSD ---
Also it's too early to talk about such switch because symbol elimination mechanism from dmd pull #2550 [1] have never worked correctly as it fails to detect symbol instantiating module (see (already fixed) Issue 11069 and Issue 11114) and thus can't be included in releases at least until known regressions are fixed.

[1] https://github.com/D-Programming-Language/dmd/pull/2550

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11284



--- Comment #10 from github-bugzilla@puremagic.com 2013-10-23 10:01:44 PDT ---
Commits pushed to master at https://github.com/D-Programming-Language/dmd

https://github.com/D-Programming-Language/dmd/commit/a1f9f08b78f6a358e98ac7a4b0423b0ae779e943 fix Issue 11284 - add -allinst compiler switch

https://github.com/D-Programming-Language/dmd/commit/58355d7481b53e5a1e88c12f72075832c0c8c04a Merge pull request #2680 from WalterBright/fix11284

fix Issue 11284 - add -allinst compiler switch

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------