https://issues.dlang.org/show_bug.cgi?id=18774

hsteoh@quickfur.ath.cx changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hsteoh@quickfur.ath.cx

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #1 from hsteoh@quickfur.ath.cx ---
AIUI, all template functions that are instantiated will end up in the object file, even if they are only used during CTFE (i.e. the instantiation is triggered by CTFE but not anywhere else).  This is a problem that's been bothering me too, which has led me to insert `if (__ctfe)` blocks into my CTFE functions.  But if you're using Phobos functions, then you're out of luck.

I agree that the compiler should have some way to track whether a particular template function / instantiation / etc., is only referenced by CTFE code, so that said functions are omitted from the object file when emitting code. Possibly something like a boolean flag in the template symbol to indicate whether the symbol is referenced by runtime constructs.

I suspect, though, that this is far from trivial to implement, because code that gets run by CTFE is essentially just one step away from being emitted as object code, and for all intents and purposes is considered as "runtime" code by many parts of the compiler.  So you get tricky things like whether a function called by a CTFE function should be marked as CTFE-only or should be included in the object file (since that function call would have been resolved before the CTFE engine even sees it, so it would already have been considered as "runtime", even if in actuality it will only ever be called from CTFE).

Still, it's a nice-to-have.

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #2 from hsteoh@quickfur.ath.cx ---
P.S. and the `if (__ctfe)` hack I use only saves on executable size, it doesn't suppress the symbol itself from appearing in the object file, so it's probably not good enough for your use case.

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #3 from Manu <turkeyman@gmail.com> ---
Well in this case, the symbol is emit in a .di file, which means it's never written to any object file; hence the unresolved external...

But that's actually kind-of irrelevant. The big question I have here is, where is the link error coming from?! Why is there a reference in main.obj to Zip!(...)'s init value? Where could that symbol reference possibly be coming from? There's no runtime calls to zip().

I don't understand how the link error is even emerging, because there should be nothing making such a reference.

--

https://issues.dlang.org/show_bug.cgi?id=18774

Jonathan M Davis <issues.dlang@jmdavisProg.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |issues.dlang@jmdavisProg.co
                   |                            |m

--- Comment #4 from Jonathan M Davis <issues.dlang@jmdavisProg.com> ---
If the .di file is being used, then the code that's importing it would expect the symbol, but if the object file is then generated from the .d file, the symbol isn't there, thus I would expect a linker error. AFAIK, the only way around that would be if the compiler were smart enough to only use the symbol for as long as necessary to do the CTFE call, because it's only used during CTFE, and I don't think that it's that smart, much as I think that we can all agree that we want it to be that smart.

I'd suggest that you provide the actual compilation commands that you're using so that it's 100% clear how to reproduce what you're doing.

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #5 from Manu <turkeyman@gmail.com> ---
> dmd -m64 main.d

Emits:

  main.obj : error LNK2019: unresolved external symbol _D4test9test_funcFiiZv
referenced in function _Dmain  (expected)
  main.obj : error LNK2001: unresolved external symbol
_D3std5range__T3ZipTAAyaTQfZQn6__initZ                (WTF?)


I think you've missed my point, there is NO REFERENCE to Zip!().init in main.d,
at least, there shouldn't be... so why the link error?
Where is the reference coming from?

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #6 from Jonathan M Davis <issues.dlang@jmdavisProg.com> ---
(In reply to Manu from comment #5)
> I think you've missed my point, there is NO REFERENCE to Zip!().init in
> main.d, at least, there shouldn't be... so why the link error?
> Where is the reference coming from?

You imported test.di. test.di references Zip!().init, and it references it when creating the function that you're calling from main.d. So, it's being used when generating main.d even if main.d doesn't use it directly. I'd guess that the problem relates to that. It's likely inserting the symbol when it imports test.di. I don't see what else the problem could be. Also, remember that you're actually generating template instantations with that code. It's not just referencing existing code like it would with a normal function. Template instantations don't live with the module that they come from in the same way that other functions do, because they can't, since they're not compiled with the module. I'd guess that that's part of the problem and that it results in the template instantiations being inserted into the resultant binary.

Curiously, if I compile your example on FreeBSD (with clang as the C/C++ compiler, though I think that it's actually using GNU's linker), I only get an undefined reference for `_D4test9test_funcFiiZv', so it appears that the problem with Zip may be system-dependent. Maybe one linker ignores it because it isn't used directly in main.d, and the other doesn't? I don't know. Certainly, the fact that it appears to differ from system to system doesn't give me a warm, fuzzy feeling.

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #7 from Manu <turkeyman@gmail.com> ---
Orly? Doesn't happen for you? That's even worse.

Just the presence of the declaration shouldn't result in a link error.
You can declare all the undefined functions you like, you don't get link errors
unless there is a reference to them that the linker tries (and can't) resolve.
I don't understand what reference there could possibly be to Zip!().init, even
if it was generated and emit to the symbol table when it was instantiated by
CTFE.

Your hunch that it's emit into some sort of template-instantiation namespace rather than into a module is probably on the money. It smells like the problem could be tangled in that complexity... but at the end of the day, I just can't imagine where the reference is that the linker could be upset about.

Anyway, this was discovered by a colleague at Blizzard who's exploring D. It would certainly be reassuring if we could fix it promptly.

--

https://issues.dlang.org/show_bug.cgi?id=18774

--- Comment #8 from Jonathan M Davis <issues.dlang@jmdavisProg.com> ---
Well, regardless of what's actually causing the problem, I completely agree that it should be fixed. However, not being a compiler dev, I have no clue how likely that is to be quick or easy.

--

https://issues.dlang.org/show_bug.cgi?id=18774

kinke@gmx.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kinke@gmx.net

--- Comment #9 from kinke@gmx.net ---
Symbol table for Win64 with a manual declaration, without importing test.di
(main.d renamed to current.d, main() renamed to foo()):

010 00000000 SECT5  notype ()    External     | _D7current3fooFZi
011 00000000 SECT6  notype       Static       | $unwind$_D7current3fooFZi
012 00000000 SECT7  notype       Static       | $pdata$_D7current3fooFZi
013 00000000 SECT4  notype       External     | _D7current12__ModuleInfoZ
014 00000000 UNDEF  notype ()    External     | _D7current9test_funcFiiZv

When importing test.di, some TypeInfos incl. accompanying functions
__xopEquals() & __xtoHash() are wrongly emitted into the module, incl. a
reference to the non-emitted struct init symbol in its TypeInfo (emitted into
current.obj), leading to the observed linking error (and referencing a bunch of
symbols in druntime):

040 00000000 SECT5  notype ()    External     | _D7current3fooFZi
041 00000000 SECT6  notype       Static       | $unwind$_D7current3fooFZi
042 00000000 SECT7  notype       Static       | $pdata$_D7current3fooFZi
043 00000000 SECT8  notype       External     | _D11TypeInfo_ya6__initZ
044 00000000 SECT9  notype ()    External     |
_D3std5array__T8AppenderTAyaZQo4Data11__xopEqualsFKxSQBzQBy__TQBvTQBpZQCdQBqKxQBaZb
045 00000000 SECTA  notype       Static       |
$unwind$_D3std5array__T8AppenderTAyaZQo4Data11__xopEqualsFKxSQBzQBy__TQBvTQBpZQCdQBqKxQBaZb
046 00000000 SECTB  notype       Static       |
$pdata$_D3std5array__T8AppenderTAyaZQo4Data11__xopEqualsFKxSQBzQBy__TQBvTQBpZQCdQBqKxQBaZb
047 00000000 SECTC  notype ()    External     |
_D3std5array__T8AppenderTAyaZQo4Data9__xtoHashFNbNeKxSQCaQBz__TQBwTQBqZQCeQBrZm
048 00000000 SECTD  notype       Static       |
$unwind$_D3std5array__T8AppenderTAyaZQo4Data9__xtoHashFNbNeKxSQCaQBz__TQBwTQBqZQCeQBrZm
049 00000000 SECTE  notype       Static       |
$pdata$_D3std5array__T8AppenderTAyaZQo4Data9__xtoHashFNbNeKxSQCaQBz__TQBwTQBqZQCeQBrZm
04A 00000000 SECTF  notype       External     |
_D44TypeInfo_S3std5array__T8AppenderTAyaZQo4Data6__initZ
04B 00000000 SECT10 notype ()    External     |
_D3std5range__T3ZipTAAyaTQfZQn11__xopEqualsFKxSQBtQBs__TQBpTQBoTQBsZQCbKxQBbZb
04C 00000000 SECT11 notype       Static       |
$unwind$_D3std5range__T3ZipTAAyaTQfZQn11__xopEqualsFKxSQBtQBs__TQBpTQBoTQBsZQCbKxQBbZb
04D 00000000 SECT12 notype       Static       |
$pdata$_D3std5range__T3ZipTAAyaTQfZQn11__xopEqualsFKxSQBtQBs__TQBpTQBoTQBsZQCbKxQBbZb
04E 00000000 SECT13 notype ()    External     |
_D3std5range__T3ZipTAAyaTQfZQn9__xtoHashFNbNeKxSQBuQBt__TQBqTQBpTQBtZQCcZm
04F 00000000 SECT14 notype       Static       |
$unwind$_D3std5range__T3ZipTAAyaTQfZQn9__xtoHashFNbNeKxSQBuQBt__TQBqTQBpTQBtZQCcZm
050 00000000 SECT15 notype       Static       |
$pdata$_D3std5range__T3ZipTAAyaTQfZQn9__xtoHashFNbNeKxSQBuQBt__TQBqTQBpTQBtZQCcZm
051 00000000 SECT16 notype       External     |
_D38TypeInfo_S3std5range__T3ZipTAAyaTQfZQn6__initZ
052 00000000 SECT17 notype       External     |
_D39TypeInfo_xS3std5range__T3ZipTAAyaTQfZQn6__initZ
053 00000000 SECT18 notype       External     | _D11TypeInfo_xm6__initZ
054 00000000 SECT19 notype       External     | _D12TypeInfo_xAa6__initZ
055 00000000 SECT1A notype       External     | _D11TypeInfo_xb6__initZ
056 00000000 SECT1B notype       External     | _D13TypeInfo_xAya6__initZ
057 00000000 SECT1C notype       External     | _D14TypeInfo_AxAya6__initZ
058 00000000 SECT1D notype       External     | _D14TypeInfo_xAAya6__initZ
059 00000000 SECT1E notype       External     |
_D36TypeInfo_E3std5range14StoppingPolicy6__initZ
05A 00000000 SECT1F notype       External     |
_D37TypeInfo_xE3std5range14StoppingPolicy6__initZ
05B 00000000 SECT4  notype       External     | _D7current12__ModuleInfoZ
05C 00000000 UNDEF  notype ()    External     | _D7test1239test_funcFiiZv
05D 00000000 UNDEF  notype       External     | _D18TypeInfo_Invariant6__vtblZ
05E 00000000 UNDEF  notype       External     | _D10TypeInfo_a6__initZ
05F 00000000 UNDEF  notype ()    External     |
_D6object__T8__equalsTxaTxaZQqFNaNbNiNfAxaQdZb
060 00000000 UNDEF  notype       External     | _D15TypeInfo_Struct6__vtblZ
061 00000000 UNDEF  notype       External     |
_D3std5array__T8AppenderTAyaZQo4Data6__initZ
062 00000000 UNDEF  notype ()    External     |
_D6object__T8__equalsTxAyaTxQfZQtFNaNbNiNfAxQvQeZb
063 00000000 UNDEF  notype       External     |
_D3std5range__T3ZipTAAyaTQfZQn6__initZ
064 00000000 UNDEF  notype       External     | _D14TypeInfo_Const6__vtblZ
065 00000000 UNDEF  notype       External     | _D10TypeInfo_m6__initZ
066 00000000 UNDEF  notype       External     | _D12TypeInfo_Axa6__initZ
067 00000000 UNDEF  notype       External     | _D10TypeInfo_b6__initZ
068 00000000 UNDEF  notype       External     | _D12TypeInfo_Aya6__initZ
069 00000000 UNDEF  notype       External     | _D14TypeInfo_Array6__vtblZ
06A 00000000 UNDEF  notype       External     | _D13TypeInfo_Enum6__vtblZ
06B 00000000 UNDEF  notype       External     | _D10TypeInfo_i6__initZ

--