Walter wrote:
> The problem is that when templates are instantiated, they are inserted into
> an object file record called a COMDAT. COMDATs have the necessary feature
> that if multiple object files have a COMDAT with the same name, the
> duplicates are discarded.
> 
> This feature is necessary for template instantiations, since the compiler
> compiling one module doesn't know about the same templates being
> instantiated by another module.
> 
> An unfortunate side effect is that the one cannot pull object files out of a
> library by referencing COMDATs alone, one must reference something else in
> that object file, too.
> 
> The dmdscript.lib library has the same problem with protoerror.d. I solved
> it with the kludge of inserting:
> 
>     int foo;
> 
> into the file, and then referencing it in dobject.d with:
> 
>     int* pfoo = &dmdscript.protoerror.foo;
> 
> Not too pretty, but it works.
> 
> 

I really appreciate this explanation, Walter.  I figured you would know what was going on!  You may have saved me a few hairs on my scalp.

- JJR

Walter escribió:
> The problem is that when templates are instantiated, they are inserted into
> an object file record called a COMDAT. COMDATs have the necessary feature
> that if multiple object files have a COMDAT with the same name, the
> duplicates are discarded.
> 
> This feature is necessary for template instantiations, since the compiler
> compiling one module doesn't know about the same templates being
> instantiated by another module.
> 
> An unfortunate side effect is that the one cannot pull object files out of a
> library by referencing COMDATs alone, one must reference something else in
> that object file, too.
> 
> The dmdscript.lib library has the same problem with protoerror.d. I solved
> it with the kludge of inserting:
> 
>     int foo;
> 
> into the file, and then referencing it in dobject.d with:
> 
>     int* pfoo = &dmdscript.protoerror.foo;
> 
> Not too pretty, but it works.
> 
> 

Could you add it somewhere in the docs? Something like, "if you have a template like this and use it like this, then do this to make it work".

While I've experienced that problem (and I think Ben too), I have no idea what you're talking about. Maybe checking dmdscript might help.

-- 
Carlos Santander Bernal

Walter wrote:
> I'm not sure how that could work.
> 
> "Nick" <Nick_member@pathlink.com> wrote in message
> news:d6aq0f$2chg$1@digitaldaemon.com...
> 
>>Interesting. But would it be possible to make the compiler auto detect
> 
> these
> 
>>cases and insert a hidden reference "behind the scenes"? It seems like a
>>straight forward workaround, and straight forward workarounds like to be
>>automated :)
> 
> 
> 

This has the potential for being a quite troublsome problem.  Dool has several template-only object files in the lib.  Problems with the other objects in the lib haven't shown up because they are being used yet in a project. :-(  Linux is free of this problem, thankfully.

So there is no possibility of a fix for this?  Is it quite complicated to repair? Perhaps it's a problem with the legacy OMF format.  Your technique fixed the problem like you said, but I'd have to start adding it to several object files in the library; dool has minimum coupling, so in order to fix the problems I'd have to reverse this fact and start making spaghetti imports in the library.

Thank you very much for the solution, though.  I can get by for now.

-JJR

"John Reimer" <brk_6502@yahoo.com> wrote in message news:d6bg0g$2ufn$1@digitaldaemon.com...
> This has the potential for being a quite troublsome problem.  Dool has several template-only object files in the lib.  Problems with the other objects in the lib haven't shown up because they are being used yet in a project. :-(  Linux is free of this problem, thankfully.
>
> So there is no possibility of a fix for this?  Is it quite complicated to repair? Perhaps it's a problem with the legacy OMF format.  Your technique fixed the problem like you said, but I'd have to start adding it to several object files in the library; dool has minimum coupling, so in order to fix the problems I'd have to reverse this fact and start making spaghetti imports in the library.
>
> Thank you very much for the solution, though.  I can get by for now.

It's a problem with the OMF format. It isn't easilly fixable.

Walter wrote:
> "John Reimer" <brk_6502@yahoo.com> wrote in message
> news:d6bg0g$2ufn$1@digitaldaemon.com...
>> This has the potential for being a quite troublsome problem.  Dool has
>> several template-only object files in the lib.  Problems with the other
>> objects in the lib haven't shown up because they are being used yet in a
>> project. :-(  Linux is free of this problem, thankfully.

I just ran into this while merging some additional Mango code into Ares.  Fortunately, this thread was fairly easy to track down, but implementing the fix has required changing almost 10 files so far, and Ares includes a very minimal portion of Mango.  I have no idea why Mango itself doesn't have this problem, but it seems I'm stuck with it.  And while this is merely annoying so far, I can see this becoming a real maintenance problem down the road.

>> So there is no possibility of a fix for this?  Is it quite complicated
>> to repair? Perhaps it's a problem with the legacy OMF format.  Your
>> technique fixed the problem like you said, but I'd have to start adding
>> it to several object files in the library; dool has minimum coupling, so
>> in order to fix the problems I'd have to reverse this fact and start
>> making spaghetti imports in the library.
>>
>> Thank you very much for the solution, though.  I can get by for now.
> 
> It's a problem with the OMF format. It isn't easilly fixable.

Why isn't this a problem with C++ template code?  A module is roughly the same as a C++ translation unit, so I imagine the same problem must occur there as well.

Here's Walter's description of the problem for reference:
----------

The problem is that when templates are instantiated, they are inserted into
an object file record called a COMDAT. COMDATs have the necessary feature
that if multiple object files have a COMDAT with the same name, the
duplicates are discarded.

This feature is necessary for template instantiations, since the compiler
compiling one module doesn't know about the same templates being
instantiated by another module.

An unfortunate side effect is that the one cannot pull object files out of a
library by referencing COMDATs alone, one must reference something else in
that object file, too.

The dmdscript.lib library has the same problem with protoerror.d. I solved
it with the kludge of inserting:

    int foo;

into the file, and then referencing it in dobject.d with:

    int* pfoo = &dmdscript.protoerror.foo;

Not too pretty, but it works.

This problem is not limited to OMF files -- it shows up on linux too, where ELF format is used.

Sean asked why this issue does not crop up with DMC, or other C++ compilers supporting templates? I'd like to understand that too. Would be nice if this (serious) difficulty were to get some lovin'

- Kris


Sean Kelly wrote:
> Walter wrote:
> 
>> "John Reimer" <brk_6502@yahoo.com> wrote in message
>> news:d6bg0g$2ufn$1@digitaldaemon.com...
>>
>>> This has the potential for being a quite troublsome problem.  Dool has
>>> several template-only object files in the lib.  Problems with the other
>>> objects in the lib haven't shown up because they are being used yet in a
>>> project. :-(  Linux is free of this problem, thankfully.
> 
> 
> I just ran into this while merging some additional Mango code into Ares.  Fortunately, this thread was fairly easy to track down, but implementing the fix has required changing almost 10 files so far, and Ares includes a very minimal portion of Mango.  I have no idea why Mango itself doesn't have this problem, but it seems I'm stuck with it.  And while this is merely annoying so far, I can see this becoming a real maintenance problem down the road.
> 
>>> So there is no possibility of a fix for this?  Is it quite complicated
>>> to repair? Perhaps it's a problem with the legacy OMF format.  Your
>>> technique fixed the problem like you said, but I'd have to start adding
>>> it to several object files in the library; dool has minimum coupling, so
>>> in order to fix the problems I'd have to reverse this fact and start
>>> making spaghetti imports in the library.
>>>
>>> Thank you very much for the solution, though.  I can get by for now.
>>
>>
>> It's a problem with the OMF format. It isn't easilly fixable.
> 
> 
> Why isn't this a problem with C++ template code?  A module is roughly the same as a C++ translation unit, so I imagine the same problem must occur there as well.
> 
> Here's Walter's description of the problem for reference:
> ----------
> 
> The problem is that when templates are instantiated, they are inserted into
> an object file record called a COMDAT. COMDATs have the necessary feature
> that if multiple object files have a COMDAT with the same name, the
> duplicates are discarded.
> 
> This feature is necessary for template instantiations, since the compiler
> compiling one module doesn't know about the same templates being
> instantiated by another module.
> 
> An unfortunate side effect is that the one cannot pull object files out of a
> library by referencing COMDATs alone, one must reference something else in
> that object file, too.
> 
> The dmdscript.lib library has the same problem with protoerror.d. I solved
> it with the kludge of inserting:
> 
>     int foo;
> 
> into the file, and then referencing it in dobject.d with:
> 
>     int* pfoo = &dmdscript.protoerror.foo;
> 
> Not too pretty, but it works.

kris wrote:

> This problem is not limited to OMF files -- it shows up on linux too, where ELF format is used.
> 
> Sean asked why this issue does not crop up with DMC, or other C++ compilers supporting templates? I'd like to understand that too. Would be nice if this (serious) difficulty were to get some lovin'
> 
> - Kris

Note that this has been reported before:

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951

Walter promised to check out it, but I haven't noticed any replies. Sitting around #D, I've gotten the impression that Mr Chancellor and others consider D's templates to be thouroughly broken due to these issues.

Lars Ivar Igesund wrote:
> kris wrote:
> 
>> This problem is not limited to OMF files -- it shows up on linux too,
>> where ELF format is used.
>>
>> Sean asked why this issue does not crop up with DMC, or other C++
>> compilers supporting templates? I'd like to understand that too. Would
>> be nice if this (serious) difficulty were to get some lovin'
>>
>> - Kris
> 
> Note that this has been reported before:
> 
> http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951
> 
> Walter promised to check out it, but I haven't noticed any replies. Sitting
> around #D, I've gotten the impression that Mr Chancellor and others
> consider D's templates to be thouroughly broken due to these issues.

It "works" in C++ because the compiler regenerates the template instantiation in *every* object file where it is used. The C++ compiler doesn't know about other object files.

In D, when multiple sources are given on the command line, the compiler doesn't generate redundant template instantiations.

Walter Bright wrote:
> Lars Ivar Igesund wrote:
>> kris wrote:
>>
>>> This problem is not limited to OMF files -- it shows up on linux too,
>>> where ELF format is used.
>>>
>>> Sean asked why this issue does not crop up with DMC, or other C++
>>> compilers supporting templates? I'd like to understand that too. Would
>>> be nice if this (serious) difficulty were to get some lovin'
>>
>> Note that this has been reported before:
>>
>> http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951
>>
>> Walter promised to check out it, but I haven't noticed any replies. Sitting
>> around #D, I've gotten the impression that Mr Chancellor and others
>> consider D's templates to be thouroughly broken due to these issues.
> 
> It "works" in C++ because the compiler regenerates the template instantiation in *every* object file where it is used. The C++ compiler doesn't know about other object files.
> 
> In D, when multiple sources are given on the command line, the compiler doesn't generate redundant template instantiations.

But Ares is built one object at a time via a makefile, so there are no "multiple sources given on the command line."  Did you mean, perhaps, that this is somehow related to how DMD treats modules vs. how DMC treats source files?  ie. that the #include model effectively copy/pastes code while importing modules does not?


Sean

Sean Kelly wrote:
> Walter Bright wrote:
>> Lars Ivar Igesund wrote:
>>> kris wrote:
>>>
>>>> This problem is not limited to OMF files -- it shows up on linux too,
>>>> where ELF format is used.
>>>>
>>>> Sean asked why this issue does not crop up with DMC, or other C++
>>>> compilers supporting templates? I'd like to understand that too. Would
>>>> be nice if this (serious) difficulty were to get some lovin'
>>>
>>> Note that this has been reported before:
>>>
>>> http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5951
>>>
>>> Walter promised to check out it, but I haven't noticed any replies. Sitting
>>> around #D, I've gotten the impression that Mr Chancellor and others
>>> consider D's templates to be thouroughly broken due to these issues.
>>
>> It "works" in C++ because the compiler regenerates the template instantiation in *every* object file where it is used. The C++ compiler doesn't know about other object files.
>>
>> In D, when multiple sources are given on the command line, the compiler doesn't generate redundant template instantiations.
> 
> But Ares is built one object at a time via a makefile, so there are no "multiple sources given on the command line."  Did you mean, perhaps, that this is somehow related to how DMD treats modules vs. how DMC treats source files?  ie. that the #include model effectively copy/pastes code while importing modules does not?

Let me clarify a bit further.  The problem I ran into when importing Mango code into Ares was that when modules with non-template code instantiated template code also defined in the library, then I'd get a link error when attempting to use that non-template library code in an application.  I speculated that perhaps the template code simply wasn't being generated for some reason so I ran obj2asm on the library object files and found I was wrong--so far as I could tell, the template code had been instantiated into the object file of every module using the code.  But that's as far as I got.  I wondered if perhaps the code was perhaps being thrown out by the 'lib' tool, but didn't make much progress inspecting the library code itself.  Further, it seems kind of odd that the same problem seems to occur on Linux, which has a different object file format AFAIK.

Aside from the workaround you suggest a while back--adding 'fake' non-template data and referencing it in the appropriate places--are there any other options for library writers?  Your proposed solution works (or it did for me at any rate), but it's a nasty hack that I don't see being well-accepted in the long term.  Implementation complexity aside, I would be happy with even "broken" code generation similar to how C++ object files are created if it would actually solve this problem without introducing a host of new ones.  But I still don't entirely understand what's going on.  Is the problem truly in code generation or does it occur later during library assembly or during linking?  And are there any alternatives such as different object file formats that might correct this as well?  From your discussion of COMDATs I don't understand why this would be a problem on Linux as well, unless perhaps the object files have a similarly bad design?


Sean