kris wrote:
> Walter Bright wrote:
> 
>> kris wrote:
>>
>>> I've been messing with the response file handed to the librarian (via lib @foo); moving modules around here and there, reordering big chunks etc. Have yet to see a notable change in the resulting exe after relinking against each lib version.
>>
>>
>>
>> Then look at the .map file to see what was linked in to the larger file that wasn't in the smaller. Remove that module from the library. Link again, and see what was unresolved. Rinse, repeat, and you'll see what was pulling it all in.
> 
> 
> That's exactly what I'm doing, and I agree there seems to be something odd going on here. With ~200 modules, it's no slam-dunk to isolate it :)


OK: narrowed it down to one obj file. The vast sea of data is coming from the 'locale' package, which is stuffed to the gills with I18N content.

There's a half dozen modules in 'locale', none of which are used or referenced by any other code in Tango, or by the example at hand. It is an entirely isolated package (for obvious reason).

Yes, there's the /potential/ for symbolic collision between 'locale' and some other module/package. Let's consider that in a moment.

In the meantime, I whittled the dependency down to one single module listed right at the very end of the lib-response file (the very last one to be added to the lib). This module is called Core.

When Core is added to the lib, the linker emits "missing symbol" errors since the rest of the locale package is missing. When Core is removed, there are no link errors. This indicates some kind of symbolic collision; one that is triggered by the very last module added to the lib?

So, sifting through the obj2asm output for Core, I see all the publics are correctly prefixed by the package name. Thus, each symbol exposed appears to be unique across the entire library. Except for one that stands out. It is noted as an 'extern', yet is also listed amongst the full set of publics in the obj2asm output; and it looks a bit suspicious. Here's a small snippet from the ocean of public symbols in Core:

====================
_D5tango4text6locale4Core18DaylightSavingTime5_ctorMFS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8TimeSpanZC5tango4text6locale4Core18DaylightSavingTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core18DaylightSavingTime5startMFZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core18DaylightSavingTime3endMFZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core18DaylightSavingTime6changeMFZS5tango4text6locale4Core8TimeSpan COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime9getSundayMFiiiiiiiiZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone12getLocalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone16getUniversalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone12getUtcOffsetMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8TimeSpan COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone20isDaylightSavingTimeMFS5tango4text6locale4Core8DateTimeZb COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone7currentFZC5tango4text6locale4Core8TimeZone COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8TimeZone5_ctorMFZC5tango4text6locale4Core8TimeZone COMDAT flags=x0 attr=x0 align=x0
_D47TypeInfo_C5tango4text6locale4Core12NumberFormat6__initZ	COMDAT flags=x0 attr=x10 align=x0
_D49TypeInfo_C5tango4text6locale4Core14DateTimeFormat6__initZ	COMDAT flags=x0 attr=x10 align=x0
_D5tango4text6locale4Core14__T7arrayOfTiZ7arrayOfFAiXAi	COMDAT flags=x0 attr=x10 align=x0
_D5tango4text6locale4Core15__T7arrayOfTAaZ7arrayOfFAAaXAAa	COMDAT flags=x0 attr=x10 align=x0
_D12TypeInfo_AAa6__initZ	COMDAT flags=x0 attr=x10 align=x0
__D5tango4text6locale4Core9__modctorFZv	COMDAT flags=x0 attr=x0 align=x0
__D5tango4text6locale4Core9__moddtorFZv	COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core8__assertFiZv	COMDAT flags=x0 attr=x0 align=x0
_D5tango4text6locale4Core7__arrayZ	COMDAT flags=x0 attr=x0 align=x0

====================

You see the odd one out? That cursed _D12TypeInfo_AAa6__initZ again?

kris wrote:
> kris wrote:
>> Walter Bright wrote:
>>
>>> kris wrote:
>>>
>>>> I've been messing with the response file handed to the librarian (via lib @foo); moving modules around here and there, reordering big chunks etc. Have yet to see a notable change in the resulting exe after relinking against each lib version.
>>>
>>>
>>>
>>> Then look at the .map file to see what was linked in to the larger file that wasn't in the smaller. Remove that module from the library. Link again, and see what was unresolved. Rinse, repeat, and you'll see what was pulling it all in.
>>
>>
>> That's exactly what I'm doing, and I agree there seems to be something odd going on here. With ~200 modules, it's no slam-dunk to isolate it :)
> 
> 
> OK: narrowed it down to one obj file. The vast sea of data is coming from the 'locale' package, which is stuffed to the gills with I18N content.
> 
> There's a half dozen modules in 'locale', none of which are used or referenced by any other code in Tango, or by the example at hand. It is an entirely isolated package (for obvious reason).
> 
> Yes, there's the /potential/ for symbolic collision between 'locale' and some other module/package. Let's consider that in a moment.
> 
> In the meantime, I whittled the dependency down to one single module listed right at the very end of the lib-response file (the very last one to be added to the lib). This module is called Core.
> 
> When Core is added to the lib, the linker emits "missing symbol" errors since the rest of the locale package is missing. When Core is removed, there are no link errors. This indicates some kind of symbolic collision; one that is triggered by the very last module added to the lib?
> 
> So, sifting through the obj2asm output for Core, I see all the publics are correctly prefixed by the package name. Thus, each symbol exposed appears to be unique across the entire library. Except for one that stands out. It is noted as an 'extern', yet is also listed amongst the full set of publics in the obj2asm output; and it looks a bit suspicious. Here's a small snippet from the ocean of public symbols in Core:
> 
> ====================
> _D5tango4text6locale4Core18DaylightSavingTime5_ctorMFS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8TimeSpanZC5tango4text6locale4Core18DaylightSavingTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core18DaylightSavingTime5startMFZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core18DaylightSavingTime3endMFZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core18DaylightSavingTime6changeMFZS5tango4text6locale4Core8TimeSpan COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime9getSundayMFiiiiiiiiZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone12getLocalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone16getUniversalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone12getUtcOffsetMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8TimeSpan COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone20isDaylightSavingTimeMFS5tango4text6locale4Core8DateTimeZb COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone7currentFZC5tango4text6locale4Core8TimeZone COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8TimeZone5_ctorMFZC5tango4text6locale4Core8TimeZone COMDAT flags=x0 attr=x0 align=x0
> _D47TypeInfo_C5tango4text6locale4Core12NumberFormat6__initZ    COMDAT flags=x0 attr=x10 align=x0
> _D49TypeInfo_C5tango4text6locale4Core14DateTimeFormat6__initZ    COMDAT flags=x0 attr=x10 align=x0
> _D5tango4text6locale4Core14__T7arrayOfTiZ7arrayOfFAiXAi    COMDAT flags=x0 attr=x10 align=x0
> _D5tango4text6locale4Core15__T7arrayOfTAaZ7arrayOfFAAaXAAa    COMDAT flags=x0 attr=x10 align=x0
> _D12TypeInfo_AAa6__initZ    COMDAT flags=x0 attr=x10 align=x0
> __D5tango4text6locale4Core9__modctorFZv    COMDAT flags=x0 attr=x0 align=x0
> __D5tango4text6locale4Core9__moddtorFZv    COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core8__assertFiZv    COMDAT flags=x0 attr=x0 align=x0
> _D5tango4text6locale4Core7__arrayZ    COMDAT flags=x0 attr=x0 align=x0
> 
> ====================
> 
> You see the odd one out? That cursed _D12TypeInfo_AAa6__initZ again?

Sorry if I'm stating the obvious, but it seems to me that the linker is finding this typeinfo COMDAT in Core first, rather than somewhere else, and is thereby forcing the inclusion of the rest of it's containing module.

Does moving core.obj to the end of the .lib solve the problem?

-- 
- EricAnderton at yahoo

Pragma wrote:
> kris wrote:
> 
>> kris wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> kris wrote:
>>>>
>>>>> I've been messing with the response file handed to the librarian (via lib @foo); moving modules around here and there, reordering big chunks etc. Have yet to see a notable change in the resulting exe after relinking against each lib version.
>>>>
>>>>
>>>>
>>>>
>>>> Then look at the .map file to see what was linked in to the larger file that wasn't in the smaller. Remove that module from the library. Link again, and see what was unresolved. Rinse, repeat, and you'll see what was pulling it all in.
>>>
>>>
>>>
>>> That's exactly what I'm doing, and I agree there seems to be something odd going on here. With ~200 modules, it's no slam-dunk to isolate it :)
>>
>>
>>
>> OK: narrowed it down to one obj file. The vast sea of data is coming from the 'locale' package, which is stuffed to the gills with I18N content.
>>
>> There's a half dozen modules in 'locale', none of which are used or referenced by any other code in Tango, or by the example at hand. It is an entirely isolated package (for obvious reason).
>>
>> Yes, there's the /potential/ for symbolic collision between 'locale' and some other module/package. Let's consider that in a moment.
>>
>> In the meantime, I whittled the dependency down to one single module listed right at the very end of the lib-response file (the very last one to be added to the lib). This module is called Core.
>>
>> When Core is added to the lib, the linker emits "missing symbol" errors since the rest of the locale package is missing. When Core is removed, there are no link errors. This indicates some kind of symbolic collision; one that is triggered by the very last module added to the lib?
>>
>> So, sifting through the obj2asm output for Core, I see all the publics are correctly prefixed by the package name. Thus, each symbol exposed appears to be unique across the entire library. Except for one that stands out. It is noted as an 'extern', yet is also listed amongst the full set of publics in the obj2asm output; and it looks a bit suspicious. Here's a small snippet from the ocean of public symbols in Core:
>>
>> ====================
>> _D5tango4text6locale4Core18DaylightSavingTime5_ctorMFS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8DateTimeS5tango4text6locale4Core8TimeSpanZC5tango4text6locale4Core18DaylightSavingTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core18DaylightSavingTime5startMFZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core18DaylightSavingTime3endMFZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core18DaylightSavingTime6changeMFZS5tango4text6locale4Core8TimeSpan COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone18getDaylightChangesMFiZC5tango4text6locale4Core18DaylightSavingTime9getSundayMFiiiiiiiiZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone12getLocalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone16getUniversalTimeMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8DateTime COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone12getUtcOffsetMFS5tango4text6locale4Core8DateTimeZS5tango4text6locale4Core8TimeSpan COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone20isDaylightSavingTimeMFS5tango4text6locale4Core8DateTimeZb COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone7currentFZC5tango4text6locale4Core8TimeZone COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8TimeZone5_ctorMFZC5tango4text6locale4Core8TimeZone COMDAT flags=x0 attr=x0 align=x0
>> _D47TypeInfo_C5tango4text6locale4Core12NumberFormat6__initZ    COMDAT flags=x0 attr=x10 align=x0
>> _D49TypeInfo_C5tango4text6locale4Core14DateTimeFormat6__initZ    COMDAT flags=x0 attr=x10 align=x0
>> _D5tango4text6locale4Core14__T7arrayOfTiZ7arrayOfFAiXAi    COMDAT flags=x0 attr=x10 align=x0
>> _D5tango4text6locale4Core15__T7arrayOfTAaZ7arrayOfFAAaXAAa    COMDAT flags=x0 attr=x10 align=x0
>> _D12TypeInfo_AAa6__initZ    COMDAT flags=x0 attr=x10 align=x0
>> __D5tango4text6locale4Core9__modctorFZv    COMDAT flags=x0 attr=x0 align=x0
>> __D5tango4text6locale4Core9__moddtorFZv    COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core8__assertFiZv    COMDAT flags=x0 attr=x0 align=x0
>> _D5tango4text6locale4Core7__arrayZ    COMDAT flags=x0 attr=x0 align=x0
>>
>> ====================
>>
>> You see the odd one out? That cursed _D12TypeInfo_AAa6__initZ again?
> 
> 
> Sorry if I'm stating the obvious, but it seems to me that the linker is finding this typeinfo COMDAT in Core first, rather than somewhere else, and is thereby forcing the inclusion of the rest of it's containing module.
> 
> Does moving core.obj to the end of the .lib solve the problem?
> 

Heya Eric

That's what it seems like and (as noted above) core.obj is already the very last one added to the lib ;)

The only way to resolve at this point is to remove core.obj entirely.

kris wrote:
> _D12TypeInfo_AAa6__initZ    COMDAT flags=x0 attr=x10 align=x0

TypeInfo's don't get the module prefix because it would cause duplication of code (i.e. bloat) to have it that way. There is no difference between the TypeInfo for char[][] in one module, and the TypeInfo for char[][] in another, so the TypeInfo names should match exactly.

kris wrote:
> Pragma wrote:
>> Sorry if I'm stating the obvious, but it seems to me that the linker is finding this typeinfo COMDAT in Core first, rather than somewhere else, and is thereby forcing the inclusion of the rest of it's containing module.
>>
>> Does moving core.obj to the end of the .lib solve the problem?
>>
> 
> Heya Eric
> 
> That's what it seems like and (as noted above) core.obj is already the very last one added to the lib ;)
> 
> The only way to resolve at this point is to remove core.obj entirely.

Did you try putting it at the front of the lib? You never know, maybe it picks the last one instead of the first one.

Unless it just happens to be the only module to define _D12TypeInfo_AAa6__initZ ...

Walter Bright wrote:
>> It does, but increases the exe size of the first example from 180kb to 617kb!
>  > 180kb is when compiled using build/rebuild/jake etc (no library) and the 617kb
>  > is when using dmd+lib only. Same flags in both cases: none at all
> 
> Let's say you have a template instance, TI. It is declared in two modules, M1 and M2:
> 
> -----------M1------------
> TI
> A
> -----------M2------------
> TI
> B
> -------------------------
> 
> M1 also declares A, and M2 also declares B. Now, the linker is looking to resolve TI, and the first one it finds is one in M1, and so links in M1. Later on, it needs to resolve B, and so links in M2. The redundant TI is discarded (because it's a COMDAT).
> 
> However, suppose the program never references A, and A is a chunk of code that pulls in lots of other bloat. This could make the executable much larger than if, in resolving TI, it had picked M2 instead.

For some reason I thought an optimizing linker worked at a segment level, but I suppose that is not true for data in a library?  In other words, since libraries are indexed by module name, I suppose this means they are necessarily dealt with at module granularity instead?

Walter Bright wrote:
> kris wrote:
> 
>> _D12TypeInfo_AAa6__initZ    COMDAT flags=x0 attr=x10 align=x0
> 
> 
> TypeInfo's don't get the module prefix because it would cause duplication of code (i.e. bloat) to have it that way. There is no difference between the TypeInfo for char[][] in one module, and the TypeInfo for char[][] in another, so the TypeInfo names should match exactly.

well, ok ... but it is responsible for what happened here? If not, what is?

Frits van Bommel wrote:
> kris wrote:
>> Pragma wrote:
>>> Sorry if I'm stating the obvious, but it seems to me that the linker is finding this typeinfo COMDAT in Core first, rather than somewhere else, and is thereby forcing the inclusion of the rest of it's containing module.
>>>
>>> Does moving core.obj to the end of the .lib solve the problem?
>>>
>>
>> Heya Eric
>>
>> That's what it seems like and (as noted above) core.obj is already the very last one added to the lib ;)
>>
>> The only way to resolve at this point is to remove core.obj entirely.
> 
> Did you try putting it at the front of the lib? You never know, maybe it picks the last one instead of the first one.
> 
> Unless it just happens to be the only module to define _D12TypeInfo_AAa6__initZ ...

I thought about that too, but that doesn't seem to be the case. As Kris also stated, the lib compiles when Core is removed.  That implies that the TypeInfo mentioned lives elsewhere - and it very likely does, as it's a "char[char[]]".

Just a hunch: does the .lib's dictionary play a role in OPTLINK's use of finding COMDAT symbol matches in a .lib file? Maybe there's some non-.obj-order-dependent behavior going on between the two.

-- 
- EricAnderton at yahoo

Frits van Bommel wrote:
> kris wrote:
> 
>> Pragma wrote:
>>
>>> Sorry if I'm stating the obvious, but it seems to me that the linker is finding this typeinfo COMDAT in Core first, rather than somewhere else, and is thereby forcing the inclusion of the rest of it's containing module.
>>>
>>> Does moving core.obj to the end of the .lib solve the problem?
>>>
>>
>> Heya Eric
>>
>> That's what it seems like and (as noted above) core.obj is already the very last one added to the lib ;)
>>
>> The only way to resolve at this point is to remove core.obj entirely.
> 
> 
> Did you try putting it at the front of the lib? You never know, maybe it picks the last one instead of the first one.
> 
> Unless it just happens to be the only module to define _D12TypeInfo_AAa6__initZ ...

No change, Frits

Pragma wrote:
> Frits van Bommel wrote:
> 
>> kris wrote:
>>
>>> Pragma wrote:
>>>
>>>> Sorry if I'm stating the obvious, but it seems to me that the linker is finding this typeinfo COMDAT in Core first, rather than somewhere else, and is thereby forcing the inclusion of the rest of it's containing module.
>>>>
>>>> Does moving core.obj to the end of the .lib solve the problem?
>>>>
>>>
>>> Heya Eric
>>>
>>> That's what it seems like and (as noted above) core.obj is already the very last one added to the lib ;)
>>>
>>> The only way to resolve at this point is to remove core.obj entirely.
>>
>>
>> Did you try putting it at the front of the lib? You never know, maybe it picks the last one instead of the first one.
>>
>> Unless it just happens to be the only module to define _D12TypeInfo_AAa6__initZ ...
> 
> 
> I thought about that too, but that doesn't seem to be the case. As Kris also stated, the lib compiles when Core is removed.  That implies that the TypeInfo mentioned lives elsewhere - and it very likely does, as it's a "char[char[]]".
> 

Perhaps a question is this: why the heck is that symbol exposed from Core, when it should instead be exposed via the TypeInfo class for char[][] instead ... linked via the TypeInfo classes in Object.d? A large number of those are present in every D executable.