kris wrote:
> Walter Bright wrote:
>> kris wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>>
>>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
> 
> After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.
> 
> For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257 
> 
> 
> The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:
> 
> 1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.
> 
> 2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.
> 
> 
>  > Then the typeinfo for char[][] is being generated by another module. I
>  > suggest it would be helpful to find that module. Grep is a handy tool
>  > for finding it.
> 
> What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.
> 
> To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.

It's a long-term proposition, but what about delaying the generation of TypeInfo until link-time?  The MS linker already optionally performs code generation to allow for more optimized executables, so I assume such a thing is definitely possible.  It would obviously require a new linker, but I don't see any other way to address these "silent dependency" issues, etc.  If I had the time I'd try it out, but as things stand there's no way that could happen before September.


Sean

Sean Kelly escribió:
> 
> It's a long-term proposition, but what about delaying the generation of TypeInfo until link-time?  The MS linker already optionally performs code generation to allow for more optimized executables, so I assume such a thing is definitely possible.  It would obviously require a new linker, but I don't see any other way to address these "silent dependency" issues, etc.  If I had the time I'd try it out, but as things stand there's no way that could happen before September.
> 
> 
> Sean

Unless I'm missing something, I don't think a new linker would be required. The compiler could just check if -c is passed. If it is, don't generate those TypeInfos, otherwise, do.

-- 
Carlos Santander Bernal

Carlos Santander wrote:
> Sean Kelly escribió:
>>
>> It's a long-term proposition, but what about delaying the generation of TypeInfo until link-time?  The MS linker already optionally performs code generation to allow for more optimized executables, so I assume such a thing is definitely possible.  It would obviously require a new linker, but I don't see any other way to address these "silent dependency" issues, etc.  If I had the time I'd try it out, but as things stand there's no way that could happen before September.
>>
>>
>> Sean
> 
> Unless I'm missing something, I don't think a new linker would be required. The compiler could just check if -c is passed. If it is, don't generate those TypeInfos, otherwise, do.

What about build utilities that compile each module separately (using -c), and then invoke the linker directly?

	-- Daniel

-- 
Unlike Knuth, I have neither proven or tried the above; it may not even make sense.

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/

Sean Kelly wrote:
> kris wrote:
>> Walter Bright wrote:
>>> kris wrote:
>>>
>>>> Walter Bright wrote:
>>>>
>>>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>>>
>>>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
>>
>> After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.
>>
>> For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257 
>>
>>
>> The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:
>>
>> 1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.
>>
>> 2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.
>>
>>
>>  > Then the typeinfo for char[][] is being generated by another module. I
>>  > suggest it would be helpful to find that module. Grep is a handy tool
>>  > for finding it.
>>
>> What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.
>>
>> To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.
> 
> It's a long-term proposition, but what about delaying the generation of TypeInfo until link-time?  The MS linker already optionally performs code generation to allow for more optimized executables, so I assume such a thing is definitely possible.  It would obviously require a new linker, but I don't see any other way to address these "silent dependency" issues, etc.  If I had the time I'd try it out, but as things stand there's no way that could happen before September.

I was going to say: new linker == our problem.  I'm in the same boat on the no time thing, but maybe we can get a group effort going later on.  If for no other reason, we could investigate what a native 64-bit toolchain for D on Windows might look like.

-- 
- EricAnderton at yahoo

kris wrote:
> kris wrote:
>> Walter Bright wrote:
>>
>>> kris wrote:
>>>
>>>> Walter Bright wrote:
>>>>
>>>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>>>
>>>>
>>>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
>>
>>
>> After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.
>>
>> For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257 
>>
>>
>> The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:
>>
>> 1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.
>>
>> 2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.
>>
>>
>>  > Then the typeinfo for char[][] is being generated by another module. I
>>  > suggest it would be helpful to find that module. Grep is a handy tool
>>  > for finding it.
>>
>> What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.
>>
>> To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.
> 
> In fact, it is so brittle and fragile that I now cannot reproduce what's noted above. Back to square one where it does not matter if Core.obj is listed first or last in the the lib "response file" -- it always gets linked, resulting in a wildly bloated binary.
> 
> With Core.obj removed from the lib, the resultant binary is still 60KB bigger than it should be, so the problem simply moves to a different bad link-chain instead.

I made a pass at trying to reproduce this the last time out, with no success - that got swept under the rug as I quickly got caught up in a handful of other things.

If only there was some way to diagnose what the linker was doing, as it happened, we could easily map out what the suspect dependencies are.

-- 
- EricAnderton at yahoo

Pragma wrote:
> Sean Kelly wrote:
>> kris wrote:
>>> Walter Bright wrote:
>>>> kris wrote:
>>>>
>>>>> Walter Bright wrote:
>>>>>
>>>>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>>>>
>>>>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
>>>
>>> After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.
>>>
>>> For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257 
>>>
>>>
>>> The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:
>>>
>>> 1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.
>>>
>>> 2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.
>>>
>>>
>>>  > Then the typeinfo for char[][] is being generated by another module. I
>>>  > suggest it would be helpful to find that module. Grep is a handy tool
>>>  > for finding it.
>>>
>>> What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.
>>>
>>> To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.
>>
>> It's a long-term proposition, but what about delaying the generation of TypeInfo until link-time?  The MS linker already optionally performs code generation to allow for more optimized executables, so I assume such a thing is definitely possible.  It would obviously require a new linker, but I don't see any other way to address these "silent dependency" issues, etc.  If I had the time I'd try it out, but as things stand there's no way that could happen before September.
> 
> I was going to say: new linker == our problem.  I'm in the same boat on the no time thing, but maybe we can get a group effort going later on.  If for no other reason, we could investigate what a native 64-bit toolchain for D on Windows might look like.

I've been wondering how far your work with DDL goes towards writing a linker? Certainly the work you've done with making sense of the OMF and ELF specs, and parsing the obj files, seems to be a huge chunk of the task. A lot of code would be common to both tasks, surely?

Don Clugston wrote:
> Pragma wrote:
>> Sean Kelly wrote:
>>> kris wrote:
>>>> Walter Bright wrote:
>>>>> kris wrote:
>>>>>
>>>>>> Walter Bright wrote:
>>>>>>
>>>>>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>>>>>
>>>>>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
>>>>
>>>> After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.
>>>>
>>>> For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257 
>>>>
>>>>
>>>> The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:
>>>>
>>>> 1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.
>>>>
>>>> 2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.
>>>>
>>>>
>>>>  > Then the typeinfo for char[][] is being generated by another module. I
>>>>  > suggest it would be helpful to find that module. Grep is a handy tool
>>>>  > for finding it.
>>>>
>>>> What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.
>>>>
>>>> To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.
>>>
>>> It's a long-term proposition, but what about delaying the generation of TypeInfo until link-time?  The MS linker already optionally performs code generation to allow for more optimized executables, so I assume such a thing is definitely possible.  It would obviously require a new linker, but I don't see any other way to address these "silent dependency" issues, etc.  If I had the time I'd try it out, but as things stand there's no way that could happen before September.
>>
>> I was going to say: new linker == our problem.  I'm in the same boat on the no time thing, but maybe we can get a group effort going later on.  If for no other reason, we could investigate what a native 64-bit toolchain for D on Windows might look like.
> 
> I've been wondering how far your work with DDL goes towards writing a linker? Certainly the work you've done with making sense of the OMF and ELF specs, and parsing the obj files, seems to be a huge chunk of the task. A lot of code would be common to both tasks, surely?

Well, the OMF loader needs some polish and some subtle refactoring (read:Tango) but I have thought the same myself.  The big stumbling block (for me at least) is understanding the "E" in "PE/COFF" for emitting .exe and .dll files.  The intermediate part, matching dependencies, is really kind of simple only up until you get into various forms of optimization: culling unused segments & symbols, deep dependency analysis, *fast* linking, minimize size, optimize for speed, etc.  Typing "link /?" in your console will quickly cast a very large shadow over this area.

ELF is another issue.  Flectioned is light-years ahead of DDL on that front.  But combined with a (future) upgraded DDL, we'll pretty much have "libtools for D".

Now if we had a library that allowed for reading *and* writing of COFF files per 100% of the specification, then I'd imagine that this wouldn't be too far out of reach.

-- 
- EricAnderton at yahoo