Lionello Lunesu wrote:
> "Walter Bright" <newshound@digitalmars.com> wrote in message news:erie2v$kad$2@digitalmars.com...
>> Frits van Bommel wrote:
>>> GNU ld seems to be perfectly happy working at the section level (with --gc-sections).
>> Yeah, well, try linking D programs with --gc-sections, and you'll get a crashing executable.
> 
> Thomas has suggested some fixes for that in bugzilla #879.

Yes, I know, and I'll probably implement them. But they are a hack. A much better solution would be a change to the ELF format to allow sections to be marked as "don't gc this section", but I doubt that'll happen.

On Wed, 21 Feb 2007 16:12:10 -0800, Walter Bright wrote:

> It's important to work with existing tools (like linkers and librarians), which (among other things) helps ensure that D programs can link with the output of other compilers (like gcc).

Has anyone tried successfully to use a Windows linker, other than OptLink, to handle D .obj files?

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Justice for David Hicks!"
22/02/2007 12:02:31 PM

Walter Bright wrote:
> kris wrote:
> 
>> Walter Bright wrote:
>>
>>> Some strategies:
>>>
>>> 1) minimize importing of modules that are never used
>>>
>>> 2) for modules with a lot of code in them, import them as a .di file rather than a .d
>>>
>>> 3) create a separate module that defines the relevant typeinfo's, and put that first in the library
>>
>>
>>
>> 1) Tango takes this very seriously ... more so than Phobos, for example.
> 
> 
> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.

This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.

> 
>> 2) That is something that could be used in certain scenario's, but is not a general or practical solution for widespread use of D.
> 
> 
> The compiler can automatically generate .di files. You're probably going to want to do that anyway as part of polishing the library - it speeds compilation times, aids proper encapsulation, etc. That's why the gc does it, and I've been meaning to do it for other bulky libraries like std.regexp.

You may remember that many of us find .di files to be something "less" than an effective approach to library interfacing? As to it making smaller, faster compiliations -- try it on the Win32 header files ... it makes them bigger and noticably slower to parse.

This is neither a valid or practical solution.


> 
> I wish to point out that the current scheme does *work*, it generates working executables. In the age of demand paged executable loading (which both Linux and Windows do), unused code in the executable never even gets loaded into memory. The downside to size is really in shipping code over a network (also in embedded systems).
> 
> So I disagree with your characterization of it as impractical.

Oh, ok. It all depends on what one expects from a toolset. Good point

> 
> For professional libraries, it is not unreasonable to expect some extra effort in tuning the libraries to minimize dependency. This is a normal process, it's been going on at least for the 25 years I've been doing it. Standard C runtime libraries, for example, have been *extensively* tweaked and tuned in this manner, and that's just boring old C. They are not just big lumps of code.
> 
>> 3) Hack around an undocumented and poorly understood problem in developer-land. Great.
> 
> 
> I think you understand the problem now, and the solution. Every developer of professional libraries should understand this problem, it crops up with most every language. If a developer doesn't understand it, one winds up with something like Java where even the simplest hello world winds up pulling in the entire Java runtime library, because dependencies were not engineered properly.

This is a problem with the toolchain, Walter. Plain and simple. The linker picks up an arbitrary, yes arbitrary, module from the library because the D language-design is such that it highlights a deficiency in the existing toolchain. See below:

You can claim all you like that devs should learn to deal with it, but the fact remains that it took us more than a day to track down this obscure problem to being a char[][] decl. It will take just as long for the next one, and perhaps longer. Where does the cycle end?

The toolchain currently operates in a haphazard fashion, linking in /whatever/ module-chain happens to declare a typeinfo for char[][]. And it does this because of the way D generates the typeinfo. The process is broken, pure and simple. We should accept this and try to figure out how to resolve it instead.


> 
>> you might as well add:
>>
>> 4) have the user instantiate a pointless and magic char[][] in their own program, so that they can link with the Tango library?
> 
> 
> I wouldn't add it, as I would expect the library developer to take care of such things by adding them to the Tango library as part of the routine process of optimizing executable size by minimizing dependencies.
> 


Minimizing dependencies? What are you talking about? Those deps are produces purely by the D compiler, and not the code design.



>> None of this is not gonna fly in practice, and you surely know that?
> 
> 
> For features like runtime time identification, etc., that are generated by the compiler (instead of explicitly by the programmer), then the dependencies they generate are a fact of life.
> 
> Optimizing the size of a generated program is a routine programming task. It isn't something new with D. I've been doing this for 25 years.

Entirely disingenuous. This is not about "optimization" at all ... it about a broken toolchain. Nothing more.

I hope you'll find a way to progress this forward toward a resolution instead of labeling it something else.

Derek Parnell wrote:
> On Wed, 21 Feb 2007 16:12:10 -0800, Walter Bright wrote:
> 
> 
>>It's important to work with existing tools (like linkers and librarians), which (among other things) helps ensure that D programs can link with the output of other compilers (like gcc).
> 
> 
> Has anyone tried successfully to use a Windows linker, other than OptLink,
> to handle D .obj files? 
> 


I tried the 'modified' Watcom linker the other night, but it found some problems with snn.lib. That was a 2003 release, and the tools have been updated significantly since then. The recent release may well have better luck (the trunk supports OMF). Would be very interested if someone were to have a go at it.

kris wrote:
> Walter Bright wrote:
>> kris wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> Some strategies:
>>>>
>>>> 1) minimize importing of modules that are never used
>>>>
>>>> 2) for modules with a lot of code in them, import them as a .di file rather than a .d
>>>>
>>>> 3) create a separate module that defines the relevant typeinfo's, and put that first in the library
>>>
>>>
>>>
>>> 1) Tango takes this very seriously ... more so than Phobos, for example.
>>
>>
>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
> 
> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
> 
>>
>>> 2) That is something that could be used in certain scenario's, but is not a general or practical solution for widespread use of D.
>>
>>
>> The compiler can automatically generate .di files. You're probably going to want to do that anyway as part of polishing the library - it speeds compilation times, aids proper encapsulation, etc. That's why the gc does it, and I've been meaning to do it for other bulky libraries like std.regexp.
> 
> You may remember that many of us find .di files to be something "less" than an effective approach to library interfacing? As to it making smaller, faster compiliations -- try it on the Win32 header files ... it makes them bigger and noticably slower to parse.
> 
> This is neither a valid or practical solution.
> 
> 
>>
>> I wish to point out that the current scheme does *work*, it generates working executables. In the age of demand paged executable loading (which both Linux and Windows do), unused code in the executable never even gets loaded into memory. The downside to size is really in shipping code over a network (also in embedded systems).
>>
>> So I disagree with your characterization of it as impractical.
> 
> Oh, ok. It all depends on what one expects from a toolset. Good point
> 
>>
>> For professional libraries, it is not unreasonable to expect some extra effort in tuning the libraries to minimize dependency. This is a normal process, it's been going on at least for the 25 years I've been doing it. Standard C runtime libraries, for example, have been *extensively* tweaked and tuned in this manner, and that's just boring old C. They are not just big lumps of code.
>>
>>> 3) Hack around an undocumented and poorly understood problem in developer-land. Great.
>>
>>
>> I think you understand the problem now, and the solution. Every developer of professional libraries should understand this problem, it crops up with most every language. If a developer doesn't understand it, one winds up with something like Java where even the simplest hello world winds up pulling in the entire Java runtime library, because dependencies were not engineered properly.
> 
> This is a problem with the toolchain, Walter. Plain and simple. The linker picks up an arbitrary, yes arbitrary, module from the library because the D language-design is such that it highlights a deficiency in the existing toolchain. See below:
> 
> You can claim all you like that devs should learn to deal with it, but the fact remains that it took us more than a day to track down this obscure problem to being a char[][] decl. It will take just as long for the next one, and perhaps longer. Where does the cycle end?
> 
> The toolchain currently operates in a haphazard fashion, linking in /whatever/ module-chain happens to declare a typeinfo for char[][]. And it does this because of the way D generates the typeinfo. The process is broken, pure and simple. We should accept this and try to figure out how to resolve it instead.
> 
> 
>>
>>> you might as well add:
>>>
>>> 4) have the user instantiate a pointless and magic char[][] in their own program, so that they can link with the Tango library?
>>
>>
>> I wouldn't add it, as I would expect the library developer to take care of such things by adding them to the Tango library as part of the routine process of optimizing executable size by minimizing dependencies.
>>
> 
> 
> Minimizing dependencies? What are you talking about? Those deps are produces purely by the D compiler, and not the code design.
> 
> 
> 
>>> None of this is not gonna fly in practice, and you surely know that?
>>
>>
>> For features like runtime time identification, etc., that are generated by the compiler (instead of explicitly by the programmer), then the dependencies they generate are a fact of life.
>>
>> Optimizing the size of a generated program is a routine programming task. It isn't something new with D. I've been doing this for 25 years.
> 
> Entirely disingenuous. This is not about "optimization" at all ... it about a broken toolchain. Nothing more.
> 
> I hope you'll find a way to progress this forward toward a resolution instead of labeling it something else.

I'm not trying to pick a fight with any of the people who have been discussing this serious issue, but I have some thoughts I'd like to add. Feel free to take my words as the ramblings of an idiot...

My theory is that Walter and you (Kris and everyone else who is trying to talk some sense into Walter) are operating on different wavelengths. (I may be on yet another wavelength.) When I read Kris's complaint, I think "Wow, that sounds like a problem that needs fixing". When I read Walter's response, I think "Hmmm, that makes sense, too. What was Kris's problem again?". And that cycle repeats for me. Walter seems to still think he understands the problem, but perhaps we could benefit from a simple illustration of the problem. Or just a restatement of the problem situation. I'm sure that I don't understand what's going on.

It's something about the compiler is generating the TypeInfo for char[][], and it's bringing in all of "Core" (but we don't need all of "Core"). And we especially don't need the "locale" package since it's bloated (and unneeded), but the whole package (including all of "Core" and "locale") is brought in because the compiler is generating TypeInfo for the char[][]. (But if the "locale" package is so bloated and unneeded, then why is it being compiled at all? Is "locale" part of "Core"?) Is any of that right? I'm so confused.

(Perhaps part of the problem is that Walter isn't that familiar with the Tango library and what it's all about. I suspect that I know more about Tango than Walter does -- and I'm afraid that I know barely anything about it -- so that could be part of the problem, too.)

-- 
jcc7

On Thu, 22 Feb 2007 12:05:22 +1100, Derek Parnell wrote:

> On Wed, 21 Feb 2007 16:12:10 -0800, Walter Bright wrote:
> 
>> It's important to work with existing tools (like linkers and librarians), which (among other things) helps ensure that D programs can link with the output of other compilers (like gcc).
> 
> Has anyone tried successfully to use a Windows linker, other than OptLink, to handle D .obj files?
>


What does gcc have to do with windows dmd tools?  And what good is it to use these tools if they aren't working properly?  That's why this whole discussion started, right?

I don't get what Walter is saying.

-JJR

"Walter Bright" <newshound@digitalmars.com> wrote in message

: It's important to work with existing tools (like linkers
and
: librarians), which (among other things) helps ensure that
D programs can
: link with the output of other compilers (like gcc).
:

What does that mean ?

D produce omf obj files, right ?
and omf is not supported by any of the gnu tools working
on the win32 platform.

gcc is not the right example.

Is it a good idea to suggest that D produce coff obj ?
This will probably give you some (or a lot of) work
to add the necessary support to your tool set,

but, D is a modern compiler, using an obj format that
is not a standard anymore.

Justin C Calvarese wrote:
> I'm not trying to pick a fight with any of the people who have been discussing this serious issue, but I have some thoughts I'd like to add. Feel free to take my words as the ramblings of an idiot...
> 
> My theory is that Walter and you (Kris and everyone else who is trying to talk some sense into Walter) are operating on different wavelengths. (I may be on yet another wavelength.) When I read Kris's complaint, I think "Wow, that sounds like a problem that needs fixing". When I read Walter's response, I think "Hmmm, that makes sense, too. What was Kris's problem again?". And that cycle repeats for me. Walter seems to still think he understands the problem, but perhaps we could benefit from a simple illustration of the problem. Or just a restatement of the problem situation. I'm sure that I don't understand what's going on.
> 
> It's something about the compiler is generating the TypeInfo for char[][], and it's bringing in all of "Core" (but we don't need all of "Core"). And we especially don't need the "locale" package since it's bloated (and unneeded), but the whole package (including all of "Core" and "locale") is brought in because the compiler is generating TypeInfo for the char[][]. (But if the "locale" package is so bloated and unneeded, then why is it being compiled at all? Is "locale" part of "Core"?) Is any of that right? I'm so confused.
> 
> (Perhaps part of the problem is that Walter isn't that familiar with the Tango library and what it's all about. I suspect that I know more about Tango than Walter does -- and I'm afraid that I know barely anything about it -- so that could be part of the problem, too.)
> 

Well said, Justin. I'm personally feeling like there's either some vast misunderstanding or there's a lot of smoke billowing about. I'll try to recapture the issue and see where it goes. Let me know if I fail to explain something?

The problem space
-----------------

1) This is not about templates anymore. We're currently past that bridge and into different territory. A common territory that every developer using D will have to face in one way or another.

2) This is not specific to Tango at all. It is a generic problem and Tango just happens to trip it in an obvious manner.

3) In a nutshell, the linker is binding code from the library that has no business being attached to the executable. Let's call this the "redundant code"?

4) Given the last set of comments from Walter, he appears to think the the redundant code is somehow imported; either by the example program or indirectly via some chain of imports within the library itself. This is where the disconnect lies, I suspect.

5) There is /no/ import chain explicitly manifested anywhere in the code in question. This should be obvious form the fact that the example links perfectly cleanly when said redundant code is deliberately removed from the library.

6) The dependency that /does/ cause the problem is one generated by the D compiler itself. It generates and injects false 'dependencies' across object modules. Specifically, that's effectively how the linker treats them.

7) These fake dependencies are responsible for, in this case, the entire "locale" package to be bound to the example app, resulting in a 350% increase in size.

8) Fake dependencies are injected in the form of typeinfo. In this case, the typeinfo is for a char[][]. This is not part of the "prepackaged" set of typeinfo, so the compiler makes it up on the fly. Trouble is, this is "global" information -- it should be in one location only.

9) The Fake dependencies cause the linker to pick up and bind whatever module happens to satisfy it's need for the typeinfo resolution. In this case, the linker sees Core.obj with a char[][] decl exposed, so it say "hey, this is the place!" and binds it. Along with everything else that Core.obj actually requires.

10) The linker is entirely wrong, but you can't really blame it since the char[][] decl is scattered throughout the library modules. It thinks it get's the /right/ one, but in fact it could have chosen *any* of them. This is now getting to the heart of the problem.

11) If there's was only one exposed decl for char[][], e.g. like int[], there would be no problem. In fact you can see all the prepackaged typeinfo bound to any D executable. There's lots of them. However, because the compiler injects this typeinfo into a variety of objects (apparently wherever char[][] is used), then the linker is boondoggled.

12) If the linker were smart, and could link segments instead of entire object modules, this would still be OK (a segment is an isolated part of the object module). But the linker is not smart. It was written to be fast, in pure assembler, decades ago.


Why is this a problem now?
-------------------------

Well, it's always been a problem to an extent, over the years. The key here is that in the past, the problem was generated principally by the developers/coder by introducing duplicate symbols and so on. Because it was in the hand of the developer, it could be resolved reasonably well.

With D, that is still potentially the case. However, the /real/ problem is this: the compiler generates the duplicate symbols all by itself. So, the developer has no means to rectify the situation ... it is entirely out of their hands. What's worse is this: there are no useful messages involved ... all you get is some bizzare and arcane message from the linker that generally misguides you instead.

Case in point: you have to strip the library down by hand, and very very carefully sift through the symbols and literally hundreds of library builds until you finally get lucky enough to stumble over the problem.

Walter asserts that the linker can be tricked into doing the right thing. This seems to show a lack of understanding on his part about the problem and the manner in which the lib and linker operate.

The linker cannot be fooled or tricked in a dependendable manner, since the combinations of redundant symbols for it to choose from are nigh impossible for a human to track on a regular basis, and the particular module resolved depends very much on where the linker currently is in it's process. As you can imagine, in a large library with a large number of compiler-generated duplicates, that's potentially a very large explosion of combinations? The notion that a developer be responsible for tricking the linker, to cover up for these injected duplicates is simply absurd :)

As was pointed out to me, OMF librarians actually uses a two-level hashmap to index the library entries. This is used by the linker to locate missing symbols. I think it's clear that this is not a linear lookup mechanism as had been claimed, and is borne out by experiments that show the process cannot be controlled, and the linker cannot be faked in usable or dependable manner.

To hammer the overall issue home, consider this: in the example application I added a dummy /magical/ declaration of

char[][] huh = [];

When linked against the lib, my executable shrank from ~620kb to ~180kb.  Where did I get this magic from? Well, it took an exceddingly long and tedious process to discover it. Rinse and repeat for the next related error.



A word about Tango
------------------

Contrary to various implications made recently, Tango is rather well organized with very limited inter-module dependencies. For example, interfaces and pure-abstract classes are deliberately used to decouple the implementation of one module from those of others. You won't see that kind of thing in many libs, and certainly not in Phobos. Tango is designed and build by people who actually care about such things, crazy as that may sound :)

The "bloat" injected into the example executable comes entirely from an isolated package. It is the "locale" package, which supports a truly extensive array of I18N tools and calanders. I think it has 7 or 8 different calander systems alone? It captures all the monetary, time and date preferences and idioms for every recongnized locale in the world. In short, it is an exceptional piece of work (from David Chapman). The equivalent out there is perhaps a good chunk of the IBM ICU project. The minimum size for that is a 7MB DLL. Typically 10MB instead.

I feel it important to point out that this powerful I18N package is, in no way, at fault here. The D compiler simply injected the wrong symbol into the wrong module at the wrong time, in the wrong order, and the result is that this package gets linked when it's not used or imported in any fashion by the code design. Instead, the dependency is created entirely by the compiler. That's a problem. It is a big problem. And it is a problem every D developer will face, at some point, when using DM tools. But it is not a problem rooted in the Tango code.

On Wed, 21 Feb 2007 16:12:10 -0800, Walter Bright wrote:

> Derek Parnell wrote:
>>> Most of the complexity in a linker stems from:
>>> 1) trying to make it fast
>> How fast is fast enough?
> 
> It's never fast enough. I know a fellow who made his fortune just writing a faster linker than MS-LINK. (You can guess the name of that linker!) Borland based their whole company's existence on fast compile-link times. Currently, ld is pig slow, it's a big bottleneck on the edit-compile-link-debug cycle on Linux.
> 
> 


That's not a good argument. ld is pig slow?  I'm sorry but I don't get that.  It works; it works as intended; and, strangely, I don't hear people complain about its apparent lack of speed.

So what if a linker is blitzingly fast. If it's outdated and broken, there's not much to get excited about.  I'll choose the slow working one any day.

-JJR

On Wed, 21 Feb 2007 15:56:31 +0000, John Reimer wrote:

> On Wed, 21 Feb 2007 01:47:53 -0800, kris wrote:
> 
>> kris wrote:
>>> Walter Bright wrote:
>>> 
>>>> kris wrote:
>>>>
>>>>> Walter Bright wrote:
>>>>>
>>>>>> For a quick & dirty test, try reversing the order the object files are presented to lib.
>>>>>
>>>>>
>>>>>
>>>>> there's a couple of hundred :-D
>>>>
>>>>
>>>>
>>>> Do the ones that were giving the undefined reference before the changes to lib.
>>> 
>>> 
>>> The lib itself is actually built via a single D module, which imports all others. This is then given to Build to construct the library. Thus I don't have direct control over the ordering. Having said that, it appears Build does something different when the modules are reordered; Likely changing the order in which modules are presented to the lib.
>>> 
>>> By moving things around, I see a change in size on the target executable between -4kb to +5kb
>>> 
>> 
>> I've been messing with the response file handed to the librarian (via lib @foo); moving modules around here and there, reordering big chunks etc. Have yet to see a notable change in the resulting exe after relinking against each lib version.
> 
> 
> Is build really a reliable means of testing this?  I mean, it's produced unusual differences in binary size in the past (granted not of that magnitude).  Of course, this is a different case too, in which a library is being created.
> 
> Just out of curiousity, does rebuild do the same thing?
> 
> -JJR


I obviously misunderstood the whole issue here.  After reading the responses, I begin to see the magnitude of the problem described.

-JJR