< SNIP good post from Kris >

After reading this, it began to dawn on me what has become one of the huge obstacles for D.

Tango represents one the largest contributions to the D world.  It
may be considered the first large commercial grade library to make it's
presence here.  It's an opportunity to prove "that D has the right stuff."

But there's a traitor in our midst, and that traitor resides in the d
tool chain, no less: the withered grip of optlink -- decades old,
astonishly speedy, tried and true, vain and boastful -- stays the progress
of D in an absurd way. We're led to believe that we need nothing better for
a modern language that steadily pushes into new territory and tests new
ideas. And yet we are ultimately held hostage by this very tool.

Does no one see the imbalance, the inconsistancy of all this? A powerful language like D has worked so hard to ease the pains of C++ users, to separate us from the stench of C++ clutter, by implementing a more palatable grammar. And yet despite all this, we are forsaken. Has the imagination and concern for progress vanished? Isn't the toolchain also one of the most critical aspects of acceptance of a language?  Why should anyone adopt a powerful and clean language when the tools just drag it deeper into the sewer.  What benefit is a new language with old tools? Cutting corners for the sake of efficiency does D no good.  Make it all good, language and tools, and the users will clamour for it.

optlink may just be the bane for D acceptance. And Tango gets the pitiful opportunity of demonstrating why D is NOT ready for prime-time in the commercial realm: the DM support tools it relies on are bogged down in the past, reflecting D's lopsided existance on yet another level: a strong language relying on a fragile, outdated, and poorly fit tool set.

-JJR

John Reimer wrote:

> That's not a good argument. ld is pig slow?  I'm sorry but I don't get
> that.  It works; it works as intended; and, strangely, I don't hear people
> complain about its apparent lack of speed. 
> 
> So what if a linker is blitzingly fast. If it's outdated and broken,
> there's not much to get excited about.  I'll choose the slow working one
> any day.

I've find OPTLINK to hang and crash a lot when linking the wxD programs
on Windows XP. But every time I try to reproduce it, it goes away... :-(

So now I just run the "make" like three times in a row, and it usually
succeeds in building everything. And yeah, it's rather fast in doing so.

But I prefer the MinGW gdc/ld, since it works the first time but slower?
(well that and that I have problems getting DMC to work with SDL / GL)

--anders

kris wrote:
> 9) The Fake dependencies cause the linker to pick up and bind whatever module happens to satisfy it's need for the typeinfo resolution. In this case, the linker sees Core.obj with a char[][] decl exposed, so it say "hey, this is the place!" and binds it. Along with everything else that Core.obj actually requires.

just a thought:
assuming the linker is working at obj file level, isn't that a
set-cover-problem and therefore NP-complete then?
given several obj files (non-disjoint sets), find the minimum number of
obj files that cover all symbols needed by the program.

why i think that it's a set-cover:
i understand that the dependency for some typeinfo can only arise if one
of the modules that are imported needs it. hence, there is always a
module with that TI that provides at least one other needed symbol.
therefore it cannot happen, that the set-cover includes a module of
which only TIs are needed.

if that is correct, one actually wouldn't want the linker to solve this correctly on object file level, rather than use a faster heuristic to approach a solution (which is what it seems to be doing now).

maybe it helps the discussion a bit if one knows that solving the problem in the toolchain necessarily involves making the linker work at segment level.

On Thu, 22 Feb 2007 09:49:23 +0200, kris <foo@bar.com> wrote:
[snip]
> 9) The Fake dependencies cause the linker to pick up and bind whatever module happens to satisfy it's need for the typeinfo resolution. In this case, the linker sees Core.obj with a char[][] decl exposed, so it say "hey, this is the place!" and binds it. Along with everything else that Core.obj actually requires.
>
> 10) The linker is entirely wrong, but you can't really blame it since the char[][] decl is scattered throughout the library modules. It thinks it get's the /right/ one, but in fact it could have chosen *any* of them. This is now getting to the heart of the problem.
>
> 11) If there's was only one exposed decl for char[][], e.g. like int[], there would be no problem. In fact you can see all the prepackaged typeinfo bound to any D executable. There's lots of them. However, because the compiler injects this typeinfo into a variety of objects (apparently wherever char[][] is used), then the linker is boondoggled.
>
> 12) If the linker were smart, and could link segments instead of entire object modules, this would still be OK (a segment is an isolated part of the object module). But the linker is not smart. It was written to be fast, in pure assembler, decades ago.
[snip]

As long as the linker will operate at the .obj file level, the linker will pull in some bloat to the executable, in practice. The question is how much the executable will be bloated.

And if the compiler generates false dependencies, the size of the bloat will enlarge.

So, a solution would be a new linker operating at the section level. (Not necessary *the* solution, but *a* solution.)

Oh, the linker was written in assembly, how hardcore. :) I don't think there's much of a point in writing a new linker (that is, if someone will do that) in assembly... not that anyone was considering using assembly... <g> If the linking times were a bit (or two) slower (because of more complex algorithm), I think it would be okay for a lot of people (all of them?). (If linking times will be a issue, for example when building debug executables, the old linker could be used for that.) Hmm, I'm wondering how much slower the current linker would be if it had been written in C/C++/D instead of assembly. I mean, today processors are so much faster than a decade ago, and hard disks had not got any faster (well, not significally). Usually the hard disk is the bottleneck, not the processor.

== Quote from Frits van Bommel (fvbommel@REMwOVExCAPSs.nl)'s article
> kris wrote:
> > Isn't there some way to isolate the typeinfo such that only a segment is linked, rather than the entire "hosting" module (the one that just happened to be found first in the lib) ?
> The obvious solution would be to always generate typeinfo even if it can be determined imported modules will already supply it. The current approach seems to confuse the linker, causing it to link in unrelated objects that happen to supply the symbol even though the compiler "meant" for another object file to supply it.
>
> Yes, that will "bloat" object files, but the current approach apparently bloats applications. Care to guess which are distributed most often? ;)

I think your idea could work. It makes sense to me, but I'd like to go one better: Let's have DMD postpone creating TypeInfo until an .exe or .dll is being created and only include them with the .obj for the "main" module (i.e. the module with the main or DllMain function).

Surely, the compiler can figure out which TypeInfo's it needs at the point of compiling an .exe or .dll. If not, even if we have to wait for linker to spit out a list of missing TypeInfo's and then generate the TypeInfo (trial-and-error), I think that would be a small price to pay for eliminating all of this bloat of unneeded module that Kris has discovered.

There seems to be a lot more concern around here about .exe-size than there is about the speed of compiling and linking. Let's fix what's broken -- even if we have to give up a little compile/link speed

My idea seems too obvious to be the solution. Does anyone else think this would work?

By the way, Kris has a thorough description of the problem here: http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=49257

jcc7

jcc7 wrote:
> == Quote from Frits van Bommel (fvbommel@REMwOVExCAPSs.nl)'s article
>> kris wrote:
>>> Isn't there some way to isolate the typeinfo such that only a
>>> segment is linked, rather than the entire "hosting" module (the
>>> one that just happened to be found first in the lib) ?
>> The obvious solution would be to always generate typeinfo even if it
>> can be determined imported modules will already supply it. The
>> current approach seems to confuse the linker, causing it to link in
>> unrelated objects that happen to supply the symbol even though the
>> compiler "meant" for another object file to supply it.
>>
>> Yes, that will "bloat" object files, but the current approach
>> apparently bloats applications. Care to guess which are distributed
>> most often? ;)
> 
> I think your idea could work. It makes sense to me, but I'd like to go one better:
> Let's have DMD postpone creating TypeInfo until an .exe or .dll is being created
> and only include them with the .obj for the "main" module (i.e. the module with
> the main or DllMain function).

Not all libraries may have a DllMain, IIRC it's completely optional.
On Windows it's required for D DLLs if you want to use the GC from within the DLL, or have static constructors/destructors in the DLL -- but otherwise you may get by without. I think if you write C-style D you may well get away without it.

> Surely, the compiler can figure out which TypeInfo's it needs at the point of
> compiling an .exe or .dll.

Not necessarily. Any modules that are linked in but not called by other modules (e.g. code only reachable from static constructors and/or static destructors) may not be seen when main/DllMain is compiled, if there even is one of these (see above point about DllMain being optional).

> If not, even if we have to wait for linker to spit out
> a list of missing TypeInfo's and then generate the TypeInfo (trial-and-error), I
> think that would be a small price to pay for eliminating all of this bloat of
> unneeded module that Kris has discovered.

This would mean you can't "manually" link stuff together, using optlink/ld/whatever directly. I don't know how many people want to do this, but Walter has made it pretty clear he wants to be able to use a generic linker[1] (i.e. one that doesn't require specialized knowledge of D) and I agree with that.
Consider this: if every (or even more than one) language required a special way of linking, that would mean you couldn't link together code written in those languages without writing a linker (or perhaps wrapper) that supports both...
Though arguably the situation with DMD/Windows is already worse when it comes to that, since almost nobody else uses OMF anymore...

> There seems to be a lot more concern around here about .exe-size than there is
> about the speed of compiling and linking. Let's fix what's broken -- even if we
> have to give up a little compile/link speed

I agree.

> My idea seems too obvious to be the solution. Does anyone else think this would work?

For above-mentioned reasons, I don't think it will work for all (corner)cases.

> By the way, Kris has a thorough description of the problem here:
> http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=49257

I've seen it.

John Reimer wrote:
> On Wed, 21 Feb 2007 16:12:10 -0800, Walter Bright wrote:
> 
>> Derek Parnell wrote:
>>>> Most of the complexity in a linker stems from:
>>>> 1) trying to make it fast
>>> How fast is fast enough?
>> It's never fast enough. I know a fellow who made his fortune just writing a faster linker than MS-LINK. (You can guess the name of that linker!) Borland based their whole company's existence on fast compile-link times. Currently, ld is pig slow, it's a big bottleneck on the edit-compile-link-debug cycle on Linux.
> 
> That's not a good argument. ld is pig slow?  I'm sorry but I don't get
> that.  It works; it works as intended; and, strangely, I don't hear people
> complain about its apparent lack of speed. 
> 
> So what if a linker is blitzingly fast. If it's outdated and broken,
> there's not much to get excited about.  I'll choose the slow working one
> any day.

Ideally, perhaps a linker could provide both options: link fast and potentially bloat the exe or link carefully (and slowly) for a lean exe.  I'd use the fast link for debugging and the slow link for releases. Assuming, of course, that the linker were reliable enough that there was no risk of changing app behavior between the two.


Sean

Sean Kelly wrote:
> Ideally, perhaps a linker could provide both options: link fast and potentially bloat the exe or link carefully (and slowly) for a lean exe.  I'd use the fast link for debugging and the slow link for releases. Assuming, of course, that the linker were reliable enough that there was no risk of changing app behavior between the two.

That might not be the case here: if a module's object file is pulled in, that module's static constructors and destructors are called at runtime, right? So if different modules are pulled in with those options, different static constructors/destructors get called.
(Same goes for unit tests, if enabled, by the way)

kris wrote:
[snip]
> 7) These fake dependencies are responsible for, in this case, the entire "locale" package to be bound to the example app, resulting in a 350% increase in size.
[snip]

Is it not possible to have a tool which strips the executable of dead code after its finished, using an external tool?

-Joel

janderson wrote:
> kris wrote:
> [snip]
>> 7) These fake dependencies are responsible for, in this case, the entire "locale" package to be bound to the example app, resulting in a 350% increase in size.
> [snip]
> 
> Is it not possible to have a tool which strips the executable of dead code after its finished, using an external tool?

Presumably this would leave static constructors/destructors intact? If not, you'll have some problems with code that depends on them.

Now, assuming you do: tango.text.locale.Core contains 3 static constructors and a static destructor. The static destructor sets some variables to null, but the static constructors do a bit more. One of them directly allocates some class instances. That pulls in the vtable for that class, which in turn pulls in all virtual member functions. Those instantiate at least two other classes (not counting Exception)...

This kind of stuff can easily cascade into a lot of code, all because a single module got pulled in and its static constructors get called.

So all is not as simple as it seems, unfortunately. To get this right[1], you'd basically have to do the dependency analysis all over again. You might as well have the linker get it right the first time...
(IMHO, to get it to do that without requiring a special linker and/or object format, the compiler needs to somehow make sure the linker won't pick the wrong object file to link to)


[1]: i.e. don't include unnecessary code, but also include all code that _is_ necessary.