Thread overview
Re: Possible solution to template bloat problem?
Aug 19, 2013
H. S. Teoh
Aug 19, 2013
Dicebot
Aug 19, 2013
Dicebot
Aug 19, 2013
Andrej Mitrovic
August 19, 2013
On Mon, Aug 19, 2013 at 10:37:35PM +0200, Dicebot wrote:
> Just two words: "separate compilation".
> 
> Any solution that is going to address template problem needs to improve current state with such compilation model, not make it even worse.

I thought about that, actually. (I'm a fan of separate compilation, though the current state of D makes it more advantageous to compile everything in one go where possible.)

There should be *one* static library where template instantiations are dumped. Let's call it templates.a for lack of a better name. Inside this archive, template instantiations are organized according to the module hierarchy -- i.e., if you import std.stdio and use writeln, the instantiation of writeln will be put into std/stdio/writeln_*.o in templates.a, NOT the root module.

Furthermore, if a particular instantiation is already present in templates.a, then it is not duplicated, but simply replaced (I was going to say ignored, but replace is better in case templates.a contains leftovers from a previous compilation run). This eliminates duplicate instantiations of the same templates in multiple modules.


> As an alternative, I have proposed one of two approaches (or both):
> 
> 1) Stop dumping all symbols into root module supplied from the command line. Emit symbols to object files that match modules they were instantiated from. If symbol has no valid source point (== constraint or CTFE) than don't emit it at all.

+1.

Plus, DMD really needs to stop stripping paths from object files *by default* -- now that we have package.d, this will blow up in ugly ways once you have templates in package.d that get instantiated and there are multiple packages with package.d.


> 2) Create object files in format that allows usage of `ld --gc-sections` (garbage collection of unused symbols upon linking). Don't know if similar thing exists for Windows.

Well, the reason I went for bare linker essentials was precisely to avoid platform-specific issues like this. But OTOH, on platforms that support the equivalent of --gc-sections, we could just emit template instantiations into separate sections instead of separate modules in a static library. The static library approach can serve as a fallback on platforms that don't support this.


> Latter should be relatively easy to do but it is not cross-platform and it does not help build systems with tracking rebuild conditions.
> 
> Former feels like a proper approach and I have been working on it (also @eskimor) for some time. But it is rather hard as relevant code does not seem to track required information at all and probably no one but Walter knows all minor details about its design. To sum it up - I have not been able to produce a pull request that passes the test suite so far (though I have put it on hold for some time, going to return to this).

Hmm. It'd be nice if this could be made to work. One thing I'm not very happy with in D is the sheer size of the executables even for relatively simple programs. If unused symbols could be stripped during linking, this would help a lot.


T

-- 
Political correctness: socially-sanctioned hypocrisy.
August 19, 2013
On Monday, 19 August 2013 at 21:33:08 UTC, H. S. Teoh wrote:
> Inside this
> archive, template instantiations are organized according to the module
> hierarchy -- i.e., if you import std.stdio and use writeln, the
> instantiation of writeln will be put into std/stdio/writeln_*.o in
> templates.a, NOT the root module.

Ok. Now you change the signature of writeln. Incremental rebuild. Will you do reflection upon the object file in templates.a to get the list of required instances? Or rebuild all modules that import std.stdio even indirectly? Most likely latter, because it also could have been inlined in several places and those need to be rebuilt too.

To track modules that actually use that template instance one needs implementation from my proposal #1 anyway :(
August 19, 2013
On Monday, 19 August 2013 at 21:33:08 UTC, H. S. Teoh wrote:
> Hmm. It'd be nice if this could be made to work. One thing I'm not very
> happy with in D is the sheer size of the executables even for
> relatively simple programs. If unused symbols could be stripped during
> linking, this would help a lot.

TBH, I was waiting until dust with DDMD settles down a bit simply reading sources, but going to get back pretty soon. Help is welcome, I can provide plenty of information to start with ;)
August 19, 2013
On 8/19/13, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:
> Plus, DMD really needs to stop stripping paths from object files *by default*

Meanwhile give your vote to: https://github.com/D-Programming-Language/dmd/pull/1871