On 10 June 2013 11:25, Martin Nowak <code@dawg.eu> wrote:
On 06/10/2013 08:29 AM, Don Clugston wrote:
Yeah, my point was that for N tests, we get O(N^^2) templates instantiated.
How is that?

Just correlation, not causality! The million-ish instantations only comes from a thousand tests.
 
> Wow! (How many of those are unique, rather than reusing an existing instantiation?)

The number of instances is also a huge performance issue when searching for existing instantiations, because search is done linear and comparing two TemplateInstance uses the costly arrayObjectMatch on the template arguments. So in average instantiating the same template N times uses O(N^^2 / 2) comparisons.
I once improved that by computing the mangling before the search and using it as a key.
That brought down the compilation time of std.range with unittests from 10 to 2 seconds.
There is at least one blocker though http://d.puremagic.com/issues/show_bug.cgi?id=7469.

Interesting. This is looking like an area where an order of magnitude improvement in both memory consumption and speed is possible...
 

> Error: out of memory

> This has happened many times before, and we dealt with it by reducing the number of modules we compiled into each object file. We once had 30 modules per obj file. Then fifteen. Then five. But now we're at one, that workaround can no longer be used.

I ran into some issue when preparing the phobos unittests for shared libraries.
If you compile the whole library at once with unittests enabled it consumes ~6GB of RAM.
In order to find a good partition I took up the idea of clustering ( http://d.puremagic.com/issues/show_bug.cgi?id=9673#c2, https://gist.github.com/dawgfoto/5747405). Well, at least clustering by imports did only find a few useful partitions because almost every module ends up importing std.range or std.algorithm. It might be interesting to do some analysis for the template instances used in each module.

I think the biggest problem is traits.d.  It contains ultra-lightweight templates that are used everywhere and have no dependencies, things like Unqual!(). But it imports
std.typetuple (which is OK, it only imports std.traits),
std.typecons. (which imports std.array, std.conv, std.exception, std.format, std.string, std.range)
std.algorithm (which imports std.array, std.ascii, std.container, std.conv, std.exception, std.functional, std.math, std.random, std.range, std.string,
    std.uni, std.utf)
... ie, basically all of Phobos

If we could split those things out into a 'leaf' module, probably most of the import graph complexity would disappear. I'm sure most of the cyclic imports are false dependencies.