On Fri, Jul 23, 2021 at 06:53:06PM +0000, JG via Digitalmars-d-learn wrote: [...]
> The program I writing is around 3000 loc and recently I noticed a large slow down in compile time which after investigation seemed to be caused by my computer running out of memory. The compile was using more than 15GB memory.  I tried using lowmem and that did solve the memory problem but the compile still takes around 1 minute. Any suggestion on how to try and improve the build time. I am currently using dub.

3000 loc and 1 minute build time?  Sounds like you're using too many nested templates / CTFE.


> Of course one could try to use fewer templates and less meta programming but that seems to defeat the purpose of using d.

I wouldn't say use fewer templates / less meta-programming.  But I'd say look into how deeply nested templates are, and whether some templates parameters may be unnecessary.  If you have recursive templates, consider refactoring it so that it uses linear expansion instead. Shallowly-nested templates generally don't run into performance problems.

(Confession: I wrote the variadic version of cartesianProduct in std.algorithm with recursive templates. It uses an exponential number of template expansions, and so quickly brings the compiler to its knees when you try to take 4 or more cartesian products in a row.  Eventually, I refactored the most common case (no infinite ranges among its arguments) to use a linear expansion with a nested loop instead. Compile times improved by a HUGE margin.)

And avoid doing too much work in CTFE, which is known to be slow. But not as slow as overly-deeply nested templates.

Another way is to have a separate build step for expanding the most heavy templates, so that you only incur that heavy expansion once in a while when you change the relevant code.  I had a Vibe.d project where Diet templates slowed me down too much (they are super template-heavy), so I split it into several different build targets with a separate link step, so that when I'm not changing the Diet templates it doesn't slow me down so much.  Dub unfortunately won't help you here (unless you use subpackages -- but I doubt it will win much) -- I recommend using a better build system like CMake or SCons. Dub's architecture simply does not play well with staged compilation.

Alternatively, use a separate pre-compilation stage for generating code (e.g., write a utility that emits D code that then gets compiled in a subsequent step).  As much as I love D's compile-time capabilities, there comes a time when it's simply more practical to just `writefln` some D code snippets into a file and compile that into the main program, instead of trying to tame the memory-guzzling beast that is the D compiler.


T

-- 
I am Ohm of Borg. Resistance is voltage over current.

On Friday, 23 July 2021 at 19:32:08 UTC, H. S. Teoh wrote:
> And avoid doing too much work in CTFE, which is known to be slow.

Well it very much depends HOW you do it. Like the ~= operation in ctfe is awfully slow and wastes a lot of memory depending on the size of the string, but if you preallocate and copy memory in chunks it isn't too bad at all.

And if you use format!"str"() in ctfe that alone can be a real speed killer.

That's why I wanna see the code, it is possible there's a fairly simple bottleneck to look at.