Any time I see people mention the benefits of D, I see "compile times" "compile times" "compile times" over and over.

I'm using very modest amounts of templates, for a fairly small sized program (very early work toward a game), and I'm hitting ~15 seconds compile time in LDC and ~7 seconds in DMD. And I'm not even compiling with optimizations!

ldc2 -w -ofextra  extra.d molto.d helper.d editor.d common.d map.d object_t.d animation.d ini.d  -L-L. $@    -gc -d-debug=3  -de -fdmd-trace-functions

dmd -w -ofextra extra.d molto.d helper.d editor.d common.d map.d object_t.d animation.d ini.d -profile=gc  -profile  -g -debug -color -L-L.

I keep putting stuff into new files, but it feels like it's compiling everything from scratch / not getting faster the way C++ does.

And I'm not even bringing up the 800MB of RAM required because I dared to import std.regex. (On a laptop with 2 GB of RAM. RIP. If I dare to have tabs open, the compile time goes into the minutes thanks to swapping.)

On Friday, 23 November 2018 at 08:57:57 UTC, Chris Katko wrote:
> Any time I see people mention the benefits of D, I see "compile times" "compile times" "compile times" over and over.
>
> I'm using very modest amounts of templates, for a fairly small sized program (very early work toward a game), and I'm hitting ~15 seconds compile time in LDC and ~7 seconds in DMD. And I'm not even compiling with optimizations!

Templates are slow, slower than CTFE.

> ldc2 -w -ofextra  extra.d molto.d helper.d editor.d common.d map.d object_t.d animation.d ini.d  -L-L. $@    -gc -d-debug=3  -de -fdmd-trace-functions
>
> dmd -w -ofextra extra.d molto.d helper.d editor.d common.d map.d object_t.d animation.d ini.d -profile=gc  -profile  -g -debug -color -L-L.
>
> I keep putting stuff into new files, but it feels like it's compiling everything from scratch / not getting faster the way C++ does.

If you pass all the files on the command line then they all get (re)compiled.
Have separate rules for each file, although if you are using template from each file in every file that may not help a whole lot.

> And I'm not even bringing up the 800MB of RAM required because I dared to import std.regex. (On a laptop with 2 GB of RAM. RIP. If I dare to have tabs open, the compile time goes into the minutes thanks to swapping.)

Yep, on the upside the regexen are very fast at runtime. std.regex depending on how much of it you are using could also be a cause of your problem.you might want to take a look at https://blog.thecybershadow.net/2018/02/07/dmdprof/
https://github.com/CyberShadow/dmdprof to see whats taking so long

Is this code online? Its a bit difficult to give specific advice without it.

On Fri, Nov 23, 2018 at 10:00 AM Chris Katko via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:

> Any time I see people mention the benefits of D, I see "compile times" "compile times" "compile times" over and over.
>
> I'm using very modest amounts of templates, for a fairly small sized program (very early work toward a game), and I'm hitting ~15 seconds compile time in LDC and ~7 seconds in DMD. And I'm not even compiling with optimizations!
>
> ldc2 -w -ofextra  extra.d molto.d helper.d editor.d common.d map.d object_t.d animation.d ini.d  -L-L. $@    -gc -d-debug=3 -de -fdmd-trace-functions
>
> dmd -w -ofextra extra.d molto.d helper.d editor.d common.d map.d object_t.d animation.d ini.d -profile=gc  -profile  -g -debug -color -L-L.
>
> I keep putting stuff into new files, but it feels like it's compiling everything from scratch / not getting faster the way C++ does.
>
> And I'm not even bringing up the 800MB of RAM required because I dared to import std.regex. (On a laptop with 2 GB of RAM. RIP. If I dare to have tabs open, the compile time goes into the minutes thanks to swapping.)
>
>
AFAIK debug builds are slower? Can you share your files with us? So I can try to find out what is the main problem

On Friday, 23 November 2018 at 08:57:57 UTC, Chris Katko wrote:
> Any time I see people mention the benefits of D, I see "compile times" "compile times" "compile times" over and over.
>
> [...]

If you can share the code privately I can use my custom profiling build of dmd to analyze the problem. just send me a mail (uplink{dot}coder{at}gmail{dot}com)

On Fri, Nov 23, 2018 at 08:57:57AM +0000, Chris Katko via Digitalmars-d-learn wrote:
> Any time I see people mention the benefits of D, I see "compile times" "compile times" "compile times" over and over.

D is extremely fast at compilation ... of C-like code. :-D

Anything involving heavy use of templates and/or CTFE will quickly slow things down, sometimes by a lot.


> I'm using very modest amounts of templates, for a fairly small sized program (very early work toward a game), and I'm hitting ~15 seconds compile time in LDC and ~7 seconds in DMD. And I'm not even compiling with optimizations!

Are you using template-heavy Phobos functions?  Some parts of Phobos are known to be extremely slow, e.g., std.format (which is indirectly imported by std.stdio), due to the sheer amount of templates it uses.


[...]
> I keep putting stuff into new files, but it feels like it's compiling everything from scratch / not getting faster the way C++ does.

Are you still compiling everything in one command, or separately compiling?  There's not much point (as far as compile times are concerned) in splitting up into new files if you're still compiling everything each time.


> And I'm not even bringing up the 800MB of RAM required because I dared to import std.regex. (On a laptop with 2 GB of RAM. RIP. If I dare to have tabs open, the compile time goes into the minutes thanks to swapping.)

Yeah, std.regex is known to be pretty nasty in terms of compile times / memory usage, because its internals uses a LOT of templates.  There have been efforts to fix / improve this, but I haven't kept up with the developments, so I'm not sure where things are at now.

One caveat about std.regex, though: you may actually want to use runtime regexen instead of ctRegex, in spite of the supposed performance improvements; I recall reading somewhere that ctRegex actually isn't as fast as you might think, yet it comes at a HUGE compile-time cost (and I do mean HUGE...  it can significantly increase compile times for only marginal or sometimes even negative runtime performance -- because of template bloat).  I've stopped using ctRegex altogether and just been initializing regexen with `static this()` instead.  Or sometimes even just pure runtime regexen, because the cost of initializing once is insignificant compared to subsequent repeated usage.


T

-- 
Too many people have open minds but closed eyes.

On Friday, 23 November 2018 at 17:21:46 UTC, H. S. Teoh wrote:
> Are you using template-heavy Phobos functions?

Merely importing a Phobos module is liable to cost you a quarter second or more of compile time.

I just got my gui lib (22,000 lines of raw source, dscanner -sloc reports 12,000) compiling in 0.2s on Linux, down from 1.0s, by removing ALL the phobos imports. It is 0.6s on Windows, probably because it imports so many of the Windows headers.


I like to say C style and Java style code compiles really fast... but you wanna beware of imports of the standard library, since it brings in a LOT of code, some of which is really slow to run (parts of it are slower than others, like std.algorithm is actually pretty fast to compile+use, but std.regex is brutally slow)

On Fri, Nov 23, 2018 at 05:37:46PM +0000, Adam D. Ruppe via Digitalmars-d-learn wrote:
> On Friday, 23 November 2018 at 17:21:46 UTC, H. S. Teoh wrote:
> > Are you using template-heavy Phobos functions?
> 
> Merely importing a Phobos module is liable to cost you a quarter second or more of compile time.

Yes, I'm aware of that.  It used to be the case that the mere importing of std.format would add 2-3 seconds to your compile time, but I think the worst of it has been fixed.  It's still not cheap to use std.format, but it's improved from what it used to be.


> I just got my gui lib (22,000 lines of raw source, dscanner -sloc reports 12,000) compiling in 0.2s on Linux, down from 1.0s, by removing ALL the phobos imports. It is 0.6s on Windows, probably because it imports so many of the Windows headers.

Yeah, I find that avoiding certain Phobos modules like std.format or std.regex really does help a lot in improving compile times.


> I like to say C style and Java style code compiles really fast... but you wanna beware of imports of the standard library, since it brings in a LOT of code, some of which is really slow to run (parts of it are slower than others, like std.algorithm is actually pretty fast to compile+use, but std.regex is brutally slow)

Well yes, std.algorithm is practically all templates, and relatively small ones (except for bears like cartesianProduct, no thanks to yours truly :-/), so even though std.algorithm.* is pretty big in terms of LOC, you only pay for what you use.  But std.regex has basically template-driven internals, and IIRC has static tables that are generated by templates and/or CTFE, so it can really slow the compiler down.

The other big offender is std.uni, which imports some pretty large templated tables that adds a few seconds to compile times.

All in all, though, the fact that we're complaining about extra seconds in compilation times still does show just how fast D compilation can be. In the old days, compiling large C++ codebases usually means 30 mins to 2 hours, and a few extra seconds won't even be noticed.  I haven't checked C++ compile times recently, though -- things may have improved since I last seriously used C++.


T

-- 
IBM = I'll Buy Microsoft!

On Friday, November 23, 2018 11:13:24 AM MST H. S. Teoh via Digitalmars-d- learn wrote:
> All in all, though, the fact that we're complaining about extra seconds in compilation times still does show just how fast D compilation can be. In the old days, compiling large C++ codebases usually means 30 mins to 2 hours, and a few extra seconds won't even be noticed.  I haven't checked C++ compile times recently, though -- things may have improved since I last seriously used C++.

When I was a student, I worked for a while at a company that had a large C++ code base that took over 3 hours to compile from scratch (incremental builds were an absolute must). I ended up working somewhere else for a while and then coming back again, and in the interim, they had begun redoing the program in Java. The build process then took about 10 minutes, and folks were complaining about it taking too long. After having been away for a while, my perspective was that it was a _huge_ improvement, but since they'd been working with it for a while, the 10 minutes was annoying. So, a lot of it comes down to perspective.

D is often a _huge_ improvement when you first switch to it, but depending on what your code does, over time, it can go from under a second to a few seconds in compilation time, and for some folks that becomes maddening in spite of the fact that the overall build times are a huge improvement over what they would have had in another language - and usually, the features that slow down the build the most are ones that don't even exist in other languages (or if they do, are far less powerful). That being said, if we can reasonably improve the compiler and standard library such that D code in general builds faster with all of the CTFE and templates, we definitely should.

- Jonathan M Davis

On Friday, 23 November 2018 at 10:00:17 UTC, Nicholas Wilson wrote:
>
> If you pass all the files on the command line then they all get (re)compiled.

How are you supposed include files if not passing them to the compiler?

I'm only using std.regex in one file, IIRC, so whatever the "proper" way to only compile changed files should improve it drastically.

Almost all of my templates are incredibly simple like using generic arguments for taking float and doubles, instead of hardcoded float.

I am using both Allegro and DAllegro (a D binding for Allegro5). But until recently, the compile times have been very fast. So it's definitely been D features cutting things down.

And as for "compile-time doesn't matter" arguments, that's plain silly. I'm not a junior programmer. I've had builds that took over 30 minutes to compile and as we all (should!) know, the longer the build time (especially over 10 seconds), the quicker the mind loses its train-of-thought and the more difficulty the brain has with establishing cause-and-effect relationships between code and bugs. When our builds hit 30 minutes, we ended up so disconnected from the project we'd end up playing a short game of Duke Nukem 3-D inbetween builds. (Ha! Builds. Build engine = the Duke Nukem 3D engine. A super-niche pun.) We were incredibly unproductive until I re-wrote the entire thing in a different language. It ran in less than 10 seconds and suddenly we were powering through new problem after new problem. (A huge data conversion project between a discontinued, no-documentation product and a new product by a different company.)

Anyhow, if you REALLY want to look at some very WIP code with poor documentation and possibly lots of random swearing (hobby project for trying out D in a game), I'll make the repo public for now:

https://bitbucket.org/katasticvoyage/ss14/src/master/

 extra.d is a main code unit for this application.

 ini.d has regex. (and no, it's not a proper lexer. I'm probably swapping it out with JSON.)

On Sat, Nov 24, 2018 at 09:20:08AM +0000, Chris Katko via Digitalmars-d-learn wrote:
> On Friday, 23 November 2018 at 10:00:17 UTC, Nicholas Wilson wrote:
> > 
> > If you pass all the files on the command line then they all get
> > (re)compiled.
> 
> How are you supposed include files if not passing them to the compiler?
> 
> I'm only using std.regex in one file, IIRC, so whatever the "proper" way to only compile changed files should improve it drastically.
[...]

Use separate compilation:

	dmd -c module1.d
	dmd -c module2.d
	...
	dmd module1.o module2.o ... -of/usr/bin/myprogram

Put this in a makefile or your build system of choice, and let it do incremental builds.


T

-- 
Государство делает вид, что платит нам зарплату, а мы делаем вид, что работаем.