Recently I studied the performance of building a vibe.d example:
https://github.com/rejectedsoftware/vibe.d/issues/208

I wrote a few tools in the process, perhaps someone might find them useful as well.

However, I'd also like to discuss a related matter:

I noticed that compiling D programs in the usual manner (rdmd) is as much as 40% slower than it can be.

This is because before rdmd knows a full list of modules to be built, it must run dmd with -v -o-, and read its verbose output. Then, it feeds that output back to the compiler again, and passes all modules on the command line of the second run.

The problem with this approach is that DMD needs to parse, lex, run CTFE, instantiate templates, etc. etc. - everything except actual code generation / optimization / linking - twice. And code generation can actually be a small part of the total compilation time.

D code already compiles pretty quickly, but here's an opportunity to nearly halve that time (for some cases) - by moving some of rdmd's basic functionality into the compiler. DMD already knows which modules are used in the program, so it just needs two new options: one to enabled this behavior (say, -r for recursive compilation), and one to specify an exclusion list, to indicate which modules are already compiled and will be found in a library (e.g. -rx). The default -rx settings can be placed in sc.ini/dmd.conf. I think we should seriously consider it.

Another appealing thing about the idea is that the compiler has access to information that would allow it to recompile programs more efficiently in the future. For example, it would be possible to get a hash of a module's public interface, so that a change in one function's code would not trigger a recompile of all modules that import it (assuming no CTFE).

Vladimir Panteleev:

> D code already compiles pretty quickly, but here's an opportunity to nearly halve that time (for some cases) - by moving some of rdmd's basic functionality into the compiler.

Make the D compiler search for its modules was one of the first (unwritten) enhancement requests. That's the right default for a handy compiler (plus a compiler switch to disable that behavour). But for backwards compatibility I think that switch has to do the opposite, to enable the recursive search.

Bye,
bearophile

On Friday, 5 April 2013 at 00:39:49 UTC, bearophile wrote:
> Vladimir Panteleev:
>
>> D code already compiles pretty quickly, but here's an opportunity to nearly halve that time (for some cases) - by moving some of rdmd's basic functionality into the compiler.
>
> Make the D compiler search for its modules was one of the first (unwritten) enhancement requests. That's the right default for a handy compiler (plus a compiler switch to disable that behavour). But for backwards compatibility I think that switch has to do the opposite, to enable the recursive search.

Yes, I agree completely. D is the only programming language I know that has both a module system, and the archaic C/C++ compilation model. Even Pascal got this right!

I think Rust got off the right foot in this regard. The compiler ("rustc") is a low-level detail which most users don't interact with directly; instead, there's "rust", a tool which acts as the user-friendly interface for the compiler, package manager, documentation tool, etc. This is in contrast with D, where the tool in the spotlight and the shorter name will throw cryptic linker errors at you if you don't to specify all modules on its command line.

On Friday, 5 April 2013 at 00:39:49 UTC, bearophile wrote:
> Vladimir Panteleev:
>
>> D code already compiles pretty quickly, but here's an opportunity to nearly halve that time (for some cases) - by moving some of rdmd's basic functionality into the compiler.
>
> Make the D compiler search for its modules was one of the first (unwritten) enhancement requests. That's the right default for a handy compiler (plus a compiler switch to disable that behavour). But for backwards compatibility I think that switch has to do the opposite, to enable the recursive search.

Actually, that switch is -c. The default behavior is "compile and link"... actually, I can't think of any instances when making -r the default when only one module is specified on the command line would break anything.

On Fri, 05 Apr 2013 00:49:08 +0200
"Vladimir Panteleev" <vladimir@thecybershadow.net> wrote:

> Recently I studied the performance of building a vibe.d example: https://github.com/rejectedsoftware/vibe.d/issues/208
> 
> I wrote a few tools in the process, perhaps someone might find them useful as well.
> 
> However, I'd also like to discuss a related matter:
> 
> I noticed that compiling D programs in the usual manner (rdmd) is as much as 40% slower than it can be.
> 
> This is because before rdmd knows a full list of modules to be built, it must run dmd with -v -o-, and read its verbose output. Then, it feeds that output back to the compiler again, and passes all modules on the command line of the second run.
> 
> The problem with this approach is that DMD needs to parse, lex, run CTFE, instantiate templates, etc. etc. - everything except actual code generation / optimization / linking - twice. And code generation can actually be a small part of the total compilation time.
> 
> D code already compiles pretty quickly, but here's an opportunity to nearly halve that time (for some cases) - by moving some of rdmd's basic functionality into the compiler. DMD already knows which modules are used in the program, so it just needs two new options: one to enabled this behavior (say, -r for recursive compilation), and one to specify an exclusion list, to indicate which modules are already compiled and will be found in a library (e.g. -rx). The default -rx settings can be placed in sc.ini/dmd.conf. I think we should seriously consider it.
> 
> Another appealing thing about the idea is that the compiler has access to information that would allow it to recompile programs more efficiently in the future. For example, it would be possible to get a hash of a module's public interface, so that a change in one function's code would not trigger a recompile of all modules that import it (assuming no CTFE).

About a year or two ago, Andrei proposed a system that addressed that issue together with a substitute for a package manager. I was against it for various reasons, and for the psudo-package-manager part of it I still am. But I've since become more open to the "Make RDMD only call DMD once" part. *That* part *might* not be a bad idea, although I can't remember quite how it worked. Something about passing DMD a cmd line tool to invoke whenever it finds a newly imported module? Although if it was like that, that it could very well slow things down on Windows, as Windows is notoriously slow at launching processes.

Or just fold RDMD functionality into DMD itself, like you said...which would probably be easier anyway.

I was very unpleasantly surprised to know from that thread about dmd doing everything but object file creation during rdmd dependency tracking step. That feels very inefficient and probably only reason this has not caught any attention is generally fast dmd compilation speed. This case begs for better integration between dmd and rdmd or even moving some functionality of rdmd to front-end.

On 4/5/13 3:29 AM, Dicebot wrote:
> I was very unpleasantly surprised to know from that thread about dmd
> doing everything but object file creation during rdmd dependency
> tracking step. That feels very inefficient and probably only reason this
> has not caught any attention is generally fast dmd compilation speed.
> This case begs for better integration between dmd and rdmd or even
> moving some functionality of rdmd to front-end.

Well rdmd caches the result of the dependency collection step and it won't rebuild dependencies unless necessary. Anyhow I think it makes a lot of sense for dmd to automatically link .d files imported by the main program compiled. A bunch of language features (e.g. no classes spread across modules) and approach to modularity (e.g. separate notions of .di and .d) make that natural.

Andrei

On 4/5/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
> it must run dmd with -v -o-, and read its verbose output.

IIRC xfbuild used to work around this by both reading the verbose output and simultaneously building. In other words it used 'dmd -deps' without the -o- switch. There's a function called compileAndTrackDeps which did this, but the entire codebase was very hacky which has lead me to abandon the port.

Note that using pipes instead of File I/O to read the verbose output of DMD can speed up the entire building process (the new std.process comes in handy for this).

However I've come to the same conclusions as you, DMD could do all of this on its own in one process invocation. It won't have to re-parse and re-process any dependencies more than once, which could be a significant boost in speed.

Plus, it would be a boon for people new to D who expect things to work in a modern way (no nasty linker errors).

Preferably we could introduce another switch -rxp which is used to exclude entire packages, e.g.:

dmd -r -rx=somemodule.d -rxp=some.package (The = were just added for
readability)

On Friday, 5 April 2013 at 10:18:25 UTC, Andrej Mitrovic wrote:

(agreed with everything not quoted)

> dmd -r -rx=somemodule.d -rxp=some.package (The = were just added for
> readability)

I think -r is redundant, and should be the default action if only one module is given on DMD's command line. I can't think of plausible situations where this could be a problem.

Considering that you can't have a module with the same name as a package, the same syntax for excluding both can be used, e.g. "-rxcrc32 -rxstd".

On Friday, 5 April 2013 at 00:51:38 UTC, Vladimir Panteleev wrote:
> On Friday, 5 April 2013 at 00:39:49 UTC, bearophile wrote:
>> Vladimir Panteleev:
>>
>>> D code already compiles pretty quickly, but here's an opportunity to nearly halve that time (for some cases) - by moving some of rdmd's basic functionality into the compiler.
>>
>> Make the D compiler search for its modules was one of the first (unwritten) enhancement requests. That's the right default for a handy compiler (plus a compiler switch to disable that behavour). But for backwards compatibility I think that switch has to do the opposite, to enable the recursive search.
>
> Yes, I agree completely. D is the only programming language I know that has both a module system, and the archaic C/C++ compilation model. Even Pascal got this right!
>

Yep, units exist since UCSD Pascal.