View mode: basic / threaded / horizontal-split · Log in · Help
July 25, 2012
Re: What is the compilation model of D?
On Wednesday, July 25, 2012 14:57:23 Andrej Mitrovic wrote:
> Hell I can't believe how outdated the compiler technology is. I can
> play incredibly realistic and interactive 3D games in real-time with
> practically no input lag, but I have to wait a dozen seconds for a
> tool to convert lines of text into object code? From a syntax
> perspective D has moved forward but from a compilation perspective it
> hasn't innovated at all.

And dmc and dmd are lightning fast in comparison to most compilers. I think 
that a lot of it comes down to the fact that optimizing code is _expensive_, 
and doing a lot of operations on an AST isn't necessarily all that cheap 
either. dmd is actually _lightning_ fast at processing text. That's not what's 
slow. It's everything which is after that which is.

And for most compilers, the speed of the resultant code matters a lot more 
than the speed of compilation. Compare this to games which need to maintain a 
certain number of FPS. They optimize _everything_ towards that goal, which is 
why they achieve it. There's also no compiler equivalent of parallelizing 
optimizations to the AST or asm like games have parallelizing geometric 
computations and the like with GPUs. The priorities are completely different, 
what they're doing is very different, and what they have to work with is very 
different. As great as it would be if compilers were faster, it's an apples to 
oranges comparison.

- Jonathan M Davis
July 25, 2012
Re: What is the compilation model of D?
On Wednesday, July 25, 2012 17:35:09 David Piepgrass wrote:
> > I find it shocking that anyone would consider 15 seconds slow
> > to compile for a
> > large program. Yes, D's builds are lightning fast in general,
> > and 15 seconds
> > is probably a longer build, but calling 15 seconds
> > "slow-to-compile" just
> > about blows my mind. 15 seconds for a large program is _fast_.
> > If anyone
> > complains about a large program taking 15 seconds to build,
> > then they're just
> > plain spoiled or naive. I've dealt with _Java_ apps which took
> > in the realm of
> > 10 minutes to compile, let alone C++ apps which take _hours_ to
> > compile. 15
> > seconds is a godsend.
> 
> I agree with Andrej, 15 seconds *is* slow for a edit-compile-run
> cycle, although it might be understandable when editing code that
> uses a lot of CTFE and static foreach and reinstantiates
> templates with a crapton of different arguments.
> 
> I am neither spoiled nor naive to think it can be done in under
> 15 seconds. Fully rebuilding all my C# code takes less than 10
> seconds (okay, not a big program, but several smaller programs).

Sure, smaller programs should should build quickly, and having build times get 
slower as the program grows can definitely be a problem. I'm not about to argue 
with that. But having a _large_ application build in 15 seconds is arguably a 
luxory. Large applications just aren't the sort of thing that builds quickly. 
But that's the sort of project that's usually commercial (either that or a 
major open source one), and I don't think that D's been used in that domain a 
lot yet.

While D compiles far faster than C++, the kind of application which takes 
hours to compile in C++ and the one that takes 10+ seconds in D are on a 
completely different level in terms of amount of source code and the level of 
complexity, even if D _would_ probably only take minutes on a similar project 
instead of hours.

- Jonathan M Davis
July 25, 2012
Re: What is the compilation model of D?
Thanks for the very good description, Nick! So if I understand 
correctly, if

1. I use an "auto" return value or suchlike in a module Y.d
2. module X.d calls this function
3. I call "dmd -c X.d" and "dmd -c Y.d" as separate steps

Then the compiler will have to fully parse Y twice and fully 
analyze the Y function twice, although it generates object code 
for the function only once. Right? I wonder how smart it is about 
not analyzing things it does not need to analyze (e.g. when Y is 
a big module but X only calls one function from it - the compiler 
has to parse Y fully but it should avoid most of the semantic 
analysis.)

What about templates? In C++ it is a problem that the compiler 
will instantiate templates repeatedly, say if I use 
vector<string> in 20 source files, the compiler will generate and 
store 20 copies of vector<string> (plus 20 copies of 
basic_string<char>, too) in object files.

1. So in D, if I compile the 20 sources separately, does the same 
thing happen (same collection template instantiated 20 times with 
all 20 copies stored)?
2. If I compile the 20 sources all together, I guess the template 
would be instantiated just once, but then which .obj file does 
the instantiated template go in?

> $rdmd --build-only (any other flags) main.d
>
> Then, RDMD will figure out *all* of the source files needed 
> (using
> the full compiler's frontend, so it never gets fooled into 
> missing
> anything), and if any of them have been changed, it will 
> automatically
> pass them *all* into DMD for you. This way, you don't have to
> manually keep track of all your files and pass them all into
> DMD youself. Just give RDMD your main file and that's it, 
> you're golden.
>
> Side note: Another little trick with RDMD: Omit the 
> --build-only and it will compile AND then run your program:

> Yes. (Unless you never import anything from in phobos...I 
> think.) But
> it's very, very fast to parse. Lightning-speed if you compare 
> it to C++.

I don't even want to legitimize C++ compiler speed by comparing 
it to any other language ;)

>> - Is there any concept of an incremental build?
>
> Yes, but there's a few "gotcha"s:
>
> 1. D compiles so damn fast that it's not nearly as much of an 
> issue as
> it is with C++ (which is notoriously ultra-slow compared
> to...everything, hence the monumental importance of C++'s 
> incremental
> builds).

I figure as CTFE is used more, especially when it is used to 
decide which template overloads are valid or how a mixin will 
behave, this will slow down the compiler more and more, thus 
making incremental builds more important. A typical example would 
be a compile-time parser-generator, or compiled regexes.

Plus, I've heard some people complaining that the compiler uses 
over 1 GB RAM, and splitting up compilation into parts might help 
with that.

BTW, I think I heard the compiler uses multithreading to speed up 
the build, is that right?

> It keeps diving deeper and deeper to find anything it can 
> "start" with.
> One it finds that, it'll just build everything back up in 
> whatever
> order is necessary.

I hope someone can give more details about this.

>> - In light of the above (that the meaning of D code can be 
>> interdependent with other D code, plus the presence of mixins 
>> and all that), what are the limitations of 
>> __traits(allMembers...) and other compile-time reflection 
>> operations, and what kind of problems might a user expect to 
>> encounter?
>
> Shouldn't really be an issue. Such things won't get evaluated 
> until the
> types/identifiers involved are *fully* analyzed (or at least to 
> the
> extent that they need to be analyzed). So the results of things 
> like
> __traits(allMembers...) should *never* change during 
> compilation, or
> when changing the order of files or imports (unless there's some
> compiler bug). Any situation that *would* result in any such 
> ambiguity
> will get flagged as an error in your code.

Hmm. Well, I couldn't find an obvious example... for example, you 
are right, this doesn't work, although the compiler annoyingly 
doesn't give a reason:

struct OhCrap {
	void a() {}
	// main.d(72): Error: error evaluating static if expression
	//             (what error? syntax error? type error? c'mon...)
	static if ([ __traits(allMembers, OhCrap) ].length > 1) {
		auto b() { return 2; }
	}
	void c() {}
}

But won't this be a problem when it comes time to produce 
run-time reflection information? I mean, when module A asks to 
create run-time reflection information for all the functions and 
types in module A.... er, I naively thought the information would 
be created as a set of types and functions *in module A*, which 
would then change the set of allMembers of A. But, maybe it makes 
more sense to create that stuff in a different module (which A 
could then import??)

Anyway, I can't even figure out how to enumerate the members of a 
module A; __traits(allMembers, A) causes "Error: import Y has no 
members".

Aside: I first wrote the above code as follows:

// Shouldn't this be in Phobos somewhere?
bool contains(alias pred = "a == b", R, E)(R haystack, E needle)
    if (isInputRange!R &&
        is(typeof(binaryFun!pred(haystack.front, needle)) : bool))
{
	return !(find!(pred, R, E)(haystack, needle).empty);
}

struct OhCrap {
	void a() {}
	static if ([ __traits(allMembers, OhCrap) ].contains("a")) {
		auto b() { return 2; }
	}
	void c() {}
}

But it causes a series of 204 error messages that I don't 
understand.
July 25, 2012
Re: What is the compilation model of D?
On Wednesday, 25 July 2012 at 19:54:31 UTC, David Piepgrass wrote:
>> It keeps diving deeper and deeper to find anything it can 
>> "start" with.
>> One it finds that, it'll just build everything back up in 
>> whatever
>> order is necessary.
>
> I hope someone can give more details about this.

TDPL chapter 11 "Scaling Up".
July 25, 2012
Re: What is the compilation model of D?
> If you use rdmd to compile (instead of dmd), you *just* give it
> your *one* main source file (typically the one with your 
> "main()"
> function). This file must be the *last* parameter passed to 
> rdmd:
>
> $rdmd --build-only (any other flags) main.d
>
> Then, RDMD will figure out *all* of the source files needed 
> (using
> the full compiler's frontend, so it never gets fooled into 
> missing
> anything), and if any of them have been changed, it will 
> automatically
> pass them *all* into DMD for you. This way, you don't have to
> manually keep track of all your files and pass them all into
> DMD youself. Just give RDMD your main file and that's it, 
> you're golden.

I meant to ask, why would it recompile *all* of the source files 
if only one changed? Seems like it only should recompile the 
changed ones (but still compile them together as a unit.) Is it 
because of bugs (e.g. the template problem you mentioned)?
July 25, 2012
Re: What is the compilation model of D?
>> I hope someone can give more details about this.
>
> TDPL chapter 11 "Scaling Up".

That's where I was looking. As I said already, TDPL does not 
explain how compilation works, especially not anything about the 
low-level semantic analysis which has me most curious.
July 25, 2012
Re: What is the compilation model of D?
On Wednesday, 25 July 2012 at 20:25:19 UTC, David Piepgrass wrote:
>>> I hope someone can give more details about this.
>>
>> TDPL chapter 11 "Scaling Up".
>
> That's where I was looking. As I said already, TDPL does not 
> explain how compilation works, especially not anything about 
> the low-level semantic analysis which has me most curious.

Strange, because it seems to me this chapter answers all your 
previous questions. What exact details are you interested in?
July 25, 2012
Re: What is the compilation model of D?
On Wednesday, 25 July 2012 at 08:06:23 UTC, Nick Sabalausky wrote:
> Yea, my understanding is that full-build times measured in days 
> are (or
> used to be, don't know if they still are) also typical of 
> high-budget
> C++-based videogames.

You must be thinking of full data rebuilds, not code recompiles.

There's no way a game could take over a day to compile and still 
produce an executable that would fit on a console.

Several minutes is more typical. Maybe up to 30 minutes in bad 
cases.
July 25, 2012
Re: What is the compilation model of D?
On Wed, 25 Jul 2012 23:20:04 +0200
"Peter Alexander" <peter.alexander.au@gmail.com> wrote:

> On Wednesday, 25 July 2012 at 08:06:23 UTC, Nick Sabalausky wrote:
> > Yea, my understanding is that full-build times measured in days 
> > are (or
> > used to be, don't know if they still are) also typical of 
> > high-budget
> > C++-based videogames.
> 
> You must be thinking of full data rebuilds, not code recompiles.
> 
> There's no way a game could take over a day to compile and still 
> produce an executable that would fit on a console.
> 
> Several minutes is more typical. Maybe up to 30 minutes in bad 
> cases.

Yea, you're probably right. I meant "full project", which almost
certainly involves going through gigabytes of assets.
July 25, 2012
Re: What is the compilation model of D?
On Wed, 25 Jul 2012 22:18:37 +0200
"David Piepgrass" <qwertie256@gmail.com> wrote:
> 
> I meant to ask, why would it recompile *all* of the source files 
> if only one changed? Seems like it only should recompile the 
> changed ones (but still compile them together as a unit.) Is it 
> because of bugs (e.g. the template problem you mentioned)?

I'm not 100% certain, but, yes, I think it's a combination of that, and
the fact that nobody's actually gone and tried to make that change to
RDMD yet.

AIUI, The original motivating purpose for RDMD was to be able to execute
a D source file as if it were a script. So finding all relevant source
files, passing them to DMD, etc, was all just necessary steps towards
that end. Which turned out to also be useful in many cases for general
project building.
1 2 3
Top | Discussion index | About this forum | D home