On Wednesday, July 25, 2012 14:57:23 Andrej Mitrovic wrote:
> Hell I can't believe how outdated the compiler technology is. I can play incredibly realistic and interactive 3D games in real-time with practically no input lag, but I have to wait a dozen seconds for a tool to convert lines of text into object code? From a syntax perspective D has moved forward but from a compilation perspective it hasn't innovated at all.

And dmc and dmd are lightning fast in comparison to most compilers. I think that a lot of it comes down to the fact that optimizing code is _expensive_, and doing a lot of operations on an AST isn't necessarily all that cheap either. dmd is actually _lightning_ fast at processing text. That's not what's slow. It's everything which is after that which is.

And for most compilers, the speed of the resultant code matters a lot more than the speed of compilation. Compare this to games which need to maintain a certain number of FPS. They optimize _everything_ towards that goal, which is why they achieve it. There's also no compiler equivalent of parallelizing optimizations to the AST or asm like games have parallelizing geometric computations and the like with GPUs. The priorities are completely different, what they're doing is very different, and what they have to work with is very different. As great as it would be if compilers were faster, it's an apples to oranges comparison.

- Jonathan M Davis

On Wednesday, July 25, 2012 17:35:09 David Piepgrass wrote:
> > I find it shocking that anyone would consider 15 seconds slow
> > to compile for a
> > large program. Yes, D's builds are lightning fast in general,
> > and 15 seconds
> > is probably a longer build, but calling 15 seconds
> > "slow-to-compile" just
> > about blows my mind. 15 seconds for a large program is _fast_.
> > If anyone
> > complains about a large program taking 15 seconds to build,
> > then they're just
> > plain spoiled or naive. I've dealt with _Java_ apps which took
> > in the realm of
> > 10 minutes to compile, let alone C++ apps which take _hours_ to
> > compile. 15
> > seconds is a godsend.
> 
> I agree with Andrej, 15 seconds *is* slow for a edit-compile-run cycle, although it might be understandable when editing code that uses a lot of CTFE and static foreach and reinstantiates templates with a crapton of different arguments.
> 
> I am neither spoiled nor naive to think it can be done in under 15 seconds. Fully rebuilding all my C# code takes less than 10 seconds (okay, not a big program, but several smaller programs).

Sure, smaller programs should should build quickly, and having build times get slower as the program grows can definitely be a problem. I'm not about to argue with that. But having a _large_ application build in 15 seconds is arguably a luxory. Large applications just aren't the sort of thing that builds quickly. But that's the sort of project that's usually commercial (either that or a major open source one), and I don't think that D's been used in that domain a lot yet.

While D compiles far faster than C++, the kind of application which takes hours to compile in C++ and the one that takes 10+ seconds in D are on a completely different level in terms of amount of source code and the level of complexity, even if D _would_ probably only take minutes on a similar project instead of hours.

- Jonathan M Davis

Thanks for the very good description, Nick! So if I understand correctly, if

1. I use an "auto" return value or suchlike in a module Y.d
2. module X.d calls this function
3. I call "dmd -c X.d" and "dmd -c Y.d" as separate steps

Then the compiler will have to fully parse Y twice and fully analyze the Y function twice, although it generates object code for the function only once. Right? I wonder how smart it is about not analyzing things it does not need to analyze (e.g. when Y is a big module but X only calls one function from it - the compiler has to parse Y fully but it should avoid most of the semantic analysis.)

What about templates? In C++ it is a problem that the compiler will instantiate templates repeatedly, say if I use vector<string> in 20 source files, the compiler will generate and store 20 copies of vector<string> (plus 20 copies of basic_string<char>, too) in object files.

1. So in D, if I compile the 20 sources separately, does the same thing happen (same collection template instantiated 20 times with all 20 copies stored)?
2. If I compile the 20 sources all together, I guess the template would be instantiated just once, but then which .obj file does the instantiated template go in?

> $rdmd --build-only (any other flags) main.d
>
> Then, RDMD will figure out *all* of the source files needed (using
> the full compiler's frontend, so it never gets fooled into missing
> anything), and if any of them have been changed, it will automatically
> pass them *all* into DMD for you. This way, you don't have to
> manually keep track of all your files and pass them all into
> DMD youself. Just give RDMD your main file and that's it, you're golden.
>
> Side note: Another little trick with RDMD: Omit the --build-only and it will compile AND then run your program:

> Yes. (Unless you never import anything from in phobos...I think.) But
> it's very, very fast to parse. Lightning-speed if you compare it to C++.

I don't even want to legitimize C++ compiler speed by comparing it to any other language ;)

>> - Is there any concept of an incremental build?
>
> Yes, but there's a few "gotcha"s:
>
> 1. D compiles so damn fast that it's not nearly as much of an issue as
> it is with C++ (which is notoriously ultra-slow compared
> to...everything, hence the monumental importance of C++'s incremental
> builds).

I figure as CTFE is used more, especially when it is used to decide which template overloads are valid or how a mixin will behave, this will slow down the compiler more and more, thus making incremental builds more important. A typical example would be a compile-time parser-generator, or compiled regexes.

Plus, I've heard some people complaining that the compiler uses over 1 GB RAM, and splitting up compilation into parts might help with that.

BTW, I think I heard the compiler uses multithreading to speed up the build, is that right?

> It keeps diving deeper and deeper to find anything it can "start" with.
> One it finds that, it'll just build everything back up in whatever
> order is necessary.

I hope someone can give more details about this.

>> - In light of the above (that the meaning of D code can be interdependent with other D code, plus the presence of mixins and all that), what are the limitations of __traits(allMembers...) and other compile-time reflection operations, and what kind of problems might a user expect to encounter?
>
> Shouldn't really be an issue. Such things won't get evaluated until the
> types/identifiers involved are *fully* analyzed (or at least to the
> extent that they need to be analyzed). So the results of things like
> __traits(allMembers...) should *never* change during compilation, or
> when changing the order of files or imports (unless there's some
> compiler bug). Any situation that *would* result in any such ambiguity
> will get flagged as an error in your code.

Hmm. Well, I couldn't find an obvious example... for example, you are right, this doesn't work, although the compiler annoyingly doesn't give a reason:

struct OhCrap {
	void a() {}
	// main.d(72): Error: error evaluating static if expression
	//             (what error? syntax error? type error? c'mon...)
	static if ([ __traits(allMembers, OhCrap) ].length > 1) {
		auto b() { return 2; }
	}
	void c() {}
}

But won't this be a problem when it comes time to produce run-time reflection information? I mean, when module A asks to create run-time reflection information for all the functions and types in module A.... er, I naively thought the information would be created as a set of types and functions *in module A*, which would then change the set of allMembers of A. But, maybe it makes more sense to create that stuff in a different module (which A could then import??)

Anyway, I can't even figure out how to enumerate the members of a module A; __traits(allMembers, A) causes "Error: import Y has no members".

Aside: I first wrote the above code as follows:

// Shouldn't this be in Phobos somewhere?
bool contains(alias pred = "a == b", R, E)(R haystack, E needle)
    if (isInputRange!R &&
        is(typeof(binaryFun!pred(haystack.front, needle)) : bool))
{
	return !(find!(pred, R, E)(haystack, needle).empty);
}

struct OhCrap {
	void a() {}
	static if ([ __traits(allMembers, OhCrap) ].contains("a")) {
		auto b() { return 2; }
	}
	void c() {}
}

But it causes a series of 204 error messages that I don't understand.

On Wednesday, 25 July 2012 at 19:54:31 UTC, David Piepgrass wrote:
>> It keeps diving deeper and deeper to find anything it can "start" with.
>> One it finds that, it'll just build everything back up in whatever
>> order is necessary.
>
> I hope someone can give more details about this.

TDPL chapter 11 "Scaling Up".

> If you use rdmd to compile (instead of dmd), you *just* give it
> your *one* main source file (typically the one with your "main()"
> function). This file must be the *last* parameter passed to rdmd:
>
> $rdmd --build-only (any other flags) main.d
>
> Then, RDMD will figure out *all* of the source files needed (using
> the full compiler's frontend, so it never gets fooled into missing
> anything), and if any of them have been changed, it will automatically
> pass them *all* into DMD for you. This way, you don't have to
> manually keep track of all your files and pass them all into
> DMD youself. Just give RDMD your main file and that's it, you're golden.

I meant to ask, why would it recompile *all* of the source files if only one changed? Seems like it only should recompile the changed ones (but still compile them together as a unit.) Is it because of bugs (e.g. the template problem you mentioned)?

>> I hope someone can give more details about this.
>
> TDPL chapter 11 "Scaling Up".

That's where I was looking. As I said already, TDPL does not explain how compilation works, especially not anything about the low-level semantic analysis which has me most curious.

On Wednesday, 25 July 2012 at 20:25:19 UTC, David Piepgrass wrote:
>>> I hope someone can give more details about this.
>>
>> TDPL chapter 11 "Scaling Up".
>
> That's where I was looking. As I said already, TDPL does not explain how compilation works, especially not anything about the low-level semantic analysis which has me most curious.

Strange, because it seems to me this chapter answers all your previous questions. What exact details are you interested in?

On Wednesday, 25 July 2012 at 08:06:23 UTC, Nick Sabalausky wrote:
> Yea, my understanding is that full-build times measured in days are (or
> used to be, don't know if they still are) also typical of high-budget
> C++-based videogames.

You must be thinking of full data rebuilds, not code recompiles.

There's no way a game could take over a day to compile and still produce an executable that would fit on a console.

Several minutes is more typical. Maybe up to 30 minutes in bad cases.

On Wed, 25 Jul 2012 23:20:04 +0200
"Peter Alexander" <peter.alexander.au@gmail.com> wrote:

> On Wednesday, 25 July 2012 at 08:06:23 UTC, Nick Sabalausky wrote:
> > Yea, my understanding is that full-build times measured in days
> > are (or
> > used to be, don't know if they still are) also typical of
> > high-budget
> > C++-based videogames.
> 
> You must be thinking of full data rebuilds, not code recompiles.
> 
> There's no way a game could take over a day to compile and still produce an executable that would fit on a console.
> 
> Several minutes is more typical. Maybe up to 30 minutes in bad cases.

Yea, you're probably right. I meant "full project", which almost certainly involves going through gigabytes of assets.

On Wed, 25 Jul 2012 22:18:37 +0200
"David Piepgrass" <qwertie256@gmail.com> wrote:
> 
> I meant to ask, why would it recompile *all* of the source files if only one changed? Seems like it only should recompile the changed ones (but still compile them together as a unit.) Is it because of bugs (e.g. the template problem you mentioned)?

I'm not 100% certain, but, yes, I think it's a combination of that, and the fact that nobody's actually gone and tried to make that change to RDMD yet.

AIUI, The original motivating purpose for RDMD was to be able to execute a D source file as if it were a script. So finding all relevant source files, passing them to DMD, etc, was all just necessary steps towards that end. Which turned out to also be useful in many cases for general project building.