On Sunday, 10 March 2013 at 13:40:04 UTC, Rainer Schuetze wrote:
> My usual estimate is about twice as fast,

That's still a huge difference. For one compiler to beat another 200% at something that isn't a microbenchmark isn't something you hear about often.

> but it depends on what you compile. It doesn't have a huge effect on running the test suite, my guess is that the runtime initialization for the MS build is slightly slower than for the dmc build, and there are a large number of small files to compile there.

But we're looking at the combined compilation numbers!

> Also, it's quite difficult to get accurate and reproducable benchmarking numbers these days, with the (mobile) processors continuously changing their performance.

You don't need an accurate benchmark to notice something is about 3 times as fast as something else....

Am Sat, 09 Mar 2013 23:29:33 -0500
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> I figure http://d.puremagic.com/issues/show_bug.cgi?id=9673 it's a great relatively confined project of good utility. We preapproved it, if anyone wants to snatch it please come forward.
> 
> Also, any comments to the design are welcome.
> 
> 
> Thanks,
> 
> Andrei

That makes me think of a hierarchy immediately:

                main.d+
               /      \
          abc.d+       def.d
         /     \
    hij.d*      klm.d

A change in hij.d bubbles up to abc.d and finally to main.d. This is much simpler than a full analysis of whether the change in hij.d really has effects up to main.d.

"dmd -c main.d abc.d hij.d" would then rebuild the set.

Caveat: A directed graph with no cycles is over-simplistic. Someone with a large project could perhaps tell what the average % of rebuilt modules would be.

-- 
Marco

On Sunday, 10 March 2013 at 15:04:42 UTC, Marco Leise wrote:
> Caveat: A directed graph with no cycles is over-simplistic.
> Someone with a large project could perhaps tell what the
> average % of rebuilt modules would be.

It is 100% as long as this bug exists : http://d.puremagic.com/issues/show_bug.cgi?id=9571

On Sunday, 10 March 2013 at 15:11:06 UTC, deadalnix wrote:
> On Sunday, 10 March 2013 at 15:04:42 UTC, Marco Leise wrote:
>> Caveat: A directed graph with no cycles is over-simplistic.
>> Someone with a large project could perhaps tell what the
>> average % of rebuilt modules would be.
>
> It is 100% as long as this bug exists : http://d.puremagic.com/issues/show_bug.cgi?id=9571

Doesn't this bug make incremental compilation (in projects where it is encountered) impossible, rather than simply impractical?

On Sunday, 10 March 2013 at 15:30:13 UTC, Vladimir Panteleev wrote:
> On Sunday, 10 March 2013 at 15:11:06 UTC, deadalnix wrote:
>> On Sunday, 10 March 2013 at 15:04:42 UTC, Marco Leise wrote:
>>> Caveat: A directed graph with no cycles is over-simplistic.
>>> Someone with a large project could perhaps tell what the
>>> average % of rebuilt modules would be.
>>
>> It is 100% as long as this bug exists : http://d.puremagic.com/issues/show_bug.cgi?id=9571
>
> Doesn't this bug make incremental compilation (in projects where it is encountered) impossible, rather than simply impractical?

It really depend on the coding style. If you do some C-like D, everything is fine, but if you use complex feature of D, then incremental compilation become clearly impossible (I spent a fair amount of time on the problem).

On Sunday, 10 March 2013 at 15:38:42 UTC, deadalnix wrote:
> It really depend on the coding style. If you do some C-like D, everything is fine, but if you use complex feature of D, then incremental compilation become clearly impossible (I spent a fair amount of time on the problem).

I think this is a serious problem. I hadn't thought of it before, but if we are designing our tools to work around implementation issues in the compiler, I think we're doing something wrong. Rather than meddling with a crippled incremental compilation model for rdmd that'll get obsoleted by a fixed compiler, how about attacking the problem directly?

It doesn't help that the problems surrounding incremental compilation (I mean the general case with incrementally compiling a few modules at once, not deadalnix's bug) don't seem to be well-defined. Do we have a filed issue with a reproducible test case?

Am Sun, 10 Mar 2013 16:44:36 +0100
schrieb "Vladimir Panteleev" <vladimir@thecybershadow.net>:

> I think this is a serious problem. I hadn't thought of it before, but if we are designing our tools to work around implementation issues in the compiler, I think we're doing something wrong. Rather than meddling with a crippled incremental compilation model for rdmd that'll get obsoleted by a fixed compiler, how about attacking the problem directly?

+1, Mono-D is currently in the middle of a refactoring to use rdmd to better handle project builds. Incremental builds were possible in earlier versions, but caused the known problems.

> It doesn't help that the problems surrounding incremental compilation (I mean the general case with incrementally compiling a few modules at once, not deadalnix's bug) don't seem to be well-defined. Do we have a filed issue with a reproducible test case?

Maybe this _class_ of bug wasn't considered before.
You just need to have one module with a template and another
one using it. If you change the template, the template module
will be recompiled (generating no code to speak of), while the
other file that actually instantiates the template remains
untouched.
Incremental builds end up with either outdated template
instances or linker errors until you force a rebuild.

-- 
Marco

On 3/10/13 12:42 PM, Marco Leise wrote:
> Maybe this _class_ of bug wasn't considered before.
> You just need to have one module with a template and another
> one using it. If you change the template, the template module
> will be recompiled (generating no code to speak of), while the
> other file that actually instantiates the template remains
> untouched.
> Incremental builds end up with either outdated template
> instances or linker errors until you force a rebuild.

But the module using the template will transitively depend on the one defining the template, so it will be rebuilt as well.

Andrei

On 3/10/13 11:44 AM, Vladimir Panteleev wrote:
> On Sunday, 10 March 2013 at 15:38:42 UTC, deadalnix wrote:
>> It really depend on the coding style. If you do some C-like D,
>> everything is fine, but if you use complex feature of D, then
>> incremental compilation become clearly impossible (I spent a fair
>> amount of time on the problem).
>
> I think this is a serious problem. I hadn't thought of it before, but if
> we are designing our tools to work around implementation issues in the
> compiler, I think we're doing something wrong. Rather than meddling with
> a crippled incremental compilation model for rdmd that'll get obsoleted
> by a fixed compiler, how about attacking the problem directly?
>
> It doesn't help that the problems surrounding incremental compilation (I
> mean the general case with incrementally compiling a few modules at
> once, not deadalnix's bug) don't seem to be well-defined. Do we have a
> filed issue with a reproducible test case?

Agreed. We need a workable item here.

Andrei

On 3/10/13 7:25 AM, Rainer Schuetze wrote:
> I don't have support for building on multiple cores, but trying it on
> visuald itself (48 files) yields
>
> - combined compilation 6s
> - single file compilation 1min4s
>
> You'd need a lot of cores to be better off with single file compilation.
>
> These are only the plugin files, not anything in the used libraries
> (about 300 more files). Using dmd compiled with dmc instead of cl makes
> these times 17s and 1min39s respectively)
>
> Almost any change causes a lot of files to be rebuilt (just tried one,
> took 49s to build).

I understand. However, I don't think that's not enough support for generalization.

Phobos is 197KLOC. At work I work on a C++ project that has k more lines of code, where k is a small number. Phobos uses separate compilation for its unittests, which are quite thorough, and compiling and running them all with make -j8 takes under two minutes on one single laptop. (The use of -j is crucial.) Building from scratch the C++ project is a small tectonic event involving dozens of machines and lasting much longer than k times more.

A large project is also more likely to manage dependencies in ways that reduce impact of individual file changes, something that apparently isn't the case for visuald.

What I'm saying here is that incremental builds are a valid choice for certain projects, and is possibly a gating factor to building large codebases with rdmd.


Andrei