Am Sun, 10 Mar 2013 13:04:06 -0400
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> But the module using the template will transitively depend on the one defining the template, so it will be rebuilt as well.
> 
> Andrei

That's true. I was referring to the status-quo, not the imagined solution.

-- 
Marco

On 3/10/13, Rainer Schuetze <r.sagitario@gmx.de> wrote:
> I think this is because without using di-files, a
> lot more code has to be analyzed for each compilation unit.

I used to think otherwise, and I wasn't wrong, however DMD has gotten progressively faster at compiling all at once:

2.050:
E:\dev\projects\WindowsAPI>timeit build.exe fullbuild
Elapsed Time:     0:00:28.370

2.050:
E:\dev\projects\WindowsAPI>timeit build.exe multicore
Elapsed Time:     0:00:14.210

It would seem like parallel builds are the way to go. But if you use a newer compiler:

2.062:
E:\dev\projects\WindowsAPI>timeit build.exe fullbuild
Elapsed Time:     0:00:14.971

2.062:
E:\dev\projects\WindowsAPI>timeit build.exe multicore
Elapsed Time:     0:00:15.061

So now full builds have become faster.

The repository: https://github.com/AndrejMitrovic/WindowsAPI

Here's the build script, which you first have to compile with a relatively recent compiler (2.050 didn't have std.parallelism or lambda syntax): http://dpaste.dzfl.pl/083247a2

Also pasted here:

import std.algorithm;
import std.array;
import std.exception;
import std.file;
import std.parallelism;
import std.path;
import std.process;
import std.string;

alias std.string.join join;

void main(string[] args)
{
    args.popFront();
    enforce(args.length);

    string[] mods = map!(a => a.name)(dirEntries(r".\win32",
SpanMode.shallow)).array;
    string flags = "-version=Unicode -version=WindowsXP";

    if (args.front == "multicore")
    {
        foreach (mod; parallel(mods))
        {
            string cmd = format("dmd -c %s %s", mod, flags);
            system(cmd);
        }

        auto objs = map!(a => a.baseName.setExtension(".obj"))(mods);
        string cmd = format("dmd -lib -ofmulti_win32.lib %s %s",
flags, objs.join(" "));
        system(cmd);
    }
    else
    if (args.front == "fullbuild")
    {
        string cmd = format("dmd -lib -offull_win32.lib %s %s", flags,
mods.join(" "));
        system(cmd);
    }
}

On 10.03.2013 15:07, Vladimir Panteleev wrote:
> On Sunday, 10 March 2013 at 13:35:34 UTC, Rainer Schuetze wrote:
>> Looks pretty ok, but considering the number of modules in dfeed (I
>> count about 24) and them being not very large, that makes compilation
>> speed for each module about 1 second. It will only be faster if the
>> number of modules to compile does not exceed twice the number of cores
>> available.
>
> ~/DFeed$ cat all.txt | wc -l
> 62

A, I didn't notice the ae link. I was already suspecting that 1 second/module was a bit long.

>
>> I think it does not scale well with increasing numbers of modules.
>
> Why? Wouldn't it scale linearly? Or you mean due to the increased number
> of graph edges when increasing graph nodes?
>

I assume that while a project grows, module dependencies also increase. So each single compile will get slower while the number of modules grow. Full build scales linearly with code size, but single file compilation time increases more (as a function of code size, module count and dependencies).

> Anyway, the programmer can take steps in lessening intermodule
> dependencies to reduce incremental build times. That's not an option
> with compiling everything at once, unless you split the code manually
> into libraries.

True.

On 10.03.2013 15:11, Vladimir Panteleev wrote:
> On Sunday, 10 March 2013 at 13:40:04 UTC, Rainer Schuetze wrote:
>> My usual estimate is about twice as fast,
>
> That's still a huge difference. For one compiler to beat another 200% at
> something that isn't a microbenchmark isn't something you hear about often.

I thought that factor 2 was common knowledge. See also this compiler comparison: http://www.willus.com/ccomp_benchmark2.shtml?p18+s14

>
>> but it depends on what you compile. It doesn't have a huge effect on
>> running the test suite, my guess is that the runtime initialization
>> for the MS build is slightly slower than for the dmc build, and there
>> are a large number of small files to compile there.
>
> But we're looking at the combined compilation numbers!

Compilation itself takes only a little part of the test suite. Most of it is creating and initializing compiler processes and executing the tests.
I've redone the test suite comparison (quick test single core): 2:38 min for msc, 2:32 for dmc, so no big difference, dmc even wins. I disabled turbo boost for the i7, but temperature control still throttled the CPU, so accuracy is not very good.

On 10.03.2013 18:13, Andrei Alexandrescu wrote:
> On 3/10/13 7:25 AM, Rainer Schuetze wrote:
>> I don't have support for building on multiple cores, but trying it on
>> visuald itself (48 files) yields
>>
>> - combined compilation 6s
>> - single file compilation 1min4s
>>
>> You'd need a lot of cores to be better off with single file compilation.
>>
>> These are only the plugin files, not anything in the used libraries
>> (about 300 more files). Using dmd compiled with dmc instead of cl makes
>> these times 17s and 1min39s respectively)
>>
>> Almost any change causes a lot of files to be rebuilt (just tried one,
>> took 49s to build).
>
> I understand. However, I don't think that's not enough support for
> generalization.
>
> Phobos is 197KLOC. At work I work on a C++ project that has k more lines
> of code, where k is a small number. Phobos uses separate compilation for
> its unittests, which are quite thorough, and compiling and running them
> all with make -j8 takes under two minutes on one single laptop. (The use
> of -j is crucial.) Building from scratch the C++ project is a small
> tectonic event involving dozens of machines and lasting much longer than
> k times more.

I agree that rebuild times are usually much faster with D than in C++.

But please also consider the usual edit-debug-build cycle. At, work, where our C++ code base is maybe 100 times larger than Visual D, build times are usually below 10 seconds for edits to a few source files. My point is that import dependencies in D are more viral than C++ headers because you cannot even remove them in di-files (this would break initialization order).

>
> A large project is also more likely to manage dependencies in ways that
> reduce impact of individual file changes, something that apparently
> isn't the case for visuald.
>
> What I'm saying here is that incremental builds are a valid choice for
> certain projects, and is possibly a gating factor to building large
> codebases with rdmd.

I agree that incremental compilation would be good to have, I'm just saying that single file compilation is not the solution. We should aim at making it work to recompile all dependent files in one compiler invocation.

On 3/10/13 4:41 PM, Rainer Schuetze wrote:
> My
> point is that import dependencies in D are more viral than C++ headers
> because you cannot even remove them in di-files (this would break
> initialization order).

This is new. Could you please elaborate?

Thanks,

Andrie

On 03/11/2013 12:49 AM, Andrei Alexandrescu wrote:
> On 3/10/13 4:41 PM, Rainer Schuetze wrote:
>> My
>> point is that import dependencies in D are more viral than C++ headers
>> because you cannot even remove them in di-files (this would break
>> initialization order).
>
> This is new. Could you please elaborate?
>
> Thanks,
>
> Andrie
>

It is not true. The initialization order is determined at run time.

On Sunday, 10 March 2013 at 16:42:52 UTC, Marco Leise wrote:
> Maybe this _class_ of bug wasn't considered before.
> You just need to have one module with a template and another
> one using it. If you change the template, the template module
> will be recompiled (generating no code to speak of), while the
> other file that actually instantiates the template remains
> untouched.
> Incremental builds end up with either outdated template
> instances or linker errors until you force a rebuild.

I don't think this is the problem that we're dealing with here. Don't quote me on it, but if I recall collectly, the problem had to do with DMD emitting template instantiations only to one arbitrary object file out of the all compiled at the moment.

On 11.03.2013 01:15, Timon Gehr wrote:
> On 03/11/2013 12:49 AM, Andrei Alexandrescu wrote:
>> On 3/10/13 4:41 PM, Rainer Schuetze wrote:
>>> My
>>> point is that import dependencies in D are more viral than C++ headers
>>> because you cannot even remove them in di-files (this would break
>>> initialization order).
>>
>> This is new. Could you please elaborate?
>>
>> Thanks,
>>
>> Andrie
>>
>
> It is not true. The initialization order is determined at run time.

An import is listed in the module info struct as a dependency, if it has static constructors (or destructors) or if it imports modules that have static constructors. If a di-file misses an import, it might not end up in the imported modules list of an importing module, so the correct order cannot be determined at runtime. ( Search the dmd source for "needmoduleinfo", especially https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L150 and https://github.com/D-Programming-Language/dmd/blob/master/src/import.c#L344 )

Also, you might run into trouble when removing static constructors/destructors from the di-file because of this.

On 03/11/2013 07:51 AM, Rainer Schuetze wrote:
>
>
> On 11.03.2013 01:15, Timon Gehr wrote:
>> On 03/11/2013 12:49 AM, Andrei Alexandrescu wrote:
>>> On 3/10/13 4:41 PM, Rainer Schuetze wrote:
>>>> My
>>>> point is that import dependencies in D are more viral than C++ headers
>>>> because you cannot even remove them in di-files (this would break
>>>> initialization order).
>>>
>>> This is new. Could you please elaborate?
>>>
>>> Thanks,
>>>
>>> Andrie
>>>
>>
>> It is not true. The initialization order is determined at run time.
>
> An import is listed in the module info struct as a dependency, if it has
> static constructors (or destructors) or if it imports modules that have
> static constructors.
> If a di-file misses an import, it might not end up
> in the imported modules list of an importing module, so the correct
> order cannot be determined at runtime.
> ( Search the dmd source for "needmoduleinfo", especially
> https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L150
> and
> https://github.com/D-Programming-Language/dmd/blob/master/src/import.c#L344
> )
>
> Also, you might run into trouble when removing static
> constructors/destructors from the di-file because of this.
>

I see, thanks! IMO this is ridiculous. I'd argue it is an implementation bug and should be fixed.