On 4/7/13, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> This is a dramatic improvement, and better build times are always well received. Do you plan to convert your work into a pull request?

I've filed the enhancement: http://d.puremagic.com/issues/show_bug.cgi?id=9896

Perhaps you could mark it as pre-approved? I'm making a pull soon.

On Sunday, 7 April 2013 at 11:45:38 UTC, Andrej Mitrovic wrote:
> Would this work?

I think so, however the main challenge is that rdmd doesn't track dependencies between each module. It just has a list of files that take part in the build. We'd need to improve either the -v output (to include inter-module dependencies) or -deps output (to include import()-ed files) to provide that information to rdmd.

On 4/7/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
> We'd need to improve either the -v
> output (to include inter-module dependencies) or -deps output (to
> include import()-ed files) to provide that information to rdmd.
>

-v is a better choice, it will allow us to use pipes later.

On 4/7/13, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> Do you plan to convert your work into a pull request?

Pull made: https://github.com/D-Programming-Language/dmd/pull/1861

On 4/7/13, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:
> On 4/7/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
>> We'd need to improve either the -v
>> output (to include inter-module dependencies) or -deps output (to
>> include import()-ed files) to provide that information to rdmd.
>>
>
> -v is a better choice, it will allow us to use pipes later.
>

Even better is to make -deps without arguments just print to stdout. I'm making a pull for this soon.

On 04/05/2013 12:18 PM, Andrej Mitrovic wrote:
> On 4/5/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
>> it must run dmd with -v -o-, and read its verbose output.
>
> IIRC xfbuild used to work around this by both reading the verbose
> output and simultaneously building. In other words it used 'dmd -deps'
> without the -o- switch. There's a function called compileAndTrackDeps
> which did this, but the entire codebase was very hacky which has lead
> me to abandon the port.
>
> Note that using pipes instead of File I/O to read the verbose output
> of DMD can speed up the entire building process (the new std.process
> comes in handy for this).
>

It would be good to use IPC instead of an a priori -rb -rx argument list.
The idea is that whenever dmd imports a module it ask rdmd (or another driver) whether this module should be compiled. The driver could then check it's cache for an existing module-object to decide this.
The driver cannot decide this a priori because it's dependency information is outdated. It could only heuristically list available packages/modules.

1.
driver has no knowledge about foo.bar's dependencies:
dmd -c foo/bar -rb -rxstd.*
----
module foo.bar;

import std.stdio, foo.baz;

// ...
----

2.
driver assumes that foo.bar depends on foo.baz and has a cached obj for foo.baz:
dmd -c foo/bar -rb -rxstd.* -rxfoo.baz
----
module foo.bar;

import std.stdio, something.else;

// ...
----

On 4/7/13, Martin Nowak <code@dawg.eu> wrote:
> 2.
> driver assumes that foo.bar depends on foo.baz and has a cached obj for
> foo.baz:
> dmd -c foo/bar -rb -rxstd.* -rxfoo.baz

How is this a problem? If foo.baz is not needed, the linker won't link it in. All the driver has to do is list the modules which shouldn't be recompiled if their modification time hasn't changed.

On 04/07/2013 07:14 PM, Andrej Mitrovic wrote:
> On 4/7/13, Martin Nowak <code@dawg.eu> wrote:
>> 2.
>> driver assumes that foo.bar depends on foo.baz and has a cached obj for
>> foo.baz:
>> dmd -c foo/bar -rb -rxstd.* -rxfoo.baz
>
> How is this a problem? If foo.baz is not needed, the linker won't link
> it in. All the driver has to do is list the modules which shouldn't be
> recompiled if their modification time hasn't changed.
>
This is a problem because 'something.else' might be a huge library you don't need to recompile.
With the current approach you're first recompiling before you know the dependencies. It's also the reason why you hardcoded std.* and core.* into the exclude list. But this will miss other prebuild libraries, e.g. vibe.

On 4/7/13, Martin Nowak <code@dawg.eu> wrote:
> This is a problem because 'something.else' might be a huge library you
> don't need to recompile.
> With the current approach you're first recompiling before you know the
> dependencies. It's also the reason why you hardcoded std.* and core.*
> into the exclude list. But this will miss other prebuild libraries, e.g.
> vibe.

So pass -rxvibe vibe.lib on the first invocation. I think we're in some kind of miscommunication..

> So pass -rxvibe vibe.lib on the first invocation. I think we're in
> some kind of miscommunication..

You don't know that something uses vibe untilyou actually compile it.
Passing every cached library to the compiler is not a feasible approach. How do you pass the relevant linker flags to the compiler? How do you support parallelization?
Using IPC to negotiate this could handle these problems. You could think of it as using the compiler as library if you want to.