On Sunday, 7 April 2013 at 00:23:06 UTC, Andrei Alexandrescu wrote:
> For larger projects I wonder if it's possible to restrict scanning only to specific directories. That way building becomes a modularity enforcer.

I'm not sure what this means. Could you expand a little, please?

On Saturday, 6 April 2013 at 17:52:40 UTC, Andrej Mitrovic wrote:
> But RDMD can track changes to dependencies, which DMD still can't do.
> I want to try implementing this feature in DMD and see the speed
> difference for incremental builds.

By "incremental builds", do you mean skipping compilation of certain modules, and re-using object files from previous builds?

Currently, rdmd deletes all generated modules after building, to save disk space, as those modules are not reused in subsequent builds. The only thing that rdmd caches, is the full list of source files in the program (so it knows which files to check to see if they've been edited), and the executable.

Actual incremental compilation with just one compiler invocation would certainly be a big speedup for large projects. As I understand, there were some problems with how dmd places data in object files that prevented implementing this, however that's been put into question recently:
http://forum.dlang.org/post/mailman.508.1365074861.4724.digitalmars-d@puremagic.com

I wonder if it's somehow related to Dicebot's findings in another recent thread:
http://forum.dlang.org/post/gxscuhrvcpnuzvgyykyp@forum.dlang.org

The compiler's verbose (-v) output, which is what rdmd currently uses, does not specify which modules depends on which - it simply lists all the files that took part in the compilation. There exists a separate option, -deps, which lists actual inter-module dependencies - judging by the identifiers, this is what rdmd used at some point. However, -deps output does not specify things such as import()-ed files. eskimor on IRC attempted to rectify that, here:
https://github.com/D-Programming-Language/dmd/pull/1839
However, as we've discussed, I think improving -v output would be better than adding a new switch to the compiler.

On 2013-04-06 16:22, Andrej Mitrovic wrote:

> It doesn't necessarily have to be dummy. I like to use an 'all.d'
> module which imports all of my library modules. I've used this
> technique to run unittest via "rdmd --main mylib\all.d". But being
> able to recursively compile a static library would be pretty nice.

Actually, just specifying a directory and having the compiler/some tool to compile all the files, recursively would be really nice to have.

-- 
/Jacob Carlborg

On 4/7/13, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> Do you plan to convert your work into a pull request?

Sure.

On 4/7/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
> By "incremental builds", do you mean skipping compilation of certain modules, and re-using object files from previous builds?

Yeah.

On 4/7/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
> Why not adapt rdmd so that it takes advantage of the new script?

Yeah, RDMD could take advantage of -rx to avoid building specific modules. It slipped my mind.

> Currently, rdmd deletes all generated modules after building, to save disk space, as those modules are not reused in subsequent builds.

Hmm.. ok so we have to add an incremental option to RDMD. If I'm not mistaken the build process for the following project would look like:

main.d -> foo.d -> bar.d -> doo.d

1. Invoke DMD to build recursively, and simultaneously fetch the dependencies to RDMD which will store the dependencies to a file:

$ dmd -rb main.d -v

2. On the second build (let's assume foo.d changed), RDMD would read the dependencies, check the modification time of each file, and knowing only foo.d has to be rebuilt it would run DMD again with:

$ dmd -rb main.d -rxbar bar.obj -rxdoo doo.obj

This assumes it keeps the object files on disk and doesn't delete them.

Would this work?

On 4/7/13, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
> As I understand, there were some problems with how dmd places data in object files that prevented implementing this, however that's been put into question recently:
> http://forum.dlang.org/post/mailman.508.1365074861.4724.digitalmars-d@puremagic.com

Yeah I can't reproduce this bug. h3r3tic also had some test-case somewhere which I couldn't reproduce[1].

[1] : https://bitbucket.org/h3r3tic/xfbuild/issue/7/make-incremental-building-reliable

On Sunday, 7 April 2013 at 07:29:26 UTC, Vladimir Panteleev wrote:
> I wonder if it's somehow related to Dicebot's findings in another recent thread:
> http://forum.dlang.org/post/gxscuhrvcpnuzvgyykyp@forum.dlang.org

I am pretty sure it is related. Current DMD logic for emitting template symbols is very naive and feels more like a hack that has several consequences in variety of cases. I am trying to create a more capable solution but have not been doing any dmd development before and this may take an eternity. Would be really glad to see some experienced dmd dev comment on topic.

On 4/7/13 12:26 AM, Vladimir Panteleev wrote:
> On Sunday, 7 April 2013 at 04:25:19 UTC, Andrei Alexandrescu wrote:
>> On 4/7/13 12:07 AM, Vladimir Panteleev wrote:
>>> On Sunday, 7 April 2013 at 02:16:10 UTC, Andrei Alexandrescu wrote:
>>>>> Why in parallel and not in one go? (-v with -rb and without -o-)
>>>>
>>>> I'm not sure how rdmd would distingish dmd's output from the output of
>>>> the running program. Oh wait, the run will be a distinct step -
>>>> awesome.
>>>
>>> I'll have a go at the rdmd side of this.
>>>
>>> Any reason not to use the deps file as the build witness?
>>
>> There was a reason, but darn I forgot it...
>
> I think I figured it out - --makedepend will update the deps file, but
> not the executable.

I think there's another one related to a bug I recently fixed. There needs to be a file touched after the executable has been built when people use -of to place the executable in a specific place.

https://github.com/D-Programming-Language/tools/commit/e14e0378375d1bb586871f8e66cc501dac64a7e1


Andrei

On 4/7/13 3:17 AM, Vladimir Panteleev wrote:
> On Sunday, 7 April 2013 at 00:23:06 UTC, Andrei Alexandrescu wrote:
>> For larger projects I wonder if it's possible to restrict scanning
>> only to specific directories. That way building becomes a modularity
>> enforcer.
>
> I'm not sure what this means. Could you expand a little, please?

I mean say we have a project consisting of business, graphics, and glue. We wouldn't want business and graphics to directly use each other, so they'd be each built with -I settings that make accidental dependencies impossible (in spite of the automatic dependency detection).

Andrei

On 4/7/13 7:45 AM, Andrej Mitrovic wrote:
> Hmm.. ok so we have to add an incremental option to RDMD. If I'm not
> mistaken the build process for the following project would look like:
>
> main.d ->  foo.d ->  bar.d ->  doo.d
>
> 1. Invoke DMD to build recursively, and simultaneously fetch the
> dependencies to RDMD which will store the dependencies to a file:
>
> $ dmd -rb main.d -v
>
> 2. On the second build (let's assume foo.d changed), RDMD would read
> the dependencies, check the modification time of each file, and
> knowing only foo.d has to be rebuilt it would run DMD again with:
>
> $ dmd -rb main.d -rxbar bar.obj -rxdoo doo.obj
>
> This assumes it keeps the object files on disk and doesn't delete them.
>
> Would this work?

Yah. One question would be how to store dependencies. One possibility is to flatly store transitive dependencies for each file (rdmd currently does that, but only for the main file). This would take care of circular dependencies automatically. One other option is to store only direct dependencies for each file, and then use a graph walker to get work done.

(I seem to recall Go is that it has a simple highly linearized organization of dependencies. If anyone knows more about that, please share.)


Andrei

On 4/7/13, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
>> 1. Invoke DMD to build recursively, and simultaneously fetch the dependencies to RDMD which will store the dependencies to a file:
>>
>> $ dmd -rb main.d -v

One thing I forgot to add were the -c and -op flags to make DMD actually generate separate object files, with paths to avoid conflicts.

On Sunday, 7 April 2013 at 11:52:54 UTC, Andrei Alexandrescu wrote:
> I think there's another one related to a bug I recently fixed. There needs to be a file touched after the executable has been built when people use -of to place the executable in a specific place.

Well, yes, that was when the witness file was added. My question was whether there was a reason for a separate file to be used, as opposed to using the .deps file. --makedepend would be the answer, as I see.