View mode: basic / threaded / horizontal-split · Log in · Help
March 21, 2009
Please integrate build framework into the compiler
1. compiler know in what situation a file need to be recompiled

Consider the file given the same header file, then the obj file of this  
will be required for linking, all other files import this file shouldn't  
require any recompilation in this case. If a file's header file changes,  
thus the interface changes, all files import this file should be  
recompiled.
Compiler can emit building command like rebuild does.

I would enjoy:

dmd -buildingcommand abc.d  > responsefile

dmd @responsefile

I think we need to eliminate useless recompilation as much as we should  
with consideration of the growing d project size.

2. maintaining the build without compiler support costs
March 21, 2009
Re: Please integrate build framework into the compiler
I don't really understand what you mean. But if you want the compiler to 
scan for dependencies, I fully agree.

I claim that we don't even need incremental compilation. It would be 
better if the compiler would scan for dependencies, and if a source file 
has changed, recompile the whole project in one go. This would be simple 
and efficient.

Here are some arguments that speak for this approach:

- A full compiler is the only piece of software that can build a 
correct/complete module dependency graph. This is because you need full 
semantic analysis to catch all import statements. For example, you can 
use a string mixin to generate import statements: mixin("import bla;"). 
No naive dependency scanner would be able to detect this import. You 
need CTFE capabilities, which require almost a full compiler. (Actually, 
dsss uses the dmd frontend for dependency scanning.)

- Speed. Incremental compilation is godawfully slow (10 times slower 
than to compile all files in one dmd invocation). You could pass all 
changed files to dmd at once, but this is broken and often causes linker 
errors (ask the dsss author for details lol). Recompiling the whole 
thing every time is faster.

- Long dependency chains. Unlike in C/C++, you can't separate a module 
into interface and implementation. Compared to C++, it's as if a change 
to one .c file triggers recompilation of a _lot_ of other .c files. This 
makes incremental compilation really look useless. Unless you move 
modules into libraries and use them through .di files.

I would even go so far to say, that dmd should automatically follow all 
imports and compile them in one go. This would be faster than having a 
separate responsefile step, because the source code needs to be analyzed 
only once. To prevent compilation of imported library headers, the 
compiler could provide a new include switch for library code. Modules 
inside "library" include paths wouldn't be compiled.

Hell, maybe I'll even manage to come up with a compiler patch, to turn 
this into reality.
March 21, 2009
Re: Please integrate build framework into the compiler
grauzone wrote:
> I don't really understand what you mean. But if you want the compiler to 
> scan for dependencies, I fully agree.
> 
> I claim that we don't even need incremental compilation. It would be 
> better if the compiler would scan for dependencies, and if a source file 
> has changed, recompile the whole project in one go. This would be simple 
> and efficient.

That's precisely what rdmd does.

Andrei
March 21, 2009
Re: Please integrate build framework into the compiler
== Quote from grauzone (none@example.net)'s article
> I claim that we don't even need incremental compilation. It would be
> better if the compiler would scan for dependencies, and if a source file
> has changed, recompile the whole project in one go. This would be simple
> and efficient.

I'm surprised that this could possibly be more efficient than incremental
compilation, but I've never worked on a project large enough for compile times to
be a major issue, so I've never really looked into this.

If incremental compilation were removed from the spec, meaning the compiler would
always know about the whole program when compiling, I assume (correct me if I'm
wrong) that would mean the following restrictions could be removed:

1.  std.traits could offer a way to get a tuple of all derived classes,
essentially the opposite of BaseTypeType.
2.  Since DMD would know about all derived classes when compiling the base class,
it would be feasible to allow templates to add virtual functions to classes.
IMHO, this would be an absolute godsend, as it is currently a _huge_ limitation of
templates.
3.  For the same reason, methods calls to classes with no derived classes could be
made directly instead of through the vtable.

Of course, these restrictions would still apply to libraries that use .di files.
If incremental compilation is actually causing more problems than it solves
anyhow, it would be great to get rid of it along with the annoying restrictions it
creates.
March 21, 2009
Re: Please integrate build framework into the compiler
Andrei Alexandrescu wrote:
> grauzone wrote:
>> I don't really understand what you mean. But if you want the compiler 
>> to scan for dependencies, I fully agree.
>>
>> I claim that we don't even need incremental compilation. It would be 
>> better if the compiler would scan for dependencies, and if a source 
>> file has changed, recompile the whole project in one go. This would be 
>> simple and efficient.
> 
> That's precisely what rdmd does.

This looks really good, but I couldn't get it to work. Am I doing 
something wrong?

--- o.d:
module o;

import tango.io.Stdout;

void k() {
    Stdout("foo").newline;
}

--- u.d:
module u;

import o;

void main() {
    k();
}



$ rdmd u.d
/tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o:(.data+0x38): 
undefined reference to `_D1o12__ModuleInfoZ'
/tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o: In function 
`_Dmain':
u.d:(.text._Dmain+0x4): undefined reference to `_D1o1kFZv'
collect2: ld returned 1 exit status
--- errorlevel 1
rdmd: Couldn't compile or execute u.d.

$ dmd|grep Compiler
Digital Mars D Compiler v1.041

> Andrei
March 21, 2009
Re: Please integrate build framework into the compiler
dsimcha wrote:
> == Quote from grauzone (none@example.net)'s article
>> I claim that we don't even need incremental compilation. It would be
>> better if the compiler would scan for dependencies, and if a source file
>> has changed, recompile the whole project in one go. This would be simple
>> and efficient.
> 
> I'm surprised that this could possibly be more efficient than incremental
> compilation, but I've never worked on a project large enough for compile times to
> be a major issue, so I've never really looked into this.

Maybe incremental compilation could be faster, but dmd has a bug that 
forces tools like dsss/rebuild to use a slower method. Instead of 
invoking the compiler once to recompile all modules that depend from 
changed files, it has to start a new compiler process for each file.

> If incremental compilation were removed from the spec, meaning the compiler would
> always know about the whole program when compiling, I assume (correct me if I'm
> wrong) that would mean the following restrictions could be removed:
> 
> 1.  std.traits could offer a way to get a tuple of all derived classes,
> essentially the opposite of BaseTypeType.
> 2.  Since DMD would know about all derived classes when compiling the base class,
> it would be feasible to allow templates to add virtual functions to classes.
> IMHO, this would be an absolute godsend, as it is currently a _huge_ limitation of
> templates.
> 3.  For the same reason, methods calls to classes with no derived classes could be
> made directly instead of through the vtable.

And you could do all kinds of interprocedural optimizations.

> Of course, these restrictions would still apply to libraries that use .di files.
> If incremental compilation is actually causing more problems than it solves
> anyhow, it would be great to get rid of it along with the annoying restrictions it
> creates.

It seems Microsoft thought the same. C# goes without incremental 
compilation. But for now, D's build model is too similar to C/C++ as 
that you'd completely remove that ability.
March 21, 2009
Re: Please integrate build framework into the compiler
Hello grauzone,

> I would even go so far to say, that dmd should automatically follow
> all imports and compile them in one go. This would be faster than
> having a separate responsefile step, because the source code needs to
> be analyzed only once. To prevent compilation of imported library
> headers, the compiler could provide a new include switch for library
> code. Modules inside "library" include paths wouldn't be compiled.

Adding that without a way to turn it off would kill D in some cases. I have 
a project where DMD uses up >30% of the available address space compiling 
one module. If I was forced to compile all modules at once, it might not 
work, end of story.

That said, for many cases, I don't see a problem with having that feature 
available.
March 21, 2009
Re: Please integrate build framework into the compiler
dsimcha wrote:
> 1.  std.traits could offer a way to get a tuple of all derived classes,
> essentially the opposite of BaseTypeType.
> 2.  Since DMD would know about all derived classes when compiling the base class,
> it would be feasible to allow templates to add virtual functions to classes.
> IMHO, this would be an absolute godsend, as it is currently a _huge_ limitation of
> templates.
> 3.  For the same reason, methods calls to classes with no derived classes could be
> made directly instead of through the vtable.

This is only if there is no dynamic linking.
March 21, 2009
Re: Please integrate build framework into the compiler
grauzone wrote:
> - Long dependency chains. Unlike in C/C++, you can't separate a module 
> into interface and implementation. Compared to C++, it's as if a change 
> to one .c file triggers recompilation of a _lot_ of other .c files. This 
> makes incremental compilation really look useless. Unless you move 
> modules into libraries and use them through .di files.

You can use interfaces for this, though that is not always possible.
March 22, 2009
Re: Please integrate build framework into the compiler
grauzone wrote:
> Andrei Alexandrescu wrote:
>> grauzone wrote:
>>> I don't really understand what you mean. But if you want the compiler 
>>> to scan for dependencies, I fully agree.
>>>
>>> I claim that we don't even need incremental compilation. It would be 
>>> better if the compiler would scan for dependencies, and if a source 
>>> file has changed, recompile the whole project in one go. This would 
>>> be simple and efficient.
>>
>> That's precisely what rdmd does.
> 
> This looks really good, but I couldn't get it to work. Am I doing 
> something wrong?
> 
> --- o.d:
> module o;
> 
> import tango.io.Stdout;
> 
> void k() {
>     Stdout("foo").newline;
> }
> 
> --- u.d:
> module u;
> 
> import o;
> 
> void main() {
>     k();
> }
> 
> 
> 
> $ rdmd u.d
> /tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o:(.data+0x38): 
> undefined reference to `_D1o12__ModuleInfoZ'
> /tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o: In function 
> `_Dmain':
> u.d:(.text._Dmain+0x4): undefined reference to `_D1o1kFZv'
> collect2: ld returned 1 exit status
> --- errorlevel 1
> rdmd: Couldn't compile or execute u.d.
> 
> $ dmd|grep Compiler
> Digital Mars D Compiler v1.041

Should work, but I tested only with D2. You may want to pass --chatty to 
rdmd and see what commands it invokes.


Andrei
« First   ‹ Prev
1 2 3 4
Top | Discussion index | About this forum | D home