Jump to page: 1 24  
Page
Thread overview
Please integrate build framework into the compiler
Mar 21, 2009
davidl
Mar 21, 2009
grauzone
Mar 21, 2009
grauzone
Mar 22, 2009
grauzone
Mar 22, 2009
davidl
Mar 22, 2009
Ary Borenszweig
Mar 23, 2009
grauzone
Mar 21, 2009
dsimcha
Mar 21, 2009
grauzone
Mar 21, 2009
Christopher Wright
Mar 21, 2009
BCS
Mar 21, 2009
Christopher Wright
Mar 22, 2009
davidl
Mar 22, 2009
grauzone
Mar 22, 2009
Kristian Kilpi
Mar 22, 2009
Christopher Wright
Mar 22, 2009
Kristian Kilpi
Mar 22, 2009
dennis luehring
Mar 23, 2009
Christopher Wright
Mar 24, 2009
Nick Sabalausky
Mar 24, 2009
bearophile
Mar 24, 2009
Nick Sabalausky
Mar 24, 2009
bearophile
Mar 24, 2009
Nick Sabalausky
Mar 24, 2009
bearophile
Mar 24, 2009
Christopher Wright
March 21, 2009
1. compiler know in what situation a file need to be recompiled

Consider the file given the same header file, then the obj file of this will be required for linking, all other files import this file shouldn't require any recompilation in this case. If a file's header file changes, thus the interface changes, all files import this file should be recompiled.
Compiler can emit building command like rebuild does.

I would enjoy:

dmd -buildingcommand abc.d  > responsefile

dmd @responsefile

I think we need to eliminate useless recompilation as much as we should with consideration of the growing d project size.

2. maintaining the build without compiler support costs

March 21, 2009
I don't really understand what you mean. But if you want the compiler to scan for dependencies, I fully agree.

I claim that we don't even need incremental compilation. It would be better if the compiler would scan for dependencies, and if a source file has changed, recompile the whole project in one go. This would be simple and efficient.

Here are some arguments that speak for this approach:

- A full compiler is the only piece of software that can build a correct/complete module dependency graph. This is because you need full semantic analysis to catch all import statements. For example, you can use a string mixin to generate import statements: mixin("import bla;"). No naive dependency scanner would be able to detect this import. You need CTFE capabilities, which require almost a full compiler. (Actually, dsss uses the dmd frontend for dependency scanning.)

- Speed. Incremental compilation is godawfully slow (10 times slower than to compile all files in one dmd invocation). You could pass all changed files to dmd at once, but this is broken and often causes linker errors (ask the dsss author for details lol). Recompiling the whole thing every time is faster.

- Long dependency chains. Unlike in C/C++, you can't separate a module into interface and implementation. Compared to C++, it's as if a change to one .c file triggers recompilation of a _lot_ of other .c files. This makes incremental compilation really look useless. Unless you move modules into libraries and use them through .di files.

I would even go so far to say, that dmd should automatically follow all imports and compile them in one go. This would be faster than having a separate responsefile step, because the source code needs to be analyzed only once. To prevent compilation of imported library headers, the compiler could provide a new include switch for library code. Modules inside "library" include paths wouldn't be compiled.

Hell, maybe I'll even manage to come up with a compiler patch, to turn this into reality.
March 21, 2009
grauzone wrote:
> I don't really understand what you mean. But if you want the compiler to scan for dependencies, I fully agree.
> 
> I claim that we don't even need incremental compilation. It would be better if the compiler would scan for dependencies, and if a source file has changed, recompile the whole project in one go. This would be simple and efficient.

That's precisely what rdmd does.

Andrei
March 21, 2009
== Quote from grauzone (none@example.net)'s article
> I claim that we don't even need incremental compilation. It would be better if the compiler would scan for dependencies, and if a source file has changed, recompile the whole project in one go. This would be simple and efficient.

I'm surprised that this could possibly be more efficient than incremental compilation, but I've never worked on a project large enough for compile times to be a major issue, so I've never really looked into this.

If incremental compilation were removed from the spec, meaning the compiler would always know about the whole program when compiling, I assume (correct me if I'm wrong) that would mean the following restrictions could be removed:

1.  std.traits could offer a way to get a tuple of all derived classes,
essentially the opposite of BaseTypeType.
2.  Since DMD would know about all derived classes when compiling the base class,
it would be feasible to allow templates to add virtual functions to classes.
IMHO, this would be an absolute godsend, as it is currently a _huge_ limitation of
templates.
3.  For the same reason, methods calls to classes with no derived classes could be
made directly instead of through the vtable.

Of course, these restrictions would still apply to libraries that use .di files. If incremental compilation is actually causing more problems than it solves anyhow, it would be great to get rid of it along with the annoying restrictions it creates.
March 21, 2009
Andrei Alexandrescu wrote:
> grauzone wrote:
>> I don't really understand what you mean. But if you want the compiler to scan for dependencies, I fully agree.
>>
>> I claim that we don't even need incremental compilation. It would be better if the compiler would scan for dependencies, and if a source file has changed, recompile the whole project in one go. This would be simple and efficient.
> 
> That's precisely what rdmd does.

This looks really good, but I couldn't get it to work. Am I doing something wrong?

--- o.d:
module o;

import tango.io.Stdout;

void k() {
    Stdout("foo").newline;
}

--- u.d:
module u;

import o;

void main() {
    k();
}



$ rdmd u.d
/tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o:(.data+0x38): undefined reference to `_D1o12__ModuleInfoZ'
/tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o: In function `_Dmain':
u.d:(.text._Dmain+0x4): undefined reference to `_D1o1kFZv'
collect2: ld returned 1 exit status
--- errorlevel 1
rdmd: Couldn't compile or execute u.d.

$ dmd|grep Compiler
Digital Mars D Compiler v1.041

> Andrei
March 21, 2009
dsimcha wrote:
> == Quote from grauzone (none@example.net)'s article
>> I claim that we don't even need incremental compilation. It would be
>> better if the compiler would scan for dependencies, and if a source file
>> has changed, recompile the whole project in one go. This would be simple
>> and efficient.
> 
> I'm surprised that this could possibly be more efficient than incremental
> compilation, but I've never worked on a project large enough for compile times to
> be a major issue, so I've never really looked into this.

Maybe incremental compilation could be faster, but dmd has a bug that forces tools like dsss/rebuild to use a slower method. Instead of invoking the compiler once to recompile all modules that depend from changed files, it has to start a new compiler process for each file.

> If incremental compilation were removed from the spec, meaning the compiler would
> always know about the whole program when compiling, I assume (correct me if I'm
> wrong) that would mean the following restrictions could be removed:
> 
> 1.  std.traits could offer a way to get a tuple of all derived classes,
> essentially the opposite of BaseTypeType.
> 2.  Since DMD would know about all derived classes when compiling the base class,
> it would be feasible to allow templates to add virtual functions to classes.
> IMHO, this would be an absolute godsend, as it is currently a _huge_ limitation of
> templates.
> 3.  For the same reason, methods calls to classes with no derived classes could be
> made directly instead of through the vtable.

And you could do all kinds of interprocedural optimizations.

> Of course, these restrictions would still apply to libraries that use .di files.
> If incremental compilation is actually causing more problems than it solves
> anyhow, it would be great to get rid of it along with the annoying restrictions it
> creates.

It seems Microsoft thought the same. C# goes without incremental compilation. But for now, D's build model is too similar to C/C++ as that you'd completely remove that ability.
March 21, 2009
Hello grauzone,

> I would even go so far to say, that dmd should automatically follow
> all imports and compile them in one go. This would be faster than
> having a separate responsefile step, because the source code needs to
> be analyzed only once. To prevent compilation of imported library
> headers, the compiler could provide a new include switch for library
> code. Modules inside "library" include paths wouldn't be compiled.

Adding that without a way to turn it off would kill D in some cases. I have a project where DMD uses up >30% of the available address space compiling one module. If I was forced to compile all modules at once, it might not work, end of story.

That said, for many cases, I don't see a problem with having that feature available.


March 21, 2009
dsimcha wrote:
> 1.  std.traits could offer a way to get a tuple of all derived classes,
> essentially the opposite of BaseTypeType.
> 2.  Since DMD would know about all derived classes when compiling the base class,
> it would be feasible to allow templates to add virtual functions to classes.
> IMHO, this would be an absolute godsend, as it is currently a _huge_ limitation of
> templates.
> 3.  For the same reason, methods calls to classes with no derived classes could be
> made directly instead of through the vtable.

This is only if there is no dynamic linking.
March 21, 2009
grauzone wrote:
> - Long dependency chains. Unlike in C/C++, you can't separate a module into interface and implementation. Compared to C++, it's as if a change to one .c file triggers recompilation of a _lot_ of other .c files. This makes incremental compilation really look useless. Unless you move modules into libraries and use them through .di files.

You can use interfaces for this, though that is not always possible.
March 22, 2009
grauzone wrote:
> Andrei Alexandrescu wrote:
>> grauzone wrote:
>>> I don't really understand what you mean. But if you want the compiler to scan for dependencies, I fully agree.
>>>
>>> I claim that we don't even need incremental compilation. It would be better if the compiler would scan for dependencies, and if a source file has changed, recompile the whole project in one go. This would be simple and efficient.
>>
>> That's precisely what rdmd does.
> 
> This looks really good, but I couldn't get it to work. Am I doing something wrong?
> 
> --- o.d:
> module o;
> 
> import tango.io.Stdout;
> 
> void k() {
>     Stdout("foo").newline;
> }
> 
> --- u.d:
> module u;
> 
> import o;
> 
> void main() {
>     k();
> }
> 
> 
> 
> $ rdmd u.d
> /tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o:(.data+0x38): undefined reference to `_D1o12__ModuleInfoZ'
> /tmp/u-1000-20-49158160-A46C236CDE107E3B9F053881E4257C2D.o: In function `_Dmain':
> u.d:(.text._Dmain+0x4): undefined reference to `_D1o1kFZv'
> collect2: ld returned 1 exit status
> --- errorlevel 1
> rdmd: Couldn't compile or execute u.d.
> 
> $ dmd|grep Compiler
> Digital Mars D Compiler v1.041

Should work, but I tested only with D2. You may want to pass --chatty to rdmd and see what commands it invokes.


Andrei
« First   ‹ Prev
1 2 3 4