On Sunday, December 22, 2013 08:21:34 Russel Winder wrote:
> There seems a dichotomy between "explicitly compile each d file (aka module) and then link" versus "compile all the files (aka modules) at once" especially since the objects for each file/module are created in both cases.
> 
> I am assuming there are also some DMD/GDC/LDC2 differences as well.
> 
> Currently Dub is "compile all source at once" and SCons is "compile each file individually and then link" (though I am currently using SCons in a "compile all source at once" as an experiment to see what needs to happen to the SCons D build support.
> 
> Is there some critical feature which means that "compile all source at once" is a better route?

1. Unless you compile each of the separate files in parallel, it's faster to compile everything at once, because you avoid having to reprocess all of the imported modules. Parallel compiling still has to do all of that work, but the gains from compiling multiple at the same time will likely offset it.

2. As I understand it, there's at least one outstanding bug where the symbols can get screwed up if you build stuff separately. Unfortunately, I don't remember the details or the bug number, but it's generally brought up when anyone tries to promote doing incremental builds.

- Jonathan M Davis

On Sun, 2013-12-22 at 01:03 -0800, Jonathan M Davis wrote: […]
> 1. Unless you compile each of the separate files in parallel, it's faster to compile everything at once, because you avoid having to reprocess all of the imported modules. Parallel compiling still has to do all of that work, but the gains from compiling multiple at the same time will likely offset it.

I can see this being so where many files are changed between builds, but I am working in a situation where I want to build and run after each change of a file. So with large numbers of files only compiling one and linking seems faster than compiling all.

> 2. As I understand it, there's at least one outstanding bug where the symbols can get screwed up if you build stuff separately. Unfortunately, I don't remember the details or the bug number, but it's generally brought up when anyone tries to promote doing incremental builds.

It would be good to uncover this one in more detail as the current SCons build strategy is always separate compilation. If anyone can add to Jonathan's point here, that would be great.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On 2013-12-22 11:44, Russel Winder wrote:

> I can see this being so where many files are changed between builds, but
> I am working in a situation where I want to build and run after each
> change of a file. So with large numbers of files only compiling one and
> linking seems faster than compiling all.

There is a big chance this will result in link errors. I don't remember the exact problems but it's something like DMD doesn't emit template symbols to all object files it should. You can search these newsgroups for "incremental complication". It was quite easy to find:

http://www.digitalmars.com/d/archives/digitalmars/D/Incremental_compilation_with_DMD_96138.html

-- 
/Jacob Carlborg

On 2013-12-22 10:03, Jonathan M Davis wrote:

> 1. Unless you compile each of the separate files in parallel, it's faster to
> compile everything at once, because you avoid having to reprocess all of the
> imported modules. Parallel compiling still has to do all of that work, but the
> gains from compiling multiple at the same time will likely offset it.

This probably also depends on how many files that need to be compiled. If there are many files it might be a good idea to compile a couple of files in parallel. It might be interesting to do some experimentation.

-- 
/Jacob Carlborg

On Sunday, December 22, 2013 10:44:15 Russel Winder wrote:
> On Sun, 2013-12-22 at 01:03 -0800, Jonathan M Davis wrote: […]
> 
> > 1. Unless you compile each of the separate files in parallel, it's faster to compile everything at once, because you avoid having to reprocess all of the imported modules. Parallel compiling still has to do all of that work, but the gains from compiling multiple at the same time will likely offset it.
> I can see this being so where many files are changed between builds, but I am working in a situation where I want to build and run after each change of a file. So with large numbers of files only compiling one and linking seems faster than compiling all.

Possibly, but dmd is so fast that the linking portion is often what takes the longest, so I wouldn't even worry about separate compilation until I had a very large project. I wouldn't expect any real gain for smaller projects, and it's definitely the case that the compiler has to do more work when compiling each file separately, so without parallelizing builds, I fully expect that incremental compilation will be slower until a project gets fairly large. But you'd have to experiment to figure out what the exact performance characteristics are.

> > 2. As I understand it, there's at least one outstanding bug where the symbols can get screwed up if you build stuff separately. Unfortunately, I don't remember the details or the bug number, but it's generally brought up when anyone tries to promote doing incremental builds.
> 
> It would be good to uncover this one in more detail as the current SCons build strategy is always separate compilation. If anyone can add to Jonathan's point here, that would be great.

Searching for posts on incremental compilation in the newsgroup will likely turn it up fairly quickly. But when you add that bug on top of the fact that incremental compilation is unlikely to garner much in the way of performance gains except for larger projects, I never bother with incremental compilation at this point and wouldn't bother with it until compilation times were getting large.

- Jonathan M Davis

On Sun, 2013-12-22 at 13:14 +0100, Jacob Carlborg wrote: […]
> There is a big chance this will result in link errors. I don't remember the exact problems but it's something like DMD doesn't emit template symbols to all object files it should. You can search these newsgroups for "incremental complication". It was quite easy to find:
> 
> http://www.digitalmars.com/d/archives/digitalmars/D/Incremental_compilation_with_DMD_96138.html

I tend to use ldc2 or gdc rather than dmd, I suspect there are a whole different set of issues?

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Sunday, 22 December 2013 at 14:39:23 UTC, Russel Winder wrote:
> I tend to use ldc2 or gdc rather than dmd, I suspect there are a whole
> different set of issues?

Yes, at least gdc has own tweaks to symbol emitting schemes and some additionals flags to influence it.

Incremental compilation (as contrary to simply separate compilation) because of template instance dependency tracking. It has improved a bit with recent change to symbol emitting but this change was incomplete and faulty on its own, still causing linker errors sometimes ;)