A quick straw poll.  Do people prefer to have all sources compiled in a single compiler call, or (more like C++) separate compilation of each object followed by a link call.

Thanks.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
> Do people prefer to have all sources compiled in a
> single compiler call

I prefer the single call.

On 12/15/12 11:55 AM, Russel Winder wrote:
> A quick straw poll.  Do people prefer to have all sources compiled in a
> single compiler call, or (more like C++) separate compilation of each
> object followed by a link call.

In phobos we use a single call for building the library. Then (at least on Posix) we use multiple calls for running unittests.

Andrei

On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
> A quick straw poll.  Do people prefer to have all sources compiled in a
> single compiler call, or (more like C++) separate compilation of each
> object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander wrote:
> On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
>> A quick straw poll.  Do people prefer to have all sources compiled in a
>> single compiler call, or (more like C++) separate compilation of each
>> object followed by a link call.
>
> Single compiler call is easier for small projects, but I worry about compile times for larger projects...


As evident by Phobos and my own project[1], for larger projects multiple concurrent calls is the only way to go. Rebuilding everything does take a bit, but a bit of thought behind the layout of the project and things work much faster when working on specific areas.

Cheers, Jakob.

[1] http://github.com/Charged/Miners

On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander wrote:
> On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
>> A quick straw poll.  Do people prefer to have all sources compiled in a
>> single compiler call, or (more like C++) separate compilation of each
>> object followed by a link call.
>
> Single compiler call is easier for small projects, but I worry about compile times for larger projects...

Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts), and compiling each source separately to an object, and then linking everything will allow easily to make the build parallel, dividing the sources to compile in various threads. Or the compiler already does that if I pass all source files in one call?

-- Renato Utsch

On Saturday, 15 December 2012 at 17:27:38 UTC, Jakob Bornecrantz wrote:
> On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
>> Single compiler call is easier for small projects, but I worry about compile times for larger projects...
>
> As evident by Phobos and my own project[1], for larger projects multiple concurrent calls is the only way to go. Rebuilding everything does take a bit, but a bit of thought behind the layout of the project and things work much faster when working on specific areas.

Phobos is only around 200kloc. I'm worrying about the really large projects (multi-million lines of code).

On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:
> On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander wrote:
> >On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
> >>A quick straw poll.  Do people prefer to have all sources compiled in a single compiler call, or (more like C++) separate compilation of each object followed by a link call.
> >
> >Single compiler call is easier for small projects, but I worry about compile times for larger projects...
> 
> Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts), and compiling each source separately to an object, and then linking everything will allow easily to make the build parallel, dividing the sources to compile in various threads. Or the compiler already does that if I pass all source files in one call?
[...]

I find that the current front-end (common to dmd, gdc, ldc) tends to work better when passed multiple source files at once. It tends to be faster, presumably because it only has to parse commonly-imported files once, and also produces smaller object/executable sizes -- maybe due to fewer duplicated template instantiations? I'm not sure of the exact reasons, but this behaviour appears consistent throughout dmd and gdc, and I presume also ldc (I didn't test that). So based on this, I'd lean toward compiling multiple files at once.

However, in very large project, clearly this won't work very well. If it takes half an hour to build the entire system, it makes the code - compile - test cycle very slow, which reduces programmer productivity.

So perhaps one possible middle ground would be to link packages separately, but compile all the sources within a single package at once. Presumably, if the project is properly organized, recompiling a single package won't take too long, and has the perk of optimizing for size within packages. This will probably also map to SCons easily, since SCons builds per-directory.


T

-- 
They pretend to pay us, and we pretend to work. -- Russian saying

On Saturday, 15 December 2012 at 17:31:19 UTC, RenatoUtsch wrote:
> On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander wrote:
>> On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
>>> A quick straw poll.  Do people prefer to have all sources compiled in a
>>> single compiler call, or (more like C++) separate compilation of each
>>> object followed by a link call.
>>
>> Single compiler call is easier for small projects, but I worry about compile times for larger projects...
>
> Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts)

I took a look at your github project, there isn't any code yet, but I like the concept. I was actually planing to do something similar, but since you are already doing it, I think my time would be better spent contributing to your project. Will there be some publicly available code in the near future?

On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:
> On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:
>> On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
>> wrote:
>> >On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder
>> >wrote:
>> >>A quick straw poll.  Do people prefer to have all sources compiled
>> >>in a single compiler call, or (more like C++) separate compilation
>> >>of each object followed by a link call.
>> >
>> >Single compiler call is easier for small projects, but I worry
>> >about compile times for larger projects...
>> 
>> Yes, I'm writing a build system for D (that will be pretty damn
>> good, I think, it has some interesting new concepts), and compiling
>> each source separately to an object, and then linking everything
>> will allow easily to make the build parallel, dividing the sources
>> to compile in various threads. Or the compiler already does that if
>> I pass all source files in one call?
> [...]
>
> I find that the current front-end (common to dmd, gdc, ldc) tends to
> work better when passed multiple source files at once. It tends to be
> faster, presumably because it only has to parse commonly-imported files
> once, and also produces smaller object/executable sizes -- maybe due to
> fewer duplicated template instantiations? I'm not sure of the exact
> reasons, but this behaviour appears consistent throughout dmd and gdc,
> and I presume also ldc (I didn't test that). So based on this, I'd lean
> toward compiling multiple files at once.

Yeah, I did read about this somewhee.

> However, in very large project, clearly this won't work very well. If it
> takes half an hour to build the entire system, it makes the code -
> compile - test cycle very slow, which reduces programmer productivity.
>
> So perhaps one possible middle ground would be to link packages
> separately, but compile all the sources within a single package at once.
> Presumably, if the project is properly organized, recompiling a single
> package won't take too long, and has the perk of optimizing for size
> within packages. This will probably also map to SCons easily, since
> SCons builds per-directory.
>
>
> T

Well, the idea is good. Small projects usually don't have much packages, so there will be just a few compiler calls. And compiling files concurrently will only have a meaningful efect if the project is large, and a large project will have a lot of packages.

Maybe adding an option to choose between compiling all sources at once, per package, or per source. For example, in development and debug builds the compilation is per file or package, but in release builds all sources are compiled at once, or various packages at once.

This way release builds will take advantage of this behavior that the frontend has, but developers won't have productivity issues. And, of couse, the behaviour will not be fixed, the devs that are using the build system will choose that.