On Saturday, 15 December 2012 at 18:24:50 UTC, jerro wrote:
> On Saturday, 15 December 2012 at 17:31:19 UTC, RenatoUtsch wrote:
>> On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander wrote:
>>> On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
>>>> A quick straw poll.  Do people prefer to have all sources compiled in a
>>>> single compiler call, or (more like C++) separate compilation of each
>>>> object followed by a link call.
>>>
>>> Single compiler call is easier for small projects, but I worry about compile times for larger projects...
>>
>> Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts)
>
> I took a look at your github project, there isn't any code yet, but I like the concept. I was actually planing to do something similar, but since you are already doing it, I think my time would be better spent contributing to your project. Will there be some publicly available code in the near future?

I expect to release a first alpha version in about 15~30 days, maybe less, it depends on how much time I will have on the rest of this month.

On Sat, Dec 15, 2012 at 07:30:52PM +0100, RenatoUtsch wrote:
> On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:
[...]
> >So perhaps one possible middle ground would be to link packages separately, but compile all the sources within a single package at once.  Presumably, if the project is properly organized, recompiling a single package won't take too long, and has the perk of optimizing for size within packages. This will probably also map to SCons easily, since SCons builds per-directory.
[...]
> Well, the idea is good. Small projects usually don't have much packages, so there will be just a few compiler calls. And compiling files concurrently will only have a meaningful efect if the project is large, and a large project will have a lot of packages.

Yes, that's the idea behind it.


> Maybe adding an option to choose between compiling all sources at once, per package, or per source. For example, in development and debug builds the compilation is per file or package, but in release builds all sources are compiled at once, or various packages at once.
> 
> This way release builds will take advantage of this behavior that the frontend has, but developers won't have productivity issues. And, of couse, the behaviour will not be fixed, the devs that are using the build system will choose that.

I forgot to mention also, that passing too many source files to the compiler may sometimes cause memory consumption issues, as the compiler has to hold everything in memory. This may not be practical for very large project, where you can't fit everything into RAM.


T

-- 
Stop staring at me like that! You'll offend... no, you'll hurt your eyes!

On Saturday, 15 December 2012 at 18:44:35 UTC, H. S. Teoh wrote:
> On Sat, Dec 15, 2012 at 07:30:52PM +0100, RenatoUtsch wrote:
>> On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:
> [...]
>> >So perhaps one possible middle ground would be to link packages
>> >separately, but compile all the sources within a single package at
>> >once.  Presumably, if the project is properly organized, recompiling
>> >a single package won't take too long, and has the perk of optimizing
>> >for size within packages. This will probably also map to SCons
>> >easily, since SCons builds per-directory.
> [...]
>> Well, the idea is good. Small projects usually don't have much
>> packages, so there will be just a few compiler calls. And compiling
>> files concurrently will only have a meaningful efect if the project
>> is large, and a large project will have a lot of packages.
>
> Yes, that's the idea behind it.
>
>
>> Maybe adding an option to choose between compiling all sources at
>> once, per package, or per source. For example, in development and
>> debug builds the compilation is per file or package, but in release
>> builds all sources are compiled at once, or various packages at once.
>> 
>> This way release builds will take advantage of this behavior that
>> the frontend has, but developers won't have productivity issues.
>> And, of couse, the behaviour will not be fixed, the devs that are
>> using the build system will choose that.
>
> I forgot to mention also, that passing too many source files to the
> compiler may sometimes cause memory consumption issues, as the compiler
> has to hold everything in memory. This may not be practical for very
> large project, where you can't fit everything into RAM.
>
>
> T

Well, so compiling by packages seem to be the best approach. When I return home I will do some tests to see what I can do.

-- Renato

On 12/15/2012 8:55 AM, Russel Winder wrote:
> A quick straw poll.  Do people prefer to have all sources compiled in a
> single compiler call, or (more like C++) separate compilation of each
> object followed by a link call.

Both are needed, and are suitable for different purposes. It's like asking if you prefer a standard or a philips screwdriver.

On Saturday, 15 December 2012 at 17:02:08 UTC, Andrei Alexandrescu wrote:
> In phobos we use a single call for building the library. Then (at least on Posix) we use multiple calls for running unittests.

This highlights the problem with giving a single answer to the question: Building a large project in one call is often impractical. It only works for Phobos library builds because many templates don't even get instantiated.

David

On 12/15/2012 9:31 AM, RenatoUtsch wrote:
> Yes, I'm writing a build system for D (that will be pretty damn good, I think,
> it has some interesting new concepts), and compiling each source separately to
> an object, and then linking everything will allow easily to make the build
> parallel, dividing the sources to compile in various threads. Or the compiler
> already does that if I pass all source files in one call?

The compiler does a little multithreading, but not enough to make a difference. I've certainly thought about various schemes to parallelize it, though.

Am 15.12.2012 17:55, schrieb Russel Winder:
> A quick straw poll.  Do people prefer to have all sources compiled in a
> single compiler call, or (more like C++) separate compilation of each
> object followed by a link call.
>
> Thanks.
>

I prefer to compile by package, like in any module supporting language.

--
Paulo

On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder wrote:
> A quick straw poll.  Do people prefer to have all sources compiled in a
> single compiler call, or (more like C++) separate compilation of each
> object followed by a link call.
>
> Thanks.

I currently use full compilation. I have no choice because the compiler isn't emitting all symbols I need and get linking errors when doing separate compilation. Compiling my code require more than 2.5Gb of RAM now and is quite slow.

To avoid multiple instantiation of the same template by the compiler, I guess the best option to me would be to compile on a by package basis.

On 12/15/2012 10:30 AM, RenatoUtsch wrote:
> Well, the idea is good. Small projects usually don't have much packages,
> so there will be just a few compiler calls. And compiling files
> concurrently will only have a meaningful efect if the project is large,
> and a large project will have a lot of packages.
>
> Maybe adding an option to choose between compiling all sources at once,
> per package, or per source. For example, in development and debug builds
> the compilation is per file or package, but in release builds all
> sources are compiled at once, or various packages at once.
>

I always thought it would be cool if my build tool could automatically determine the dependency graph of my d code and use it to intelligently break my project into small enough semi-independent chunks to compile.

On 2012-12-17 03:18:45 +0000, Walter Bright <newshound2@digitalmars.com> said:

> Whether the file format is text or binary does not make any fundamental difference.

I too expect the difference in performance to be negligible in binary form if you maintain the same structure. But if you're translating it to another format you can improve the structure to make it faster.

If the file had a table of contents (TOC) of publicly visible symbols right at the start, you could read that table of content alone to fill symbol tables while lazy-loading symbol definitions from the file only when needed.

Often, most of the file beyond the TOC wouldn't be needed at all. Having to parse and construct the syntax tree for the whole file incurs many memory allocations in the compiler, which you could avoid if the file was structured for lazy-loading. With a TOC you have very little to read from disk and very little to allocate in memory and that'll make compilation faster.

More importantly, if you use only fully-qualified symbol names in the translated form, then you'll be able to load lazily privately imported modules because they'll only be needed when you need the actual definition of a symbol. (Template instantiation might require loading privately imported modules too.)

And then you could structure it so a whole library could fit in one file, putting all the TOCs at the start of the same file so it loads from disk in a single read operation (or a couple of *sequential* reads).

I'm not sure of the speedup all this would provide, but I'd hazard a guess that it wouldn't be so negligible when compiling a large project incrementally.

Implementing any of this in the current front end would be a *lot* of work however.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/