February 27, 2019
On Tue, Feb 26, 2019 at 11:44:28AM +1030, ted via Digitalmars-d wrote: [...]
> I'd be interested in your thoughts on https://github.com/GrahamStJack/bottom-up-build
> 
> We use it here (commercial environment with deliverables into defence & commercial customers) (it was created as a response to poor existing build tools). It is rigid on preventing circularities, and deals with code-generation as part of its build cycle. It currently deals with a mixed C++/D codebase of well over 1/2 million lines. It is agnostic to the tool chain - that's part of the configuration - we just use c++ (clang & gcc), and D (dmd & ldc). Also allows codebase to be split amongst multiple repositories.
[...]

Sorry didn't get around to this until now.

At first glance, it looks like a kind of make variant -- sorry if this, or any of the following comments, is wrong, as I only briefly skimmed over the example and some of the bub source code.

It seems to follow make's model of global variables for things like compile flags and so forth. Which is convenient, but could lead to messiness in large enough projects -- i.e., it's not clear to me at first glance how to handle compiling source code with different compile options, or how to generate multiple binaries (e.g., native compile + cross compile) from the same set of sources.  Perhaps encapsulating these variables in configuration objects (e.g. SCons' "environments") might help with this.

Not scanning for dependencies until a target needs to be built is quite a clever idea. I'll have to keep that in mind if I ever decide to invent my own build system. :-D

Splitting a codebase across multiple repositories is also a nice idea. I wonder if it could be taken further: in the form of a build graph export into some kind of standard format, that potentially other build tools can import and build with equivalent results. That's the kind of "holy grail" of build tools that I envision these days, a way to unify all the divergent build tools out there and make it possible to integrate projects across different build systems.

There seems to be baked-in rules for generating executables, static / dynamic libraries, etc., and it appears that some special handling is done in these cases.  It's not clear to me how to achieve equivalent functionality just from the various bub configuration files if the user had to, for example, support a new programming language that needed specialized scanning for dependencies, or how to handle multi-input, multi-output transformations like the Java compiler (does bub support building Java?).  Would it be necessary to modify bub source code in order to extend it to handle such cases, or is it already possible with bub.cfg?

I took a quick look at bub.planner.doPlanning, and it appears to me that it has to scan the entire source tree in order to determine what file(s) have changed. Which would make the cost of a build proportional to the size of the workspace, rather than the size of the changeset.  Is this correct?  If so, I'd suggest taking a look at Tup (http://gittup.org/tup/), which uses modern OS features (inotify / the Windows equivalent which I can't remember) to detect which sources have changed, and correspondingly which subgraph of the entire project's DAG is pertinent at the next build command -- the rest of the DAG is skipped, which in a large project can greatly improve turnaround build times.

Also, are partial builds supported?


T

-- 
Programming is not just an act of telling a computer what to do: it is also an act of telling other programmers what you wished the computer to do. Both are important, and the latter deserves care. -- Andrew Morton
March 01, 2019
I'm not the writer of bub, so I'll try and answer as best I can.






On 28/2/19 2:51 am, H. S. Teoh wrote:
> On Tue, Feb 26, 2019 at 11:44:28AM +1030, ted via Digitalmars-d wrote:
> [...]
>> I'd be interested in your thoughts on
>> https://github.com/GrahamStJack/bottom-up-build
>>
>> We use it here (commercial environment with deliverables into defence
>> & commercial customers) (it was created as a response to poor existing
>> build tools). It is rigid on preventing circularities, and deals with
>> code-generation as part of its build cycle. It currently deals with a
>> mixed C++/D codebase of well over 1/2 million lines. It is agnostic to
>> the tool chain - that's part of the configuration - we just use c++
>> (clang & gcc), and D (dmd & ldc). Also allows codebase to be split
>> amongst multiple repositories.
> [...]
> 
> Sorry didn't get around to this until now.
> 
> At first glance, it looks like a kind of make variant -- sorry if this,
> or any of the following comments, is wrong, as I only briefly skimmed
> over the example and some of the bub source code.
> 
> It seems to follow make's model of global variables for things like
> compile flags and so forth. Which is convenient, but could lead to
> messiness in large enough projects -- i.e., it's not clear to me at
> first glance how to handle compiling source code with different compile
> options, or how to generate multiple binaries (e.g., native compile +
> cross compile) from the same set of sources.  Perhaps encapsulating
> these variables in configuration objects (e.g. SCons' "environments")
> might help with this.

This is handled by having multiple bub.cfg files. As the build directory is located elsewhere and the build is performed in that directory, the setup for each build directory is done once - so this has worked well in practice.

> 
> Not scanning for dependencies until a target needs to be built is quite
> a clever idea. I'll have to keep that in mind if I ever decide to invent
> my own build system. :-D
> 
> Splitting a codebase across multiple repositories is also a nice idea. I
> wonder if it could be taken further: in the form of a build graph export
> into some kind of standard format, that potentially other build tools
> can import and build with equivalent results. That's the kind of "holy
> grail" of build tools that I envision these days, a way to unify all the
> divergent build tools out there and make it possible to integrate
> projects across different build systems.

I know this build graph is available - I believe it can be output from a commandline switch - so getting it into a standard format would be quite easy.


> 
> There seems to be baked-in rules for generating executables, static /
> dynamic libraries, etc., and it appears that some special handling is
> done in these cases.  It's not clear to me how to achieve equivalent
> functionality just from the various bub configuration files if the user
> had to, for example, support a new programming language that needed
> specialized scanning for dependencies, or how to handle multi-input,
> multi-output transformations like the Java compiler (does bub support
> building Java?).  Would it be necessary to modify bub source code in
> order to extend it to handle such cases, or is it already possible with
> bub.cfg?

It is a product of the environment in which it is used. i.e.  there has been no push for anything else at this stage. I'm fairly sure that Graham would take suggestions like this on board.

> 
> I took a quick look at bub.planner.doPlanning, and it appears to me that
> it has to scan the entire source tree in order to determine what file(s)
> have changed. Which would make the cost of a build proportional to the
> size of the workspace, rather than the size of the changeset.  Is this
> correct?  If so, I'd suggest taking a look at Tup
> (http://gittup.org/tup/), which uses modern OS features (inotify / the
> Windows equivalent which I can't remember) to detect which sources have
> changed, and correspondingly which subgraph of the entire project's DAG
> is pertinent at the next build command -- the rest of the DAG is
> skipped, which in a large project can greatly improve turnaround build
> times.

On our codebase, this step takes less than a second, so it hasn't been an issue.

> 
> Also, are partial builds supported?

By partial builds, if you mean that it only builds the files that are affected by the code change just made? In that case yes.

If you mean the equivalent of a makefile target where a subset of files is defined, then no. (Again, there is no call for this within our environment)

> 
> 
> T
> 
--ted