On Thursday, 25 September 2014 at 13:50:10 UTC, Jacob Carlborg wrote:
> On 25/09/14 09:38, Atila Neves wrote:
>
>> Here's one: having to manually generate the custom main file for
>> unittest builds. There's no current way (or at least there wasn't when I
>> brought it up in the dub forum) to tell it to autogenerate a D file from
>> a dub package and list it as a dependency of the unittest build.
>
> Hmm, I haven't used Dub to run unit tests. Although, DMD has a "-main" flag that adds an empty main function.

I don't want an empty main function. I want the main function and the file it's in to be generated by the build system.

Atila

On Thursday, 25 September 2014 at 13:51:17 UTC, Jacob Carlborg wrote:
> On 25/09/14 09:38, Atila Neves wrote:
>
>> Here's one: having to manually generate the custom main file for
>> unittest builds. There's no current way (or at least there wasn't when I
>> brought it up in the dub forum) to tell it to autogenerate a D file from
>> a dub package and list it as a dependency of the unittest build.
>
> BTW, I would say that's a minor issue, far from begin enough to create a completely new build system.

I agree. That's why I haven't written a build system yet. However, larger projects need this kind of thing. If I were working on a team of 20+ devs writing D, I'd eventually need it. I'd consider SCons, CMake, etc. but I'd still need dub for package management.

Atila

On Thursday, 25 September 2014 at 14:25:25 UTC, H. S. Teoh via Digitalmars-d wrote:
> On Thu, Sep 25, 2014 at 03:47:22PM +0200, Jacob Carlborg via Digitalmars-d wrote:
>> On 25/09/14 08:39, H. S. Teoh via Digitalmars-d wrote:
> [...]
>> >But even then, I *did* run into the problem of non-reproducible
>> >builds with dmd. So there's still a blemish there. :-P  Makes me want
>> >to alias `make` to `make clean; make` just for this alone
>> 
>> I don't think the "clean" action can be completely excluded. I
>> recently tried to build a project, without running "clean", and got
>> some unexpected errors. Then I remembered I had just installed a new
>> version of the compiler.
> [...]
>
> That's the hallmark of make-based projects. SCons projects, OTOH, almost
> never needs to do that (except if you totally screwed up your SCons
> scripts). The super-complicated SCons script that I described in another
> post? I don't even remember that last *year* when I had to do the
> equivalent of a clean. I've been updating, branching, merging the
> workspace for years now, and it Just Builds -- correctly at that.

+1. I remember the days where I dealt with hand-written makefiles and running "make clean" just to be sure. I don't miss them.

If you ever _need_ to type "make clean" then the build system is broken. Simple as. It's analogous to deleting the object file to make sure the compiler does its job properly.

Atila

On 2014-09-25 16:23, H. S. Teoh via Digitalmars-d wrote:

> That's the hallmark of make-based projects.

This was Ninja actually. But how would the build system know I've updated the compiler?

-- 
/Jacob Carlborg

On 2014-09-25 16:01, Wyatt wrote:

> I might look at the "modern alternative" once it supports debugging
> 64-bit executables. :/

LLDB supports OS X, Linux and FreeBSD. 32 and 64bit on all of these platforms [1]. Are you looking for Windows support?

[1] http://lldb.llvm.org/

-- 
/Jacob Carlborg

On 9/25/14, 10:26 AM, Atila Neves wrote:
> On Thursday, 25 September 2014 at 13:50:10 UTC, Jacob Carlborg wrote:
>> On 25/09/14 09:38, Atila Neves wrote:
>>
>>> Here's one: having to manually generate the custom main file for
>>> unittest builds. There's no current way (or at least there wasn't when I
>>> brought it up in the dub forum) to tell it to autogenerate a D file from
>>> a dub package and list it as a dependency of the unittest build.
>>
>> Hmm, I haven't used Dub to run unit tests. Although, DMD has a "-main"
>> flag that adds an empty main function.
>
> I don't want an empty main function. I want the main function and the
> file it's in to be generated by the build system.

Why would be the focus on the mechanism instead of the needed outcome? -- Andrei

On Thursday, 25 September 2014 at 17:42:09 UTC, Jacob Carlborg
wrote:
> On 2014-09-25 16:23, H. S. Teoh via Digitalmars-d wrote:
>
>> That's the hallmark of make-based projects.
>
> This was Ninja actually. But how would the build system know I've updated the compiler?

The compiler is an input to the build rule.  Consider the rule:

build: $(CC) my.c -o my.o

what are the dependencies for the rule "build"?

my.c obviously.  Anything the compiler accesses during the
compilation of my.c.  And *the compiler itself*, referenced here
as $(CC).  From a dependency management standpoint, executables
are not special except as running them leads to the discovery of
more dependencies than may be statically specified.

On 9/25/14, 10:47 AM, Cliff wrote:
> On Thursday, 25 September 2014 at 17:42:09 UTC, Jacob Carlborg
> wrote:
>> On 2014-09-25 16:23, H. S. Teoh via Digitalmars-d wrote:
>>
>>> That's the hallmark of make-based projects.
>>
>> This was Ninja actually. But how would the build system know I've
>> updated the compiler?
>
> The compiler is an input to the build rule.  Consider the rule:
>
> build: $(CC) my.c -o my.o
>
> what are the dependencies for the rule "build"?
>
> my.c obviously.  Anything the compiler accesses during the
> compilation of my.c.  And *the compiler itself*, referenced here
> as $(CC).  From a dependency management standpoint, executables
> are not special except as running them leads to the discovery of
> more dependencies than may be statically specified.

Yah, it only gets odder from there. E.g. all stuff being build must also list the rule file itself as a dependency.

FWIW I've seen implications that a better build tool would improve the lot of dlang contributors. My response to that would be: I agree that Phobos' posix.mak (https://github.com/D-Programming-Language/phobos/blob/master/posix.mak) is a tad baroque and uses a couple of the more obscure features of make. I also consider a few changes to it could have been effected simpler and better. However, as far as I can tell it's not a large issue for dlang contributors, and to the extent it is, it's not because of make's faults.


Andrei

On Thu, Sep 25, 2014 at 09:18:26PM +0400, Dmitry Olshansky via Digitalmars-d wrote: [...]
> I had an "insanely" complicated one could handle 2 platforms, 3 emulators and a few custom build steps (including signing binaries), cleaning, tracking dependencies with minimal rebuilds. For instance, binary need not to be signed again if it's bit for bit the same. All of the above was done in about 50 lines, never braking as the project progressed, I think it changed only 2-3 times over a year.
>
> I was never able to make a good makefile that wouldn't shit on me in some way, like not properly cleaning something or handling errors . I'm not even talking about the 10+ LOCs for anything barely useful.

+1. Jives with my experience with SCons as well.


> 25-Sep-2014 01:12, Andrei Alexandrescu пишет:
> >(2) any build system for a project of nontrivial size needs a person/team minding it - never saw such a thing as it's just all automated and it works;
> 
> The question is amount of work, size of file and frequency of changes. For instance, Scons easily allows not to change a makefile on every single added module, I can't say if make could pull it off.

It can. Except that you can expect things will go horribly wrong if you so much as breathe in the wrong direction. And when you're dealing with a multi-person project (i.e., 99% of non-trivial software projects), expect that people will do stupid things that either breaks the makefile, or they will make breaking changes to the makefile that will screw everything up for everyone else.


> >(3) especially if the build system is not that familiar, the role of the build czar is all the more important.
> 
> Somebody (Russel?) was working on D Scons support, how about start to capitalize on this work? I'd gladly try to make a scons script for phobos/druntime if there is even a tiny chance of apporval.

D support has been merged into mainline SCons, thanks to Russel's efforts. I've been using his D module for a while now, and find it generally very C/C++-like, which is perfect for projects that need heavy interfacing between C/C++ and D. For D-only projects, it's a bit klunky because it defaults to separate compilation, whereas dmd tends to work better with whole-program compilation (or at least per-module rather than per-source-file compilation).

SCons does hit performance barriers when you get into very large projects (on the magnitude of Mozilla Firefox, or the GTK desktop suite, for example), but for something the size of dmd/druntime/phobos, you won't notice the difference at all. In fact, you'd save a lot of time by not needing to make clean; make every single time you do anything non-trivial.


> >So the reality is quite a bit more complicated than the shiny city on a hill you describe.
> 
> Most people using make/autotools say build systems are hard, it need not to be true but I'm obviously biased.
[...]

I'm biased too, but I'm inclined to think that those people say it's hard because it *is* hard when the best you got is make and autotools. :-P


T

-- 
Leather is waterproof.  Ever see a cow with an umbrella?

On Thu, Sep 25, 2014 at 07:42:08PM +0200, Jacob Carlborg via Digitalmars-d wrote:
> On 2014-09-25 16:23, H. S. Teoh via Digitalmars-d wrote:
> 
> >That's the hallmark of make-based projects.
> 
> This was Ninja actually. But how would the build system know I've updated the compiler?
[...]

The compiler and compile flags are inputs to the build rules in SCons.

In my SCons projects, when I change compile flags (possibly for a subset of source files), it correctly figures out which subset (or the entire set) of files needs to be recompiled with the new flags. Make fails, and you end up with an inconsistent executable.

In my SCons projects, when I upgrade the compiler, it recompiles everything with the new compiler. Make doesn't detect a difference, and if you make a change and recompile, suddenly you got an executable 80% compiled with the old compiler and 20% compiled with the new compiler. Most of the time it doesn't make a difference... but when it does, have fun figuring out where the problem lies. (Or just make clean; make yet again... the equivalent of which is basically what SCons would have done 5 hours ago.)

In my SCons projects, when I upgrade the system C libraries, it recompiles everything that depends on the updated header files *and* library files. In make, it often fails to detect that the .so's have changed, so it fails to relink your program. Result: your executable behaves strangely at runtime due to wrong .so being linked, but the problem vanishes once you do a make clean; make.

Basically, when you use make, you have to constantly do make clean; make just to be sure everything is consistent. In SCons, the build system does it for you -- and usually more efficiently than make clean; make because it knows exactly what needs recompilation and what doesn't.

In my SCons projects, when I checkout a different version of the sources to examine some old code and then switch back to the latest workspace, SCons detects that file contents haven't changed since the last build, so it doesn't rebuild anything. In make, it thinks the entire workspace has changed, and you have to wait another half hour while it rebuilds everything. It doesn't even detect that all intermediate products like .o's are identical to the last time, so it will painstakingly relink everything again. SCons will detect when a .o file hasn't changed from the last build (e.g., you only changed a comment in the source file), and it won't bother relinking your binaries or trigger any of the downstream dependencies' rebuilds.

Make-heads find the idea of the compiler being part of the input to a build rule "strange"; to me, it's common sense. What's strange is the fact that make doesn't (and can't) guarantee anything about the build -- you don't know for sure whether an incremental build gives you the same executable as a build from a clean workspace. You don't know if recompiling after checking out a previous release of your code will actually give you the same binaries that you shipped 2 months ago. You don't know if the latest build linked with the latest system libraries. You don't know if the linked libraries are consistent with each other. You basically have no assurance of *anything*, except if you make clean; make. But wait a minute, I thought the whole point of make is incremental building. If you have to make clean; make *every* *single* *time*, that completely defeats the purpose, and you might as well just put your compile commands in a bash script instead, and run that to rebuild everything from scratch every time. Yeah it's slow, but at least you know for sure your executables are always what you think they are! (And in practice, the bash script might actually be faster -- I lost track of how much time I wasted trying to debug something, only to go back and make clean; make and discover that the bug vanished. Had I used the shell script every single time, I could've saved how many hours spent tracking down these heisenbugs.)


T

-- 
Gone Chopin. Bach in a minuet.