On Sat, 2017-06-17 at 00:07 +0200, Mike Wey via Digitalmars-d wrote:
> On 06/16/2017 03:16 PM, Joakim wrote:
> > As for Meson, never dealt with it much, do you have an example for
> > D
> > code we can look at?
> 
> The meson files for tilix might be a good example: https://github.com/gnunn1/tilix/tree/master/experimental/meson

I use Meson for building Me TV:

https://github.com/russel/Me-TV

The CMake build is only there so as to use CLion.

> 
-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On 6/15/2017 11:30 PM, Russel Winder via Digitalmars-d wrote:
> A direct question to Walter and Andrei really.
> 
> If someone, let us say Russel Winder, create a CMake/Ninja and/or
> Meson/Ninja build for DMD, is there any chance of it being allowed to
> replace the Make system?
> 
> If the answer is no, then Russel will obviously not waste his time
> doing something that will not be accepted.

It's highly unlikely it would be accepted:

1. make is ubiquitous. It's not something we have to scrounge to find on platform X, it's already there. People already are familiar with it (even if they hate it).

2. we're in the D business, not the project build business. It's easier to get past that "first 5 minutes" if everything about D other than D itself is familiar and conventional.

3. to steal from Churchill, `make` is the worst form of build system except for all the others

4. much as I dislike make, the time spent wrestling with it is a vanishingly tiny slice of time compared to what spent on the rest of D. Getting that time slice to zero will have no effect on productivity, and I'm not convinced that a newer build system will even reduce that time slice at all (see point 5).

5. D has a more complex build process than it should. Using another build system won't make that complexity go away.

6. unlike the choice to use github, there is no clear winner in the `make` category of build tools. If the industry has moved on from make to X, then we should, too. But it has not.

7. the current makefiles for DMD suffer from over-engineering, i.e. making simple things complicated and excessively using obscure features of make. This isn't really the fault of make.

On Saturday, 17 June 2017 at 19:20:54 UTC, Walter Bright wrote:
> On 6/15/2017 11:30 PM, Russel Winder via Digitalmars-d wrote:
>> A direct question to Walter and Andrei really.
>> 
>> If someone, let us say Russel Winder, create a CMake/Ninja and/or
>> Meson/Ninja build for DMD, is there any chance of it being allowed to
>> replace the Make system?
>> 
>> If the answer is no, then Russel will obviously not waste his time
>> doing something that will not be accepted.
>
> It's highly unlikely it would be accepted:
>
> 1. make is ubiquitous. It's not something we have to scrounge to find on platform X, it's already there. People already are familiar with it (even if they hate it).
>
> 2. we're in the D business, not the project build business. It's easier to get past that "first 5 minutes" if everything about D other than D itself is familiar and conventional.
>
> 3. to steal from Churchill, `make` is the worst form of build system except for all the others
>
> 4. much as I dislike make, the time spent wrestling with it is a vanishingly tiny slice of time compared to what spent on the rest of D. Getting that time slice to zero will have no effect on productivity, and I'm not convinced that a newer build system will even reduce that time slice at all (see point 5).
>
> 5. D has a more complex build process than it should. Using another build system won't make that complexity go away.
>
> 6. unlike the choice to use github, there is no clear winner in the `make` category of build tools. If the industry has moved on from make to X, then we should, too. But it has not.
>
> 7. the current makefiles for DMD suffer from over-engineering, i.e. making simple things complicated and excessively using obscure features of make. This isn't really the fault of make.

A lot of this is simply saying Make is popular so we should just stick with it: I hate that mindset.  It is the same mindset D has to combat with C or C++, and that was exhibited when you spoke against the SDL format for dub.

I understand what you're saying, that D should pick its battles, but it's possible to do that and not just go with the status quo for everything other than D.  It is often necessary to get rid of defaults in many other categories, like Make or XML, and where possible D should try to make the best choices, without getting too far out there in NIH-land or its own D island.

I don't think choosing an outside Python build system that GNOME is in the process of switching to qualifies as either of those, as Python is almost as ubiquitous as make.  If we were to use Meson, we'd gain a lot in ease of use and lose almost nothing in platform availability.

All that said, I'll reiterate what I said earlier to Russel, and what I'd say to Atila too: don't aim to replace Make, just aim to provide an alternative in the dmd/phobos repos.  If we find that everybody is using that instead of Make, we'll just switch over to it naturally someday.

On 6/17/2017 1:25 PM, Joakim wrote:
> A lot of this is simply saying Make is popular so we should just stick with it: I hate that mindset.  It is the same mindset D has to combat with C or C++, and that was exhibited when you spoke against the SDL format for dub.

It's not quite the same. I spend 99.99% of my time programming working with code, not makefiles. Productivity gains from using a better language have a major effect. A better make simply does not.


> D should pick its battles,

That's a good summary of the situation.


> All that said, I'll reiterate what I said earlier to Russel, and what I'd say to Atila too: don't aim to replace Make, just aim to provide an alternative in the dmd/phobos repos.  If we find that everybody is using that instead of Make, we'll just switch over to it naturally someday.

Maintaining two parallel build systems is having the downsides of both and the benefits of neither. Then there's the endless vim-vs-emacs debate about which alternative build system is better. It's not a battle we need to or can afford to invest in.

Let's please stay focused on things that will move the needle.

On Saturday, 17 June 2017 at 21:49:29 UTC, Walter Bright wrote:
> It's not quite the same. I spend 99.99% of my time programming working with code, not makefiles.

I'd say somewhere around 10% of the time I've spent on D pull requests has been wasted because of the makefiles. It is a barrier to entry in new contributors adding new modules.

Though, I think it is just because the ones in there are overcomplicated and crappy. My ideal makefile is less than ten lines long and I see no reason why dmd, phobos, druntime, and the  dlang.org website can't get close to that. Heck, `dmd **/*.d` almost actually works...

On Saturday, 17 June 2017 at 21:49:29 UTC, Walter Bright wrote:
> On 6/17/2017 1:25 PM, Joakim wrote:
>> A lot of this is simply saying Make is popular so we should just stick with it: I hate that mindset.  It is the same mindset D has to combat with C or C++, and that was exhibited when you spoke against the SDL format for dub.
>
> It's not quite the same. I spend 99.99% of my time programming working with code, not makefiles. Productivity gains from using a better language have a major effect. A better make simply does not.

I agree that most time spent coding doesn't involve the build system, but as Adam says, you say that as someone with a long history with Make.  For noobs, it represents a barrier to contributing, one that we should strive to lower as much as possible.

>> D should pick its battles,
>
> That's a good summary of the situation.
>
>
>> All that said, I'll reiterate what I said earlier to Russel, and what I'd say to Atila too: don't aim to replace Make, just aim to provide an alternative in the dmd/phobos repos.  If we find that everybody is using that instead of Make, we'll just switch over to it naturally someday.
>
> Maintaining two parallel build systems is having the downsides of both and the benefits of neither. Then there's the endless vim-vs-emacs debate about which alternative build system is better. It's not a battle we need to or can afford to invest in.
>
> Let's please stay focused on things that will move the needle.

There's an easy way out of that morass.  Anybody who wants to submit an alternate build file for their build system must also provide a point of contact/support for that build system, so that any questions or update requests would be directed at them.

Make would remain the default and you make clear that it's the _only_ one officially supported.  Eventually, you prune out all the build files that don't work that well and aren't being supported by their proponents.  If they like, the project contributors can eventually decide to switch over to the new build system as the default, if it has been proven for some time to both work better and be well-supported.

All this would require you to do nothing, just perhaps make a decision someday down the line if and when a build system has been proven and others want to make it the default.  Would you be okay with starting this pragmatic approach, by okaying the commit of a Meson or Reggae build file to these repos and seeing where it goes?

On Saturday, 17 June 2017 at 21:49:29 UTC, Walter Bright wrote:
> It's not quite the same. I spend 99.99% of my time programming working with code, not makefiles.

Martin and Sebastian can correct me if I'm wrong, but unless I am, Martin, Sebastian, and myself spend a scary amount of time wrestling with makefiles. We have had bad production issues due to makefiles, such as missing dependencies causing inconsistent or broken builds (notably observable when something works only without -j), typos or undefined variables resulting in bugs being silently ignored (hey, I filed a PR to fix one of those not half an hour ago - dmd#6916), CI checks being accidentally completely switched off (which happened more than once!), the win32.mak / win64.mak / posix.mak mess, super-ugly hacks due to limitations of Make that slow down the build unconditionally and/or make everything much more fragile and complicated... and I could go on.

Which is not to say that I think we need to get off makefiles ASAP. Migrating to anything else is going to undoubtedly incur a massive migration cost and for a long time, a big maintenance cost (especially if it involves dogfooding). I agree with many of your arguments as well.

I think you're not encountering much of the problems with makefiles because you're mainly dealing with DMD, whose build process is relatively simple - and even there, the build process is mostly maintained these days by other people. In comparison, the build process for dlang.org has been getting much more complicated with time (partially because of some technical debt and fragmentation, but also simply because we want to do more things like nice runnable examples, various pre-processing / post-processing / validation, building the spec from the compiler source, etc.

If anyone wants to SERIOUSLY propose a plan to migrate from makefiles, I'd be interested to look at it. However, its benefits obviously must outweigh the maintenance costs of current makefiles, as well as not raising the contribution barrier too much.

Some requirements would be:

- Obvious syntax - making a change such as adding a new target or dependency should be obvious just from looking at the existing code.

- Performance - shouldn't have considerable overhead; but also, it shouldn't take 10 seconds to "rebuild" the build tool itself after every modification.

- Cross-platform support - this may seem obvious, but it should support at least all platforms DMD supports. One of the suggestions for replacement in a similar recent thread (http://forum.dlang.org/post/ohkkqj$21lg$1@digitalmars.com), buildsome, shows no indication of any Windows support at all, not even plans to work on it.

- Minimal dependencies - ideally it should work with as few as possible external dependencies that one would typically need to install to build D. Windows is very often overlooked when evaluating this requirement, for example although Python is ubiquitous on POSIX systems, it's much less so on Windows.

That doesn't leave many candidates. reggae might be the closest to satisfying all of the above, though I haven't looked too closely at it. For one thing, I don't really understand the need for build system generators - seems like an extra step and imposing an implementation detail on the user.

On 6/17/2017 6:20 PM, Vladimir Panteleev wrote:
> On Saturday, 17 June 2017 at 21:49:29 UTC, Walter Bright wrote:
>> It's not quite the same. I spend 99.99% of my time programming working with code, not makefiles.
> 
> Martin and Sebastian can correct me if I'm wrong, but unless I am, Martin, Sebastian, and myself spend a scary amount of time wrestling with makefiles.

I spent hours today attempting to figure out why, after I refreshed my copy of druntime with HEAD, the druntime unittests would no longer run. I am still at a dead end.

The biggest issue with understanding the makefiles is a near total lack of any helpful comments. I stepped up a bit with this:

  https://github.com/dlang/druntime/pull/1843

but so far I have not been able to figure out what is going on with the druntime/test directory. I don't even know why it exists.

Makefiles need to be documented just like code. Even simple things like what variables are inputs to the makefile. What public targets are in the makefile (rather than private targets). What those internally defined variables are for, and what are their possible values. What the sed scripts are doing. What's the difference between SRCS and MANIFEST. Etc.

Make can be a bear to use. But I believe that is mostly due to the convention of never documenting anything in it (I see this commonly, not just for our project!). I'm guilty of that myself.

David Held, a while back, did do some work to fix this. You can see the results at the beginning of:

  https://github.com/dlang/phobos/blob/master/posix.mak

though it has suffered from years of neglect, barnacles and bit rot.

We must try and do better with this. We can start by refusing to accept makefile PRs that add new constructions without documentation, just like we don't accept new functions without Ddoc comments.

On Saturday, 17 June 2017 at 19:20:54 UTC, Walter Bright wrote:
> On 6/15/2017 11:30 PM, Russel Winder via Digitalmars-d wrote:
>> A direct question to Walter and Andrei really.
>> 
>> If someone, let us say Russel Winder, create a CMake/Ninja and/or
>> Meson/Ninja build for DMD, is there any chance of it being allowed to
>> replace the Make system?
>> 
>> If the answer is no, then Russel will obviously not waste his time
>> doing something that will not be accepted.
>
> It's highly unlikely it would be accepted:
>
> 1. make is ubiquitous. It's not something we have to scrounge to find on platform X, it's already there. People already are familiar with it (even if they hate it).
>
> 2. we're in the D business, not the project build business. It's easier to get past that "first 5 minutes" if everything about D other than D itself is familiar and conventional.
>
> 3. to steal from Churchill, `make` is the worst form of build system except for all the others
>
> 4. much as I dislike make, the time spent wrestling with it is a vanishingly tiny slice of time compared to what spent on the rest of D. Getting that time slice to zero will have no effect on productivity, and I'm not convinced that a newer build system will even reduce that time slice at all (see point 5).
>
> 5. D has a more complex build process than it should. Using another build system won't make that complexity go away.
>
> 6. unlike the choice to use github, there is no clear winner in the `make` category of build tools. If the industry has moved on from make to X, then we should, too. But it has not.
>
> 7. the current makefiles for DMD suffer from over-engineering, i.e. making simple things complicated and excessively using obscure features of make. This isn't really the fault of make.

These reasons are terrible.

1. So! Just because something is "universal" doesn't mean it is good. Look at any major religion. Usually those that believe in it tend to use as proof that there are so many other believers.

2. If you are in the D business you are in the make business.

3. Which, you are saying the exact same as make is the best. Yet, those others are just make clones that are not any worse(as they make it better). It is a contradictory argument.

4. You are not taking in to account everyone elses time. Sure, you might only save a penny, but if 7B people saved a penny, that would be 70 million dollars. That is not an insignificant amount.

5. It might, how do you know, have you tried?

6. This is not logical, they haven't moved precisely because they are thinking like you are. "If X hasn't moved from Y then why should we" everyone exclaims.

7. Then come up with something better instead of making excuses... which is what it all boils down to. You could have simply said, in a more logical way: "I do not want to spent the time to create/switch to something new and better because it is not worth it to me"... as that is all you have said(obfuscating in to a 7 point bullet list of meaningless il-arguments doesn't change that).


My suggestion is, that, if someone else wants to "waste" their time creating something that is new and better then they should be "allowed" and "supported" in doing so with the only caveat that they must prove, at the end of the day, that it is better. If it is not, then they "wasted" their time. If it is, then everyone is happier and more productive. The question is, are we fighting over pennies or millions?

On Sat, 2017-06-17 at 12:20 -0700, Walter Bright via Digitalmars-d wrote:
> 
[…]
> It's highly unlikely it would be accepted:

So it is highly unlikely I am going to offer to do any work on it then.

I am extremely disappointed in the attitude displayed by your reply, it shows a lack of interest in doing things better. This is ironic in the extreme given it is the build infrastructure for a compiler for programming language supposedly to help make things better for programmers.

> 1. make is ubiquitous. It's not something we have to scrounge to find
> on
> platform X, it's already there. People already are familiar with it
> (even if
> they hate it).

"is ubiquitous" is a lazy answer. Python is ubiquitous to the same extent. Of course you still have to install make so actually it isn't as ubiquitous as the comment implies.

> 2. we're in the D business, not the project build business. It's
> easier to get
> past that "first 5 minutes" if everything about D other than D itself
> is
> familiar and conventional.

You may be in the D business and already know the Make-based framework intimately, but it is a barrier to anyone else wanting to contribute. The inference must thus be that you don't want new contributors to the compiler. That's fine, but a dead end for the future of D.

> 3. to steal from Churchill, `make` is the worst form of build system
> except for
> all the others

Argumentation by false analogy.

> 4. much as I dislike make, the time spent wrestling with it is a
> vanishingly
> tiny slice of time compared to what spent on the rest of D. Getting
> that time
> slice to zero will have no effect on productivity, and I'm not
> convinced that a
> newer build system will even reduce that time slice at all (see point
> 5).

As you have no experience you have no data point. Everyone else, who does have actual data is moaning like crazy about staying with a pure Make/Shell system.

The world has moved forward in build in the last 40 years, can I suggest you try and learn a little bit about it.

> 5. D has a more complex build process than it should. Using another
> build system
> won't make that complexity go away.

Yes it will.

> 6. unlike the choice to use github, there is no clear winner in the
> `make`
> category of build tools. If the industry has moved on from make to X,
> then we
> should, too. But it has not.

Yes it has. I suspect you are focussing too much on the D compiler internals and not enough on the programmer development environment and tools progress that has been made in the last 20 years.

> 7. the current makefiles for DMD suffer from over-engineering, i.e.
> making
> simple things complicated and excessively using obscure features of
> make. This
> isn't really the fault of make.

Again faulty reasoning. Replacing a system is a way of getting rid of crap and getting something better.

I am not a fan of fashion or fadism in software development, but neither am I a fan of ignoring all progress so as pretend nothing has changed. There is nothing wrong with a project changing its build system on a regular, although not too frequent, basis.

D should be there at the cutting edge of software development in its own environment as an integral part of the pitch to the developers. Rust and Go have done this – well Go has a problem with vendoring, but like most of the Go community, we'll pretend it isn't a big problem.

Reggae is D's pitch in the CMake and Meson class of meta-build tools. Why aren't all the D compiler and tool developments using it? Or if Reggae is a step too far because it is a D program offering D (or Python, Lua,…) specifications of build, use Meson.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder