On 6/13/20 8:59 PM, Dennis wrote:
> On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:
>> Now we require dmd to build dmd...
> 
> We do anyway ever since the code base got converted from C++ to D.

As I mentioned, that was done carefully and beautifully by an AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken.

Did I mention I appreciate good engineering?

On Sunday, 14 June 2020 at 01:26:02 UTC, Andrei Alexandrescu wrote:
> On 6/13/20 8:59 PM, Dennis wrote:
>> On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:
>>> Now we require dmd to build dmd...
>> 
>> We do anyway ever since the code base got converted from C++ to D.
>
> As I mentioned, that was done carefully and beautifully by an AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken.
>
> Did I mention I appreciate good engineering?

make -f posix.mak AUTO_BOOTSTRAP is still a thing.

On 6/13/20 9:30 PM, Stefan Koch wrote:
> On Sunday, 14 June 2020 at 01:26:02 UTC, Andrei Alexandrescu wrote:
>> On 6/13/20 8:59 PM, Dennis wrote:
>>> On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:
>>>> Now we require dmd to build dmd...
>>>
>>> We do anyway ever since the code base got converted from C++ to D.
>>
>> As I mentioned, that was done carefully and beautifully by an AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken.
>>
>> Did I mention I appreciate good engineering?
> 
> make -f posix.mak AUTO_BOOTSTRAP is still a thing.

Not when I tried it: https://forum.dlang.org/post/r5e8uj$i1b$1@digitalmars.com

On 6/13/2020 6:44 AM, kinke wrote:
> Side note: LDC has been using GNUmake on Windows for years, incl. extending some upstream Makefiles to work with and test Windows targets too.
> While GNUmake itself is surprisingly portable (source comes with a Visual Studio project working out of the box, resulting in a little 330 KB Win64 stand-alone executable),
DM make is 52K :-)

On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:

> Where. Is. The. Progress.

Progress would be to fix the makefiles so that they are usable out-of-the-box with the newer versions of VS. This is abysmal:

VCDIR=\Program Files (x86)\Microsoft Visual Studio 10.0\VC
SDKDIR=\Program Files (x86)\Microsoft SDKs\Windows\v7.0A

...

CC=$(VCDIR)\bin\amd64\cl
LD=$(VCDIR)\bin\amd64\link
AR=$(VCDIR)\bin\amd64\lib

On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:
> The original Andrei, just like today's Andrei, has an appreciation for good engineering. I didn't feel the need to add to provide detail because (a) most regulars in this forum already knew what I was going to say, and (b) nobody save for a few would share my opinion.
>
> But, I'll bite again, again to regret it.

I'll reply with similar regrets, because I agree that this discussion isn't helping anyone.
tl;dr: I agree with you that build.d isn't the ideal engineering piece that D was supposed to enable, but it did solve a few real problems and using D without dependency was really the only option to move forward:

In Detail:

1) DigitalMars Make was a massive real problem (and still is for Druntime+Phobos)
2) Duplication into many similar Makefiles with mismatching definitions (posix.mak, win32.mak, win64.mak, osmodel.mak, ...)
3) There were many things we couldn't model properly in Make (e.g. version detection shell script, documentation build, ...)
4) Especially the Windows Makefiles were missing a lot of target and options (e.g. no LTO/PGO builds, no library builds, ...)

https://github.com/dlang/dmd/blob/v2.080.0/src/posix.mak
https://github.com/dlang/dmd/blob/v2.080.0/src/win32.mak
https://github.com/dlang/dmd/blob/v2.080.0/src/win64.mak

5) You and Walter repeatedly struck down all attempts to use any other build tool (like e.g. reggae for Phobos - https://github.com/dlang/phobos/pull/4194)
6) The Windows Makefiles looked liked this:

---
$G/nteh.obj : $C\rtlsym.h $C\nteh.c
	$(CC) -c -o$@ $(MFLAGS) $C\nteh

$G/os.obj : $C\os.c
	$(CC) -c -o$@ $(MFLAGS) $C\os

$G/out.obj : $C\out.c
	$(CC) -c -o$@ $(MFLAGS) $C\out

$G/outbuf.obj : $C\outbuf.h $C\outbuf.c
	$(CC) -c -o$@ $(MFLAGS) $C\outbuf

$G/pdata.obj : $C\pdata.c
	$(CC) -c -o$@ $(MFLAGS) $C\pdata

$G/ph2.obj : $C\ph2.c
	$(CC) -c -o$@ $(MFLAGS) $C\ph2
---

7) Also, complexity in general increased over time as C++ header generation or CXX compatibility tests got added. There are a couple of other new features that are quite helpful for CIs (like e.g. the very detailed failure message on build failures).

In build.d there definitely are a few relicts from the Makefiles times as it was a lot easier to do a step-by-step migration of the targets than migrating everything in one bulk, so I'm not denying that the build script source code could be refactored and improved, but so could be almost any code in the D ecosystem.

Lastly, if you really want to know why build.d grew from 250 lines initially to its current size, the answer is rather simple. It's because of many, many shortcomings and bugs in Druntime and Phobos.
As a prime example I encourage you to read through is:

https://github.com/dlang/dmd/pull/10611 (GitHub hides most of the discussion by default)

> * Then, we have main() dealing with a lot of stuff. (Why?)

Not sure, what's a "a lot of stuff" for you, but that's the main method as of now:

---
int main(string[] args)
{
    try
    {
        runMain(args);
        return 0;
    }
    catch (BuildException e)
    {
        // Ensure paths are relative to the root directory
        // s.t. error messages are clickable in most IDE's
        writeln(e.msg.replace(buildPath("dmd", ""), buildPath("src", "dmd", "")));
        return 1;
    }
}
---

To be fair, runMain does more things (gettings arguments + starting thread pool) and of course, the showHelp() could have easily been moved outside, but I really don't see how this is a critical problem for D?
Especially considering how big main() is at "your" rdmd:

https://github.com/dlang/tools/blob/master/rdmd.d#L62

> * The rules using makeRule are a confusing mix of code and data starting at line 249, after the reader has browsed through a bunch of implementation details.

1) Those are all rules. The implementation details are the rules...
2) The "reader" has a nice overview of all rules and targets at the beginning (before implementation)
3)) A typical "reader" is much more likely to execute ./build.d -h (or --help) than reading the source

> An interesting side question  would be what language features
> would make such things easier to write; e.g. a mixin string
> in conjunction with a DSL that implements a small subset of make may be nicer.

Maybe a standard library with isn't riddled with hundreds of bugs and old/outdated/unusable code?

> But also becomes ironic - we started by going away from make.)

It's not ironic. WE CAN'T USE MAKE ON WINDOWS.

> * Hundreds of lines of rules follow.

Most of these rules are to support ugly things like e.g. the complicated ddoc or to workaround shortcomings in the standard library (e.g. no safe cross-platform scheduler).

> Did build.d attain its objective?

Absolutely. One big example is that people can now build DMD on Windows (this was _very_ hard/near to impossible before).

> * The source files are hardcoded somewhere starting at line 1160. I wish I was kidding.

build.d used globbing initially. It was removed, because Walter explicitly requested hard-coding.
I can't find the PR with the request at the moment, but here's one of the PRs which removed the globbing in favor of hard-coding:

https://github.com/dlang/dmd/pull/10212

> * A lot more support code follows, which is fine for a single-file build tool. I wish this were a two-files affair a la build.d and buildinfo.d, or if single file have an obvious separation:

While we could have done a separation for rdmd which you struck done a while ago [1] which btw lead to the only two people interested in contributing to rdmd loosing their interest.

Now, to answer your question fully I have to add that one of those two people (Jonathan Marler) has since then decided to create a better rdmd: rund [2]. Rund fixes one of the main shortcomings of rdmd (the need to compile a file twice with dmd and thus having an almost doubled run time) and adds a couple of other handy features like being able to specify execution flags, e.g.

---
#!/usr/bin/env rund
//!importPath <path>
---

If rdmd would have such a feature, we could split build.d into multiple files. Unfortunately, it doesn't and thus it must remain a single file as `./build.d` needs to work out of the box.

[1] https://github.com/dlang/tools/pull/344
[2] https://github.com/dragon-lang/rund

> or if single file have an obvious separation:

The separation is already there [1] , but I guess one could make it more obious.

[1] https://github.com/dlang/dmd/blob/master/src/build.d#L768

> I've often said gmake should be used for Windows to build dmd. Now we require dmd to build dmd...

Yes, how else is one going to build D code since 2.067?

> Of course that ruined the carefully constructed AUTO_BOOTSTRAP option that allows building dmd on a fresh system.

Nope. In fact, AUTO_BOOTSTRAP is still being used by some of our own CIs, e.g. Buildkite still does

```
make -f posix.mak AUTO_BOOTSTRAP=1
```

Some other do this to get an active D compiler in your environment:

```
source $(curl https://dlang.org/install.sh | bash -s install -a)
```

> Not when I tried it: https://forum.dlang.org/post/r5e8uj$i1b$1@digitalmars.com

To be fair that's not a fault of build.d. You aren't able to compile anything with an outdated compiler on OSX. For details see some of the responses to your thread:

https://forum.dlang.org/post/rimiuawxnzobcgfuwxez@forum.dlang.org
https://forum.dlang.org/post/zvsjvhvgprhgacybvctb@forum.dlang.org

On Sunday, 14 June 2020 at 06:07:40 UTC, Max Samukha wrote:
> On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:
>
>> Where. Is. The. Progress.
>
> Progress would be to fix the makefiles so that they are usable out-of-the-box with the newer versions of VS. This is abysmal:
>
> VCDIR=\Program Files (x86)\Microsoft Visual Studio 10.0\VC
> SDKDIR=\Program Files (x86)\Microsoft SDKs\Windows\v7.0A
>
> ...
>
> CC=$(VCDIR)\bin\amd64\cl
> LD=$(VCDIR)\bin\amd64\link
> AR=$(VCDIR)\bin\amd64\lib

For DMD, the Makefiles are only there for legacy CI reasons or AUTO_BOOTSTRAP and they might even be fully removed soonish.

So at least to build DMD, everything should work out of the box on any platform when executing `./src/build.d` (or `rdmd src\build.d` on Windows) and there should be no need to fight with the hard-coded paths to Visual Studio 10.
If not, please open issues on Bugzilla (and/or report here).

On Sunday, 14 June 2020 at 04:26:00 UTC, Walter Bright wrote:
> On 6/13/2020 6:44 AM, kinke wrote:
>> Side note: LDC has been using GNUmake on Windows for years, incl. extending some upstream Makefiles to work with and test Windows targets too.
>> While GNUmake itself is surprisingly portable (source comes with a Visual Studio project working out of the box, resulting in a little 330 KB Win64 stand-alone executable),
>
> DM make is 52K :-)

Size doesn't matter if it doesn't work.

On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
> On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:
>> The original Andrei, just like today's Andrei, has an appreciation for good engineering. I didn't feel the need to add to provide detail because (a) most regulars in this forum already knew what I was going to say, and (b) nobody save for a few would share my opinion.
>>
>> But, I'll bite again, again to regret it.
>
> I'll reply with similar regrets, because I agree that this discussion isn't helping anyone.

It's helping me.

[big snip of clarifying context and concise refutations of many earlier assertions]

Thanks Seb, for the context, the refutations, and for the references to alternative tools (rund, reggae).

On 6/14/20 1:05 PM, Bruce Carneal wrote:
> On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
>> On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:
>>> The original Andrei, just like today's Andrei, has an appreciation for good engineering. I didn't feel the need to add to provide detail because (a) most regulars in this forum already knew what I was going to say, and (b) nobody save for a few would share my opinion.
>>>
>>> But, I'll bite again, again to regret it.
>>
>> I'll reply with similar regrets, because I agree that this discussion isn't helping anyone.
> 
> It's helping me.
> 
> [big snip of clarifying context and concise refutations of many earlier assertions]
> 
> Thanks Seb, for the context, the refutations, and for the references to alternative tools (rund, reggae).

Most of said refutations are reducible to simple misunderstandings. A few are due to my oversights. A few are Seb's. My core point stays: build.d is not Good Work(tm), and can't be talked into it. It must be worked into it.

I don't want the makefiles back. I want Good Work, which build.d has a good way to go toward. A good part of my critique can be addressed by refactoring build.d in the ways I suggested. Another part of the critique would be achieved by a pass through its design, with an eye for making dependency syntax tolerable.