Although it took longer than I expected to get around to it, I'm working on a release-generator tool for DMD. I'm finding that a very significant amount of the effort involved (much more than I expected) is discovering and dealing with all the fun little differences between the posix and win32 makefiles (and now we have some win64 makefiles as well).

Efforts can be made to decrease these differences, but simply having them separate makefiles in the first place (let alone using completely different "make"s: GNU make vs DM make) is a natural invitation for divergence.

No disrespect intended to Digital Mars Make, but since GNU make appears to be more feature-rich, have wider overall adoption, and is freely available on Windows as a pre-built binary <http://gnuwin32.sourceforge.net/packages/make.htm>: Would it be acceptable to use gmake as *the* make for DMD? Ie, either convert the windows makefiles to gmake, or expand the posix makefiles to support windows?

I'd be willing to give it a shot myself, and I could trivially
write a small batch utility to download Win gmake and put it on the
current PATH, so that nobody has to go downloading/installing it
manually. I would do this *after* finishing the release-generator tool,
but afterwords it would allow the tool's implantation to be greatly
simplified.

Is this something that would be acceptable, or does building DMD for Windows need to stay as DM make?

(Sorry for the accidental cross-post to "announce", can someone delete
that one?)

On Saturday, August 10, 2013 14:35:04 Nick Sabalausky wrote:
> Is this something that would be acceptable, or does building DMD for Windows need to stay as DM make?

I don't see any problem with it, but that doesn't mean that Walter won't.

Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.

- Jonathan M Davis

On Sat, 10 Aug 2013 16:21:45 -0700
Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> 
> Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.
> 

Yea, while I do like that too, it would make bootstrapping difficult.

But then again, if parts of DMD start being written in D (as there has been some talk about), then that would have to deal with the exact same bootstrapping issue anyway.

Although, if that does happen (parts of DMD written in D), then I'd imagine it may help a lot to do it *starting* from a really good solid makefile instead of the inconsistent makefiles we have now. *Then* we could transition to a D-based buildscript if we really wanted, but I think starting with a D-based buildscript, or the current posix/win makefiles, could just make everything messier.

The posix makefiles actually aren't too bad at this point (the "generated" directories phobos/druntime use on posix are a big improvement) but the windows makefiles seem to be lagging behind. At the very least, I'd like to see that situation engineered away via common posix/windows makefiles - which of course requires using the same "make".

On Sat, Aug 10, 2013 at 04:21:45PM -0700, Jonathan M Davis wrote:
> On Saturday, August 10, 2013 14:35:04 Nick Sabalausky wrote:
> > Is this something that would be acceptable, or does building DMD for Windows need to stay as DM make?
> 
> I don't see any problem with it, but that doesn't mean that Walter won't.
> 
> Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.
[...]

I'm all for ditching make.  What about this:

- We write a small D app that automatically scans all dependencies and
  generates a shell script / .BAT file / whatever the target platform
  uses, that contains compile commands that builds DMD and a make
  replacement written in D. This is for bootstrapping.

- The make replacement written in D can then be used to rebuild DMD,
  build druntime, Phobos, etc..

The first step is what makes this all work, 'cos you'll need to already have a working D compiler before step 2 is usable. (Either that, or ship binaries, but then you'll get people complaining about their platform of choice not being supported, the binaries being incompatible with their quirky installation of system libraries, etc..) Once DMD is built, we can junk the script / .BAT file and use the D make-replacement from then on.


T

-- 
People tell me that I'm skeptical, but I don't believe it.

On Saturday, August 10, 2013 20:06:32 Nick Sabalausky wrote:
> On Sat, 10 Aug 2013 16:21:45 -0700
> 
> Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.
> 
> Yea, while I do like that too, it would make bootstrapping difficult.
> 
> But then again, if parts of DMD start being written in D (as there has been some talk about), then that would have to deal with the exact same bootstrapping issue anyway.

Yeah, it introduces a circular dependency, but it's one that we're planning it introduce anyway.

> Although, if that does happen (parts of DMD written in D), then I'd imagine it may help a lot to do it *starting* from a really good solid makefile instead of the inconsistent makefiles we have now. *Then* we could transition to a D-based buildscript if we really wanted, but I think starting with a D-based buildscript, or the current posix/win makefiles, could just make everything messier.
> 
> The posix makefiles actually aren't too bad at this point (the "generated" directories phobos/druntime use on posix are a big improvement) but the windows makefiles seem to be lagging behind. At the very least, I'd like to see that situation engineered away via common posix/windows makefiles - which of course requires using the same "make".

I actually think that a build script written in D could be quite clean, but it would obviously not be even vaguely standard, which could be viewed as a definite con. Really, I don't care all that much how the build scripts are put together just so long as they're reasonably maintainable, and what we have right now isn't, especially on the Windows side.

If I had to tackle the problem though, I'd likely tackle it with a script written in D. But I've got enough on my plate already without worrying about the built system. Thanks for looking into it.

- Jonathan M Davis

On Saturday, August 10, 2013 17:14:35 H. S. Teoh wrote:
> On Sat, Aug 10, 2013 at 04:21:45PM -0700, Jonathan M Davis wrote:
> > On Saturday, August 10, 2013 14:35:04 Nick Sabalausky wrote:
> > > Is this something that would be acceptable, or does building DMD for Windows need to stay as DM make?
> > 
> > I don't see any problem with it, but that doesn't mean that Walter won't.
> > 
> > Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.
> 
> [...]
> 
> I'm all for ditching make.  What about this:
> 
> - We write a small D app that automatically scans all dependencies and
>   generates a shell script / .BAT file / whatever the target platform
>   uses, that contains compile commands that builds DMD and a make
>   replacement written in D. This is for bootstrapping.
> 
> - The make replacement written in D can then be used to rebuild DMD,
>   build druntime, Phobos, etc..
> 
> The first step is what makes this all work, 'cos you'll need to already have a working D compiler before step 2 is usable. (Either that, or ship binaries, but then you'll get people complaining about their platform of choice not being supported, the binaries being incompatible with their quirky installation of system libraries, etc..) Once DMD is built, we can junk the script / .BAT file and use the D make-replacement from then on.

Since we're going to be porting dmd to D, which will force you to have a D compiler to build the D compiler anyway, I don't see any reason to jump through hoops to make anything bootstrappable. In the long run, you're going to have to either start with a dmd which compiled with C++ or cross-compile it from a machine which already has dmd, which is exactly the same boat that C/C++ are in. It's just that they've been around a lot longer, are supported on more platforms, and don't change as much.

- Jonathan M Davis

On Sat, 10 Aug 2013 17:14:35 -0700
"H. S. Teoh" <hsteoh@quickfur.ath.cx> wrote:
> 
> I'm all for ditching make.  What about this:
> 
> - We write a small D app that automatically scans all dependencies and
>   generates a shell script / .BAT file / whatever the target platform
>   uses, that contains compile commands that builds DMD and a make
>   replacement written in D. This is for bootstrapping.
> 
> - The make replacement written in D can then be used to rebuild DMD,
>   build druntime, Phobos, etc..
> 


That's a very interesting idea. Couple thoughts:

- It sounds a lot like RDMD, just with shell-script output added
  (actually, I think RDMD can already generate makefiles). So I'm
  wondering how much of RDMD could be leveraged for this. But maybe not
  much since RDMD is specially-tailored to scanning D-based projects,
  not C/C++ ones like DMD (but druntime and phobos OTOH...). I guess
  the tool you're talking about would be specifically-designed to scan
  DMD's sources. Or is there some existing C/C++ tool we could/should
  leverage?

- What happens when DMD starts using D-language sources? The generated
  bootstrapper shell scripts would no longer be able to compile DMD
  because DMD hasn't yet been built. So it wouldn't solve *that* issue,
  but I suppose we can just shell-bootstrap the last pure-C/C++ DMD as
  an "origin" DMD and use that to compile the next DMD (or the most
  recent one it's capable of compiling), and then use that to compile
  the next-in-line DMD, and so on up to whatever's the latest. (Which
  is probably what would have to happen *anyway* even without the tool
  you suggest.)

Sounds like that should work, and fairly well, too (as long as DMD's D-based sources are careful to be compilable with *sufficiently* older versions of itself). And it eliminates any point in bothering to make any big improvements on the makefiles.

On Sat, 10 Aug 2013 17:23:08 -0700
Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Saturday, August 10, 2013 17:14:35 H. S. Teoh wrote:
> > On Sat, Aug 10, 2013 at 04:21:45PM -0700, Jonathan M Davis wrote:
> > > On Saturday, August 10, 2013 14:35:04 Nick Sabalausky wrote:
> > > > Is this something that would be acceptable, or does building DMD for Windows need to stay as DM make?
> > > 
> > > I don't see any problem with it, but that doesn't mean that Walter won't.
> > > 
> > > Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.
> > 
> > [...]
> > 
> > I'm all for ditching make.  What about this:
> > 
> > - We write a small D app that automatically scans all dependencies and generates a shell script / .BAT file / whatever the target platform uses, that contains compile commands that builds DMD and a make replacement written in D. This is for bootstrapping.
> > 
> > - The make replacement written in D can then be used to rebuild DMD,
> >   build druntime, Phobos, etc..
> > 
> > The first step is what makes this all work, 'cos you'll need to already have a working D compiler before step 2 is usable. (Either that, or ship binaries, but then you'll get people complaining about their platform of choice not being supported, the binaries being incompatible with their quirky installation of system libraries, etc..) Once DMD is built, we can junk the script / .BAT file and use the D make-replacement from then on.
> 
> Since we're going to be porting dmd to D, which will force you to have a D compiler to build the D compiler anyway, I don't see any reason to jump through hoops to make anything bootstrappable. In the long run, you're going to have to either start with a dmd which compiled with C++ or cross-compile it from a machine which already has dmd, which is exactly the same boat that C/C++ are in. It's just that they've been around a lot longer, are supported on more platforms, and don't change as much.
> 

A fair point, although cross-compiling can be a real pain. But maybe that's not too big of an issue: after all, how often would it need to be done? And we could always still resort to building H.S.'s idea if such became necessary.

I guess the big question is (largely posed at D's leaders, top dogs and
OS-package builders):

Have we reached a point where we would be comfortable ditching the makefiles, given a sufficiently well-written D-based alternative?

On 8/10/2013 4:21 PM, Jonathan M Davis wrote:
> On Saturday, August 10, 2013 14:35:04 Nick Sabalausky wrote:
>> Is this something that would be acceptable, or does building DMD for
>> Windows need to stay as DM make?
>
> I don't see any problem with it, but that doesn't mean that Walter won't.

Tools built for Unix never work right on Windows. It's why, for example, I run git on Linux and don't use the Windows badly ported versions of git. Tiresome problems revolve around failure to adapt to \ path separators, ; in PATH, CRLF line endings, Windows SEH, case insensitive file names, no symbolic links, etc., no Perl installed, etc.

DMD and Phobos are fairly unusual in how well adapted they are to both Windows and Linux.


> Another suggestion that I kind of liked was to just build them all with a
> single script written in D and ditch make entirely, which would seriously
> reduce the amount of duplication across platforms. But that's obviously a much
> bigger change and would likely be much more controversial than simply using a
> more standard make.

I don't see much point in that. The dmd build is straightforward, and I see no particular gain from reinventing that wheel.

On Sunday, 11 August 2013 at 05:48:19 UTC, Walter Bright wrote:
> On 8/10/2013 4:21 PM, Jonathan M Davis wrote:
>> On Saturday, August 10, 2013 14:35:04 Nick Sabalausky wrote:
>>> Is this something that would be acceptable, or does building DMD for
>>> Windows need to stay as DM make?
>>
>> I don't see any problem with it, but that doesn't mean that Walter won't.
>
> Tools built for Unix never work right on Windows. It's why, for example, I run git on Linux and don't use the Windows badly ported versions of git. Tiresome problems revolve around failure to adapt to \ path separators, ; in PATH, CRLF line endings, Windows SEH, case insensitive file names, no symbolic links, etc., no Perl installed, etc.
>

Things can be wonky from a vanilla windows command prompt, which is why I never use any Linux tools there. MSYS makes all those problems go away. I use git exclusively on windows, but via gitbash, which is built on top of MSYS.

Of course, it would be silly to require MSYS or Cygwin to build on Windows, but there's always CMake. A number of open source projects use it these days. Ship a configuration file with the source, then the user can generate Makefiles for a number of compilers and platforms.