Here's[1] another attempt at converting the Makefile for POSIX systems to D using reggae[2]. I first tried my hand at writing a parser for Makefiles using Pegged and machine translating to D. I learned a lot about parsing and GNU make in the process and also that:

1. It was about two orders of magnitude more work than I thought to support the subset of GNU make used by posix.mak
2. The generated D code wasn't exactly pretty

I concluded that worse is better and hand translated all of posix.mak instead. It isn't as pretty as possible either, but that's on purpose - since I'll have to mentally apply any further diffs and hand translate, I tried to make it as close as possible to the existing Makefile. That way the edits should be obvious.

AFAICT by testing manually, it does everything that the original Makefile does, including all the different unit test targets and documentation. Then again, I only tested on Linux. I've been trying to bend the auto-tester to my will to make sure all posix.mak systems are still green but I ran into configuration issues.

I'll carry on maintaining the D description as updates to posix.mak happen for a year. If we haven't switched by then we probably never will.

To try it out yourself, get & build/run reggae from dub in this branch at the root dir of phobos selecting the backend of choice with `-b <backend>`. I suggest the binary one since one of the points of this is to remove dependencies from other build systems. Then run make/ninja/tup/./build with the same targets as usual. If that doesn't work... open an issue on github.

Destroy!

Atila


[1] https://github.com/atilaneves/phobos/blob/reggae/reggaefile.d
[2] https://github.com/atilaneves/reggae

On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:
> Here's[1] another attempt at converting the Makefile for POSIX systems to D using reggae[2].

Great work! This transition cannot happen soon enough. Maintaining the make files is a huge pain and something always falls through the cracks.

Am Mon, 18 Apr 2016 15:15:26 +0000
schrieb Atila Neves <atila.neves@gmail.com>:

Just remember that makefiles are well integrated with other systems (i.e. package managers) and there are some conventions, about what certain environment variables mean. To name a few:

MAKEOPTS=-j4  (parallel build with N processes)
DC=/opt/ldc2-0.17/bin/ldc2  (Dlang compiler)
DC_FLAGS=-O5  (flags for that compiler)
LDFLAGS=-L=-O1  (flags for the linker)
PREFIX=/usr  (installation prefix)

Dlang projects already have some make files that handle linker
flag prefixes while others will pass them verbatim to gcc and
some make files accept a DMD (ldmd2, gdmd, dmd) and others a
DC (ldc2, gdc, dmd). The more things we can override, the
better.
As a practical example, on Gentoo, the package management sets
all kinds of environment variables, most importantly PREFIX,
LDFLAGS, CFLAGS and CXXFLAGS. It would be great if any
replacement system would also honor those (translated to D).
Even better if it could handle GDC and LDC2 directly (since
gdmd and ldmd2 don't do all the way).
LDFLAGS need translation from the default (GCC) syntax to DMD
and LDC2. I currently do this in a helper script and would not
depend on the build script doing that.

-- 
Marco

On Monday, 18 April 2016 at 23:17:39 UTC, Marco Leise wrote:
> Am Mon, 18 Apr 2016 15:15:26 +0000
> schrieb Atila Neves <atila.neves@gmail.com>:
>
> Just remember that makefiles are well integrated with other systems (i.e. package managers) and there are some conventions, about what certain environment variables mean. To name a few:
>
> MAKEOPTS=-j4  (parallel build with N processes)

This is make-dependent though. ninja and the reggae binary backend run with the number of cores by default.

> DC=/opt/ldc2-0.17/bin/ldc2  (Dlang compiler)

DC, CC, and CXX all work already.

> DC_FLAGS=-O5  (flags for that compiler)
> LDFLAGS=-L=-O1  (flags for the linker)
> PREFIX=/usr  (installation prefix)

These I need to add. Well, if there's enough of a use-case anyway.

> Dlang projects already have some make files that handle linker
> flag prefixes while others will pass them verbatim to gcc and
> some make files accept a DMD (ldmd2, gdmd, dmd) and others a
> DC (ldc2, gdc, dmd). The more things we can override, the
> better.

The phobos makefile only has DMD.

> As a practical example, on Gentoo, the package management sets
> all kinds of environment variables, most importantly PREFIX,
> LDFLAGS, CFLAGS and CXXFLAGS. It would be great if any
> replacement system would also honor those (translated to D).

I didn't even know CFLAGS was a default!

> Even better if it could handle GDC and LDC2 directly (since
> gdmd and ldmd2 don't do all the way).

Here it'd be better to improve gdmd and ldmd2, I think. In any case it's a leaky abstraction. I've never seen a CMake build that doesn't have ifs for what compiler is being used in order to choose the appropriate flags.

> LDFLAGS need translation from the default (GCC) syntax to DMD
> and LDC2. I currently do this in a helper script and would not
> depend on the build script doing that.

Why does it need translating? If you use the system linker directly, use LDFLAGS as is. If not, use whatever linker flags are appropriate for the compiler being used (-L-l... for instance).

Atila

Am Tue, 19 Apr 2016 17:02:43 +0000
schrieb Atila Neves <atila.neves@gmail.com>:

> > DC_FLAGS=-O5  (flags for that compiler)
> > LDFLAGS=-L=-O1  (flags for the linker)
> > PREFIX=/usr  (installation prefix)
> 
> These I need to add. Well, if there's enough of a use-case anyway.

It's just normal on Gentoo, to be able to set all the compiler and linker flags and have atypical configurations for their software. One guy with a security enhanced Linux and GCC installation required -fPIC to be passed. This is easy when the build system provides a facility for it as otherwise I need to maintain patches.

> > Dlang projects already have some make files that handle linker flag prefixes while others will pass them verbatim to gcc and some make files accept a DMD (ldmd2, gdmd, dmd) and others a DC (ldc2, gdc, dmd). The more things we can override, the better.
> 
> The phobos makefile only has DMD.

Well, it has the compiler that ships with the sources anyways. Point taken.

> > As a practical example, on Gentoo, the package management sets all kinds of environment variables, most importantly PREFIX, LDFLAGS, CFLAGS and CXXFLAGS. It would be great if any replacement system would also honor those (translated to D).
> 
> I didn't even know CFLAGS was a default!

It is for makefiles. These conventions that grew over time
make them convenient to the expert. But they are hardly
visible so easily missed when seeking to replace make files.
https://en.wikipedia.org/wiki/CFLAGS
On a typical Gentoo "ricer" system they'd include e.g.
  -O3 -flto -march=native

> > Even better if it could handle GDC and LDC2 directly (since
> > gdmd and ldmd2 don't do all the way).
> 
> Here it'd be better to improve gdmd and ldmd2, I think. In any case it's a leaky abstraction. I've never seen a CMake build that doesn't have ifs for what compiler is being used in order to choose the appropriate flags.

Maybe you are right. I hope passing "native" arguments to gdc and ldc2 through their wrappers works reliably and gdmd stops reading dmd.conf.

> > LDFLAGS need translation from the default (GCC) syntax to DMD and LDC2. I currently do this in a helper script and would not depend on the build script doing that.
> 
> Why does it need translating? If you use the system linker directly, use LDFLAGS as is. If not, use whatever linker flags are appropriate for the compiler being used (-L-l... for instance).

I can only speak for Gentoo, where this env. variable is assumed to be passed as flags to GCC in the linking step, so the options are already prefixed with "-Wl,". For passing them to the linker the prefixes needs to be stripped and for passing them to ldc2 they need to be replaced by -L=. I have code in place in the compiler selection logic for Dlang packages that performs this translation. I don't know how exactly you plan to handle this in your script. E.g. translate LDFLAGS form GCC to DMD style, expect the user to set them in DMD style or handle all three compilers by translating to each of them.

-- 
Marco

On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:
> Here's[1] another attempt at converting the Makefile for POSIX systems to D using reggae[2].
...
> Destroy!
>
> Atila

I know you your intention was to keep it similar to makefile, but for me it looks unnecessarily complex.

What whould you say about different approach e.g.:

void configure()
{
    with (targets["linux shared library"])
    {
        inheritConfig("linux 32");
        sources = "lib/*.d";
        dependency ~= "objects";
        output = "somthing.so";
    }
}


Piotrek

On Friday, 22 April 2016 at 17:00:21 UTC, Piotrek wrote:
> On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:
>> Here's[1] another attempt at converting the Makefile for POSIX systems to D using reggae[2].
> ...
>> Destroy!
>>
>> Atila
>
> I know you your intention was to keep it similar to makefile, but for me it looks unnecessarily complex.
>
> What whould you say about different approach e.g.:
>
> void configure()
> {
>     with (targets["linux shared library"])
>     {
>         inheritConfig("linux 32");
>         sources = "lib/*.d";
>         dependency ~= "objects";
>         output = "somthing.so";
>     }
> }
>
>
> Piotrek

I think simpler would be better and that it could definitely be made simpler.

But none of that matters unless we can switch, and we won't be able to if I can't maintain it in parallel.

It's complex because posix.mak is complex.

Atila

On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:
> Here's[1] another attempt at converting the Makefile for POSIX systems to D using reggae[2]. I first tried my hand at writing a parser for Makefiles using Pegged and machine translating to D. I learned a lot about parsing and GNU make in the process and also that:
>
> 1. It was about two orders of magnitude more work than I thought to support the subset of GNU make used by posix.mak
> 2. The generated D code wasn't exactly pretty
>
> I concluded that worse is better and hand translated all of posix.mak instead. It isn't as pretty as possible either, but that's on purpose - since I'll have to mentally apply any further diffs and hand translate, I tried to make it as close as possible to the existing Makefile. That way the edits should be obvious.
>
> AFAICT by testing manually, it does everything that the original Makefile does, including all the different unit test targets and documentation. Then again, I only tested on Linux. I've been trying to bend the auto-tester to my will to make sure all posix.mak systems are still green but I ran into configuration issues.
>
> I'll carry on maintaining the D description as updates to posix.mak happen for a year. If we haven't switched by then we probably never will.
>
> To try it out yourself, get & build/run reggae from dub in this branch at the root dir of phobos selecting the backend of choice with `-b <backend>`. I suggest the binary one since one of the points of this is to remove dependencies from other build systems. Then run make/ninja/tup/./build with the same targets as usual. If that doesn't work... open an issue on github.
>
> Destroy!
>
> Atila
>
>
> [1] https://github.com/atilaneves/phobos/blob/reggae/reggaefile.d
> [2] https://github.com/atilaneves/reggae

Phobos posix.mak in D:

https://github.com/atilaneves/phobos/blob/new_reggae/reggaefile.d

I've given up on making it look like the makefile for maintenance reasons and tried to write it "properly". At the end of the day, that phobos posix.mak just does a lot of stuff so even though I tried getting rid of cruft it's still rather large.

Notice however how there are no explicit listings of packages or converting them or anything like that. This isn't the best I can do, but it was a lot of work already so I thought I'd share.

This new attempt does everything posix.mak at that point in time did except for running unit tests per package. I think that's the test runner's job anyway. I also changed the individual unit test targets from e.g. std/array.test to std.array.test because, well, it makes a lot more sense.

Atila