Okay, so I'm prepping up a SCons build of Phobos. It comes along rather nicely, I've replicated most of posix.mak paraphernalia in ~150 LOC that does work for both Windows and Linux, the rest of POSIX to follow shortly.

Some make targets are trivial to translate (ddocs, zips etc.) but unittest runner and SO/PIC build flags is an arcane mess. So I'm asking if anybody know what's the heck it tries to do and can just tell me to save the time on reverse-enginering.

What I know(?) so far:
1. First we build library in one go - trivial to reproduce.
2. Then we compile each unittest with -c and -deps to dump actual dependencies.
3. Then we run a bunch of sed/sort/uniq to extract module names from verbose output of compiler (red flag IMHO, but anyway).
https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L325
4. We promptly ignore these files afterwards...
5. We build a unittester from Druntime (pulling sources out of tree - omg) with ALL of object files:
https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L334
6. Run it passing a specific module to unittest:
https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L355

I hope I got it right. And I'm not entirely sure where SO/DLL fits (except for setting the PIC flag which is quite straight-forward).

My hopes are that once I get over these obstacle and replicate whatever we wanted here in the first, we could get down to 2 files for all of our platform builds:
1. SConstruct - setup environment, detect OS, fiddle with parameters and in general defines "how" to build stuff (estimate: 160-180 LOCs)
2. SConscript - a laconic file showing what to build, doing so in simple cross platform, with plain *.d globs. (estimate: about 60-70 LOCs)

That means in general that SConstruct is changed only for new platform/compiler and SConscript only when source/targets structure radically changes.


-- 
Dmitry Olshansky

On Thu, Nov 27, 2014 at 11:17:34PM +0300, Dmitry Olshansky via Digitalmars-d wrote:
> Okay, so I'm prepping up a SCons build of Phobos.

Hooray!


> It comes along rather nicely, I've replicated most of posix.mak paraphernalia in ~150 LOC that does work for both Windows and Linux, the rest of POSIX to follow shortly.

Very nice!


> Some make targets are trivial to translate (ddocs, zips etc.) but unittest runner and SO/PIC build flags is an arcane mess.

Well, building SO/PIC itself is an arcane mess. ;-)


> So I'm asking if anybody know what's the heck it tries to do and can just tell me to save the time on reverse-enginering.

I've tried looking into that before... unfortunately I didn't have the patience to figure it out either.


> What I know(?) so far:
> 1. First we build library in one go - trivial to reproduce.
> 2. Then we compile each unittest with -c and -deps to dump actual
> dependencies.
> 3. Then we run a bunch of sed/sort/uniq to extract module names from verbose
> output of compiler (red flag IMHO, but anyway).
> https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L325
> 4. We promptly ignore these files afterwards...

Wait, what? What's the point of running -deps and sed/sort/whatever if we don't actually care about it in the first place?


> 5. We build a unittester from Druntime (pulling sources out of tree -
> omg) with ALL of object files:
> https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L334

Well, the current makefile *does* assume druntime is in ../druntime (it does strange things if that's not true, and that's not just for the unittester), so what can you say...


> 6. Run it passing a specific module to unittest: https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L355

Whoa, really?! Why can't we do per-module test drivers?


> I hope I got it right. And I'm not entirely sure where SO/DLL fits (except for setting the PIC flag which is quite straight-forward).

I think you have to do a second pass, i.e., recompile the entire Phobos with -pic, -lib, etc., to get the .so.

Also, don't forget the 'html' target that builds the docs. Currently it's quite broken (e.g., std.windows.charset is not being built, so on dlang.org this is a dangling hyperlink). But basically, you just compile the entire Phobos all over again with -D, -Dd, etc., and include the .ddoc files on each command-line. Shouldn't be too hard to do in SCons, though.

My personal advice is to run this as a *separate* step (`dmd -o- -D -Dd...`), and don't try to "optimize" by combining ddoc generation with actual compilation, since that will make your SCons dependency tree *really* hairy, and you might end up with subtle bugs if you don't insert the correct Depends() lines or if you don't specify *all* files output by dmd. Plus, for the doc-generation build, you want to generate docs for *all* source files, even those not intended for the host platform, so your list of input files will be different from the actual build. Better keep the two separate.


> My hopes are that once I get over these obstacle and replicate
> whatever we wanted here in the first, we could get down to 2 files for
> all of our platform builds:
> 1. SConstruct - setup environment, detect OS, fiddle with parameters
> and in general defines "how" to build stuff (estimate: 160-180 LOCs)
> 2. SConscript - a laconic file showing what to build, doing so in
> simple cross platform, with plain *.d globs. (estimate: about 60-70
> LOCs)
> 
> That means in general that SConstruct is changed only for new platform/compiler and SConscript only when source/targets structure radically changes.
[...]

Sounds like an excellent idea!


T

-- 
By understanding a machine-oriented language, the programmer will tend to use a much more efficient method; it is much closer to reality. -- D. Knuth

On Thu, 27 Nov 2014 23:17:34 +0300
Dmitry Olshansky via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> Okay, so I'm prepping up a SCons build of Phobos. It comes along rather nicely, I've replicated most of posix.mak paraphernalia in ~150 LOC that does work for both Windows and Linux, the rest of POSIX to follow shortly.
> 
> Some make targets are trivial to translate (ddocs, zips etc.) but unittest runner and SO/PIC build flags is an arcane mess. So I'm asking if anybody know what's the heck it tries to do and can just tell me to save the time on reverse-enginering.
> 
> What I know(?) so far:
> 1. First we build library in one go - trivial to reproduce.
> 2. Then we compile each unittest with -c and -deps to dump actual
> dependencies.
> 3. Then we run a bunch of sed/sort/uniq to extract module names from
> verbose output of compiler (red flag IMHO, but anyway).
> https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L325
> 4. We promptly ignore these files afterwards...
> 5. We build a unittester from Druntime (pulling sources out of tree -
> omg) with ALL of object files:
> https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L334
> 6. Run it passing a specific module to unittest:
> https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L355
> 
> I hope I got it right. And I'm not entirely sure where SO/DLL fits (except for setting the PIC flag which is quite straight-forward).
> 
> My hopes are that once I get over these obstacle and replicate whatever
> we wanted here in the first, we could get down to 2 files for all of our
> platform builds:
> 1. SConstruct - setup environment, detect OS, fiddle with parameters and
> in general defines "how" to build stuff (estimate: 160-180 LOCs)
> 2. SConscript - a laconic file showing what to build, doing so in simple
> cross platform, with plain *.d globs. (estimate: about 60-70 LOCs)
> 
> That means in general that SConstruct is changed only for new platform/compiler and SConscript only when source/targets structure radically changes.
> 
does this mean that 'make' will be eventually dropped? oh, noes...

On Thu, Nov 27, 2014 at 11:30:49PM +0200, ketmar via Digitalmars-d wrote:
> On Thu, 27 Nov 2014 23:17:34 +0300
> Dmitry Olshansky via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> 
> > Okay, so I'm prepping up a SCons build of Phobos. It comes along rather nicely, I've replicated most of posix.mak paraphernalia in ~150 LOC that does work for both Windows and Linux, the rest of POSIX to follow shortly.
[...]
> does this mean that 'make' will be eventually dropped? oh, noes...

One idea I had, which is easily done in SCons, is to auto-generate makefiles for each platform. On the dev box, run scons with a particular virtual target, say `scons genmake` or something like that, and it will iterate over each supported platform and spit out a makefile using the dependency tree it already knows. The generated makefile will be a list of hard-coded rules for building stuff, with no macros and what-not, so it will Just Work, but not do much more than that. These generated makefiles can then be shipped as part of the source distribution, but they need not (and probably should not!) be in the git tree.


T

-- 
Many open minds should be closed for repairs. -- K5 user

On Thu, 27 Nov 2014 13:39:37 -0800
"H. S. Teoh via Digitalmars-d" <digitalmars-d@puremagic.com> wrote:

> makefiles can then be shipped as part of the source distribution, but they need not (and probably should not!) be in the git tree.
hope this will not happen soon. i used to build DMD from git head, but i don't want to install python for that. ;-)

On 11/28/2014 6:39 AM, H. S. Teoh via Digitalmars-d wrote:
>
> One idea I had, which is easily done in SCons, is to auto-generate
> makefiles for each platform. On the dev box, run scons with a particular
> virtual target, say `scons genmake` or something like that, and it will
> iterate over each supported platform and spit out a makefile using the
> dependency tree it already knows. The generated makefile will be a list
> of hard-coded rules for building stuff, with no macros and what-not, so
> it will Just Work, but not do much more than that. These generated
> makefiles can then be shipped as part of the source distribution, but
> they need not (and probably should not!) be in the git tree.
>

I would think that premake or CMake would be better for that sort of thing. In addition to the pregenerated Makefiles, you could also keep premake.lua (or the CMake stuff) in the source tree and then anyone who wanted could generate Visual Studio project files, customized makefiles or whatever else they needed.

"H. S. Teoh via Digitalmars-d"  wrote in message news:mailman.2384.1417124501.9932.digitalmars-d@puremagic.com...

> > does this mean that 'make' will be eventually dropped? oh, noes...
>
> One idea I had, which is easily done in SCons, is to auto-generate
> makefiles for each platform. On the dev box, run scons with a particular
> virtual target, say `scons genmake` or something like that, and it will
> iterate over each supported platform and spit out a makefile using the
> dependency tree it already knows. The generated makefile will be a list
> of hard-coded rules for building stuff, with no macros and what-not, so
> it will Just Work, but not do much more than that. These generated
> makefiles can then be shipped as part of the source distribution, but
> they need not (and probably should not!) be in the git tree.

I think we should be ok adding the generated files to source control, so long as the autotester is set up to verify they have been updated. 

On 2014-11-27 23:07, ketmar via Digitalmars-d wrote:

> hope this will not happen soon. i used to build DMD from git head, but i
> don't want to install python for that. ;-)

There's usually no problems with Python on Posix, but on Windows, I really don't want that. I really like that DMD has so few dependencies.

-- 
/Jacob Carlborg

I never liked SCons for some reason. I prefer CMake over it. Waf is IMHO better than SCons too. Maybe it is more fair to compare SCons and Waf as they both are Python-based.

Anyhow, use whatever works for you. :)

On Friday, 28 November 2014 at 08:45:30 UTC, Jacob Carlborg wrote:
>
> There's usually no problems with Python on Posix, but on Windows, I really don't want that. I really like that DMD has so few dependencies.

Same here. I prefer the current situation where we build DMD and runtime using Make.