On Wed, Feb 21, 2018 at 11:03:13AM +0000, Russel Winder wrote: [...]
> I have been (still am?) a SCons fan but it has some serious problems
> in some workflows.
> 
> I think all of the things you mention are solved in a system that has a general purpose programming language to describe the project and the build. There is clearly a tension between specifying a project in a purely declarative way (cf. Cargo, Dub, Maven) where all build and deploy activities are effectively hardwired (though Maven has plugins to amend things) versus systems that use a programming language.
[...]

I think the ideal situation straddles the divide between declarative build specs and a full-fledged general programming language.  You don't want it to get too general, lest you end up with the build equivalent of spaghetti code where the build script becomes unreadable and unmaintainable.  OTOH a purely declarative approach is limited by how well the DSL is designed.  An insufficiently-expressive declarative language leads to much frustration when you find yourself unable to express something that you need to do with your build.

Ultimately, at the bottom level, we're just dealing with DAGs, so at some level the build spec should simply be a bunch of node specs, where each node consists of some set of inputs, some set of outputs, and a computational black box (the build action) to get from the former to the latter. This black box can be anything at all, and should not be unnecessarily constrained.

On a higher level, though, specifying individual nodes may not always be desirable (e.g., you want to be able to express things like "all .d files in this directory", or "all recursive import dependencies of source file F").  Typically you'd want a slightly higher-level language for generating DAG node specs.  Here is where some serious thought needs to be put into designing a language that's both straightforward for common tasks, yet expressive enough to handle unusual tasks.  To abuse a metaphor, DAG node specs are the assembly language of build systems, and generally you want to "program" your builds in a higher-level language, but you could either end up with the build equivalent of C with the build analogues of unchecked array bounds, uncontrolled pointer arithmetic, and hundreds of gotchas completely non-obvious to the uninitiated, resulting in ugly, unmaintainable build scripts, or the build equivalent of D with its expressive power yet packaged in a clean syntax, resulting in clean build scripts that are comfortable to write and maintain.

So far I've not yet decided which approach is better: using a full-fledged programming language as the high-level API, like SCons does, or a more constrained but more easily controlled (and implemented!) declarative DSL that is not necessarily Turing-complete. Either way, I prefer baking in as little special behaviour as possible, in the sense that any special capabilities like build-in source code dependency scanning or linker flag control ought to be built on top of primitives that are exposed to user build scripts, so that at least in theory, user build scripts could also attain to equivalent capability without needing to hack the build tool.  Generally, I dislike any disparity between what built-in constructs can do "magically" vs. what user code (build scripts) can achieve, because that usually means that things will work well as long as you remain within the parameters the author has conceived, but should you ever encounter a situation the author didn't anticipate, you're stuck up the creek without a paddle.


T

-- 
Philosophy: how to make a career out of daydreaming.

On 2/21/18 10:30 AM, H. S. Teoh wrote:
> I think the ideal situation straddles the divide between declarative
> build specs and a full-fledged general programming language.  You don't
> want it to get too general, lest you end up with the build equivalent of
> spaghetti code where the build script becomes unreadable and
> unmaintainable.  OTOH a purely declarative approach is limited by how
> well the DSL is designed.  An insufficiently-expressive declarative
> language leads to much frustration when you find yourself unable to
> express something that you need to do with your build.

Working in the Java world, I was extremely happy when I discovered Gradle. It looks declarative thanks to the Groovy language, but you can easily mix 'n' match more imperative code inline.

For a taste of Gradle, here's a Java-centric build file from their getting-started guides [1]:

```
apply plugin: 'java'
apply plugin: 'application'

repositories {
    jcenter()
}

dependencies {
    compile 'com.google.guava:guava:21.0'
    testCompile 'junit:junit:4.12'
}

mainClassName = 'App'
```

And here's a C++ one:

```
apply plugin: 'cpp'

model {
    components {
        main(NativeExecutableSpec)
    }
}
```

Of course in the real world build files get bigger and more complex, but to me they tend to remain very readable.

Comparing Java's Maven and Gradle (and in the JS world, Grunt and Gulp) have given me a strong preference for code-based build scripts, as long as they remain readable.

[1]: https://gradle.org/guides/#getting-started

On Wed, 2018-02-21 at 12:05 -0700, David Gileadi via Digitalmars-d wrote:
> […]
> 
> Working in the Java world, I was extremely happy when I discovered
> Gradle. It looks declarative thanks to the Groovy language, but you
> can
> easily mix 'n' match more imperative code inline.

It is worth pointing out that traditionally Gradle scripts have been Groovy scripts using the ever evolving and improving Gradle DSL, but that there is now the possibility of using the Kotlin Gradle DSL. Still a programming language with a internal DSL, but using a static compiled language with full type checking and IDE support. Groovy is a dynamic language and so IDE support is not as complete.

> For a taste of Gradle, here's a Java-centric build file from their getting-started guides [1]:
> 
> ```
> apply plugin: 'java'
> apply plugin: 'application'
> 
> repositories {
>      jcenter()
> }
> 
> dependencies {
>      compile 'com.google.guava:guava:21.0'
>      testCompile 'junit:junit:4.12'
> }

For truly reproducible builds exact version numbers are good, but as with Dub, etc. Gradle has a notation for allowing less exact version specification. Exact version numbers require just one access to JCenter (as here, but also Maven Central, or any other properly structured repository) to cache the artefact. Using the variability will cause a version check to be made at each build. There is though a Gradle offline mode to avoid this if need be.

> mainClassName = 'App'
> ```
> 
> And here's a C++ one:
> 
> ```
> apply plugin: 'cpp'
> 
> model {
>      components {
>          main(NativeExecutableSpec)
>      }
> }
> ```
> 
> Of course in the real world build files get bigger and more complex,
> but
> to me they tend to remain very readable.

Indeed. The Gant and GPars builds need a serious rewrite to get rid of
a whole raft of crud. This is the danger of a flexible and open build
specification that does not happen with systems that use a purely
declarative project specification, e.g. Dub and Cargo. But the openness
and flexibility allows for a wider range of possibilities. I shall
stick my neck out and say that the current Cargo way of working is
going to prove inadequate and extensions, or even a replacement, will
happen. The Make → Ant → Maven → Gradle path provides a lot of support
for this view regarding Cargo, and I believe Dub.

> Comparing Java's Maven and Gradle (and in the JS world, Grunt and
> Gulp)
> have given me a strong preference for code-based build scripts, as
> long
> as they remain readable.

Me too, which is why I still have a fondness for SCons. Meson has taken the "fixed DSL" road, so not a declarative project statement, but not a full programming language. Fortunately you can still escape to do a bit of Python so not all is lost.


> 
> [1]: https://gradle.org/guides/#getting-started
-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Wed, 2018-02-21 at 09:30 -0800, H. S. Teoh via Digitalmars-d wrote:
> 
> […]

> I think the ideal situation straddles the divide between declarative
> build specs and a full-fledged general programming language.  You
> don't
> want it to get too general, lest you end up with the build equivalent
> of
> spaghetti code where the build script becomes unreadable and
> unmaintainable.  OTOH a purely declarative approach is limited by how
> well the DSL is designed.  An insufficiently-expressive declarative
> language leads to much frustration when you find yourself unable to
> express something that you need to do with your build.

Gradle with Groovy or Kotlin scripts, and SCons with Python scripts are at the extreme, one I actually like. You have all the openness and flexibility of a programming language to describe the project and the build. But yes you have all the dangers of a programming language. But with Python and Groovy, there is a philosophy of trusting the programmer to do things in the simplest and most declarative way. Obviously there are a lot of untrustworthy programmers out there, but that can be solved with training and mentoring.

Dub, Cargo, and to a great extent Maven, give you tools to specify declaratively the project and nothing else. All actions are pre-defined and you have to work within the convention or not at all. The tool allows no trust of the people working on the project to do the right thing.

Meson (I'm not sure about CMake, I do not like its DSL) uses a Python- like fixed DSL written in Python, so not an internal DSL, but really an external DSL. Fortunately it does allow escaping to Python, but I fear they may close this incredibly useful loophole.

Reggae allows for a number of different programming language to be used but I am not sure if it is SCons-style DSL or Meson style DSL, I need to do more work with it to find out.

Dub chose to use JSON (personal comment: yuk) and then SDL (personal comment: why not TOML) so as to have only key:value declaration of project properties. OK so there is a Maven-esque project lifecycle with hooks, but unlike Maven there appears no way of providing new lifecycles. This plugin feature is the only way Maven survives. With Gradle having Groovy (and Kotlin) specification obviates the need for most plugins and yet there is also a plugin system. Gradle is a superb example in my view of how to get it right. A declarative DSL for describing all the project properties, which is all you need for a project working with the standard convention.

A core question is should D be the language of specification. Reggae certainly allows it and shows a way forward. But is it arguably better to use Python or JavaScript (personal opinion: please no, use Python) exactly because it is a dynamic language.

Taking all the lessons of Dub and Reggae is clearly important, but then is the strategy to amend Dub and merge Reggae or start a new codebase?

> Ultimately, at the bottom level, we're just dealing with DAGs, so at
> some level the build spec should simply be a bunch of node specs,
> where
> each node consists of some set of inputs, some set of outputs, and a
> computational black box (the build action) to get from the former to
> the
> latter. This black box can be anything at all, and should not be
> unnecessarily constrained.
> 
> On a higher level, though, specifying individual nodes may not always
> be
> desirable (e.g., you want to be able to express things like "all .d
> files in this directory", or "all recursive import dependencies of
> source file F").  Typically you'd want a slightly higher-level
> language
> for generating DAG node specs.  Here is where some serious thought
> needs
> to be put into designing a language that's both straightforward for
> common tasks, yet expressive enough to handle unusual tasks.  To
> abuse a
> metaphor, DAG node specs are the assembly language of build systems,
> and
> generally you want to "program" your builds in a higher-level
> language,
> but you could either end up with the build equivalent of C with the
> build analogues of unchecked array bounds, uncontrolled pointer
> arithmetic, and hundreds of gotchas completely non-obvious to the
> uninitiated, resulting in ugly, unmaintainable build scripts, or the
> build equivalent of D with its expressive power yet packaged in a
> clean
> syntax, resulting in clean build scripts that are comfortable to
> write
> and maintain.

I think SCons, Gradle, Reggae, and likely Dub, actually have all these issues covered. Important to make them explicit, but not anything that should be problematic.

> So far I've not yet decided which approach is better: using a
> full-fledged programming language as the high-level API, like SCons
> does, or a more constrained but more easily controlled (and
> implemented!) declarative DSL that is not necessarily Turing-
> complete.
> Either way, I prefer baking in as little special behaviour as
> possible,
> in the sense that any special capabilities like build-in source code
> dependency scanning or linker flag control ought to be built on top
> of
> primitives that are exposed to user build scripts, so that at least
> in
> theory, user build scripts could also attain to equivalent capability
> without needing to hack the build tool.  Generally, I dislike any
> disparity between what built-in constructs can do "magically" vs.
> what
> user code (build scripts) can achieve, because that usually means
> that
> things will work well as long as you remain within the parameters the
> author has conceived, but should you ever encounter a situation the
> author didn't anticipate, you're stuck up the creek without a paddle.

I still find Meson constraining for anything not fitting that standard project model. Indeed it was the Meson developers who stated that for one of my projects I had to use the "drop down to Python", there was no other way of handling things.

Whilst I will use Meson for Debian related builds when I can (Meson is approved, SCons is not) I am still an advocate for internal DSL in a programming language, and make sure that good build people do the most declarative, least complex thing.

> 
> T
> 
-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Thu, Feb 22, 2018 at 09:24:05AM +0000, Russel Winder via Digitalmars-d wrote:
> On Wed, 2018-02-21 at 09:30 -0800, H. S. Teoh via Digitalmars-d wrote:
[...]
> Dub, Cargo, and to a great extent Maven, give you tools to specify declaratively the project and nothing else. All actions are pre-defined and you have to work within the convention or not at all. The tool allows no trust of the people working on the project to do the right thing.

And this is what I find frustrating about this way of implementing the declarative approach.  It's an all-or-nothing proposition: either you confine yourself to work within the model set down by the author, or you're left out in the cold. There is no compromise.

IMO, the correct approach is to *empower* the user, not to force the user to conform to a prefabricated mold.  I think this is actually orthogonal to the whole declarative/imperative divide; I can conceive of models where a declarative approach empowers, rather than restricts, the user.  It all depends on what is laid down in the foundations.  A good design ought to consist of a small number of orthogonal primitives with obvious semantics, but the combinations of which give you maximal expressive power.

SCons comes pretty close to this: while it has a whole bunch of very useful built-in functionality, it also exposes the lower-level primitives used to build those functionalities in the first place. E.g., scanners, generators, etc..  This is what I mean by empowering the user. It lets you use the same tools the author used to create the higher-level constructs, as opposed to saying "we don't support X; if you want X, you'll just have to write it from scratch in bare Python". Which is better than not being able to do anything at all outside of the author's prefabricated model (e.g., dub), but still essentially amounting to "what you want to do is not our department; here's the escape hatch to leave the building, you're on your own".

You don't necessarily need a full-fledged programming language to work with -- my SConscripts quite often consist of custom Environment methods that are basically just abstractions of a bunch of calls to Depends and Command. Very few things actually require full-out Python.


[...]
> Taking all the lessons of Dub and Reggae is clearly important, but then is the strategy to amend Dub and merge Reggae or start a new codebase?

I think it depends on how fundamental of a change we're looking at.  So far, I've not found a single build tool that meets my ideal of a small number of orthogonal yet powerful primitives that, combined together, yields maximal expressive power.  SCons comes pretty close, but it does have its inherent limitations.


[...]
> I still find Meson constraining for anything not fitting that standard project model. Indeed it was the Meson developers who stated that for one of my projects I had to use the "drop down to Python", there was no other way of handling things.

So that probably means Meson is not for me.  I can't stand having to work within arbitrarily-defined standard project models that are imposed without any clear reason why it is absolutely indispensible.  Saying "drop down to Python" to me is basically showing you the door and saying "you're on your own".  At least it's better than having no door at all, like in dub where you're either in, or out in the cold; there is no compromise, not even an escape hatch.


> Whilst I will use Meson for Debian related builds when I can (Meson is approved, SCons is not) I am still an advocate for internal DSL in a programming language, and make sure that good build people do the most declarative, least complex thing.
[...]

The good thing about using a programming language as DSL is that if said language is well-known, it alleviates the need for the user to learn yet another DSL.  In this day and age of new DSLs popping up like dandelions in a field, people just don't want to have to learn something new unless they absolutely have to.  In that vein, since we're talking about builds in D, one good idea might be to have a DSL that's essentially a subset of D.  I.e., actually a different language, but one that happens to look like a subset of D, with comparable semantics.  That saves us from having to deal with Turing-completeness in a build script, yet at the same time retain to some degree the comfort of not needing to learn Yet Another Different Language.


T

-- 
First Rule of History: History doesn't repeat itself -- historians merely repeat each other.