In all the discussion of Dub to date, it hasn't been pointed out that JVM building merged dependency management and build a long time ago. Historically:

  Make → Ant → Maven → Gradle

and Gradle can handle C++ as well as JVM language builds.

So the integration of package management and build as seen in Go, Cargo, and Dub is not a group of outliers. Could it be then that it is the right thing to do. After all package management is a dependency management activity and build is a dependency management activity, so why separate them, just have a single ADG to describe the whole thing.

SCons, CMake, and Meson (also Reggae?) are traditional build tools, but they assume all package dependency management is handled elsewhere, i.e. they state what is required for a build but assume some other tool provides those packages, usually OS package management, but note with C++ Conan is rapidly becoming a big player. JFrog via Bintray and Artifactory do appear to be the leaders in repository management and Gradle works bery well with it.

Rust, Ceylon, and D have to date chosen to eschew systems like Bintray to create their own language specific versions for whatever reason. This leads to language specific dependency management and build. Go is also in this category really except that Git, Mercurial, and Breezy (née Bazaar) repositories are the only package storage used.

Then, is a DevOps world, there is deployment, which is usually a dependency management task. Is a totally new tool doing ADG manipulation really needed for this?

The lessons from all the tools from SCons to Gradle is that it is all about ADG manipulation and constraint satisfaction. SCons is really quite restricted even though it does it very well (*).  Gradle really tries hard not just to solve the problems of Maven (**), but to do end- to-end project management well. In a sense it is the antithesis of each tool does one thing and one thing only model, it is the "there is one and only one ADG to describe the life of the project". Maven and Gradle, and to a lesser extent Cargo and Go, emphasise project management as a wholistic thing, rather than making people deside on each item of the tool chain. Gradle proves a good plugin system to allow changes to the default standard project lifecycle.

Gradle uses Groovy scripts or Kotlin scripts for project specifications. Most projects are easily described by a purely declarative specification using internal DSLs. However, for those awkwards bits for some projects a bit of programming in the specification solves the problem. So this goes against the "a project specification must be purely declarative so use TOML/JSON/SDL" but is easy DevOps worth it.

Atila's Reggae has already shown how easy it is to use D (or Python, Ruby, JavaScript, Lua) to define a build in an internal DSL.

Merging ideas from Dub, Gradle, and Reggae, into a project management tool for D (with C) projects is relatively straightforward of plan albeit really quite a complicated project. Creating the core ADG processing is the first requirement. It has to deal with external dependencies, project build dependencies, and deployment dependencies. The initial problem is creating an ADG from what is potentially a CDG and then doing constraint satisfaction to generate actions. SCons and Gradle have a lot to offer on this.

Having been obsessed by build and project management since about 1976, I'd be interested in doing some work on this.



(*) The O(N) vs. O(n), SCons vs. Tup thing that T raised in another
thread is important, but actually it is an implementation thing of how
do you detect change, it isn't an algorithmic issue at a system design
level. But it is important.

(**) Which many people ignore because Maven remains the major project management tool in the JVM-verse.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:

> Having been obsessed by build and project management since about 1976, I'd be interested in doing some work on this.

This is interesting to me, though I don't know how helpful I can be. At a minimum I will be able to do testing on different platforms. I will also write some code if competent to do so.

On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:
> In all the discussion of Dub to date, it hasn't been pointed out that JVM building merged dependency management and build a long time ago. Historically:
>
>   Make → Ant → Maven → Gradle
>
> and Gradle can handle C++ as well as JVM language builds.
>
> So the integration of package management and build as seen in Go, Cargo, and Dub is not a group of outliers. Could it be then that it is the right thing to do. After all package management is a dependency management activity and build is a dependency management activity, so why separate them, just have a single ADG to describe the whole thing.
>
> SCons, CMake, and Meson (also Reggae?) are traditional build tools, but they assume all package dependency management is handled elsewhere,

Indeed. That's the big problem of these non-specialiazed build tools.

On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:
> In all the discussion of Dub to date, it hasn't been pointed out that JVM building merged dependency management and build a long time ago. Historically:
>
>   Make → Ant → Maven → Gradle
>
> and Gradle can handle C++ as well as JVM language builds.
>
> So the integration of package management and build as seen in Go, Cargo, and Dub is not a group of outliers. Could it be then that it is the right thing to do. {snip}

As an additional data point, the main project management tool for Clojure is [Leiningen](https://leiningen.org/). I don't understand the details under the hood, but it makes use of Maven infrastructure and can install packages from the Clojure online package repo, as well as from [Maven Central](http://search.maven.org/) (the online repo of JVM jars).

Lein installs packages into the same place maven does, into a "local repo" in an ~/.m2/repository directory. This makes it very easy to `rm -fr` it to start afresh if needed. Lein keeps multiple versions of packages when your various projects use different versions of the same library.

The lein project config file is itself written in Clojure. See a [sample lein config file](https://github.com/technomancy/leiningen/blob/master/sample.project.clj) if interested.

When I first encountered lein I was confused. Was it for project creation, dependency management, building and running your app, or running tests? Turns out it does all of those, and also supports plug-ins to allow it to do all kinds of other project-related tasks that aren't built-into the tool (for example, generating docs --- see the [full list of lein plug-ins](https://github.com/technomancy/leiningen/wiki/Plugins)).

On Monday, 12 February 2018 at 10:35:06 UTC, Russel Winder wrote:
> In all the discussion of Dub to date, it hasn't been pointed out that JVM building merged dependency management and build a long time ago. Historically:
>
>   Make → Ant → Maven → Gradle
>
> and Gradle can handle C++ as well as JVM language builds.
>
> So the integration of package management and build as seen in Go, Cargo, and Dub is not a group of outliers. Could it be then that it is the right thing to do. After all package management is a dependency management activity and build is a dependency management activity, so why separate them, just have a single ADG to describe the whole thing.
>
> SCons, CMake, and Meson (also Reggae?) are traditional build tools, but they assume all package dependency management is handled elsewhere, i.e. they state what is required for a build but assume some other tool provides those packages, usually OS package management, but note with C++ Conan is rapidly becoming a big player. JFrog via Bintray and Artifactory do appear to be the leaders in repository management and Gradle works bery well with it.
>
> Rust, Ceylon, and D have to date chosen to eschew systems like Bintray to create their own language specific versions for whatever reason. This leads to language specific dependency management and build. Go is also in this category really except that Git, Mercurial, and Breezy (née Bazaar) repositories are the only package storage used.
>
> Then, is a DevOps world, there is deployment, which is usually a dependency management task. Is a totally new tool doing ADG manipulation really needed for this?
>
> The lessons from all the tools from SCons to Gradle is that it is all about ADG manipulation and constraint satisfaction. SCons is really quite restricted even though it does it very well (*).  Gradle really tries hard not just to solve the problems of Maven (**), but to do end- to-end project management well. In a sense it is the antithesis of each tool does one thing and one thing only model, it is the "there is one and only one ADG to describe the life of the project". Maven and Gradle, and to a lesser extent Cargo and Go, emphasise project management as a wholistic thing, rather than making people deside on each item of the tool chain. Gradle proves a good plugin system to allow changes to the default standard project lifecycle.
>
> Gradle uses Groovy scripts or Kotlin scripts for project specifications. Most projects are easily described by a purely declarative specification using internal DSLs. However, for those awkwards bits for some projects a bit of programming in the specification solves the problem. So this goes against the "a project specification must be purely declarative so use TOML/JSON/SDL" but is easy DevOps worth it.
>
> Atila's Reggae has already shown how easy it is to use D (or Python, Ruby, JavaScript, Lua) to define a build in an internal DSL.
>
> Merging ideas from Dub, Gradle, and Reggae, into a project management tool for D (with C) projects is relatively straightforward of plan albeit really quite a complicated project. Creating the core ADG processing is the first requirement. It has to deal with external dependencies, project build dependencies, and deployment dependencies. The initial problem is creating an ADG from what is potentially a CDG and then doing constraint satisfaction to generate actions. SCons and Gradle have a lot to offer on this.
>
> Having been obsessed by build and project management since about 1976, I'd be interested in doing some work on this.
>
>
>
> (*) The O(N) vs. O(n), SCons vs. Tup thing that T raised in another
> thread is important, but actually it is an implementation thing of how
> do you detect change, it isn't an algorithmic issue at a system design
> level. But it is important.
>
> (**) Which many people ignore because Maven remains the major project management tool in the JVM-verse.

Lot's of stuff here.  I would love for a build/package management tool to emerge and take over like git did with source control.  However, my guess is that this problem is very hard and that's why there's so many tools that have their own pros and cons.  But it sounds like you're trying to sift through them all and find the gems in each tool.

I myself have made attempts to tackle the problem.  I created a tool called "dbuild" that allowed you to use D code to describe your project and then build it for you (kinda like Raggae I suppose).  And my latest project I call "bidmake" which is more general.  It implements the model of what I call "build contracts".  A project describes a "contract" and various "contractors" can be used to fulfill the contract.  You could have a "build contractor" that builds a contract, or a "documentation contractor" that generates documentation, etc.  Anyway, I currently use the tool successfully on a few projects but I'm still not sure how much potential it has.  Maybe with your experience you can take a look at it and see if there's any substance to the idea. (https://github.com/marler8997/bidmake)

ADG? Google doesnt find anything relevant

On Tuesday, 13 February 2018 at 10:06:43 UTC, welkam wrote:
> ADG? Google doesnt find anything relevant

Acyclic directed graph

On Tue, 13 Feb 2018 10:21:49 +0000, Abdulhaq wrote:

> On Tuesday, 13 February 2018 at 10:06:43 UTC, welkam wrote:
>> ADG? Google doesnt find anything relevant
> 
> Acyclic directed graph

A.K.A "DAG" - directed acyclic graph.

I am also interested in build systems, and am currently convinced that the problem space is complex enough to warrant build systems being separate from package management systems.

Maybe a compromise position would be for a package management system to define an interface through which it can do things like:

* Discover what the external dependencies are,

* Provide those external dependencies, and

* Invoke a full build.

Then, any number of build systems (and deployment systems?) could be adapted to work with the package management system.

On Thursday, 15 February 2018 at 04:11:51 UTC, Graham St Jack wrote:
> Maybe a compromise position would be for a package management system to define an interface through which it can do things like:
>
> * Discover what the external dependencies are,
>
> * Provide those external dependencies, and
>
> * Invoke a full build.
>
> Then, any number of build systems (and deployment systems?) could be adapted to work with the package management system.

That is exactly what GNU Guix offers. With support for isolated builds, continuous integration testing, and containers thrown in, if you want that. People misunderstand Guix somewhat because it presents itself as a 'package manager' and even distribution in its own right. But actually it is a dependency manager that can run on top of any system.

I am writing a BLOG on how to use Guix for Python development using its package managers and even dependency injection (say a choice of BLAS libraries or LLVM in the case of D). When that is done I could do a D writeup if there is enough interest.

I am sure some people roll their eyes when I mention GNU Guix. But, hey, if you are an Emacs or gcc user you may be able to afford to pay attention to long running GNU projects.

To anticipate real criticism: there are currently two main issues: (1) GNU Guix runs on Linux and (2) the default requires a running build daemon.

About (1) since this is a libre project the focus is on Linux and Hurd. It is actually fairly straightforward to port Guix to other OS's. Nix, which shares the build daemon with Guix, runs on OSX and Windows.

About (2) the daemon can run unprivileged. I have written documents about running Guix without root access (I need it on HPC). It is just a little more involved.

Anyway, I don't really care who uses Guix or who uses something else. I expect 99.9% of people to ignore these ideas. Just to say I simply use it to make my own life easier. The point here is that I understand what it took to create Guix and it is non-trivial. We can reuse this functionality very easily and take control over the dependency graph. You get reproducible builds, easy mixing of LLVM versions and many other features. Fixing (1) above is much easier than recreating something like Guix from scratch. And since Guix is distribution agnostic you can use it on any old CentOS, Debian, Ubuntu... you name it. The only thing Guix uses is the *running* kernel API. Even glibc and the linked library loader come with Guix (and you can easily run multiple versions of said libraries). That is a full and deep dependency graph.

With Guix I do not need dub or pip or gems. It is trivially easy to manage dependencies. Still I can use those package managers if I want to. In a controlled fashion.

PS The JVM world has the advantage of being a clear and isolated system. Good news is that Guix supports that too and it provides an awesome libre bootstrap from source. If you care about the free and open in FOSS that is a huge selling point in a world that appears to increasingly bootstrap from binary blobs.

It is easy to write-off such ideas. But if you don't try it, you don't understand it. Similar to the D vs other language discussions. I am not going to say Guix is easy. Similar to the fact that D is not easy. But you can gradually get in and learn to appreciate the great engineering under the hood. That goes for Nix too. Guix and Nix still share the build daemon, though they have become completely different systems with different characteristics.

If you want to try Guix - I am here to support you.

On Thursday, 15 February 2018 at 07:21:24 UTC, Pjotr Prins wrote:
> On Thursday, 15 February 2018 at 04:11:51 UTC, Graham St Jack wrote:
>> Maybe a compromise position would be for a package management system to define an interface through which it can do things like:
>>
>> * Discover what the external dependencies are,
>>
>> * Provide those external dependencies, and
>>
>> * Invoke a full build.
>>
>> Then, any number of build systems (and deployment systems?) could be adapted to work with the package management system.
>
> That is exactly what GNU Guix offers. With support for isolated builds, continuous integration testing, and containers thrown in, if you want that. People misunderstand Guix somewhat because it presents itself as a 'package manager' and even distribution in its own right. But actually it is a dependency manager that can run on top of any system.

It's a bit confusing since the first thing [the Guix webpage](https://www.gnu.org/software/guix/) talks about "GuixSD", rather than the Guix tool in its own right.

Wow, I didn't realize how established Guix is. It's [packages page](https://www.gnu.org/software/guix/packages/) boasts 6868 available packages!

The [Guix manual](https://www.gnu.org/software/guix/manual/) appears to be comprehensive.

Are any other languages using Guix for their 3rd-party online package repo? If not, why?

> I am writing a BLOG on how to use Guix for Python development using its package managers and even dependency injection (say a choice of BLAS libraries or LLVM in the case of D). When that is done I could do a D writeup if there is enough interest.
>
> I am sure some people roll their eyes when I mention GNU Guix.

Not at all. Seems like an amazing tool!