On Thursday, 5 February 2015 at 16:32:26 UTC, Sönke Ludwig wrote:
>> I think the only build script I had to write was for RABCDAsm, which
>> probably can't be reimplemented as a dub.json file:
>>
>> http://forum.dlang.org/post/peurhbfebwdskcballzk@forum.dlang.org
>
> Okay, things like testing for DMD bugs and signing stuff would indeed still have to be done outside of DUB using a separate pre(Build/Generate)Command (a failing command should automatically stop the build), but at least the dependency management part (haveLZMA) should map fine to DUB packages.

The haveLZMA thing isn't needed to detect D packages, but the presence of the C library, similar to what a ./configure script does.

> Fun fact: If you register a GIT working copy using "dub add-path/add-local" and it doesn't have an exact tag checked out, then any package that uses it, will have something like this in its dub.selections.json file: '"vibe-d": "0.7.22+commit.17.g75ecb6b"', where "0.7.22" is the latest version tag on that branch and the suffix is the current commit hash.
>
> This information is currently not used to drive GIT and automatically check out that same commit before building. But we could of course implement that in one form or another to have fully reproducible builds with commit granularity.

That's nice. Having the build system generate a file with the dependencies' commits should work as well as git submodules, if it can be expected that the build system will be invoked at least once before committing.

On Thu, 2015-02-05 at 16:52 +0100, Sönke Ludwig via Digitalmars-d wrote:
> 
[…]
> GIT master now supports generating VisualD, CMake and Sublime Text project files. "Make" is also a commonly requested one that would be easy to add, but hasn't been done yet. Of course it also supports RDMD using "dub build --rdmd".

What, no SCons.

Also there needs to be a collaboration between Dub developers and Kingsley Hendrickse with his IntelliJ IDEA plugin so as to get IDEA project generation from Dub specs.

Actually probably need to ensure CLion as well as IDEA, but this ought to come for free.

Whatever the Eclipse, Netbeans, IDEA/PyCharm/CLion/Android-Studio history, the current (and very sensible in my opinion) approach is for there to be a separate project and dependency specification, that is then used by the IDE to create a project – and for there to be updatability. If there is also a non-IDE way of building as well even better.

All (some better than others) support Gradle specifications. Especially true now that the Android toolchain is Gradle/Android- Studio. Well except for Go folk who find Gradle an abomination.

Dub, to be a player, has to prove better than Gradle for the C/C++/D combination. In a similar position is Go, Rust/Cargo, Ceylon, so it is not all Gradle.

I got a vibe from last Tuesday evening's first London D Users meeting that there is an opportunity for D to compete with the OCAML/Haskell/Python/C++ game in finance industry, well at least the hedge fund and quant end of it.

I think having a really good dependency and build systems is way, way, way more important than improving DMD/GDC/LDC in the next few months. Getting a "this is the new thing about using D" out there is key.


-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Thu, 05 Feb 2015 18:12:56 +0100, Sönke Ludwig wrote:

>> so i'll use "make" for everything else too. "make" can be wordy, but at least i'll have only one build tool to care about, not two.
> 
> If you just want to have a build tool, that's fine (if you want to do all the dependency tracking and cross platform/compiler compatibility stuff by hand). But the main point is that you can seamlessly use foreign packages, which is not really possible with make (well, of course it is...).

so strip out all the building duties from dub, and let it be just a package manager. and then include dub package manager in D distribution and teach rdmd how to use it. bingo! now D has a package manager, a simple build tool, and a way to use it's package manager in other build tools.

>> there is nothing wrong in making easy things easy ;-), but dub has no way to make hard things possible. i can't teach it new tricks, and i definitely don't want "build systems explosion" in my projects. and dub can't be easily used like "pkg-config", so even it's package management part is not good for non-dub people.
>>
>>
> It will gain C/C++ build support, so that particular issue should at that point be a thing of the past.

that is not what dub needs, i believe. not another hard-coded support for some language, but a mechanics to add support for *any* language and tool without hard-coding it.

> The "preBuildCommands" feature *is* a way to make hard things possible.

it's in no way making possible building the things without resorting to another build tool. so it's twice as much pain for project author: now he has to learn both tools instead of only one.

> It may not hit the sweet spot in your
> particular case, because it requires another tool to achieve the job,
> but it's definitely doable.

everything dub does is doable with shell scripts too. hm. maybe dub is wrongly designed from the start, and it should be just a set of shell scripts for various tasks...

> The development approach has been to try to get those features in first that enable the most common use cases.

most of the time simply invoking rdmd with corresponding arguments is enough. augmenting rdmd with package management is cool. having another rdmd is not cool.

> but it should be no problem to push the boundary of
> what's possible within the package description format to a level where
>  >99% of people can be happy.

...or simply decouple package descriptions from package build instructions, and stop using dub for package building. the ONLY thing dub has to know is "execute this command to build the package". that's all. ah, and there must be a way to ask dub about environment, package list and so on.

On Thu, 2015-02-05 at 18:01 +0100, Sönke Ludwig via Digitalmars-d wrote:
> 
[…]
>  I'm now just left wondering what it is that is "fundamentally
> flawed". Everything mentioned so far is either
> bugs or missing functionality, or rather just mostly missing
> convenience.

The core problem is the use of a JSON file to express the definition of the project. This sort of specification is fine for dependency but not sufficient for build.

Make is a fine external DSL for dependencies, but everything else relies on the action language, generally Bash. GNU Make adds macros and a whole slew of things to try and make things better, all of them adding more imperative aspects.

Autotools is a wonderful M4 edifice, but not a solution to the problems of project management Make despite many years of people actually succeeding in doing it.

CMake has a DSL for describing build, not so good on the project description side, but in the end the language is ugly, underpowered and reliant on Make.

SCons is not perfect, but it beats Make and CMake by creating an internal DSL on Python. Waf takes aspects of this further and makes Autotools look a bit ill. There is a natural balance

Go delegates to a DVCS all dependency issues, everything build-wise is totally by convention. It works but the hassle people are finding with repeatable builds leads to conventions as per gopkg.in.

Ant, hummm… started out as a platform independent version on Make, used XML for build specification and claimed to be declarative. Looking at Ant scripts these days, the hoops people go through to construct imperative programs in XML are amazing. The only alternative is to write a new task, which lots of people have to do.

Maven tried to go the project specification, and partly succeeded. Artefact repository approach a huge win. Specification in XML, yuk. Attempt at totally convention based build only partly worked. Lots of people have to write plugins to get the process they need for their goals.

Gradle fixed the transitive dependency tracking problems of Maven and has Groovy specifications. Yes there is a lot of purely declarative convention-based stuff, quite right too, but the ability to trivially add the extra bit of imperative in the build specification, stops the mess of Maven, Ant, CMake, Make.

SBT tries to use Scala as a project and build specification language. For simple projects, a few declarative lines are all that is needed. However for anything complicated you end up going outside the convention-only, but like Gradle and unlike Maven you can do this. SBTs major problem is Scala compile time for non-trivial project specification. Hence the need for a build daemon, a route Gradle has had to go as well due to infrastructure start-up time.

I could turn this into a better essay, but the above gives the main threads of the argument: project specification can be declarative, build specification is best as declarative as possible, convention- based, but not totally declarative.

(I now I should mention tup here but I still haven't used it enough to know anything real about it.)

> Things may not (yet) be ideal, but fundamentally DUB is nothing more
> than a tool built on a defined abstract package description format
> (vs. a procedural build description). If you (or anyone else for
> that matter)
> want to say that using a descriptive format is fundamentally flawed
> then
> I'd be grateful for some additional substantiation. Otherwise there
> is nothing "fundamental" in the system that I know of.

The fundamental problem here is that project specification and build specification is not either declarative or imperative.  From 1977 onwards we have a trail of evidence that shows it is part declarative (as much as possible), but part imperative (as little as possible, but necessary for non-trivial projects).

Sticking with JSON as the only specification notation for Dub will inevitably lead people to have to hack or find another tool. The solution would be to either:

– Treat the JSON specification for dependency management only and use something else for build description so as not to be constrained by the only convention.

– switch to an internal DSL so as to have mostly declarative but imperative as needed description of project, dependency and build process.


-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Thu, 05 Feb 2015 16:52:53 +0100, Sönke Ludwig wrote:

>> alas, without this feature dub is not a build system. i understand that there are alot of things to do and the resources are limited. that's why i think that dub should be separated to two projects: package manager and build system.
>>
>> the package manager part should have good "machine" interface to allow it's usage in various scripts and other build systems. and build system... oh, well, it can simply use package manager. this way we can discuss "dub-pkg" and "dub-build" as separate entities, and flaws of one project will not automatically propagate on another project.
>>
> That's how it is meant to be!

yet this is not clear now. some people looking at dub as "build system with crappy package manager included", other as "package manager with crappy build system included" (and some other as "chimera made of crappy package manager and crappy build system" ;-).

clear separation of two projects will immediately show the goals of each one, so there will be less space for misunderstanding and speculations.

I believe only truly practical approach is to design a D build tool as a library (and, ideally, make it a Phobos module). But at the same time it is important to keep it generic and not tied to D or building application in general.

There are two important bits here (important for me):

1)

It must be designed in terms of "target - script - dependency" abstractions, similar to make or tup. I will never use anything that pretends to be a build tool but keeps imagining build process as "compile and link bunch of files and libraries". Good litmus test is this makefile target chain we have in one of projects (very simplified):

bin/app: src/app/main.d protocol.o
    # compile app

src/app/main.d: $(D_SOURCES)

protocol.d: protocol.h
    # use dstep

protocol.h protocol.c: protocol.proto
    # use proto-c, protocol buffer C compiler

Anything that does not allow me to express such dependency chain in native straightforward manner without resorting to external scripts is simply not good enough. dub fails this, CMake fails this.

2)

D is only cross-platfrom scripting language we can rely on. This is probably biggest problem of make (apart from bunch of legacy syntax crap) - any real build system needs relatively complicated build rules for target transformation. However we can reasonably expect working D compiler from anyone wanting to compile D project - which is both perfectly cross-platform and does not request installation of any additional binaries.

"Dicebot"  wrote in message news:listanyegqyanevsnvlv@forum.dlang.org... 

> D is only cross-platfrom scripting language we can rely on. This is probably biggest problem of make (apart from bunch of legacy syntax crap) - any real build system needs relatively complicated build rules for target transformation. However we can reasonably expect working D compiler from anyone wanting to compile D project - which is both perfectly cross-platform and does not request installation of any additional binaries.

Sometimes you make a lot of sense.

On Friday, 6 February 2015 at 12:30:45 UTC, Daniel Murphy wrote:
> "Dicebot"  wrote in message news:listanyegqyanevsnvlv@forum.dlang.org...
>
>> D is only cross-platfrom scripting language we can rely on. This is probably biggest problem of make (apart from bunch of legacy syntax crap) - any real build system needs relatively complicated build rules for target transformation. However we can reasonably expect working D compiler from anyone wanting to compile D project - which is both perfectly cross-platform and does not request installation of any additional binaries.
>
> Sometimes you make a lot of sense.

LOL

+1 to all of this. Except for the part about CMake relying on Make. It doesn't, Make is just one of the backends. I haven't used CMake with anything other than Ninja in a very long time.

Atila

On Friday, 6 February 2015 at 11:04:59 UTC, Russel Winder wrote:
> On Thu, 2015-02-05 at 18:01 +0100, Sönke Ludwig via Digitalmars-d wrote:
>> 
> […]
>>  I'm now just left wondering what it is that is "fundamentally flawed". Everything mentioned so far is either
>> bugs or missing functionality, or rather just mostly missing convenience.
>
> The core problem is the use of a JSON file to express the definition
> of the project. This sort of specification is fine for dependency but
> not sufficient for build.
>
> Make is a fine external DSL for dependencies, but everything else
> relies on the action language, generally Bash. GNU Make adds macros
> and a whole slew of things to try and make things better, all of them
> adding more imperative aspects.
>
> Autotools is a wonderful M4 edifice, but not a solution to the
> problems of project management Make despite many years of people
> actually succeeding in doing it.
>
> CMake has a DSL for describing build, not so good on the project
> description side, but in the end the language is ugly, underpowered
> and reliant on Make.
>
> SCons is not perfect, but it beats Make and CMake by creating an
> internal DSL on Python. Waf takes aspects of this further and makes
> Autotools look a bit ill. There is a natural balance
>
> Go delegates to a DVCS all dependency issues, everything build-wise is
> totally by convention. It works but the hassle people are finding with
> repeatable builds leads to conventions as per gopkg.in.
>
> Ant, hummm… started out as a platform independent version on Make,
> used XML for build specification and claimed to be declarative.
> Looking at Ant scripts these days, the hoops people go through to
> construct imperative programs in XML are amazing. The only alternative
> is to write a new task, which lots of people have to do.
>
> Maven tried to go the project specification, and partly succeeded.
> Artefact repository approach a huge win. Specification in XML, yuk.
> Attempt at totally convention based build only partly worked. Lots of
> people have to write plugins to get the process they need for their
> goals.
>
> Gradle fixed the transitive dependency tracking problems of Maven and
> has Groovy specifications. Yes there is a lot of purely declarative
> convention-based stuff, quite right too, but the ability to trivially
> add the extra bit of imperative in the build specification, stops the
> mess of Maven, Ant, CMake, Make.
>
> SBT tries to use Scala as a project and build specification language.
> For simple projects, a few declarative lines are all that is needed.
> However for anything complicated you end up going outside the
> convention-only, but like Gradle and unlike Maven you can do this.
> SBTs major problem is Scala compile time for non-trivial project
> specification. Hence the need for a build daemon, a route Gradle has
> had to go as well due to infrastructure start-up time.
>
> I could turn this into a better essay, but the above gives the main
> threads of the argument: project specification can be declarative,
> build specification is best as declarative as possible, convention-
> based, but not totally declarative.
>
> (I now I should mention tup here but I still haven't used it enough to
> know anything real about it.)
>
>> Things may not (yet) be ideal, but fundamentally DUB is nothing more than a tool built on a defined abstract package description format (vs. a procedural build description). If you (or anyone else for that matter)
>> want to say that using a descriptive format is fundamentally flawed then
>> I'd be grateful for some additional substantiation. Otherwise there is nothing "fundamental" in the system that I know of.
>
> The fundamental problem here is that project specification and build
> specification is not either declarative or imperative.  From 1977
> onwards we have a trail of evidence that shows it is part declarative
> (as much as possible), but part imperative (as little as possible, but
> necessary for non-trivial projects).
>
> Sticking with JSON as the only specification notation for Dub will
> inevitably lead people to have to hack or find another tool. The
> solution would be to either:
>
> – Treat the JSON specification for dependency management only and use
> something else for build description so as not to be constrained by
> the only convention.
>
> – switch to an internal DSL so as to have mostly declarative but
> imperative as needed description of project, dependency and build
> process.

> There are two important bits here (important for me):

The approach I'm thinking of agrees 100% with your two important bits.

> Anything that does not allow me to express such dependency chain in native straightforward manner without resorting to external scripts is simply not good enough. dub fails this, CMake fails this.

CMake doesn't fail at this, it's just not easy. It's what add_custom_command is for.

> D is only cross-platfrom scripting language we can rely on.

Which is why that's the language I want to express the build in. Methinks I have some code to write and some emailing to do with Soenke.

Atila