On Tuesday, 3 February 2015 at 20:45:03 UTC, Dicebot wrote:
> On Tuesday, 3 February 2015 at 20:18:31 UTC, Paolo Invernizzi wrote:
>> Don't forget tup!!! [1]
>>
>> [1] http://gittup.org/tup/
>
> This is first time I hear about tup but I must admit it looks really cool. It has all the good thing I love about make but with more thinking being put into it and no awkward legacy.
>
> And it is honest natively compiled program with no extra annoying dependencies.
>
> It would tempting to include something like that in Phobos as a library - sadly, it is licenced under GPL and thus this option is not available :(

I'm reading the paper on it now, looks very interesting.  Tup may be licensed but I'm sure we could incorporate the algorithms it uses into phobos somehow...I'll know more after I finish reading.

This is really good feedback, thanks. Would you be interested in discussing this further privately? I've already got some ideas brewing in my head.

Atila

On Tuesday, 3 February 2015 at 18:00:31 UTC, Russel Winder wrote:
> On Mon, 2015-02-02 at 16:50 +0000, Atila Neves via Digitalmars-d wrote:
> […]
>> 
>> I have ideas that go beyond this, but this is useful input. I wonder how to proceed now about gathering actual requirements from D devs and seeing which ones are important.
>
> Learn lessons from SBT vs. SCons/Gant/Gradle: SBT is a Scala build
> system using Scala programs as input.
>
> Some Scala folk got religious about Scala being the only right language
> for Scala builds, and created what has become SBT. Because of the way
> statically typed Scala works as a domain specific languages there are
> performance and expression issues that means SBT is getting a reputation
> in the very companies that should be supporting it that it is "the
> enemy".
>
> A D build system must trivially support C, C++(, and Fortran?).
>
> Transitive dependency management needs careful consideration. I haven't
> investigated whether Dub does this right, it may well do. Maven does
> not, Gradle does.
>
> Define package, artefact, dependency carefully. Go has a strong model
> (even if it is wrong). Ceylon improves on Java/Scal/etc. Python got it
> woefully wrong: eggs were a mess, and PyPI is not curated well enough
> (the vast majority of stuff on PyPI is unusable rubbish).
>
> Worry about platform specific packagers: MacPorts, Homebrew, Debian,
> Fedora, FreeBSD, OpenBSD, etc. will (hopefully) want to package. Java is
> generally a disaster for them, and Python isn't that much better. Go is
> ignoring al this and creating a monoculture based on Git and Mercurial
> as packages are shipped as source. I see Debian and Fedora trying to
> package Go things and predict the same mess as Java and Python.
>
> Support principally a declarative DSL, but with imperative structure
> possible. A build specification is a meta-program which when executed
> creates the program and hence the build process.
>
> Have no clearly silly constraints, cf. the needs for SBT specification
> lines always to be separated by blank lines.
>
> Be convention over configuration, but allow configuration.
>
> Use BinTray and Artifactory as well as the current Dub repository.
>
> Do not be afraid to change the meta-data specification so that. Dub may
> well be a good start, but it may need work. Gradle has changed it's
> meta-data specifications many times despite being constrained by
> compatibility with Maven POMs and Ivy specification.
>
> Stand on the shoulders of giants, but do not assume they are always
> right, be prepared to change stuff for the better.
>
> Get alpha testers in early and let them drive things – albeit within the
> boundaries of your vision for the system. Dub did well here if I
> remember correctly and created a group of people who did and emailed
> rather than just emailing.
>
> I better finish my essay at this point I have a London D User Group
> meeting to go to :-)
>
> http://www.meetup.com/London-D-Programmers/

On Tue, Feb 03, 2015 at 08:41:38PM +0000, Jonathan Marler via Digitalmars-d wrote:
> On Tuesday, 3 February 2015 at 20:18:31 UTC, Paolo Invernizzi wrote:
> >[1] http://gittup.org/tup/
> 
> I don't know who wrote that site but they are quite hilarious :)
> 
> "Unfortunately, tup is so fast that your chair mounted jousting might suffer. I apologize in advance if someone besmirches your honor and you are unable to properly defend yourself as a result."
> 
> "In tup, the arrows go up. This is obviously true because it rhymes. "

Don't be fooled, though. Tup contains some revolutionary ideas that would-be implementors of the Next Great Build System would do well to take note.  The paper linked to by the website explains the difference between "first generation" build algorithms vs. "second generation" build algorithms.

In brief, first generation build algorithms are centered around linearly scanning dependencies to update, which is not scalable, because it is O(n), and as the size of the source tree grows large, linear scanning becomes prohibitively expensive. Your code-compile-test cycle gets bottlenecked at the compile step because every single time the poor build tool has to scan the entire source tree of hundreds or even thousands of source files, and rebuilding the graph from scratch. Furthermore, any attempt to reduce the cost of the linear scan (e.g., invoke make from a subdirectory) breaks build correctness, because the algorithm is unable to reliably determine the smallest consistent subgraph that must be updated when some node(s) change unless it scans everything.

Second generation build algorithms are centered around *not* scanning, but taking advantage of modern OSes providing APIs for file change notification. Rather than scan the whole source tree every time, it takes the changeset as input -- either from the OS, or from some other source of information. By leveraging OS features, we can obtain this info on an as-needed basis instead of an O(n) scan. Upon this, a modified algorithm with O(log n) complexity is built, that at the same time also guarantees correctness -- by the modified representation of the dependency graph, it is possible to derive the smallest consistent subgraph that needs to be updated when some changes are made.

In addition to an overhaul of the core algorithm, tup also has some clever tricks, like wrapping libc of executed commands, so that all *actual* inputs to the compiler (and other build commands) are included in the dependency graph -- if the compiler reads /usr/include/stdio.h, for example, tup will know that the source file depends on stdio.h -- and this *without* needing to scan the source file manually or asking the user to specify such tedious things.

In fact, tup is just a proof-of-concept tool (that's surprisingly usable for what it is!); the author expects that build tools of the future will develop its ideas further. As for me, I get the feeling that tup has the right ideas, even if it isn't quite "it" yet -- the implementation isn't quite what I would've chosen. But nevertheless, it holds a lot of promise for the future, unlike most other alternative build tools, which are just more-of-the-same-rehashed-the-nth-time.


T

-- 
What are you when you run out of Monet? Baroque.

On Tuesday, 3 February 2015 at 20:18:31 UTC, Paolo Invernizzi wrote:
> On Tuesday, 3 February 2015 at 17:34:05 UTC, Russel Winder wrote:
>> The issue is though that Dub was created because someone did something
>> rather than just talking (and emailing) about it. As Hans and colleagues
>> took knowledge and experience and created Gradle, somebody, or group of
>> people, need to take the practical experience of Make, CMake, Jam,
>> SCons, Waf, Ant, Gant, Maven, Gradle and Dub, and do something.
>>
>> I will also say that Bintray and Artifactory may be better
>> artefact/dependency stores that trying to build something D-specific
>> from scratch.
>
> Don't forget tup!!! [1]
>
> [1] http://gittup.org/tup/

I didn't forget it myself, but thanks for bringing it up. I'm considering having a tup backend actually.
Tup is cool but unfortunately is still "too manual", for lack of a better term. I'd much rather write "add_executable" and not worry about the details than write the commands explicitly myself. But it's certainly interesting as an alternative backend to, say, Ninja.

Atila

On Tuesday, 3 February 2015 at 03:20:49 UTC, Martin Nowak wrote:
> On Tuesday, 3 February 2015 at 02:39:56 UTC, Vladimir Panteleev wrote:
> With dub you do this instead.
>
> dub fetch digger
> dub run digger -- --help

This is the killer feature for me! As a user I don't really care about how it is built.

On Tuesday, 3 February 2015 at 21:05:38 UTC, Atila Neves wrote:
> I didn't forget it myself, but thanks for bringing it up. I'm considering having a tup backend actually.
> Tup is cool but unfortunately is still "too manual", for lack of a better term. I'd much rather write "add_executable" and not worry about the details than write the commands explicitly myself. But it's certainly interesting as an alternative backend to, say, Ninja.

I would warn against this attitude. Trying to do too much magic is one of reasons I ignore all of modern build tools and still keep my makefiles. There is huge benefit in knowing that your build tool can express any dependency tree based workflow in uniform manner - be it compiling sometithing, downloading remote artifacts or generating a package.

With a good base "smart" solutions can be built on top. This is actually how we use make in Sociomantic - by having a set of standard makefiles with D-specific rules that allow to define build target as simple as this:

$B/appname: $C/src/appname/main.d

all += $B/appname

(yes, that is all that needs to be in actual makefile)

On 02/03/2015 04:20 AM, Martin Nowak wrote:
> On Tuesday, 3 February 2015 at 02:39:56 UTC, Vladimir Panteleev wrote:
>> I might revisit Dub again once some of the fixable issues mentioned
>> here are fixed.
>
> Another very important argument is consistency in using packages.
> I recently tried digger which is a really great tool, but it took me
> about 5 min to get it running.
>

Just used Digger again and I'm going to copy my console output here, because it nicely illustrates my point.

➜ ~ cd Build/Digger/
➜ Digger git:(master) ls
ae/       bisect.ini         cache.d       common.d  custom.d  digger.d           digger-web.d  repo/   result/  win32/
bisect.d  bisect.ini.sample  CHANGELOG.md  current/  digger* digger.ini.sample  README.md     repo.d  web/
➜ Digger git:(master) git pull
rdremote: Counting objects: 25, done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 25 (delta 11), reused 0 (delta 0)
Unpacking objects: 100% (25/25), done.
From https://github.com/CyberShadow/Digger
   8f33627..b6d06be  master     -> origin/master
Fetching submodule ae
md --buiFirst, rewinding head to replay your work on top of it...
Fast-forwarded master to b6d06be2346986a74937c918ed1d4ac121a9819f.
➜ Digger git:(master) ✗ rdmd --build-only digger
digger.d(12): Error: module main is in file 'ae/utils/main.d' which cannot be read
import path[0] = .
import path[1] = /usr/include/dmd/phobos
import path[2] = /usr/include/dmd/druntime/import
Failed: ["dmd", "-v", "-o-", "digger.d", "-I."]
➜ Digger git:(master) ✗ ls
ae/       bisect.ini         cache.d       common.d  current/  digger*  digger.ini.sample  README.md  repo.d   web/
bisect.d  bisect.ini.sample  CHANGELOG.md  config.d  custom.d  digger.d  digger-web.d       repo/      result/  win32/
➜ Digger git:(master) ✗ git submodule update
fatal: reference is not a tree: 318ccc76669e9840f43b88e63bcfb1e461ce6d61
Unable to checkout '318ccc76669e9840f43b88e63bcfb1e461ce6d61' in submodule path 'ae'
➜ Digger git:(master) ✗ git submodule fetch
usage: git submodule [--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]
   or: git submodule [--quiet] status [--cached] [--recursive] [--] [<path>...]
   or: git submodule [--quiet] init [--] [<path>...]
   or: git submodule [--quiet] deinit [-f|--force] [--] <path>...
   or: git submodule [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--reference <repository>] [--recursive] [--] [<path>...]
   or: git submodule [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
   or: git submodule [--quiet] foreach [--recursive] <command>
   or: git submodule [--quiet] sync [--recursive] [--] [<path>...]
➜ Digger git:(master) ✗ git submodule status
+c58d5b75242fcab15872acc5d922e103d3ae9423 ae (remotes/upstream/HEAD)
 7a476ca6139869e89943bcf21e0b27904bdf9491 win32 (heads/master)
➜ Digger git:(master) ✗ gitk
➜ Digger git:(master) ✗ git gui
➜ Digger git:(master) ✗ git reset --hard
HEAD is now at b6d06be Detect absence of ae library
➜ Digger git:(master) ✗ git diff
diff --git a/ae b/ae
index 318ccc7..c58d5b7 160000
--- a/ae
+++ b/ae
@@ -1 +1 @@
-Subproject commit 318ccc76669e9840f43b88e63bcfb1e461ce6d61
+Subproject commit c58d5b75242fcab15872acc5d922e103d3ae9423
➜ Digger git:(master) ✗ git reset --hard
HEAD is now at b6d06be Detect absence of ae library
➜ Digger git:(master) ✗ rdmd --build-only digger
digger.d(12): Error: module main is in file 'ae/utils/main.d' which cannot be read
import path[0] = .
import path[1] = /usr/include/dmd/phobos
import path[2] = /usr/include/dmd/druntime/import
Failed: ["dmd", "-v", "-o-", "digger.d", "-I."]
➜ Digger git:(master) ✗ cd ae
➜ ae gitk
➜ ae cd ..
➜ Digger git:(master) ✗ git submodule sync
Synchronizing submodule url for 'ae'
Synchronizing submodule url for 'win32'
➜ Digger git:(master) ✗ cat .gitmodules
[submodule "ae"]
	path = ae
	url = git://github.com/CyberShadow/ae.git
[submodule "win32"]
	path = win32
	url = https://github.com/CS-svnmirror/dsource-bindings-win32
➜ Digger git:(master) ✗ cat .git
.git/  .gitignore  .gitmodules
➜ Digger git:(master) ✗ rdmd --build-only digger
digger.d(12): Error: module main is in file 'ae/utils/main.d' which cannot be read
import path[0] = .
import path[1] = /usr/include/dmd/phobos
import path[2] = /usr/include/dmd/druntime/import
Failed: ["dmd", "-v", "-o-", "digger.d", "-I."]
➜ Digger git:(master) ✗ git submodule update
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 8 (delta 5), reused 4 (delta 1)
Unpacking objects: 100% (8/8), done.
From git://github.com/CyberShadow/ae
   217b080..66502df  master     -> origin/master
   e8d867c..0b1cfc0  new-net    -> origin/new-net
 * [new branch]      old-net    -> origin/old-net
Submodule path 'ae': checked out '318ccc76669e9840f43b88e63bcfb1e461ce6d61'
➜ Digger git:(master) rdmd --build-only digger
➜ Digger git:(master)

No idea why git submodule update worked the second time.

On 02/03/2015 10:02 PM, H. S. Teoh via Digitalmars-d wrote:
> Rather than scan the whole source tree every time, it
> takes the changeset as input -- either from the OS, or from some other
> source of information.

Indeed a good idea, although according to his graphs, this only starts to matter for +1000 files.

On Wed, Feb 04, 2015 at 02:30:23AM +0100, Martin Nowak via Digitalmars-d wrote:
> On 02/03/2015 10:02 PM, H. S. Teoh via Digitalmars-d wrote:
> >Rather than scan the whole source tree every time, it takes the changeset as input -- either from the OS, or from some other source of information.
> 
> Indeed a good idea, although according to his graphs, this only starts to matter for +1000 files.

The size of software projects these days tend to dwarf that easily.  For a personal website project of mine, for example, SCons is taking noticeably long just to scan the entire directory tree for changes. (But then again, I do have a *lot* of intermediate build targets.) If a one-man project is already running into these performance limits, how much faster enterprise projects?


T

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth

On Wednesday, 4 February 2015 at 01:04:42 UTC, Martin Nowak wrote:
> On 02/03/2015 04:20 AM, Martin Nowak wrote:
>> On Tuesday, 3 February 2015 at 02:39:56 UTC, Vladimir Panteleev wrote:
>>> I might revisit Dub again once some of the fixable issues mentioned
>>> here are fixed.
>>
>> Another very important argument is consistency in using packages.

Some people are more familiar with git, some with Dub. As a Dub developer, I'm not surprised you're more familiar with it. The two simple commands you quoted look as alien to me as the supplied build instructions would look to someone who never used git - meanwhile, they are self-explanatory to everyone who's used git submodules before.

I'm sure I or anyone could at will produce similar logs of failing to figure out dub basics and it spitting mysterious error messages.

Frankly, I don't see this as a considerable argument.