On Monday, 2 February 2015 at 16:26:45 UTC, Atila Neves wrote:
> This thread has just made me decide to write a D build system, in D and configured in D that builds on dub and its packages intead of reinventing the wheel. I might ask the community for inputs on requirements. Maybe this will be my DConf 2015 submission.

Why not building on cmake? You have experience with it. trentforkert made very good progress with D support in cmake and has good chances to merge it upstream [1].

If we look at the Vision/2015H1 goals, we see a couple of points, where cmake already has the necessary support:

- C++ integration (hybrid projects)
- alternative compilers
- embedded systems

This features would take a lot of effort/time to be implemented in a pure D solution, nevertheless any of these points is a niche in the D community.

Apart from the ugly script language, cmake would be a pretty good fit.


[1] - https://github.com/trentforkert/cmake

On 02/02/15 11:13, Vladimir Panteleev via Digitalmars-d wrote:
> On Monday, 2 February 2015 at 10:09:00 UTC, Joseph Rushton Wakeling wrote:
>> Well, as long as the requirements are expressed in the form,
>>
>>     "package-name": ">=1.2.3"
>
> This will allow Dub to pick a new major version with incompatible changes, no?

Sorry, I was taking for granted that you'd understand that placing an upper bound on versions was also possible.  Thanks to everyone who chipped in with examples.

> As I said in my reply to Mathias, what dub does breaks the module path and file
> path consistency when modules/subpackages lie in the repository root.

Well, frankly, I think putting modules or subpackages in the repository root is a bad idea.  I'm not sure why I should mind that dub enforces not doing such things.

On 02/02/15 15:40, Dicebot via Digitalmars-d wrote:
> Quick summary of my opinion can look like this:
>
> 1) dub is a horrible build tool
> 2) git submodules is a horrible way for dependency management
>
> First thing is something we can improve on in future and, actually, dub supports
> using rdmd as a build tool already.
>
> Second thing is an inherent issue that won't change without some fundamental
> breaking changes in how hit submodules are implemented.
>
> Because of that I am trying to push dub despite the fact I don't like lot of
> things about it personally.

Very well put.  If there are issues with dub, we can and should address them -- not turn to a tool that isn't actually intended to solve the problem we are trying to address.

> 4) Inherently coupled with specific DVCS
>
> git and GitHub are most common (and officially endorsed) ways of version control
> in D world but those are not the only one. dub registry allows to use GitHub,
> GitBucket and source tarball packages in the same project - in simple uniform
> matter. It is a 100% in-house tool that does no depend on decisions of external
> project maintainers that provides a common API for everything else to interoperate

Yup.  It would be very presumptuous to assume that just because git works well for the core D project, that everyone out there should use it for their projects.

By the way, talking of other VCS, I recently discovered a neat little trick for visualizing a complex git history in a "clean" way.  You need to install bzr, and the qbzr and bzr-git plugins (this might also require bzrtools).  Then, in a git branch, type,

    bzr qlog

... and you'll have a rather beautiful linear history of the master branch, with the option to expand out any individual merge that you want and review the commits therein.

It'll take a little longer to load a large history, because it's solving a more complex problem than the standard git log tools, but I've found it a rather nice way of simplifying the view of complex, multi-branch history.  I know some people have complained about the allegedly tangled mess of dmd/druntime/phobos commit history, and this might be a useful trick to help with that.

I'm not aware of any git history visualizer that supports this functionality, which is a shame, because it must be possible.

On Monday, February 02, 2015 22:37:13 Joseph Rushton Wakeling via Digitalmars-d wrote:
> On 02/02/15 15:40, Dicebot via Digitalmars-d wrote:
> > Quick summary of my opinion can look like this:
> >
> > 1) dub is a horrible build tool
> > 2) git submodules is a horrible way for dependency management
> >
> > First thing is something we can improve on in future and, actually, dub supports using rdmd as a build tool already.
> >
> > Second thing is an inherent issue that won't change without some fundamental breaking changes in how hit submodules are implemented.
> >
> > Because of that I am trying to push dub despite the fact I don't like lot of things about it personally.
>
> Very well put.  If there are issues with dub, we can and should address them -- not turn to a tool that isn't actually intended to solve the problem we are trying to address.

I've long thought that it some point, dub would be forced to support arbitrary build systems in order to actually work in the general case. There are far too many special cases and more complicated things that build scripts and the like frequently have to deal with for dub's simple approach to work in many cases. Just look at what something like cmake it can do. It blows dub out of the water when it comes to power.

The fact that we have package management via dub is great, and the way it builds stuff works for many common cases (especially when you're talking about libraries), but I really don't think that its build system is powerful enough for the long run. I doubt that Sonke has time to deal with it given how much is on his plate, but we really need to look at making it so that dub supports a much more powerful build system - possibly even making it so that it can support projects building with existing build systems such as cmake. I'm sure that that sort of thing comes with its own types of problems, but what dub does right now is just too simplistic.

A prime example that I've wondered about is how to deal with cases where you need to create a C or C++ wrapper for something and use it in your project. If dub supported makefiles, then that would be trivial, but as it stands, it can't do anything even close to that. Anything that's C or C++ has to already be built no the target system, be in a standard position for the linker to find it, etc., or you can't use it in a dub project. Given our need for C/C++ compatability, I think that that's a serious flaw. Dub is new, and we've gotten by thus far with what it can do, but if it's going to really be our standard build system for D, it needs to improve by quite a bit.

- Jonathan M Davis

On 02/02/2015 03:09 AM, Vladimir Panteleev wrote:
> 1a. rdmd and D's module system:
>[...]
> In contrast, Dub's default modus operandi is to blindly send to the
> compiler all *.d files found in the "src" folder, whether they're
> actually used or not. Not only can this be slower if not all modules are
> always used, but it also won't work if the source code contains multiple
> entry points, forcing you to write complicated configuration files (i.e.
> do the computer's work for it).
>

This is one of my biggest beefs with dub, too, and constantly causes me trouble. (I love that dub exists as a package manager, but I REALLY dislike that it tries to be build system too. Especially since I don't like the way its build functionality works.)

In theory, dub does have the --rdmd switch to make it select source files in a sane manner (by deferring to rdmd), but unfortunately it doesn't get well-tested and frequently breaks:

https://github.com/D-Programming-Language/dub/issues/492

> 1b. rdmd and D's search path
>
> rdmd does not have any additional parameters to set up for where it
> needs to look for source files, because it relies on the compiler's
> search mechanism. Thus, if you can build your program with rdmd, "dmd
> -o- program" will succeed, and usually vice versa.
>
> In contrast, Dub builds its own search path using its JSON configuration
> files, and has no equivalent of "dmd -o-".
>
> There is no simple way to syntax-check just one file in a project when
> using Dub. I rely on this a lot in my workflow - I configured a
> syntax-check key in my editor, which I use almost as often as I save. A
> syntax check (dmd -o-) is much faster than a full build, as it skips
> parsing other parts of the project, code generation, and linking.
>

My #1 beef with Dub: It desperately needs a way to *just* obtain the -o/-version/etc args to be passed directly into dmd/rdmd/ldmd/gdmd/etc. Without that, dub is useless as a MERE package manager. It's either packaging AND building, or nothing. (Ever try to use a recent version of vibe.d in a project that *doesn't* use dub as its build system? PITA. It isn't even recommended to do so.)

I tried to add that feature one time, but I had trouble grokking the relevant section of dub's source :(

There's also one other big thing I don't like about it: It needlessly reinvents and renames dmd's entire set of command switches. That isn't even needed for ldc/gdc anyway since, last I heard, the ldmd and gdmd wrappers exist. I pushed to get the *actual* compiler switched accepted by dub, and the feature made it in, but trying to use it generates a big giant warning for all your library's users complaining that the feature *shouldn't* be used. Thus completely defeating the point. Argh.

Again, I love that Dub (and especially it's package repository) exists. But dub just tries to hard to impose it's own way.

On 02/02/2015 05:15 PM, Nick Sabalausky wrote:
>
> My #1 beef with Dub: It desperately needs a way to *just* obtain the
> -o/-version/etc args to be passed directly into dmd/rdmd/ldmd/gdmd/etc.

Actually I meant -I, not -o. (But I guess maybe -o too.)

On Monday, 2 February 2015 at 21:55:21 UTC, Jonathan M Davis wrote:
> I've long thought that it some point, dub would be forced to support
> arbitrary build systems in order to actually work in the general case. There
> are far too many special cases and more complicated things that build
> scripts and the like frequently have to deal with for dub's simple approach
> to work in many cases. Just look at what something like cmake it can do. It
> blows dub out of the water when it comes to power.
>
> The fact that we have package management via dub is great, and the way it
> builds stuff works for many common cases (especially when you're talking
> about libraries), but I really don't think that its build system is powerful
> enough for the long run. I doubt that Sonke has time to deal with it given
> how much is on his plate, but we really need to look at making it so that
> dub supports a much more powerful build system - possibly even making it so
> that it can support projects building with existing build systems such as
> cmake. I'm sure that that sort of thing comes with its own types of
> problems, but what dub does right now is just too simplistic.
>
> A prime example that I've wondered about is how to deal with cases where you
> need to create a C or C++ wrapper for something and use it in your project.
> If dub supported makefiles, then that would be trivial, but as it stands, it
> can't do anything even close to that. Anything that's C or C++ has to
> already be built no the target system, be in a standard position for the
> linker to find it, etc., or you can't use it in a dub project. Given our
> need for C/C++ compatability, I think that that's a serious flaw. Dub is
> new, and we've gotten by thus far with what it can do, but if it's going to
> really be our standard build system for D, it needs to improve by quite a
> bit.
>
> - Jonathan M Davis

I think it's the moment to point out the latest P.R. merged into dub:
https://github.com/D-Programming-Language/dub/pull/489

It does support Makefiles in a way (prebuiltcommand), but I agree it's rather simplistic.

So. I've reported lots of bugs in DUB and fixed some, and since using it I never looked back. It does seem more stable these days.

On Monday, 2 February 2015 at 08:09:39 UTC, Vladimir Panteleev wrote:
> So, in order to start using Dub, I'd need to:
>
> - restructure the directory layout of my library (breaking change)

I was about to say "yes" but perhaps that's fixable in DUB.
Currently, using ae checkouted in an ae/ directory means having an "import leak" meaning that you have to import a path outside the checkouted repositery. That's somehow unhygienic like, er, a global namespace. But yeah DUB could isolate checkouts one directory deeper (ae-1.0.0/ae/) to make this a non-problem, this would support libraries like ae/ and make DScanner work.

> - update all projects which use this library to use Dub instead
And drop their current build system for something hopefully more uniform and not that complicated.

> - give up quick syntax checking
Fixed it you push code one directory further.

> - give up commit-granularity versioning
Not necessarily if you make tags at each commit (and most of the time it's not really necessary).

> - begin maintaining JSON configuration files
It's not that bad honestly. Actually mandatory field are few and between. If using sub-packages it does take some learning though.

> - begin versioning libraries by hand
Yes, but with the benefit of:
- helping your users choosing a version, and protecting them from churn (eg. broken build)
- choosing if they want breaking changes.
- see also: http://forum.dlang.org/post/nknzdktahezyztzdvwpz@forum.dlang.org for the diamond dependency problem.

git submodules work well when you don't use third-party libs but let's be honest, not everyone understand git submodules let alone want to. I also know some D programmers that don't want to touch git.

> No thanks.
>
> I could invest time in improving Dub to fix or ameliorate some of the above points, but I don't see a compelling reason to. In fact, I think we should integrate rdmd into dmd - dmd clearly already knows which source files participate in compilation, as all rdmd does is basically take dmd's output and feed it back to it. This will greatly speed up compilation, too.
>
> Change my view.

The DUB way does take more work and requires some learning but it makes things super simple for library _users_ and imho this benefits the ecosystem as a whole. If you want more users then DUB is nicer to them.

I've also found it changed my way to develop, I'm much more likely to reuse something from the registry, and much more likely to bundle breaking changes together.

On Monday, 2 February 2015 at 08:09:39 UTC, Vladimir Panteleev wrote:
> ...

Sorry if it has been answered, but one of the points brought up was non-D project management. Is it possible for Dub to i.e, compile C files for me and link them with my D files? I'm currently using makefiles for this.

> - restructure the directory layout of my library (breaking change)

That's likely solveable. Haven't seen anyone putting sources in the root dir for ages though, mostly because it contains readmes and configs.

> - update all projects which use this library to use Dub instead

If we can solve the root Dir issue, then you can use both in parallel.

> - give up quick syntax checking

What keeps you from calling DMD?

> - give up commit-granularity versioning

You should be careful with breaking changes, then the granularity isn't needed.

> - begin maintaining JSON configuration files

It's not zero configuration, but we recently pimped the init command to take dependencies. And there is going to be SDL as alternative to JSON.

> - begin versioning libraries by hand

Semantic versioning is a good thing, because it allows sensible dependency updates.
You also need library versions for your changlog and issues.

> - install Dub on all my computers, servers, and virtual machines

We'll distribute it with DMD soon.
>
> No thanks.
>
> I could invest time in improving Dub to fix or ameliorate some of the above points, but I don't see a compelling reason to. In fact, I think we should integrate rdmd into dmd - dmd clearly already knows which source files participate in compilation, as all rdmd does is basically take dmd's output and feed it back to it. This will greatly speed up compilation, too.

We could also add a lot of the rdmd workflow to dub.
https://github.com/D-Programming-Language/dub/issues/103

There seems to be a general scepticism against dub and I wonder what the reasons are.
A lack of good documentation and guides is clearly the main hurdle for starters, but there seems to be more, so I'm glad that you shared this.