On 11/02/2018 11:40 AM, Jonathan M Davis wrote:
> On Sunday, February 11, 2018 11:26:30 rikki cattermole via Digitalmars-d
> wrote:
>> On 11/02/2018 11:18 AM, Russel Winder wrote:
>>> Clearly though there is a problem with Dub as a build system for many
>>> of it's users – or rather people who try and reject.
>>
>> Put simply, they expect far too much.
>> Dub's scope is limited, lets not forget that.
> 
> The problem with that is that if dub is the way to build D projects in
> general, then it needs to be able to do pretty much whatever you need to do
> for pretty much any project - even if that involves backdoors. You need to
> be able to do arbitrary stuff with your builds.
> 
> It's not as critical for applications so long as dub provides an easy way to
> link in any libraries that it pulls in, but dub needs to be able to build
> libraries no matter what crazy stuff you need to do, otherwise, those
> libraries can't interact with the dub ecosystem, and dub is how D projects
> in general pull in their dependencies.
> 
> So, for instance, if your D library has to build C or C++ code and link that
> in as part of what it does, that needs to be possible with dub, even if dub
> itself doesn't handling building code that isn't D. Also, if you need to
> generate code as part of your build and then build those files, that needs
> to be possible. And the way that dub is set up at this point, that sort of
> stuff is rather difficult to do.
> 
> dub wouldn't have to be all that powerful if it were simply a handy build
> tool for the average use case, but when it's tied in to package management
> for D libraries and is the primary way that D projects pull in libraries, it
> needs to be far more than a simple build tool. And right now, it's not far
> enough away from being a simple build tool.
> 
> - Jonathan M Davis

Dub can do everything that you have described.
You are fully free to run cmake if you wish before the build. Will it result in binaries that are decent? Probably not for most use cases.

Its a hard problem to solve, I just wish people respected dub's scope more. Because it is very decently well scoped for its job already.

On 02/11/2018 01:54 AM, Pjotr Prins wrote:
> Dub is getting some flak here. This is unsurprising because it is really hard to write a good package manager and build system. I use a lot of languages and not one has a satisfactory package manager. Mostly they try to do too much and get in the way or they do too little and people complain (I prefer the second option). And when there are 100+ dependencies, like with Go and Node, it just becomes impossible to say anything about the state of the system (security, anyone?).
> 
> Package management is mostly dependency management. This I handle with GNU Guix (and Nix) package managers. They are great at that.

Ugh, system-level package managers. That's the #1 reason I keep hopping around distros, never finding one I like:

They're all completely fucking non-standard. Every fucking distro has it's own damn package manager (and then there's mac and win and the mobiles), and on top of that, most of those system managers are almost completely worthless *when* you need something other then "the latest version" let alone multiple versions (because "newer is always better" is software's #1 sacred cow). Those are *exactly* the reasons why programming langauges have NEEDED to start rolling their own damn package managers, because relying on the goddamn system managers for libs is a complete fucking non-starter.

(Hmm, Can you tell I'm not happy with system package managers? ;) )

The langauge-based package managers just simply need to keep "buildsystem" OUT of the package manager's scope. That's all. THAT is why system-level packages can be built with whatever tool: because the one thing the system package managers actually do get right is NOT rolling their own mandatory buildsystem.

One of the things on my master list of "project's I'll probably never get around to" is a CLI front-end to unify all this fractured apt/pacman/yum crap.

> This also leaves people to choose any old build system. Inside GNU Guix the build system is consistent, which is really nice. I'll write a blog some time this year.
> 
> What you really want is to be able to discover packages (i.e., a website such as Dub provides), pull them into your tree (which is just a path and can be handled by git submodules, though I don't like those much either),

Agreed.

> and when you distribute: add them to Guix or Nix and provide those packages with build system and as binary deployments to others. 

And fracture the hell out of your user-base, forcing even Linux alone to be treated as 100 different fucking incompatable operating systems for which each package only supports one or two? Ugh, god no. Spare me. Even dub, for all it's shortcomings at LEAST has absolutely no trouble treating nearly all of Linux as ONE unified target.

On 02/11/2018 06:18 AM, Russel Winder wrote:
> 
> Clearly though there is a problem with Dub as a build system for many
> of it's users – or rather people who try and reject.
> 

The problem isn't just "dub as a buildsystem". The other equally big problem here is that "dub as a package manager" pretty much forces "dub as a buildsystem" for library authors. Yes, *officially* it doesn't, but realistically, yea it does. And I don't see how dub can fix in any sane way without a major, fundamental redesign.

On 02/11/2018 06:47 AM, rikki cattermole wrote:
> 
> Dub can do everything that you have described.

No it can't. Not if you value your time and sanity.

> You are fully free to run cmake if you wish before the build. Will it result in binaries that are decent? Probably not for most use cases.
> 

I *have* taken exactly that approach. It caused me literally years of unending greif, even single-handedly turned me away from doing much in D for a long while, and ultimately just proved intractable. Abandoning dub *entirely* is actually *easier* than trying to use it *only* as a package manager. But if you're putting out a D lib, you can't do that because people expect a lib to be part of a package system (for good reason).

On Monday, 12 February 2018 at 01:45:42 UTC, Nick Sabalausky (Abscissa) wrote:
> But if you're putting out a D lib, you can't do that because people expect a lib to be part of a package system (for good reason).

Yeah, this drives me nuts too. Most "bugs" I hear are just dub being weird (like mine I put as "Sourcelibrary" and it works fine if you use one... but if it is included by some other lib as some other build, it now conflicts. What I wish it did was just realize the package is already a dependency and reuse it! But it doesn't seem to work that way.)

And dub's search is kinda bad and the index is just flat-out awful, which does more harm than good. People do the code.dlang.org search, something doesn't come up, so they assume it isn't there even when it is.

I've considered doing a new package website with a new search, using dub's official stuff just as a backend api, coupled with docs available online. I think I can fix a bunch of those frontend problems with it. But I just a lot of higher priority stuff to do (like actually writing my libs!) so I haven't gotten around to it.

On Sunday, February 11, 2018 20:30:19 Nick Sabalausky  via Digitalmars-d wrote:
> The langauge-based package managers just simply need to keep "buildsystem" OUT of the package manager's scope. That's all. THAT is why system-level packages can be built with whatever tool: because the one thing the system package managers actually do get right is NOT rolling their own mandatory buildsystem.

I'm not sure that it's entirely a bad thing that dub includes a build system in it, since it's the ease of use of its build system that is part of why it's so worth using. It's just that its simplicity is also part of why it doesn't work well once you get out of the basic situation of "here's a group of D files that need to be compiled together as a library or application."

So, I do agree that in principle, we would be much better off if dub weren't so tightly coupled with its build system. If dub were defined in a way that allowed packages to use whatever build system they wanted so long as they provided the build artifacts in a standard way to the package manager so that they could be linked into other dub projects, we'd have something a lot more flexible. And if done right, I think that it would be much better. dub's default build system could still be used, but it wouldn't be forced.

The problem with that approach though (and part of why I think dub doesn't do that) is that if dub really is set up with the idea that you use whatever build system you want, then you have the issue of whether the system doing the build has all of the necessary software to do the build. Dub doesn't completely avoid that as-is, since you can depend on certain libraries being on the system, and you can't write scripts to be run as part of the build which could then require who-knows-what on the system (they're just difficult to get to work with dub in a number of cases because of how much dub assumes). But if dub itself were designed with the idea that you could use cmake, scons, make, or whatever build system anyone felt like putting together, then there's a fairly high chance when you go to pull in a package from dub that it's going to require stuff to build that you don't have on your system, and since you frequently end up pulling in packages recursively such that you could be using dub packages that you've never even heard of, it could become difficult to ensure that you have everything you need to build everything in the dub package chain.

So, dub's solution of targeting a single build system side steps that issue to a great extent - not entirely, but enough that it doesn't tend to be a problem. And on some level, that's a good thing, but it comes at a definite cost when you actually need something fancier.

So, I don't know what the right solution is. I'd like to see dub do better interacting with arbitrary build systems, and if that can be done well, I think that dub would be much better, but it's a hard problem. Ultimately though, dub seems to have been designed to solve the common case while not really making the general case very tractable (maybe not impossible, but definitely difficult), but its place as the way the way to pull in 3rd party libraries in D means that it arguably needs to be doing more than just solve the common case, or 3rd party libraries that don't work with the common case become intractable, which is definitely not good.

We would probably be better off if someone could come up with an alternate package manager that was at least compatible enough with dub to use libraries that use dub so that it could both work with dub and compete with it and maybe ultimatelly get things right, but package and build management is one of those hard problems that no one wants to work on, and plenty of folks like to complain about.

As with too much around here, a big part of the issue is ultimately man power. Even if we could all agree on exactly how D's build and package management situation should be solved, actually get the work done is a huge problem.

- Jonathan M Davis

On Sunday, February 11, 2018 19:01:09 Jonathan M Davis via Digitalmars-d wrote:
> As with too much around here, a big part of the issue is ultimately man power. Even if we could all agree on exactly how D's build and package management situation should be solved, actually get the work done is a huge problem.

On that note, I feel that I really should thank Sonke and those few that have helped him write dub. As much as I think that some aspects of dub need to be redesigned, at least they actually went and did something and managed to get it to take off enough that there are quite a few D libraries available using dub. So, while the solution may not be what we want, at least we have a solution.

And even if someone else does come along and put in the time and effort like the dub folks did but comes up with a solution that we like better, the new solution will undoubtedly benefit from seeing what has and hasn't worked for us with dub.

And maybe dub itself will be reworked to be where we want things to do be, but either way, it requires more manpower, and at least right now, we have _something_, even if it's far from perfect.

- Jonathan M Davis

On 02/10/2018 07:35 AM, Timon Gehr wrote:
> 
> TL;DR: Parametrically polymorphic functions have /runtime/ type parameters. inout can be interpreted as a dependent function of type "{type y | y.among(x, const(x), immutable(x)) } delegate(type x)" and an inout function can be thought of as a function that takes inout as an argument and produces the function as the return value. This formulation is more powerful than what the inout syntax can capture, and this is what causes problems with type safety. In particular, 'inout' does not support proper lexical scoping.
> 
> TS;NM: https://gist.github.com/tgehr/769ac267d76b74109a195334ddae01c3 (Some version of this was originally intended to go to the D blog, but I wanted to wait until inout has an obviously type safe definition. It also highlights other inout issues than just type unsafety and shows how all of them might be fixed in principle by adding polymorphism.)
> 
> ---
> 
> I'll first explain parametric polymorphism, and then what the inout problem is.
> 

Ahh, thanks. I'm still not *completely* 100%, but those explanations definitely helped a lot. Very insightful.

Side questions:

- Does this mean that fixing the issues with inout (even if done via some replacement of inout) would necessarily involve some runtime processing in at least certain cases?

- Though I realize this may contradict the definition of "parametric polymorphism" as you've described (and also ignoring the matter of virtual functions): Would it be theoretically possible to have parametric polymorphism that's *implemented* entirely via compile-time mechanisms such as templates? Or does our template system as D has it already represent the fundamental limits of such an approach?

Incidentally, I've felt for a long while it would've been really nice if D had first-class types (for both runtime and compile time). I think a system like that, if done right, could have gone a long way to alleviating all the awkwardness and asymmetries of reflection and type handling in D. It's also why I can understand people coming from certain dynamic languages who may find our metaprogramming combersome.

On Monday, 12 February 2018 at 02:31:38 UTC, Jonathan M Davis wrote:
> On Sunday, February 11, 2018 19:01:09 Jonathan M Davis via Digitalmars-d wrote:
>> As with too much around here, a big part of the issue is ultimately man power. Even if we could all agree on exactly how D's build and package management situation should be solved, actually get the work done is a huge problem.
>
> On that note, I feel that I really should thank Sonke and those few that have helped him write dub. As much as I think that some aspects of dub need to be redesigned, at least they actually went and did something and managed to get it to take off enough that there are quite a few D libraries available using dub. So, while the solution may not be what we want, at least we have a solution.

Absolutely. Dub gets criticism because it is used which is an indicator of a successful piece of work. Kudos for anyone who pulls that off. Now we just find it is hard to do this well.

As someone involved with software distribution and deployment in production systems, my suggestion is to limit the scope of the package manager. Best have to do it little, but do it well and make most of existing solutions. The goal should really be package discovery and making it easy for beginners (autotools and configure does not do today, though C gets away with it). Thereafter comes a point you have to hand over the deployment issue. I strongly suggest not to make it complicated and not to try to create a homogeneous development system for all targets. I'll write up how I manage my development and deployment some time later this year, it may or may not be a good example ;)

On 02/11/2018 09:01 PM, Jonathan M Davis wrote:
> On Sunday, February 11, 2018 20:30:19 Nick Sabalausky  via Digitalmars-d
> wrote:
>> The langauge-based package managers just simply need to keep
>> "buildsystem" OUT of the package manager's scope. That's all. THAT is
>> why system-level packages can be built with whatever tool: because the
>> one thing the system package managers actually do get right is NOT
>> rolling their own mandatory buildsystem.
> 
> I'm not sure that it's entirely a bad thing that dub includes a build system
> in it, since it's the ease of use of its build system that is part of why
> it's so worth using. It's just that its simplicity is also part of why it
> doesn't work well once you get out of the basic situation of "here's a group
> of D files that need to be compiled together as a library or application."

You're right, of course. I guess I wasn't clear about this, and frankly I even lost sight of it, but if it HAD been easy and clear to just say "hey, dub, I'm bringing along my own buildsystem and opting out of yours, ok? thanks dude" and that was that, no muss, no fuss, no "tricking" dub into doing what I need, I would've been fine with that.

The problem is that, in my experience, dub's assumptions about it doing your build for you tend to be so deeply ingrained into the way it and its configuration format all work, that I just don't see attempts to separate them out as likely work out all that well. If even at all.

Heck, at this point I feel like most of my attempts to get dub to do anything I have in mind have wound up being less "what is the directive to do that?" and more "How to I nudge, persuade, or outright trick dub into doing what I need?" Even if I just have a compiler switch I want to use, or not use, I can't *just use it*. Best case scenario, dub will brand my library with a big ugly warning when anyone uses it. More likely, I have to go translate it into dub-speak (if I'm LUCKY), or, all too frequently, I find myself flailing around, trying to find just the right combination of settings to *coax* dub's behavior.

This is a completely *typical* dub experience for me, and frankly, one of the more *pleasant* ones:
https://github.com/atilaneves/unit-threaded/issues/106

Did you know dub behaves *differently* if your test configuration is named "unittest" verses when it has a different name? I didn't. Principle of most surprise, I guess.

THIS is me using dub. Flawless illustration:
https://www.youtube.com/watch?v=QvcFRgJwE2k&t=0m12s

> The problem with that approach though (and part of why I think dub doesn't
> do that) is that if dub really is set up with the idea that you use whatever
> build system you want, then you have the issue of whether the system doing
> the build has all of the necessary software to do the build.  Dub doesn't
> completely avoid that as-is, since you can depend on certain libraries being
> on the system,
> [...]But if dub itself were designed with the idea that you could
> use cmake, scons, make, or whatever build system anyone felt like putting
> together, then there's a fairly high chance when you go to pull in a package
> from dub that it's going to require stuff to build that you don't have on
> your system, and since you frequently end up pulling in packages recursively

That's why the way it *should* work is: If your build has a dependency (another .d lib, a C lib, cmake, rake, scons, whatever), you make a dub package for it if there isn't already one. Problem solved.

But once again, dub's assumption that *everything* is a set of *.d files for it to complile gets in the way here too.

> So, dub's solution of targeting a single build system side steps that issue
> to a great extent - not entirely, but enough that it doesn't tend to be a
> problem. And on some level, that's a good thing,

No, all it does it create problems and prevent workarouds, as described above. If a package needs, ex, scons, to build, it should be simple to create a dummy dub package for scons that (best case) offers to install scons for you, maybe even a local non-system-wide installation, or at the very least, tells the user, "hey, scons not detected, it's a dependncy of this package, so you need to install it."

After all, managing dependencies are what package managers are all about, and dub isn't handling it very well when it comes to tool dependencies (as opposed to "a library of .d files to be linked in" dependencies, which is really all it's designed for).

> 
> So, I don't know what the right solution is.

The right solution is to give up on dub, as it's too far down the wrong design road, and have a package manager that's designed from the ground up to be a package manager (not a simplified buildsystem that happens to also include dependency-handling features). As a package manager, *anything* the package needs, a D library, a C library, an external tool, even a compiler version and the package manager itself...is just another dependency for the package manager to handle.

I should point out, strong as I may feel about all this, I don't mean to be too much of a dick about dub. I'm well aware how much effort went into dub, and I *do* respect that. Frankly, I've always been absolutely *stunned* by just how much Sonke accomplishes. Beyond me how he manages to do all that: I know I certainly couldn't. It seems superhuman! And I am grateful there's been at least SOMETHING for D package management.

It's just, from a technical perspective looking toward the near/mid-future, I just have a hard time seeing dub getting to where we really need a D package manager to be, and I admit I've gotten to the point of real frustration.