On Wednesday, 17 April 2019 at 08:50:31 UTC, Atila Neves wrote:
>
> Could you please give an example of this?
>

Sorry I'm not clear.

I'll dig into my conversations with MrSmith33 later and will come back to you because it's not in my mind anymore (and I wasn't the one doing the work).

Let's meet at DConf at least to talk about this.

On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:

> Or perhaps people will continue to say stuff on very long email threads
> and end up doing nothing. This is the third such thread on Dub I can
> remember, the previous two ended up with no  outcomes other than lots
> of opinions in very long email threads.

This is a leadership problem. Notably absent from these discussions are Andrei and Walter. This is a critical piece of infrastructure that's not working, it's distributed with DMD, and they're completely silent because it's not something they find interesting.

If you're the leader of a large open source project, now is when you step in and ask who is willing to put in the time to take this on, so you can give one or more of them the authority to make decisions. There should be a formal proposal that can be commented on as we do with a DIP. Work begins, with a formal definition of when it's ready to go live guiding those doing the work.

Maybe we'll see something like that happen. Based on what I've seen over the last six years, it seems unlikely. Instead there will be long email threads by individuals that have no decisionmaking authority, and everyone will conclude that they'd be crazy to dump hundreds of hours into such a project. In a couple months we'll have a fourth long email thread on ways to fix Dub.

On Wednesday, 17 April 2019 at 13:41:43 UTC, bachmeier wrote:
> On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:
>
>> Or perhaps people will continue to say stuff on very long email threads
>> and end up doing nothing. This is the third such thread on Dub I can
>> remember, the previous two ended up with no  outcomes other than lots
>> of opinions in very long email threads.
>
> This is a leadership problem. Notably absent from these discussions are Andrei and Walter. This is a critical piece of infrastructure that's not working, it's distributed with DMD, and they're completely silent because it's not something they find interesting.
>
> If you're the leader of a large open source project, now is when you step in and ask who is willing to put in the time to take this on, so you can give one or more of them the authority to make decisions. There should be a formal proposal that can be commented on as we do with a DIP. Work begins, with a formal definition of when it's ready to go live guiding those doing the work.
>
> Maybe we'll see something like that happen. Based on what I've seen over the last six years, it seems unlikely. Instead there will be long email threads by individuals that have no decisionmaking authority, and everyone will conclude that they'd be crazy to dump hundreds of hours into such a project. In a couple months we'll have a fourth long email thread on ways to fix Dub.

Please be more precise. Dub does a perfect job if you have standalone D applications. There are issues, as I understand, if you have complex integration scenarios with C/C++.

Dub is built by the community for the community. I also had to create some pull reduests to get all my scenarios working and now everything works for my scenarios like a charme.

Maybe for complex integration scenarios another build tool has to be used. Or maybe Dub has to be improved to support complex scenarios.

I do not see any management issue here.
If you (to all) have issues, please consider creating pull requests.

Kind regards
Andre

On 17.04.2019 17:09, Andre Pany wrote:
> 
> Please be more precise. Dub does a perfect job if you have standalone D applications. There are issues, as I understand, if you have complex integration scenarios with C/C++.
> 
> Dub is built by the community for the community. I also had to create some pull reduests to get all my scenarios working and now everything works for my scenarios like a charme.
> 
> Maybe for complex integration scenarios another build tool has to be used. Or maybe Dub has to be improved to support complex scenarios.
> 
> I do not see any management issue here.
> If you (to all) have issues, please consider creating pull requests.
> 
> Kind regards
> Andre
> 
> 
I agree with you totally. I don't understand why W&A should appear in this thread. dub wasn't intended to be standard package manager. And it's absolutely normally that now community needs to improve it. There is no someone fault at all.

On Wednesday, 17 April 2019 at 14:09:06 UTC, Andre Pany wrote:

> Please be more precise. Dub does a perfect job if you have standalone D applications. There are issues, as I understand, if you have complex integration scenarios with C/C++.
>
> Dub is built by the community for the community. I also had to create some pull reduests to get all my scenarios working and now everything works for my scenarios like a charme.
>
> Maybe for complex integration scenarios another build tool has to be used. Or maybe Dub has to be improved to support complex scenarios.
>
> I do not see any management issue here.
> If you (to all) have issues, please consider creating pull requests.

Well clearly others disagree with you. You are saying "it works for me - make changes to the source code and submit a PR if you don't like it."

That is not an answer. It excludes anyone new to D, anyone that uses D but does not have time to work on Dub, or anyone that doesn't have the ability to make changes to Dub as needed for acceptance. No such requirement exists for using any other language to my knowledge, and I've used a lot of languages over the years. If this is the path that will be taken, there is no point speculating why the language isn't used more, especially in commercial settings. There's not even a starting point for using the language if these issues are not resolved.

Dub is an official project because it is included with DMD and it is the recommended (and for the most part only) way to distribute D code. It needs to be taken seriously.

On Wed, Apr 17, 2019 at 03:48:36AM -0400, Nick Sabalausky (Abscissa) via Digitalmars-d wrote:
> On 4/16/19 1:39 PM, H. S. Teoh wrote:
[...]
> > I was thinking the build/test/etc. command can itself be defined by a dependency.  For example, if your project uses make for builds, there could be a `make` dub package that defines command-line templates used for generating platform-specific build commands. Your package would simply specify:
> > 
> > 	"build-system": "make"
> > 
> > and the "make" package would define the exact command(s) for
> > invoking make [...etc...]
> 
> Interesting idea. I definitely like it, though I would consider it not a "basic fundamental" mechanism, but rather a really nice bonus feature that provides a convenient shortcut for having to specifying certain things manually...IF you happen to be using a buildsystem known by the package manager...or (better yet, if possible) a buildsystem that's able to tell the package manager enough about itself to make this happen.

The idea behind this is that in order to avoid baking in support for a multitude of build systems (make, scons, meson, gradle, whatever) into dub, which will inevitably be unmaintained and out-of-date 'cos is too much work to keep up with everything, we delegate the job to individual packages.  There would be one package per external build system, e.g., a 'make' package for make support, an 'scons' package for scons support, and so on.  These packages don't have to do very much, all they need is to (1) define a way of installing said build system, in a way that can be used by dub via a standard interface, and (2) export various command-line templates that dub will use for invoking the build system.

So for example, the 'make' package might contain a bunch of URLs for downloading a 'make' executable and any associated libraries and data files, and then export `/path/to/dub/cache/bin/make` as the command template for invoking the build system.  There could be OS-dependent variations, such as `C:\path\to\dub\cache\make\gmake.exe` if you're on Windows, for example.  These command flavors are defined within the 'make' dub package, and then any project that uses make can just declare a dependency on 'make', and dub will know how to do the Right Thing(tm). Dub itself wouldn't have to know the specifics of how to work with a make-based source tree; it will just execute the commands defined by the 'make' dub package.


[...]
> > If done correctly, dependency resolution is just a topological walk on a DAG.  Get this right, and everything else falls into place.
> 
> Yes, but therin lies the rub - "getting this right". AFAICT, Sonke seems to have a very good grasp on dependency resolution via DAG (or at least, certainly a far better grasp than I do). But yet, even his algorithms in DUB, he needed to occasionally patch with common-use-case short-circuiting logic (and I don't even know what else) just to fix various seemingly-infinite-loop issues (not technically infinite, but appeared to be to the user) that people were still reporting some years into DUBs lifetime.
> 
> 'Course, if it really is simpler than it seems to me (and *especially* is it *isn't*), then I'd certainly be more than happy to delegate dependency resolution out to somebody else.

It would help if somebody can provide a concrete example of a complex dependency graph that requires anything more than a straightforward topological walk.


T

-- 
MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a 16-bit patch to an 8-bit operating system originally coded for a 4-bit microprocessor, written by a 2-bit company that can't stand 1-bit of competition.

On 17.04.2019 19:03, H. S. Teoh wrote:
>>> If done correctly, dependency resolution is just a topological walk
>>> on a DAG.  Get this right, and everything else falls into place.
>>
>> Yes, but therin lies the rub - "getting this right". AFAICT, Sonke
>> seems to have a very good grasp on dependency resolution via DAG (or
>> at least, certainly a far better grasp than I do). But yet, even his
>> algorithms in DUB, he needed to occasionally patch with
>> common-use-case short-circuiting logic (and I don't even know what
>> else) just to fix various seemingly-infinite-loop issues (not
>> technically infinite, but appeared to be to the user) that people were
>> still reporting some years into DUBs lifetime.
>>
>> 'Course, if it really is simpler than it seems to me (and *especially*
>> is it *isn't*), then I'd certainly be more than happy to delegate
>> dependency resolution out to somebody else.
> 
> It would help if somebody can provide a concrete example of a complex
> dependency graph that requires anything more than a straightforward
> topological walk.
> 
> 
> T
> 

Couple of years ago I've made an example - https://github.com/drug007/dub-reduced-case
Now dub works in other way than that time but nevertheless.
Now it says:
```
drug@drug:/tmp/dub-reduced-case/rdpl$ dub build :inspector
Building package rdpl:inspector in /tmp/dub-reduced-case/rdpl/inspector/
Root package rdpl:core reference gfm:math ~>6.1.4 cannot be satisfied.
Packages causing the conflict:
	rdpl:core depends on ~>6.1.4
	timespatial depends on ~>6.0.0
	timespatial depends on ~>6.0.0
	timespatial depends on ~>6.0.0
```
shouldn't dub select gfm:math of version 6.1.4 here?

On Wednesday, 17 April 2019 at 16:03:24 UTC, H. S. Teoh wrote:
> The idea behind this is that in order to avoid baking in support for a multitude of build systems (make, scons, meson, gradle, whatever) into dub, which will inevitably be unmaintained and out-of-date 'cos is too much work to keep up with everything, we delegate the job to individual packages.  There would be one package per external build system, e.g., a 'make' package for make support, an 'scons' package for scons support, and so on.  These packages don't have to do very much, all they need is to (1) define a way of installing said build system, in a way that can be used by dub via a standard interface, and (2) export various command-line templates that dub will use for invoking the build system.
> [...]

Invoking the external tools is probably doable, the problem is consuming artifacts generated by these tools. Then there are the specialties of each tool: some distinguish between configuration (done once for a fresh build) and build, some are very good at tracking the dependencies and incremental builds, some rediscover the world on each invocation, some support cross-compiling, some use command line parameter for customizing the build, other use configuration files, some support debug builds, some support listing the build targets, some support a server mode, some require a server mode, some deprecate features over time, etc., etc. Finally, it's a mess.

There is an effort in Scala world to do something in the same direction (they have a lot of build systems): https://github.com/scalacenter/bsp/blob/master/docs/bsp.md. But this effort resumes itself to just IDE integration.

What I'm missing is the tool for building D projects, do you suggest that each project should use what it seams fit, or just stick with dub?

> It would help if somebody can provide a concrete example of a complex dependency graph that requires anything more than a straightforward topological walk.

I don't know if I understand you right, but probably having some generated code, or calling shell scripts with side-effects already complicates the problem enough.

On 4/17/19 4:06 AM, Russel Winder wrote:
> On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
> Digitalmars-d wrote:
>>
> […]
>>
>> Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
>> see
>> on this particular list, and that definitely includes your
>> suggestion. I
>> left it at "not necessarily limited to..." for good reason ;)
> […]
> 
> The problem here is that people are having lots of ideas on this email
> list, but no-one is choosing to collate them into a an actual "call to
> action".

FWIW, the only reason I was being non-canonical about desired features in a package manager is that (from experience) I wanted to avoid pie-in-the-sky up-front design and focus first on the core necessities - additional niceties can come later, on top of a solid working core.

> As ever in an ever extending email thread the energy to do
> something within the D community dissipates and nothing ends up
> happening.

Actually, I've been fired up enough about this for a good while to put my code where my mouth is (I just have a bad habit of spreading myself too thin dev-wise, and having too much other life crap getting in the way). And, personally, I find this to be D's biggest current stumbling block, even moreso than anything in the language or compiler (which is probably good that I feel that way, because I'm pretty much out-of-my-league when it comes to hacking on the compiler). So note my next post in this thread...

On Wed, Apr 17, 2019 at 04:51:57PM +0000, Dragos Carp via Digitalmars-d wrote:
> On Wednesday, 17 April 2019 at 16:03:24 UTC, H. S. Teoh wrote:
> > The idea behind this is that in order to avoid baking in support for
> > a multitude of build systems (make, scons, meson, gradle, whatever)
> > into dub, which will inevitably be unmaintained and out-of-date 'cos
> > is too much work to keep up with everything, we delegate the job to
> > individual packages.  There would be one package per external build
> > system, e.g., a 'make' package for make support, an 'scons' package
> > for scons support, and so on.  These packages don't have to do very
> > much, all they need is to (1) define a way of installing said build
> > system, in a way that can be used by dub via a standard interface,
> > and (2) export various command-line templates that dub will use for
> > invoking the build system.
> > [...]
> 
> Invoking the external tools is probably doable, the problem is consuming artifacts generated by these tools.

It's not really a problem, as far as dub is concerned.  All we need is some way to export the list of generated artifacts.  E.g., if it's a library package, all you really need is to export (1) the import paths and (2) the path(s) to the generated .so / .a / .lib / .dll files, and dub should be able to figure out how to make use of this library without actually knowing what the build command does.

In other words, we hide the complexity of the build system under one or more canned invocation commands (e.g., 'make', 'make clean', 'make docs', 'make test', for a standard set of actions you might commonly want to do), and have the wrapper package tell us what the build artifacts are.


> Then there are the specialties of each tool: some distinguish between configuration (done once for a fresh build) and build, some are very good at tracking the dependencies and incremental builds, some rediscover the world on each invocation, some support cross-compiling, some use command line parameter for customizing the build, other use configuration files, some support debug builds, some support listing the build targets, some support a server mode, some require a server mode, some deprecate features over time, etc., etc. Finally, it's a mess.

I don't think we need to support all of those cases. As far as dub-as-a-package-manager is concerned, all it really cares about is that each dependency is some black box amorphous blob of files, with an associated set of arbitrary shell commands for each standard action: build debug, build release, test, query import paths, query library paths, etc..  These actions should be standard across all codebases, and it doesn't matter how they are implemented underneath. They can be arbitrarily complex shell commands. The dirty details are abstracted away by the build system wrapper packages.

As for cross-compilation, IMO this should have built-in support as an optional feature. I.e., there should be a standard command for "cross-compile to architecture XYZ", and packages can either support this or not. If one or more dependencies don't support cross-compilation, then you cannot cross-compile (with dub).


> There is an effort in Scala world to do something in the same direction (they have a lot of build systems): https://github.com/scalacenter/bsp/blob/master/docs/bsp.md. But this effort resumes itself to just IDE integration.
> 
> What I'm missing is the tool for building D projects, do you suggest that each project should use what it seams fit, or just stick with dub?

If you're asking me about the *current* state of dub, I have to say I don't use it except to pull in code.dlang.org dependencies.  I.e., I find it only useful as a package manager and nothing else.  As a build system it does not support critical use cases that I need,[*] so I find it completely unusable.

[*] Specifically, these are not supported by the current version of dub AFAIK:

(1) Arbitrary codegen (i.e., build a subset of D files into helper
program H1, then invoke H1 on some data files to generate D code, then
compile the resulting D code into the final executable);

(2) Cross-compilation (e.g., Linux x86 host -> Android/ARM);

(3) Arbitrary preprocessing of data files (build a subset of D files into helper program H2, then invoke H2 on some data input files to produce final data files -- e.g., compress HTML/Javascript, rescale images, etc.);

(4) Building of deliverables (e.g., pack build products X,Y,Z into a
.zip file, or build/sign an APK, or build a Debian .deb package, etc.).

These are all non-trivial tasks, and I do not expect that dub will ever support all of them natively (though it would be nice if it allowed me to specify custom commands to accomplish such tasks!).  That's why I'm pushing for separating dub as a package manager from dub as a build tool.  The package manager's responsibility is to resolve dependencies, resolve version constraints, and download dependencies.  How these dependencies are built should be handled by a *real* build system that can handle (1) to (4) above.

That's why I'm proposing to encapsulate the task of building to a wrapper dub package that abstracts away the complexity.  At the end of the day, it just boils down to (1) you have a bunch of input files, (2) there's some command that transforms these input files into output files, and (3) these output files reside somewhere in the filesystem. As long as you're told where the output files in (3) are, you know enough to use this package.  What goes on inside (2) is irrelevant; said command can be arbitrarily complex -- it can be an entire build system, for example, that consists of many subcommmands -- but a package manager doesn't need to (and shouldn't) care about such details.


> > It would help if somebody can provide a concrete example of a complex dependency graph that requires anything more than a straightforward topological walk.
> 
> I don't know if I understand you right, but probably having some generated code, or calling shell scripts with side-effects already complicates the problem enough.

This is only a problem if you're conflating package management with a build system.  If you tackle solely the problem of package management, then these are not problems at all.  As far as a package manager is concerned, you have a root package P that has dependencies on some packages A, B, C, that in turn depend on other packages D, E, F.  Your job as package manager is to figure out which version(s) of A, B, C, D, E, F to download.  Each package is an arbitrary collection of files, with an associated command that transforms these files into build products.  All you need to know is a string containing the command that performs this transformation, and a list of paths to the build product(s).  You don't care what files are in the package, really.  For all you care they could be a bunch of URIs that point to remote objects, or a bunch of encrypted data files, or a .zip file of an entire OS. None of that is relevant.  All that matters is that there's a command (specified within the package itself) that tells you how to transform said files into the build products.  You just run this command and let it do its job -- how, you don't care.  As long as the command produces the build products, that's good enough.  Once it's done, you just query for the path(s) to the build products, and use that in the next node up the dependency tree.

It's not the package manager's job to stick its grubby hands into the dirty details of the actual build process.


T

-- 
ASCII stupid question, getty stupid ANSI.