April 17, 2019
On Wednesday, 17 April 2019 at 17:53:35 UTC, H. S. Teoh wrote:
> In other words, we hide the complexity of the build system
> under one or more canned invocation commands (e.g., 'make',
> 'make clean', 'make docs', 'make test', for a standard set of
> actions you might commonly want to do), and have the wrapper
> package tell us what the build artifacts are.

What you lose with this is dub's current platform independence.
That's not always present anyway, for example a project could rely
on specific cPanel or macOS features, but it would be nice to
retain it as the default.

Ada's gprbuild has project files like this:

  project Packetbl is
     for Object_Dir use "build";
     for Exec_Dir use "bin";
     for Source_Dirs use ("src");
     for Languages use ("Ada", "C");
     for Main use ("packetbl.adb");
     package Compiler is
        for Default_Switches ("Ada") use ("-O3", "-gnata", "-gnaty-m", "-gnatwa");
        for Default_Switches ("C") use ("-O3", "-Wall");
     end Compiler;
     package Linker is
        for Default_Switches ("Ada") use ("-lnfnetlink", "-lnetfilter_queue", "-lanl");
     end Linker;
  end Packetbl;

So, this project uses Ada and C languages, a binary should be built
with packetbl.adb as the main, and other language files under src/
are linked in. This project doesn't bother to list source files,
but gprbuild knows what .adb/.ads and .c files are, so it builds
them all with the appropriate compiler and then links them
together.

To know how to compile, gprbuild has a system configuration that
includes the exact compilers to use for each language, their paths,
etc. The sort of thing you'd set with 'dub init', with probable
defaults, and with compiler knowledge (of how to invoke "gcc",
"clang", etc.) built into dub.
April 17, 2019
On Wed, Apr 17, 2019 at 06:17:25PM +0000, Julian via Digitalmars-d wrote:
> On Wednesday, 17 April 2019 at 17:53:35 UTC, H. S. Teoh wrote:
> > In other words, we hide the complexity of the build system under one or more canned invocation commands (e.g., 'make', 'make clean', 'make docs', 'make test', for a standard set of actions you might commonly want to do), and have the wrapper package tell us what the build artifacts are.
> 
> What you lose with this is dub's current platform independence. That's not always present anyway, for example a project could rely on specific cPanel or macOS features, but it would be nice to retain it as the default.
[...]

Actually, this is precisely why I said that build systems would have their own wrapper packages.  The wrapper packages would specify potentially different commands to run, depending on the current platform.  Since the wrapper package would have specific knowledge about the build system it's wrapping, it could craft the build command such that it sets up the appropriate platform-dependent settings.

E.g., the 'make' wrapper would specify a command that sets up INCPATH to /usr/include on Linux, for example, and C:\blah\blah\include on Windows.

In fact, we could even use this as a mechanism to support cross compilation.  E.g., given the current platform and the target platform as parameters, the wrapper package would craft a build command that sets up the appropriate path(s) and environment variables for cross compilation.  This is an optional bonus feature, of course.

The point is that the build system wrapper packages would hide away these details from dub, so dub doesn't have to know about how to set up the build for Windows vs. Linux. It just installs the wrapper package and queries it for the right command to use.


T

-- 
Frank disagreement binds closer than feigned agreement.
April 17, 2019
On 4/17/19 12:03 PM, H. S. Teoh wrote:
> On Wed, Apr 17, 2019 at 03:48:36AM -0400, Nick Sabalausky (Abscissa) via Digitalmars-d wrote:
>> On 4/16/19 1:39 PM, H. S. Teoh wrote:
> [...]
>>> I was thinking the build/test/etc. command can itself be defined by
>>> a dependency.  For example, if your project uses make for builds,
>>> there could be a `make` dub package that defines command-line
>>> templates used for generating platform-specific build commands. Your
>>> package would simply specify:
>>>
>>> 	"build-system": "make"
>>>
>>> and the "make" package would define the exact command(s) for
>>> invoking make [...etc...]
>>
>> Interesting idea. I definitely like it, though I would consider it not
>> a "basic fundamental" mechanism, but rather a really nice bonus
>> feature that provides a convenient shortcut for having to specifying
>> certain things manually...IF you happen to be using a buildsystem
>> known by the package manager...or (better yet, if possible) a
>> buildsystem that's able to tell the package manager enough about
>> itself to make this happen.
> 
> The idea behind this is that in order to avoid baking in support for a
> multitude of build systems (make, scons, meson, gradle, whatever) into
> dub, which will inevitably be unmaintained and out-of-date 'cos is too
> much work to keep up with everything, we delegate the job to individual
> packages.

I agree, letting build system packages tell the packmangr how to use them is definitely preferable to baked-in support.

> So for example, the 'make' package might contain a bunch of URLs for
> downloading a 'make' executable and any associated libraries and data
> files, and then export `/path/to/dub/cache/bin/make` as the command
> template for invoking the build system.  There could be OS-dependent
> variations, such as `C:\path\to\dub\cache\make\gmake.exe` if you're on
> Windows, for example.  These command flavors are defined within the
> 'make' dub package, and then any project that uses make can just declare
> a dependency on 'make', and dub will know how to do the Right Thing(tm).
> Dub itself wouldn't have to know the specifics of how to work with a
> make-based source tree; it will just execute the commands defined by the
> 'make' dub package.

Yes, in general, pre-built binary tool packages that originate from outside (or inside) D-land must be supported. This has been one of my bigger issues with DUB. And a little bit extra consideration for buildsystems in particular is likely warranted.

Only change I would make is that declaring a dependency on a buildsystem shouldn't *necessarily* mean that the package manager invokes the buildsystem directly. In many cases it would, and could maybe be a good default, but it also needs to support the case where a project uses its own buildscript and the buildscript invokes something like make or such (probably, in this situation, the buildscript would invoke the buildsystem - or any other executable tools it relies on - through the package manager...or at least through information provided by the package manager via envvars, for example).

Another thing to keep in mind is that the interactions between package manager and buildsystem will likely require more than just the buildsystem telling the package manager "Here are the command(s) you use to invoke me". For example, the package manager may need to relay things like "Which configuration is being built". Although, now that I think of it, since the package manager is banned by charter from being a buildsystem, it might not need to pass nearly as much information to buildscripts/systems/etc as DUB-as-a-pack-manager would need to.


> It would help if somebody can provide a concrete example of a complex
> dependency graph that requires anything more than a straightforward
> topological walk.

Personally, in the absence of someone like Sonke who's already experienced with real-world dependency resolution popping in to help out on that, I'd really like to just simply start by adapting and re-using some already-established dependency-resolving code, such as from DUB. This is one thing I'd really want to avoid reinventing the wheel on if at all possible.

--------------------------------

In response to suggestions from NaN and Russel, I'd like to move this to a Github project.  No code yet, just because I don't have any yet, but we can use the issues/PRs/wiki features to collaborate publicly in a more structured, less transient, way:

https://github.com/Abscissa/DPak

@H. S. Teoh: I forget your Github handle. Let me know and I'll add you as a collaborator.
April 17, 2019
On 4/16/19 8:12 AM, Atila Neves wrote:
> 
> I'll take a look. In the meanwhile I started this:
> 
> https://github.com/kaleidicassociates/bud
> 
> It uses dub as a library to make sure that it works just as dub does, but bypasses what it can. The idea is to completely separate these disparate tasks:

What is the current state of this tool?
April 17, 2019
On 4/17/19 1:53 PM, H. S. Teoh wrote:
> At the end of
> the day, it just boils down to (1) you have a bunch of input files, (2)
> there's some command that transforms these input files into output
> files, and (3) these output files reside somewhere in the filesystem.
> As long as you're told where the output files in (3) are, you know
> enough to use this package.  What goes on inside (2) is irrelevant; said
> command can be arbitrarily complex -- it can be an entire build system,
> for example, that consists of many subcommmands -- but a package manager
> doesn't need to (and shouldn't) care about such details.

+1, exactly correct
April 17, 2019
On Wednesday, 17 April 2019 at 19:15:40 UTC, Nick Sabalausky (Abscissa) wrote:
> On 4/17/19 1:53 PM, H. S. Teoh wrote:
>> At the end of
>> the day, it just boils down to (1) you have a bunch of input files, (2)
>> there's some command that transforms these input files into output
>> files, and (3) these output files reside somewhere in the filesystem.
>> As long as you're told where the output files in (3) are, you know
>> enough to use this package.  What goes on inside (2) is irrelevant; said
>> command can be arbitrarily complex -- it can be an entire build system,
>> for example, that consists of many subcommmands -- but a package manager
>> doesn't need to (and shouldn't) care about such details.
>
> +1, exactly correct

In the ideal world, perhaps. I am just worried opening dub to other build systems will just open a can of worms. Packages will use different build systems, which will be a pain to set up. Right now, the advice is "to get started with D, install official DMD distribution and set up your project with dub". Afterwards, the advice may be "to get started with D, install official DMD distribution, set up your project with dub, and install git,svn,meson,scons,make,cmake so that you can build some packages. Oh and set these environmental variables for these tools. Oh and if you're on Windows you're screwed".

While dub may seem limiting to some, it offers standarization and unification. I know these are forbidden words in D community, but standards and forcing people to do things certain way are often beneficial and boost the cohesion of the ecosystem.

"I find this to be D's biggest current stumbling block, even moreso than anything in the language or compiler" - really? The biggest D stumbling block is the fact that it's not easy to set up a mixed C/C++ project with D? How many projects of that kind are there even? How about issues such as crappy std.xml long overdue for replacement (I could list more but I don't want to derail this thread)?
April 17, 2019
On Wednesday, 17 April 2019 at 20:10:08 UTC, JN wrote:
> "I find this to be D's biggest current stumbling block, even
> moreso than anything in the language or compiler" - really? The
> biggest D stumbling block is the fact that it's not easy to set
> up a mixed C/C++ project with D? How many projects of that kind
> are there even? How about issues such as crappy std.xml long
> overdue for replacement (I could list more but I don't want to
> derail this thread)?

Is there no good C or C++ XML library that you could make a std.xml
alternative out of? Maybe there'd be a dub package that you could
just grab and use for now, that wrapped one, if this were an option
for dub.

Consider std.regex and https://github.com/jrfondren/topsender-bench
The fastest std.regex option is more than 10x slower than libc
regex, which is already too slow to seriously use for anything but
once-off tasks. (I put a thread in learn about this. None of the
other suggestions there made any significant change to
performance.) When the first thing someone'd try is ** 140x **
slower than the Python script they didn't even think about
optimizing, I can't say it benefits D that nobody can say "oh yeah
just dub add pcre and change your import"
April 17, 2019
On Wednesday, 17 April 2019 at 20:32:28 UTC, Julian wrote:
> When the first thing someone'd try is ** 140x **
> slower than the Python script they didn't even think about
> optimizing

37x, rather. It's still 2s ("I can hit enter and wait for the output")
vs. over a minute ("I'd better take a 10min break while this runs.")
April 17, 2019
On 4/17/19 4:10 PM, JN wrote:
> On Wednesday, 17 April 2019 at 19:15:40 UTC, Nick Sabalausky (Abscissa) wrote:
>> On 4/17/19 1:53 PM, H. S. Teoh wrote:
>>> At the end of
>>> the day, it just boils down to (1) you have a bunch of input files, (2)
>>> there's some command that transforms these input files into output
>>> files, and (3) these output files reside somewhere in the filesystem.
>>> As long as you're told where the output files in (3) are, you know
>>> enough to use this package.  What goes on inside (2) is irrelevant; said
>>> command can be arbitrarily complex -- it can be an entire build system,
>>> for example, that consists of many subcommmands -- but a package manager
>>> doesn't need to (and shouldn't) care about such details.
>>
>> +1, exactly correct
> 
> In the ideal world, perhaps.

No, there's no "ideal world" about this. What HSTeoh describes above are just plain the cold, hard facts of the matter. No ifs, ands or buts.

The complications that we're accustomed to seeing and expecting are ones that come straight from DUB's failure to fully separate package management from building.

> I am just worried opening dub to other build systems will just open a can of worms. Packages will use different build systems, which will be a pain to set up. Right now, the advice is "to get started with D, install official DMD distribution and set up your project with dub". Afterwards, the advice may be "to get started with D, install official DMD distribution, set up your project with dub, and install git,svn,meson,scons,make,cmake so that you can build some packages. Oh and set these environmental variables for these tools. Oh and if you're on Windows you're screwed".

With what I have in mind (and HS Teoh appears to have the same thing in mind), then no, those scenarios are absolutely, definitely not a realistic result. They're just fear-based speculation at best.

In the long run, DUB may or may not remain the recommended buildsystem for newcomers, but that's all. And it least it won't actively PREVENT alternative buildsystems from coming along and proving superior enough to dethrone DUB.

> While dub may seem limiting to some, it offers standarization and unification.

Well that's just the problem - It *doesn't* provide that, it completely fails to. It tries to standardize and unify through basically the same strong-arm "submit or die" approach as Nobunaga or Napoleon, but as I predicted it would from the start, it just would up fracturing the ecosystem instead[1]. And a fractured ecosystem is by definition NEITHER standardized NOR unified.

[1] The no-so-well-known fact is, many projects are forced out of the DUB package ecosystem because of DUB buildsystem's limitations. Though that may not be easy to see, since such packages are forced under-the-radar because of their opting out of DUB as a buildsystem also forces them out of the visibility provided by code.dlang.org. And then the ones that do join the DUB side are too often forced into hardships that first of all, shouldn't even be necessary, and second of all, still fail to offer the tradeoff of unifying/standardizing the ecosystem anyway.

The *only* way to standardize and unify is to do so in a way that allows individual packages to work the way they need to.
April 17, 2019
On Wed, Apr 17, 2019 at 08:10:08PM +0000, JN via Digitalmars-d wrote: [...]
> In the ideal world, perhaps. I am just worried opening dub to other build systems will just open a can of worms. Packages will use different build systems, which will be a pain to set up. Right now, the advice is "to get started with D, install official DMD distribution and set up your project with dub". Afterwards, the advice may be "to get started with D, install official DMD distribution, set up your project with dub, and install git,svn,meson,scons,make,cmake so that you can build some packages. Oh and set these environmental variables for these tools. Oh and if you're on Windows you're screwed".

Again, this concern is addressed by my proposal of build system wrapper dub packages.  Basically, the act of declaring a dependency on, say, the 'make' wrapper package, should cause dub to download and install a working installation of make.  This can be done by structuring the 'make' wrapper package such that its "build" command is to download and install a working version of make.

In a nutshell, it would work like this:
- Project P depends on project Q, so dub downloads project Q.
- Project Q depends on 'make', so dub downloads the 'make' wrapper
  package.
- The 'make' wrapper package's "build" command is actually an installer
  that downloads and installs make on the local system. For maximum
  non-intrusion, the installation location can be somewhere in a
  standard location in the dub cache, so it won't conflict with any
  other make installation you may have on your system. So dub invokes
  this "build" command, and now you have a working version of make.
  (Note that this includes setting up any environment variables that may
  be necessary.)
- Now that project Q's dependencies are satisfied, it invokes Q's build
  command, which runs make to build everything. This now works, because
  the previous step ensures we have a working version of make installed.
- Now that project P's dependencies are satisfied, we can build P as
  usual.

This is one example of why abstracting away the "build" command of a package from the package manager can be highly beneficial: it allows you to pull in any arbitrary external package without needing dub to be aware of how said external package works.

In the above scheme, you can substitute 'make' with any build system of your choice, and it will still work with no additional effort from the end user.  This is ensured by the wrapper package abstracting away the dirty details of how to install a working version of ${buildtool} so that dub can perform the installation without needing to know the details of how to do it.

The wrapper package also hides away the dirty details of how to setup a build tool like make properly, including setting up environment variables and what-not.  Basically, it does whatever it takes for the whole process to be completely transparent to the user.


> While dub may seem limiting to some, it offers standarization and unification. I know these are forbidden words in D community, but standards and forcing people to do things certain way are often beneficial and boost the cohesion of the ecosystem.

Standards should be empowering, rather than arbitrarily limiting.  What we're proposing here is a standard way for arbitrary software packages to be used as dub dependencies.  Let me repeat that again: what we're proposing here is a *standard* way for *arbitrary* software packages (of any language, any build system, etc.) to be usable as dub dependencies.

The standard here is the standard "API" of how packages work:

1) A package can depend on one or more packages, with optional version constraints. This is the way dub already works.

2) A package can consist of any arbitrary collection of input files. What they are and how they are structured is irrelevant.

3) A package has a standard declaration of how it should be built. Basically, this is any arbitrary command that transforms said input files into output files. How this is carried out is irrelevant -- that's a detail the package is responsible for, and is none of the package manager's business.

4) A package has a standard declaration of where to find the built
products (import paths, library files, etc.).

5) A package that wraps around a build system has a standard declaration of how to formulate commands to invoke it for various standard tasks (build debug, build release, test, etc.). So dependent packages don't actually have to know how to run make on Windows vs. Linux, it uses a standard query to formulate the right command, and simply runs it. The wrapper package ensures that this command will Just Work(tm).

Just like the range API, the user of the range (the package manager)
doesn't, and shouldn't, care about how each of these "methods" work
under the hood.  As long as they conform to the API (declare items (1)
to (4) in a standard way), that's all that's necessary for dub to do the
Right Thing(tm).


> "I find this to be D's biggest current stumbling block, even moreso than anything in the language or compiler" - really? The biggest D stumbling block is the fact that it's not easy to set up a mixed C/C++ project with D?  How many projects of that kind are there even?

Given the amount of effort currently being put into C++ interop?  I'm guessing it's a lot more common than you think.


> How about issues such as crappy std.xml long overdue for replacement (I could list more but I don't want to derail this thread)?

Jonathan's dxml is already a dub package, isn't it?  You could just use that.


T

-- 
If it tastes good, it's probably bad for you.