On Mon, Apr 15, 2019 at 07:52:07PM -0400, Nick Sabalausky (Abscissa) via Digitalmars-d wrote:
[...]
> [...]  Basically, a package manager's config should need exactly three things from your project:
> 
> 1. What dependencies.
> 2. The command(s) to (build, test, etc.)
> 3. Name(s)/Location(s) of the build's output.
> 
> Then, with that information, a package manager provides services such as (but not necessarily limited to):
> 
> 1. A simple, standardized way for you and your users to obtain/build the dependencies.
> 
> 2. A simple, standardized way for buildscripts/buildsystems to obtain the information needed to include the dependencies in their own build (such as -I... include directories, paths to the now-already-built lib/exec binaries, etc.)

I'd add:

3. A standard query interface for querying a remote repository for packages (matching some names / patterns) and version numbers.


> From this, each project can naturally either just roll its own buildscripts, or depend on another package providing a builsystem.

That's a good idea.  Completely decouple package management from building.  Let the package manager do what it does best: managing packages, and leave the compilation to another tool more suited for the job.  Of course, the default build command can be set to `dub build` to keep existing users happy.

I was thinking the build/test/etc. command can itself be defined by a dependency.  For example, if your project uses make for builds, there could be a `make` dub package that defines command-line templates used for generating platform-specific build commands. Your package would simply specify:

	"build-system": "make"

and the "make" package would define the exact command(s) for invoking make (e.g., if gmake is picked up as a resolution, it could define the executable name as `gmake`, or it could define different CLI syntax for Windows vs. Linux, etc.).  The package manager would then recognize "make" as a dependency, and would download and install that package (which presumably would install an appropriate version of 'make') if it hasn't already been installed, then invoke it to build the project.

The "make" package itself would be presumably some kind of wrapper around a system utility, or an installer of some sort that installs an appropriate version of make on the system.  It could be implemented as a normal dub package whose "build" command is something that runs an attached installer.


> Some of the *details* can be quite nontrivial...like dependency resolution algorithms, or designing the interactions between package manager and buildsystem to be simple, yet effective enough to suit all parties needs.  But ultimately, it boils down conceptually to be very simple.
[...]

If done correctly, dependency resolution is just a topological walk on a DAG.  Get this right, and everything else falls into place.

To generate the DAG, there would need to be some kind of list of repositories, which defaults to code.dlang.org (but which can be customized if necessary -- e.g., add local filesystem repos for local packages, or a URL to a private additional repository).  There would also need to be some kind of version resolution scheme.  It shouldn't be too hard to do: all you need is for repositories to support three queries:

1) Fetch a list of packages matching the given name(s);

2) Filter said list by a given OS/platform identifier.

3) Filter said list by zero or more given version specs (either an exact match, or a <= or >= match, or whatever else you may want to filter version strings by.).

Repositories ought to support such queries efficiently, i.e., the package manager shouldn't have to download the entire list of 1000 packages each with 50 different version identifiers, and then perform an O(n) search on the list to find the necessary matching package.  It should behave like a database: you send it an OS identifier, a list of desired package names, each with one or more version filters, and it returns a list of matching package descriptions, including URLs from which you may download the package(s).  It should take only 1 network roundtrip to get this information.

Then once the package manager receives the URLs, it can use whatever method necessary to download said URLs (or just cache it somewhere if it's a pathname on the local filesystem), and read the package description(s) to find any additional dependencies that it may need.


T

-- 
A program should be written to model the concepts of the task it performs rather than the physical world or a process because this maximizes the potential for it to be applied to tasks that are conceptually similar and, more important, to tasks that have not yet been conceived. -- Michael B. Allen

On Tuesday, 16 April 2019 at 17:08:22 UTC, Dragos Carp wrote:
> On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
>> [...]
>
> I have a different proposal.
>
> One of the goals of D is the interoperability with C/C++ and interoperability in general. Because of that I think it really make sens to not reinvent the wheel, but to have good support for an already existing tool.
> Since a couple of months I'm using Bazel, and I like it. Not necessary the implementation (is big and Java), but the concepts behind it. So I'm already working right now on improving the D support [1]: adding more tests, supporting more compiler versions, etc.
>
> [...]

Super nice! I was just going to ask if anyone's been using bazel and what they think of it. And here you are already hacking away d rules :D

On Tuesday, 16 April 2019 at 17:08:22 UTC, Dragos Carp wrote:
> On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
>> Hi all,
>> ...
> So my plan is:
>  - add tests: supporting d libraries, exectuables, tests and C++/D hybrid applications
>  - add compiler version selection
>  - ldc and gcc support
>  - dub .json, .sdl -> bazel BUILD file converter (not foolproof). If necessary, manual intervention is acceptable.
>  - add windows support
>  - write a D starlark [2] implementation
>  - rewrite bazel in D. Probably this will never happen, but maybe the subset sufficent to cover the dub functionality will be doable.
>

Very nice, I  can add support for mir packages. Do you have an opensource working D+Bazel project and Travis config for it to use it as an example? --Ilya

On Wednesday, 17 April 2019 at 02:57:44 UTC, 9il wrote:
>
> Very nice, I  can add support for mir packages. Do you have an opensource working D+Bazel project and Travis config for it to use it as an example? --Ilya

Not yet. I'll get back to you when this is set. Travis config is part of first item on the plan. BTW the repo is https://github.com/dcarp/rules_d.

On 16.04.2019 20:39, H. S. Teoh wrote:
> If done correctly, dependency resolution is just a topological walk on a
> DAG.  Get this right, and everything else falls into place.
> 
How do you resolve conflicts using DAG only?

On 4/16/19 1:39 PM, H. S. Teoh wrote:
> On Mon, Apr 15, 2019 at 07:52:07PM -0400, Nick Sabalausky (Abscissa) >>
>> Then, with that information, a package manager provides services such
>> as (but not necessarily limited to):
>>
>> 1. A simple, standardized way for you and your users to obtain/build
>> the dependencies.
>>
>> 2. A simple, standardized way for buildscripts/buildsystems to obtain
>> the information needed to include the dependencies in their own build
>> (such as -I... include directories, paths to the now-already-built
>> lib/exec binaries, etc.)
> 
> I'd add:
> 
> 3. A standard query interface for querying a remote repository for
> packages (matching some names / patterns) and version numbers.
> 

Oh, don't get me wrong, there's a whole BUNCH of things I'd like to see on this particular list, and that definitely includes your suggestion. I left it at "not necessarily limited to..." for good reason ;)


>>  From this, each project can naturally either just roll its own
>> buildscripts, or depend on another package providing a builsystem.
> 
> That's a good idea.  Completely decouple package management from
> building.  Let the package manager do what it does best: managing
> packages, and leave the compilation to another tool more suited for the
> job.

Exactly.

> I was thinking the build/test/etc. command can itself be defined by a
> dependency.  For example, if your project uses make for builds, there
> could be a `make` dub package that defines command-line templates used
> for generating platform-specific build commands. Your package would
> simply specify:
> 
> 	"build-system": "make"
> 
> and the "make" package would define the exact command(s) for invoking
> make [...etc...]

Interesting idea. I definitely like it, though I would consider it not a "basic fundamental" mechanism, but rather a really nice bonus feature that provides a convenient shortcut for having to specifying certain things manually...IF you happen to be using a buildsystem known by the package manager...or (better yet, if possible) a buildsystem that's able to tell the package manager enough about itself to make this happen.


>> Some of the *details* can be quite nontrivial...like dependency
>> resolution algorithms, or designing the interactions between package
>> manager and buildsystem to be simple, yet effective enough to suit all
>> parties needs.  But ultimately, it boils down conceptually to be very
>> simple.
> [...]
> 
> If done correctly, dependency resolution is just a topological walk on a
> DAG.  Get this right, and everything else falls into place.

Yes, but therin lies the rub - "getting this right". AFAICT, Sonke seems to have a very good grasp on dependency resolution via DAG (or at least, certainly a far better grasp than I do). But yet, even his algorithms in DUB, he needed to occasionally patch with common-use-case short-circuiting logic (and I don't even know what else) just to fix various seemingly-infinite-loop issues (not technically infinite, but appeared to be to the user) that people were still reporting some years into DUBs lifetime.

'Course, if it really is simpler than it seems to me (and *especially* is it *isn't*), then I'd certainly be more than happy to delegate dependency resolution out to somebody else.

On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
> 
[…]
> 
> Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
> see
> on this particular list, and that definitely includes your
> suggestion. I
> left it at "not necessarily limited to..." for good reason ;)
[…]

The problem here is that people are having lots of ideas on this email list, but no-one is choosing to collate them into a an actual "call to action". As ever in an ever extending email thread the energy to do something within the D community dissipates and nothing ends up happening.

Perhaps we should make the hack day at DConf 2019 "get something actually moving on this" day.

Or perhaps people will continue to say stuff on very long email threads
and end up doing nothing. This is the third such thread on Dub I can
remember, the previous two ended up with no  outcomes other than lots
of opinions in very long email threads.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:
> On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
> Digitalmars-d wrote:
>> 
> […]
>> 
>> Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
>> see
>> on this particular list, and that definitely includes your
>> suggestion. I
>> left it at "not necessarily limited to..." for good reason ;)
> […]
>
> The problem here is that people are having lots of ideas on this email list, but no-one is choosing to collate them into a an actual "call to action". As ever in an ever extending email thread the energy to do something within the D community dissipates and nothing ends up happening.
>
> Perhaps we should make the hack day at DConf 2019 "get something actually moving on this" day.
>
> Or perhaps people will continue to say stuff on very long email threads
> and end up doing nothing. This is the third such thread on Dub I can
> remember, the previous two ended up with no  outcomes other than lots
> of opinions in very long email threads.

The people who are actually going to do the work should setup a private group somewhere. These sprawling threads on the public newsgroup dilute not just the vision but also the motivation.

On 17/04/2019 8:27 PM, NaN wrote:
> On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:
>> On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
>> Digitalmars-d wrote:
>>>
>> […]
>>>
>>> Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
>>> see
>>> on this particular list, and that definitely includes your
>>> suggestion. I
>>> left it at "not necessarily limited to..." for good reason ;)
>> […]
>>
>> The problem here is that people are having lots of ideas on this email list, but no-one is choosing to collate them into a an actual "call to action". As ever in an ever extending email thread the energy to do something within the D community dissipates and nothing ends up happening.
>>
>> Perhaps we should make the hack day at DConf 2019 "get something actually moving on this" day.
>>
>> Or perhaps people will continue to say stuff on very long email threads
>> and end up doing nothing. This is the third such thread on Dub I can
>> remember, the previous two ended up with no  outcomes other than lots
>> of opinions in very long email threads.
> 
> The people who are actually going to do the work should setup a private group somewhere. These sprawling threads on the public newsgroup dilute not just the vision but also the motivation.

Over on Discord we have the Graphics work group which is working out quite well. Still very early on work wise so we've kept quite quiet about it.

We have done our best to set aside our opinions and do research to find out the requirements before writing code with a majority in agreement before decisions / pulling of code. It would be a good model to replicate if a few people came together and decided to start work on a new (although compatible) build system.

On Tuesday, 16 April 2019 at 15:52:57 UTC, Guillaume Piolat wrote:
> On Tuesday, 16 April 2019 at 12:12:45 UTC, Atila Neves wrote:
>>
>> I'll take a look. In the meanwhile I started this:
>>
>> https://github.com/kaleidicassociates/bud
>>
>> It uses dub as a library to make sure that it works just as dub does, but bypasses what it can. The idea is to completely separate these disparate tasks:
>>
>> * Dependency version resolution (from dub.{sdl,json} -> dub.selections.json)
>> * Fetching dependencies that are not already in the file system (trivial after dub.selections.json has been generated)
>> * Extracting information from dub about source files, import paths, etc.
>> * Actually using the information from dub to build
>>
>
> Won't complain about such good news!
>
> dub.json and dub.selections.json must unify though, as dub.selections.json is not self-sufficient, what if they contradict themselves?

I'm not sure I know what you mean. Unless you edit dub.selections.json manually, it can't contradict the build recipe it came from. Version 1 of dub.selections.json isn't sufficient, but I'm leaving that as a problem to solve later.

> I think our research concluded that solving dependencies for a particular platform+configuration is (in an idealized DUB) a fundamentally different operation than generating a proper dub.selections.json (for which you would have to find a super-set for "all conf").

Could you elaborate on this please? At least for configurations, I don't see why it wouldn't be possible to generate the current JSON object in dub.selections.json, but for each configuration instead. And, possibly platform (linux, x86_64, etc.).

> The problem is that configurations/platform could appear and disappear alongside previous versions.

Could you please give an example of this?

> (For readers: this would lead the path to platform-based dependencies and configuration-based dependencies which are currently not doable, such as:
>
>     "dependencies-linux": { "x11": "~>1.0" }
> )

Yep!