On Monday, 4 March 2019 at 14:04:41 UTC, Seb wrote:
> You fix a bug in let's say Vibe.d (or any other library for that matter). You don't want to wait 6 months until your bug fix gets merged and released (if ever). I have run into this many many times.

Perfectly reasonable use-case, the question is whether the right solution is to tie DUB to direct support for git submodules.

I'll slightly rephrase my earlier remarks: this and other cases I can think of all seem to come down to:

    How does one make DUB aware of and able to handle a codebase that
    is not registered in the main DUB package repo and possibly does
    not even define its own DUB package config?

What I would suggest is that rather than building in direct support for git, which is only one VCS, we consider instead how to make it possible to specify dependencies in terms of two kinds of instructions:

  * how to fetch a dependency's code and place it in a directory determined
    either by DUB itself or by another setting in the `dub.json` file

    - this might be done by a git clone or checkout, or by fetching and
      unzipping an archive file, or by a number of other options

  * how to define DUB packaging/dependency settings if the dependency does
    not define them in its own codebase

That should be a more flexible approach that would allow the same outcomes.

On 2019-03-04 06:32, Nick Sabalausky (Abscissa) wrote:

> Notice that not only is `dub.selections.json` included in version control, but I also have another file: `dub.selections.vibe-0.8.3.json`. That's because some compiler versions require vibe v0.7.x and NOT v0.8.x, but newer compilers require v0.8.x and up. So `dub.selections.json` is carefully set (by hand) to exactly my minimum dependency versions. And `dub.selections.vibe-0.8.3.json` is carefully set to the minimum versions required by the newer compilers.

Usually `dub.selections.json` should not be committed for libraries, only for applications.

With the latest version of Dub it's now possible to specify which compiler is required [1]. That could potentially have solved your problem. I'm not sure if this means that Dub will try to pick an older version of the dependency if the compiler requirement doesn't match.

[1] https://dlang.org/changelog/2.085.0.html#toolchain_requirements

-- 
/Jacob Carlborg

On 3/4/19 12:19 PM, Jacob Carlborg wrote:
> On 2019-03-04 06:32, Nick Sabalausky (Abscissa) wrote:
> 
>> Notice that not only is `dub.selections.json` included in version control, but I also have another file: `dub.selections.vibe-0.8.3.json`. That's because some compiler versions require vibe v0.7.x and NOT v0.8.x, but newer compilers require v0.8.x and up. So `dub.selections.json` is carefully set (by hand) to exactly my minimum dependency versions. And `dub.selections.vibe-0.8.3.json` is carefully set to the minimum versions required by the newer compilers.
> 
> Usually `dub.selections.json` should not be committed for libraries, only for applications.

Well, its absolutely mandatory if you want your CI builds to be reproducible. (As they SHOULD be, unless you enjoy completely unnecessary potential problems.)

Although, come to think of it, I guess I could just rename it to `dub.selections.ci-default.json` and have my travis setup script automatically rename that to `dub.selections.json` if I haven't chosen any other alternative dub.selections for the current CI job.

But I'm not sure I see the harm in including `dub.selections.json`, though. If you're using the given project as a lib in your own project, dub doesn't actually do anything with the `dub.selections.json`s from the dependencies...or does it? 'Cause *that* would be bad...(Though I can't imagine *why* it would intentionally work that way.)

> With the latest version of Dub it's now possible to specify which compiler is required [1]. That could potentially have solved your problem.
> [1] https://dlang.org/changelog/2.085.0.html#toolchain_requirements

Yea, just saw that myself. I'm definitely thrilled to see it added (for other reasons), I wish it had been added years ago, but I don't see how it would help the cases I mentioned above.

> I'm not sure if this means that Dub will try to pick an older version of the dependency if the compiler requirement doesn't match.

Hmm, maybe, I hadn't thought of that. I certainly hope so. But in any case, the vide-d versions supported by mysql-native lack the "toolchainRequirements" directive (as they predate it), so it doesn't help mysql-native's existing cases here, at least not anytime in the near future. It also doesn't help with needing to test on the minimum
supported dependency versions.

On Monday, 4 March 2019 at 13:12:09 UTC, Joseph Rushton Wakeling wrote:
> On Monday, 4 March 2019 at 10:19:29 UTC, Atila Neves wrote:
>> This doesn't work.
>
> Exactly which bits of it?

Try starting a new dub project that depends on autowrap v0.2.0.

> I have actual projects that use git submodules to provide some dependencies for dub.  There are no nested submodules, though.

This might be the crucial difference - it might be a submodules thing.

On Tuesday, 5 March 2019 at 10:44:04 UTC, Atila Neves wrote:
> Try starting a new dub project that depends on autowrap v0.2.0.

Indeed if you just add a regular dub dependency this won't work (because dub doesn't know how to check out the pyd submodule).  But try instead:

* in your main project add a submodule for autowrap (checking out v0.2.0)

* in your dub config specify the dependency as

      "autowrap": { "path": "./path/to/autowrap_submodule" }

(or whatever the SDL equivalent is).

That works.

On Wednesday, 6 March 2019 at 12:23:07 UTC, Joseph Rushton Wakeling wrote:
> On Tuesday, 5 March 2019 at 10:44:04 UTC, Atila Neves wrote:
>> Try starting a new dub project that depends on autowrap v0.2.0.
>
> Indeed if you just add a regular dub dependency this won't work (because dub doesn't know how to check out the pyd submodule).  But try instead:
>
> * in your main project add a submodule for autowrap (checking out v0.2.0)
>
> * in your dub config specify the dependency as
>
>       "autowrap": { "path": "./path/to/autowrap_submodule" }
>
> (or whatever the SDL equivalent is).
>
> That works.

Nope it only works locally. It does not work for anyone using it via dub. As mentioned earlier, try fetching this package (or any package with submodules) from the dub registry.

On Wednesday, 6 March 2019 at 12:29:17 UTC, Seb wrote:
> Nope it only works locally. It does not work for anyone using it via dub. As mentioned earlier, try fetching this package (or any package with submodules) from the dub registry.

Yes.  But that's because you're doing something fundamentally inappropriate for the DUB package repo, which is to have source code dependencies defined somewhere (the git submodule system) that the DUB repo inherently cannot know about.

Basically if you want to have git submodules in the mix then you have to organize the entire dependency hierarchy for everything involved via git submodules rather than via the DUB package repo.  The git submodule issue isn't a blocker towards DUB being able to handle these dependencies, it's a blocker on the DUB package repo containing packages whose source code dependencies are not known to the DUB package repo.  Those are not quite the same things.

In other words, if you choose to bypass DUB's supply mechanisms and hand control to something else (like git), then you need to respect that separation of concerns: handle the supply via the tool you asked to provide supply, and just give DUB sufficient config to build the code once it has been supplied.

And it may be important to have that separation of concerns in your face -- after all, if it's trivial to just set a DUB package dependency to point at an arbitrary git repo, what's to stop me setting up a convenience library that depends on a git fork of vibe, which includes dodgy tracking components?

I trust that _your_ fork of vibe is well-motivated and only has bugfixes, but I'm not sure that I would trust that approach in general, and I am not so sure it's a good idea to open it up as a possibility in the DUB package repo.

After all, if you can't rely on upstream vibe to get out timely bugfix releases, maybe the problem is one of process rather than of DUB itself.  I'm not sure it isn't better to fix that process rather than open up a Pandora's box of dependency options to work around it.

On Wednesday, 6 March 2019 at 12:23:07 UTC, Joseph Rushton Wakeling wrote:
> On Tuesday, 5 March 2019 at 10:44:04 UTC, Atila Neves wrote:
>> Try starting a new dub project that depends on autowrap v0.2.0.
>
> Indeed if you just add a regular dub dependency this won't work (because dub doesn't know how to check out the pyd submodule).  But try instead:
>
> * in your main project add a submodule for autowrap (checking out v0.2.0)
>
> * in your dub config specify the dependency as
>
>       "autowrap": { "path": "./path/to/autowrap_submodule" }
>
> (or whatever the SDL equivalent is).
>
> That works.

I don't want to do that. I want to use dub as intended, and have it figure out versions for me. I also want to make my code available for others, have them list my package in their dub.sdl and for Bob to subsequently be their proverbial uncle.

It's bad enough I can't use dub as a build system, I'm certainly not going to manually do its job as a package manager!  Especially if one of the submodules depends on a dub package that depends on a dub package that depends on a dub package...

On Wednesday, 6 March 2019 at 14:27:56 UTC, Atila Neves wrote:
> I don't want to do that. I want to use dub as intended

I would suggest that DUB stopped being used as intended the moment someone submitted a package to the DUB repo with source code dependencies that were not also packages in the DUB repo.

On Wednesday, 6 March 2019 at 23:38:20 UTC, Joseph Rushton Wakeling wrote:
> On Wednesday, 6 March 2019 at 14:27:56 UTC, Atila Neves wrote:
>> I don't want to do that. I want to use dub as intended
>
> I would suggest that DUB stopped being used as intended the moment someone submitted a package to the DUB repo with source code dependencies that were not also packages in the DUB repo.

A fair point.

At the time I was trying to pin the pyd version because there was a bug in the dub registry that prevented me from doing so in the regular fashion. Then the bug got fixed and I moved on.