We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.

We're currently using git submodules for local dependencies. If they were migrated to dub, we'd run into the problem of what to do while working on a dependency. Often changes to a dependency only make sense in the context of the project that needs the dependency, so we have to be able to set dub to use the git branch that we're working on. However, while this can be made to work locally by using dub add-override to point dub at a folder that has the branch checked out, this cannot work on our Jenkins system because obviously that doesn't have the local folder.

It seems to me that a possible solution would be to be able to set a git repository as a dub package supplier. Dub would then check out the repository to .dub/packages/<project>/<git hash>, and build from there. dub.selections.json could store the commit hash for reproducible builds.

Since this would require a hefty amount of work on dub to support git remote repos, I wanted to ask upfront: would such a PR plausibly be accepted? And in case not, do you have alternate proposals for D dependency management that won't force us to tag a repo before using it in another project?

Thanks for your consideration!

On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
> We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.
>
> We're currently using git submodules for local dependencies. If they were migrated to dub, we'd run into the problem of what to do while working on a dependency. Often changes to a dependency only make sense in the context of the project that needs the dependency, so we have to be able to set dub to use the git branch that we're working on. However, while this can be made to work locally by using dub add-override to point dub at a folder that has the branch checked out, this cannot work on our Jenkins system because obviously that doesn't have the local folder.

A dependency can point to a local directory, including a cloned git submodule. Have a look how I did that in the DLP project.

https://github.com/jacob-carlborg/dlp/blob/9982cd75ea347170b4f13fc259789df264bcd7b3/dub.sdl#L9

—
/Jacob Carlborg

On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
> We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.
>
> We're currently using git submodules for local dependencies. If they were migrated to dub, we'd run into the problem of what to do while working on a dependency. Often changes to a dependency only make sense in the context of the project that needs the dependency, so we have to be able to set dub to use the git branch that we're working on. However, while this can be made to work locally by using dub add-override to point dub at a folder that has the branch checked out, this cannot work on our Jenkins system because obviously that doesn't have the local folder.
>
> It seems to me that a possible solution would be to be able to set a git repository as a dub package supplier. Dub would then check out the repository to .dub/packages/<project>/<git hash>, and build from there. dub.selections.json could store the commit hash for reproducible builds.
>
> Since this would require a hefty amount of work on dub to support git remote repos, I wanted to ask upfront: would such a PR plausibly be accepted? And in case not, do you have alternate proposals for D dependency management that won't force us to tag a repo before using it in another project?
>
> Thanks for your consideration!

1.

You can use path-based dependencies, you won't have to listed dependencies transitively.
You _will_ have to:
- checkout everything, you can have several package in the same repo for example
- "your builds won't be reproducible" aka "you can't regress", because I don't think you can express versions with path-based
- dub.selections.json won't be effective for such path-based stuff


2.

I think you'd better host your own DUB registry

On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
> We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.
>
> [...]

You know, dub can also retrieve dependencies from Maven? Also the file system repository provider was fixed, enabling you to use any directory/ftp/s3 folder as repository.

Regarding you question, yes, such enhancement is heavily welcome. There is also an open pr you can build on.

Kind regards
Andre

On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
> We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.
>
> [...]

Yes, of course it would be. It's a very commonly requested feature.
There even was an old PR for this: https://github.com/dlang/dub/pull/1403

On Wednesday, 31 July 2019 at 14:44:35 UTC, Seb wrote:
> On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
>> We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.
>>
>> [...]
>
> Yes, of course it would be. It's a very commonly requested feature.
> There even was an old PR for this: https://github.com/dlang/dub/pull/1403

Sweet, thank you! I'm gonna start poking away at this somewhere in the next month, probably.

Re everyone else, again, local paths will not work because they'd fail to resolve on our build server. We could, I guess, push branch builds to our dub repo, but that's kind of awkward and would depend on our build architecture in an uncomfortable way. Going directly to the source seems to me the way to go.

On Wednesday, 31 July 2019 at 15:34:56 UTC, FeepingCreature wrote:
> On Wednesday, 31 July 2019 at 14:44:35 UTC, Seb wrote:
>> On Wednesday, 31 July 2019 at 13:02:10 UTC, FeepingCreature wrote:
>>> We (Funkwerk) are currently using rake as a build system. Since rake does not offer package management, we have to list all transitive dependencies of every package, which is obviously not satisfactory. Because of this, we're looking into switching to dub and dub.selection.json as a "source of truth" for the used set of library dependencies.
>>>
>>> [...]
>>
>> Yes, of course it would be. It's a very commonly requested feature.
>> There even was an old PR for this: https://github.com/dlang/dub/pull/1403
>
> Sweet, thank you! I'm gonna start poking away at this somewhere in the next month, probably.
>
> Re everyone else, again, local paths will not work because they'd fail to resolve on our build server. We could, I guess, push branch builds to our dub repo, but that's kind of awkward and would depend on our build architecture in an uncomfortable way. Going directly to the source seems to me the way to go.

As you have written Jenkins:

Please also see the Dub enabled Jenkins pipeline library: https://forum.dlang.org/thread/ljuosdnorythkosdrtyp@forum.dlang.org

Also the generic issue scanner plugin of Jenkins is now able to read D-Scanner issue json file.

The generic code coverage plugin of Jenkins will be in the very next future be  able to read LST files ;)

Kind regards
Andre