Hi all,

I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.

I'm wondering if there's a way to isolate the part that downloads and caches all of the dependencies in dub.json without running the build.

Right now I have to run the build twice. First one, for the sake of downloading the dependencies and later for running if the cache exists. Having the build run the first time takes up a lot of time. I'm going for the "npm install" type of effect.

The reason the cache needs to be rebuilt each time is because CircleCI runs docker images. Once the deployment is finished, the image fs is destroyed so there's no way to hold onto the cache for future builds.

On 11/04/2018 1:50 AM, Clinton wrote:
> Hi all,
> 
> I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.
> 
> I'm wondering if there's a way to isolate the part that downloads and caches all of the dependencies in dub.json without running the build.
> 
> Right now I have to run the build twice. First one, for the sake of downloading the dependencies and later for running if the cache exists. Having the build run the first time takes up a lot of time. I'm going for the "npm install" type of effect.
> 
> The reason the cache needs to be rebuilt each time is because CircleCI runs docker images. Once the deployment is finished, the image fs is destroyed so there's no way to hold onto the cache for future builds.

You should be able to do this by using ``dub describe`` and ``dub fetch`` with a simple script.

But a reasonable feature request to make it default for fetch command.

On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote:
> Hi all,
>
> I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.
>
> [...]

Thanks. That's a good idea.

It would be great to have "dub fetch --all" or something like that to fetch all packages from the project dub.json.

On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote:
> Hi all,
>
> I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.
>
> [...]

If I remember correctly I used dub upgrade exactly for this purpose.

Kind regards
Andre

Am 10.04.2018 um 15:50 schrieb Clinton:
> Hi all,
> 
> I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.
> 
> I'm wondering if there's a way to isolate the part that downloads and caches all of the dependencies in dub.json without running the build.
> 
> Right now I have to run the build twice. First one, for the sake of downloading the dependencies and later for running if the cache exists. Having the build run the first time takes up a lot of time. I'm going for the "npm install" type of effect.
> 
> The reason the cache needs to be rebuilt each time is because CircleCI runs docker images. Once the deployment is finished, the image fs is destroyed so there's no way to hold onto the cache for future builds.

"dub upgrade --missing-only" should have the desired effect. It downloads everything and also resolved any possible missing version selections.

On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote:
> Hi all,
>
> I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.
>
> I'm wondering if there's a way to isolate the part that downloads and caches all of the dependencies in dub.json without running the build.
>
> Right now I have to run the build twice. First one, for the sake of downloading the dependencies and later for running if the cache exists. Having the build run the first time takes up a lot of time. I'm going for the "npm install" type of effect.
>
> The reason the cache needs to be rebuilt each time is because CircleCI runs docker images. Once the deployment is finished, the image fs is destroyed so there's no way to hold onto the cache for future builds.

As far as I understand it, `dub describe` fetches everything. Then you can cache `~/.dub/packages/`.

Alternatively you can do `dub describe --cache=local` to put the packages in the current directory. You could then use `dub add-path .` or add `--cache=local` to all future calls to use those locally fetched packages.

Even better:
% mkdir cache
% cd cache
% dub describe --root=../ --cache=local

and then either
% dub build --root=../ --cache=local
or
% dub add-path .
% cd ../
% dub build

which keeps things nice and clean

On Tuesday, 10 April 2018 at 15:31:41 UTC, John Colvin wrote:
> On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote:
>> Hi all,
>>
>> I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up.
>>
>> I'm wondering if there's a way to isolate the part that downloads and caches all of the dependencies in dub.json without running the build.
>>
>> Right now I have to run the build twice. First one, for the sake of downloading the dependencies and later for running if the cache exists. Having the build run the first time takes up a lot of time. I'm going for the "npm install" type of effect.
>>
>> The reason the cache needs to be rebuilt each time is because CircleCI runs docker images. Once the deployment is finished, the image fs is destroyed so there's no way to hold onto the cache for future builds.
>
> As far as I understand it, `dub describe` fetches everything. Then you can cache `~/.dub/packages/`.
>
> Alternatively you can do `dub describe --cache=local` to put the packages in the current directory. You could then use `dub add-path .` or add `--cache=local` to all future calls to use those locally fetched packages.
>
> Even better:
> % mkdir cache
> % cd cache
> % dub describe --root=../ --cache=local
>
> and then either
> % dub build --root=../ --cache=local
> or
> % dub add-path .
> % cd ../
> % dub build
>
> which keeps things nice and clean

As per Sönke's advice, you can replace `dub describe` with `dub upgrade --missing-only` to avoid generating the description.

On Tuesday, 10 April 2018 at 15:31:41 UTC, John Colvin wrote:
> On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote:
>> [...]
>
> As far as I understand it, `dub describe` fetches everything. Then you can cache `~/.dub/packages/`.
>
> Alternatively you can do `dub describe --cache=local` to put the packages in the current directory. You could then use `dub add-path .` or add `--cache=local` to all future calls to use those locally fetched packages.
>
> Even better:
> % mkdir cache
> % cd cache
> % dub describe --root=../ --cache=local
>
> and then either
> % dub build --root=../ --cache=local
> or
> % dub add-path .
> % cd ../
> % dub build
>
> which keeps things nice and clean

Wow, this is helpful! I thought describe only showed which packages were currently cached or just output based on the dub.json. I'm going to try this out.

On Tuesday, 10 April 2018 at 15:31:41 UTC, John Colvin wrote:
> On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote:
>> [...]
>
> As far as I understand it, `dub describe` fetches everything. Then you can cache `~/.dub/packages/`.
>
> Alternatively you can do `dub describe --cache=local` to put the packages in the current directory. You could then use `dub add-path .` or add `--cache=local` to all future calls to use those locally fetched packages.
>
> Even better:
> % mkdir cache
> % cd cache
> % dub describe --root=../ --cache=local
>
> and then either
> % dub build --root=../ --cache=local
> or
> % dub add-path .
> % cd ../
> % dub build
>
> which keeps things nice and clean

Just tried this on my CircleCI setup. Works perfectly! Thanks everyone!