After having had another day where my CI can't build because of gateway errors, this morning I had enough. (I setup an uptime robot so we can see how bad it actually is [1], it checks every 5 min and I just started it.)

I know, I can setup my CI to cache dub packages. But that is just an optimisation.

Things ought to just work.

So this morning I started on a quick fix. I took a look at the source and concluded that for dub packages hosted on gitlab and github you can simply redirect to gitlab's and github's respective archives. That is what the dub registry does anyway.

To that end I setup that logic in a cloudflare worker [2]. So for metadata it redirects to the api endpoint on code.dlang.org (which is pretty stable), and for to actual packages directly to gitlab or github.

Use it like so:

`dub build --registry=https://dub.bytecraft.nl`

Of course, this is a bandaid solution. Probably dub can be made to do this as well. And likely should. If it were up to me I would go serverless.

[1] https://stats.uptimerobot.com/6mQX4Crw2L
[2] https://github.com/skoppe/dub-registry-mirror

On Tuesday, 19 November 2019 at 13:27:31 UTC, Sebastiaan Koppe wrote:
> After having had another day where my CI can't build because of gateway errors, this morning I had enough. (I setup an uptime robot so we can see how bad it actually is [1], it checks every 5 min and I just started it.)
>
> [...]
>
> Of course, this is a bandaid solution. Probably dub can be made to do this as well. And likely should. If it were up to me I would go serverless.
>
> [1] https://stats.uptimerobot.com/6mQX4Crw2L
> [2] https://github.com/skoppe/dub-registry-mirror

I have updated the script to cache all routes. This means the registry mirror at dub.bytecraft.nl will also work when the registry at code.dlang.org is completely down, **if** your dependencies have been cached previously.

The cache works in the following manner:

1. if a cache entry exists for a given route and it is no longer than 5 minutes old: return that
2. if a cache entry is older than 5 minutes: fetch from upstream, cache it and return that
3. if a cache entry is older than 5 minutes and upstream is down: reset cache expiry and return from cache
4. if a cache entry doesn't exist: fetch from upstream, cache it and return that
5. if a cache entry doesn't exist and upstream is down: return error

On top of that there is a 5 second timeout on fetches from upstream.

Essentially a cached entry is kept up-to-date (5 minute lag) whenever upstream is alive, otherwise stale data is returned if available.

Use it like so:

`dub build --registry=https://dub.bytecraft.nl`

Code at https://github.com/skoppe/dub-registry-mirror/blob/master/worker.js

very nice, thank you