As you may have already noticed, the main registry server, code.dlang.org got unreachable yesterday. This was caused by an old VPS of mine getting terminated. The registry had already moved to a different server years ago, but, without me realizing it, the DNS entry still pointed to the old one, with a "temporary" HTTP proxy forwarding to the new server being set up.

By now the DNS entry has been corrected, an up-to-date TLS certificate is in place, and the registry is running stable. There are still reports of people not being able to access code.dlang.org, which is apparently caused by intermediate DNS servers still reporting the old IP address and should start working during the next few hours. A temporary workaround is to specify --registry=http://31.15.67.41/ on the dub command line.

Unfortunately both fallback servers have been down for a while now, so that this resulted in a total blackout. I plan to move the main registry to a powerful dedicated server in January, which will fix all memory resource related issues that sometimes show up, and could then keep the current VPS as a relatively reliable fallback server. Both together should guarantee virtually 100% uptime, although more fallback servers are of course highly desirable.

In addition to that, I plan to separate the repository polling process form the web and REST frontend, as the former appears to be the main cause for failures (a GC memory leak of some kind and a possibly codegen related crash when being compiled with DMD being the two known issues, which both need further investigation).

On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:
> As you may have already noticed, the main registry server, code.dlang.org got unreachable yesterday. This was caused by an old VPS of mine getting terminated. The registry had already moved to a different server years ago, but, without me realizing it, the DNS entry still pointed to the old one, with a "temporary" HTTP proxy forwarding to the new server being set up.
>
> By now the DNS entry has been corrected, an up-to-date TLS certificate is in place, and the registry is running stable. There are still reports of people not being able to access code.dlang.org, which is apparently caused by intermediate DNS servers still reporting the old IP address and should start working during the next few hours. A temporary workaround is to specify --registry=http://31.15.67.41/ on the dub command line.
>
> Unfortunately both fallback servers have been down for a while now, so that this resulted in a total blackout. I plan to move the main registry to a powerful dedicated server in January, which will fix all memory resource related issues that sometimes show up, and could then keep the current VPS as a relatively reliable fallback server. Both together should guarantee virtually 100% uptime, although more fallback servers are of course highly desirable.
>
> In addition to that, I plan to separate the repository polling process form the web and REST frontend, as the former appears to be the main cause for failures (a GC memory leak of some kind and a possibly codegen related crash when being compiled with DMD being the two known issues, which both need further investigation).

yay thanks for fixing this so soon.

In my experience having a background task fetching the whole time with vibe.d has nearly always been a bad idea in terms of memory for me. These days I started using cronjobs which run every so often instead and let the OS do all the memory freeing which works a lot better. This also scales a lot better because all workers just read/write to the database server and can be increased or decreased at any point.

Have you maybe also considered making the package zip downloads a separate server? It could be load balanced using nginx as well.

Am 16.12.2019 um 12:23 schrieb WebFreak001:
> On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:
>> As you may have already noticed, the main registry server, code.dlang.org got unreachable yesterday. This was caused by an old VPS of mine getting terminated. The registry had already moved to a different server years ago, but, without me realizing it, the DNS entry still pointed to the old one, with a "temporary" HTTP proxy forwarding to the new server being set up.
>>
>> By now the DNS entry has been corrected, an up-to-date TLS certificate is in place, and the registry is running stable. There are still reports of people not being able to access code.dlang.org, which is apparently caused by intermediate DNS servers still reporting the old IP address and should start working during the next few hours. A temporary workaround is to specify --registry=http://31.15.67.41/ on the dub command line.
>>
>> Unfortunately both fallback servers have been down for a while now, so that this resulted in a total blackout. I plan to move the main registry to a powerful dedicated server in January, which will fix all memory resource related issues that sometimes show up, and could then keep the current VPS as a relatively reliable fallback server. Both together should guarantee virtually 100% uptime, although more fallback servers are of course highly desirable.
>>
>> In addition to that, I plan to separate the repository polling process form the web and REST frontend, as the former appears to be the main cause for failures (a GC memory leak of some kind and a possibly codegen related crash when being compiled with DMD being the two known issues, which both need further investigation).
> 
> yay thanks for fixing this so soon.
> 
> In my experience having a background task fetching the whole time with vibe.d has nearly always been a bad idea in terms of memory for me. These days I started using cronjobs which run every so often instead and let the OS do all the memory freeing which works a lot better. This also scales a lot better because all workers just read/write to the database server and can be increased or decreased at any point.
> 
> Have you maybe also considered making the package zip downloads a separate server? It could be load balanced using nginx as well.

The zips are currently just redirects to GitHub/Bitbucket, but ultimately we should really cache them, if just to keep old versions of packages available in case they disappear from the original repository for whatever reason.

BTW, it looks like most of the CI failures that are usually attributed to the registry are in fact caused by GitHub, probably in combination with the borderline short timeout that is currently configured for dub/curl. I'd imagine that the timeout must be a frequent problem in particular for countries such as China, where the network latency adds a few hundred milliseconds on top of the server response time.

On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:
> As you may have already noticed, the main registry server, code.dlang.org got unreachable yesterday. This was caused by an old VPS of mine getting terminated. The registry had already moved to a different server years ago, but, without me realizing it, the DNS entry still pointed to the old one, with a "temporary" HTTP proxy forwarding to the new server being set up.
>
> By now the DNS entry has been corrected, an up-to-date TLS certificate is in place, and the registry is running stable. There are still reports of people not being able to access code.dlang.org, which is apparently caused by intermediate DNS servers still reporting the old IP address and should start working during the next few hours. A temporary workaround is to specify --registry=http://31.15.67.41/ on the dub command line.
>
> Unfortunately both fallback servers have been down for a while now, so that this resulted in a total blackout. I plan to move the main registry to a powerful dedicated server in January, which will fix all memory resource related issues that sometimes show up, and could then keep the current VPS as a relatively reliable fallback server. Both together should guarantee virtually 100% uptime, although more fallback servers are of course highly desirable.
>
> In addition to that, I plan to separate the repository polling process form the web and REST frontend, as the former appears to be the main cause for failures (a GC memory leak of some kind and a possibly codegen related crash when being compiled with DMD being the two known issues, which both need further investigation).

Hi Sonke,

Thanks a lot for taking care of the issues so quickly and taking actions.
I was a bit sad yesterday not being able to upgrade, but now the new registry server is a lot faster here! It makes browsing the registry a pleasure.

---------------

For all future readers, here is something you can do in case of DUB failure.

What to do if the DUB registry is down:
- Checkout all dependencies manually.

   $ git clone https://hosting.com/mydep.git mydep
   $ cd mydep
   $ git checkout <desired-tag>

- Add those dependencies as local packages:

   $ dub add-local .
   $ cd ..

- Build your project normally. The local packages should override the online packages.

   $ dub build


---------------

My ISP still serves old IP.
Thanks for such a blackout.
D is still not for production use, just a toy that may break accidentally by a will of its creators.

On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:
> My ISP still serves old IP.
> Thanks for such a blackout.
> D is still not for production use, just a toy that may break accidentally by a will of its creators.

I wish I could disagree, but from the view of large enterprise adoption, this is indeed a very bad situation.

On Monday, 16 December 2019 at 22:43:58 UTC, bachmeier wrote:
> On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:
>> My ISP still serves old IP.
>> Thanks for such a blackout.
>> D is still not for production use, just a toy that may break accidentally by a will of its creators.
>
> I wish I could disagree, but from the view of large enterprise adoption, this is indeed a very bad situation.

Interprise must always use local copy -> never blindly trust the internet for reliable!

On Tue, Dec 17, 2019 at 12:55:35AM +0000, Pham via Digitalmars-d-announce wrote:
> On Monday, 16 December 2019 at 22:43:58 UTC, bachmeier wrote:
> > On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:
> > > My ISP still serves old IP.
> > > Thanks for such a blackout.
> > > D is still not for production use, just a toy that may break
> > > accidentally by a will of its creators.
> > 
> > I wish I could disagree, but from the view of large enterprise adoption, this is indeed a very bad situation.
> 
> Interprise must always use local copy -> never blindly trust the internet for reliable!

+100.  I've never liked the idea that the possibility to build my projects depends on some random internet server somewhere out there in the ether that I have no control over.


T

-- 
Never ascribe to malice that which is adequately explained by incompetence. -- Napoleon Bonaparte

On Tuesday, 17 December 2019 at 00:55:35 UTC, Pham wrote:
> On Monday, 16 December 2019 at 22:43:58 UTC, bachmeier wrote:
>> On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:
>>> My ISP still serves old IP.
>>> Thanks for such a blackout.
>>> D is still not for production use, just a toy that may break accidentally by a will of its creators.
>>
>> I wish I could disagree, but from the view of large enterprise adoption, this is indeed a very bad situation.
>
> Interprise must always use local copy -> never blindly trust the internet for reliable!

Oh, I don't doubt that. My point was that it makes the D language project look like a small-scale open source project relying on volunteers (in this case Sonke) being generous with time and resources. What manager is going to trust a project like that with a major project? It seems unlikely that this would not be considered a major issue.

On Tuesday, 17 December 2019 at 01:34:16 UTC, bachmeier wrote:
> Oh, I don't doubt that. My point was that it makes the D language project look like a small-scale open source project relying on volunteers (in this case Sonke) being generous with time and resources. What manager is going to trust a project like that with a major project? It seems unlikely that this would not be considered a major issue.

I totally agree. I am very grateful for Sönke for everything he has and continues to do, but I believe we should move away from self hosting the dub registry. Not only will that increase the uptime, but it will also increase the bus factor and take some work off Sönke's hands into the d organisation.

In another post I outlined a very crude draft proposal https://forum.dlang.org/post/arbvxihnxnboqcifhlbm@forum.dlang.org Since no one replied I assume everyone is onboard. (just kidding)

The architecture isn't difficult, but the migration path is. In February I should have some time and I am willing to setup a proof of concept.