On Monday, 12 May 2025 at 12:07:50 UTC, Sönke Ludwig wrote:

> This is not really true, the truly static content has been moved to dub.pm a while ago and the rest is made up of dynamic views on the package database. Of course it would be possible to cache pages, but that wouldn't help against crawlers.

By the way, I've found this guy fighting back specifically to aggressive AI crawlers with a bizzarre technique: gzip bombs.

https://idiallo.com/blog/zipbomb-protection

/Paolo

On Monday, 12 May 2025 at 12:07:50 UTC, Sönke Ludwig wrote:
[...]
>> - Turn off the GC.
>
> In the situation that lead to the issues that would blow up memory usage within a very short amount of time, replacing the bad responsiveness with a non-responsive system or in the OOM killer terminating some process.

Can those processes be factored out of the vibe.d process?

>> - Run vibe.d under hard memory/cpu limit (which terminate
>>    the process if exceeded).
>
> The problem here was not hardware resources, but the fact that the GC was stopping the process for a large amount of time, as well as allocation overhead.

Why does a/the (CGI) process have to live longer than the socket
on which the response is sent to the client? The cleanup can
easily be done by the kernel.

> [...]
>> 
>> - Deploy a monitoring process which checks if vibe.d responds
>>    in say 20 ms. If not kill vibe.d with SIGKILL.
>
> There is a certain startup overhead and there are some requests that can take longer (registry dump being the longest, but also requesting information about a dependency graph, which is done by dub).

Can these time consuming startup tasks be performed before the
bind call is issued?

> I think this should really only be a last-resort approach (e.g. response time > 5s), because it replaces bad response times with failed requests.

Does all this concern only code behind the login?

>> - flock with LOCK_EX on an open fd fopend on the vibe.d binary
>>    right before vibe.d issues the bind call.
>> 
>> - Have a second vibe.d process running blocking on it's flock call
>>    (= auto restart).
>
> This would still mean that active connections will fail, which is not ideal in a situation where the restart would be frequently necessary.

Not necessarily AFAIR but I would have to look that up (socket
sharing, hot restart ...).

>> [...]
>> To me it seems that code.dlang.org is mostly not a web app but
>> a web site with static content. Have you been thinking of serving
>> that static content with apache/nginx using rate limiting
>> (mod_throttle etc.)? Or putting this content directly to CF who is
>> also a web hoster?
>
> This is not really true, the truly static content has been moved to dub.pm

With "static content" I mean data/information that only changes
slowly or rarely.

> a while ago and the rest is made up of dynamic views on the package database. Of course it would be possible to cache pages, but that wouldn't help against crawlers.

- Block them with robots.txt by their UA strings?
- Throttle request per sec by UA string/IP/Network/AS?
- Require a cookie for accessing the page. That is easy to reinvent.

> Writing out all package and package list pages and then serving them as static content would result in a huge amount of files and occupied memory and would be time consuming.

If the final pages are assembled in the user agent it will suffice
to put only the "variable" (nontheless static) content in JSON files
on the server.

Like it is done in http://wiki.c2.com/

BTW: Are we talking about less or more than 1 million files?

> This would only really make sense when the number of pages would get reduced massively (e.g. no per-version package pages, limiting the number of result pages for the popular/new/updated package lists).

How many files are changed or created per day? What amount of
memory (MB) is stored on the machine per day?

On 14/05/2025 2:13 AM, Denis Feklushkin wrote:
>     It allows us to remove a SPOF in our critical infrastructure, as a
>     Github outage would always cause a registry being unusable anyway.
> 
> It seems this approach will solidify GitHub as SPOF for us, because for now we aren't forced to use Github

If done right, all dub has is a HTTP url to download the index from.

With that capability you can host anywhere you like.

Even with a registry.