History and Problem

Now that the Editions DIP is up public comment it is time to have a frank discussion about where the D Runtime (DRT for the remainder) fits into the puzzle that Editions introduces into the D ecosystem. The reason I bring this up is that it appears that, at present, most people treat DRT as a particularly vexing extension of the compiler that nobody wants to touch. As a result, the presumption seems to be that, with the advent of editions, we should just freeze DRT wherever it is when the first edition drops and never touch it again because doing so might break something across editions. While this position is certainly expedient, I would argue that this position is going to severely limit the future of Phobos and D.

Furthermore, I would argue that this position makes a presumption that all DRT should ever be is a compiler extension library. I propose that this view is both incorrect and a significant limitation on what we can do with Phobos 3 and even future editions.

I want to call back to DConf 2023 where I spent much of conference working to undo the deprecation of the ODBC bindings in Phobos because they were being replaced with version (Windows) bindings in DRT, which is utter nonsense, ODBC can be used on POSIX quite easily (and I do every day). The explanation I was given as to why this was done is that the bindings have nothing to do with system resources, but because the bindings were provided by DRT they should be removed from Phobos because Phobos is not in the business of providing library bindings.

This creates a rule for Phobos. Phobos does not expose system resources or bindings. IMO this is a good rule. Phobos is bespoke, and that is the proper situation. The standard library really should not be in the business of exposing system level constructs.

Then why not use the DRT bindings? Because the bindings in DRT are automatically generated and are fundamentally broken because of it. At the time it was noted to me that those bindings really should not be in DRT either because DRT isn’t supposed to provide access to non-System resources like ODBC and they were only there by accident because the ODBC headers got swept into the auto-generation process.

This creates a rule for DRT. DRT is only to expose system resources, nothing more. And indeed, this is a good rule as well. The purpose of a runtime is to expose system resources in a uniform manner, and to the extent that DRT does this, it does a fantastic job.

Currently we call DRT a runtime, but when you catalog what it does, it’s really a “compiler support library that provides some runtime services.” There is no argument from me that the GC is a runtime service, the same for threads, but the rest of the listed items that DRT provides are very definitely aimed at enabling the compiler to do things like AA’s for example. AA’s are a feature that is only accessible from the language hence “compiler support”.

The most often given reason for keeping DRT as-is is that “we only need to port DRT to a new platform, Phobos is platform independent.” I submit that this is factually incorrect. There are currently 31 files in Phobos with version (WinXXXX) statements in them. 14 with version (linux). And IIRC everybody who has tried a port D to a new platform recently has just not bothered to port Phobos at all. To be sure, some of these version statements are benign, but take a look at std.stdio, I wish whoever wants to port that much luck in their endeavors. It has 32 instances version (WinXXXX) alone. And that is our basic I/O module.

Runtimes are not compiler support libraries, they are application-system interfaces. The .NET runtime does much more than threads and the GC. The Java runtime does much more than threads and the GC. The C runtime does much more than memory management. I could go on, but you get the point.

The purpose of a runtime is to provide a system resource access layer for the standard library. Consider the C Runtime and the C Standard Library. The C Runtime provides the system specific implementation, and the Standard Library provides the standardized application interface. No matter what system you are on, calling printf will print a string to the terminal via the runtime. If you get a FILE* you will work with a file on disk no matter what filesystem the system is using. These are not compiler supports. They are a universal system-application interface.

An example of why this matters is found in std.stdio. We currently use the C Runtime to do basic I/O operations. While that works and is expedient, it limits us. For example, doing colored terminal output is so difficult nobody does it, or, we are limited to the C file interface, etc. To gain access to more advanced capabilities, we need to use the system API’s, but these are diverse, and putting them in Phobos would significantly expand the code required and make maintenance a nightmare. What is interesting about this is that, in fact, D has two runtimes. The C Runtime, which we use as a universal system interface, and the DRT which is a bunch of compiler support tools that the CRT doesn’t have.

Proposed Design

My proposal is that we expand DRT into a universal system interface for the standard library. DRT would contain low-level interfaces for Terminal and File I/O, threads, event-loops, sockets, cryptography, anything that would normally access a system API. Why go to all this effort? Simple, so that we can move beyond the limitations imposed by the C Runtime. Using the CRT was certainly expedient in the early days of D, but we’ve grown beyond that and it’s time to upgrade our capabilities.

However, this will significantly expand the scope of DRT and will make porting harder. I think the answer to this problem came out of left field in a conversation I had last Thursday with an ex-Google engineer who has no prior knowledge of D. What he said was: “you need to shard your runtime.” This statement caused several things to “click” and got me started writing this missive.

Before I get into the design of the DRT I want to propose rules that will allow us to continue to evolve DRT in the future without breaking past editions. From an ABI standpoint, there are only three actions that can be taken: Add, Rename, Remove.

1. Remove. Banned. We cannot remove symbols from the DRT API. But we can rename them provided we follow the rename rules.
2. Rename. Renames include changing the parameters in any way. We can support this by offering either a shim that redirects to the new method or leaving the original method alone and building a new one next to it.
3. Add. The is probably the easiest. New symbols can be added at any time.
   If we follow these rules, then we can safely use the latest DRT version with all prior editions.

For DRT I propose a sharded design with a split between the compiler support modules and the universal system interface shards. What this means is that there will be a “Core-DRT” that only contains things that the compiler needs to function. Then, for each subsystem in Phobos, there would be a corresponding DRT subsystem that can be built and linked (statically or dynamically). This means that the only mandatory component that would have to be ported is the “Core-DRT” component for the compiler. The porting engineer would then be free to choose to port as much or as little of what remains of DRT and know that the corresponding Phobos subsystem is guaranteed to work once the port is complete. Another to view these components is as an “onion” where each one builds on the one above it. Core must exist for System, System exists for Cryptography, etc. Below is a, likely incomplete, list of potential runtime shards.

Core:

1. Compiler Hooks
2. GC
3. Threads
4. Event Loop (in later iterations)

System:

1. Console/Terminal I/O
2. File I/O

Cryptography:

1. System Cryptography Primitives
2. Optional OpenSSL extended Cryptography Primitives (to support what the OS does not).

Network:

1. Sockets
2. SSL/TLS

I’m not married to the above so please destroy. One thing that I think would be beneficial for ease-of-porting is researching if it would be possible to move Threads and the Event Loop out of Core and into the System component then provide a mechanism for notifying the GC of thread creation/deletion without actually requiring the Threads (and thus the Event Loop) implementation to be in the Core DRT so that people porting to systems that don’t have Threads (or simply don’t want to support them) don’t have to port them to get basic D code running.

The goal of Core is to be the most reduced component of the runtime that can still produce a functional program in D. The larger goal of this DRT expansion project is to eventually drop the requirement for the C Runtime. By doing so we free ourselves from the limitations of the CRT, such as being unable to handle Console Colors on systems that support them, and we remove a layer of abstraction by calling directly into the system API’s. This does mean more work for porting in terms of needing to call the system API’s directly as opposed to using the CRT, however, one option could be to provide an opt-in CRT-enabled build of DRT that uses the CRT for a limited set of features.

I want to point out that these are ideas intended to address the limitations of the current situation and the proposal to make changes is a result of looking at the work we’re going to have to do with Phobos 3. I fully understand that I can be wrong on things here, but the present situation with DRT is not going to remain tenable as we move forward with expanding Phobos.

Another angle that might be worth pursuing is building DRT up into a Runtime that can be used by many languages, similar to how the CRT is used. It would also increase the appeal of D to folks looking for a platform to start building a language on, like how an ARM backend will help the hobbyist language community. I understand that LLVM and GCC are there, but they are not simple to work with. If we can provide an ecosystem that is simple to work with for language experimentation and development, we can capture people looking for a tool to start designing languages quickly and still provide them a path to integrate with LLVM or GCC via our existing integrations with those tools.

These are ideas, not directions, so please destroy.

On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

I think DRT only needs to concern itself with supporting language features. Anything else needs to go elsewhere.

On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

We have to take into account the combinatorial explosion that comes with the rename method. For example, should you choose to change the Throwable interface (because it's old and really not good), you would need to make sure that everything that uses it is compatible with it. There are known ways to do it in other languages (https://wiki.qt.io/D-Pointer) but I think the rules you proposed are incomplete as they only work for functions, not types.

Layering things / encapsulating them is good, but you also need a way for inner layers to refer to outer layers, otherwise you severely limit the capabilities of your system.
Take for example unittests: we would like to provide a much better out-of-the box experience for unittests. You should get colors, summary, the ability to run them in parallel, or even a filewatcher that auto-recompiler them, out of the box. We might consider unittests to be in Core, but I'm sure the rest is not.

A resounding yes. Keeping the ability to easily bind to the CRuntime is useful (e.g. being able to spin up a socket and just look at the C documentation), but because it's "good enough", it was never good at all.

On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

Firstly people do colour, sometimes not very well but it's clearly not "so difficult nobody does it". There should be a higher level way of doing it shipping with the language but that's more of a question of high-level tradeoffs than anything to do with druntime.

Secondly, I think this reasoning leads towards a trap: Placing an arbitrary division between phobos and druntime, or more charitably placing that division in the wrong place, can lead to a lot of needless debate or lost productivity.

Unless the compiler (i.e. containing druntime) and phobos were in a repo together you can end up in the same situation we had before with the compiler and runtime being separate leading to have numbers of double-PRs and so on, only squared.

While we still have a runtime I don't think phobos should have all of the nuts and bolts in it but in contrast to "maintenance would be a nightmare" I posit that this be preferable to many alternatives: be glad you can do the maintenance in the first place. When you split things up across projects (or worse repos) you lose the ability to do atomic commits, test (easily, buildkite isn't easy) at the same time for example.

Aside from that I do like really the idea of ditching the C API where easy e.g. purely as a gimmick I've always liked the idea of hello world inlining down to a system call where applicable.

There are things where the C runtime does cover up a lot of hardware-specific details that we really don't need to care about but other things where it's a pile of crap and worth ditching.

On Thursday, 6 June 2024 at 18:00:56 UTC, Sebastiaan Koppe wrote:

I agree. If nothing else because you can start a new phobos yesterday whereas making changes to druntime has all kinds of other high-dimensional treadeoffs to faff around with e.g. the druntime a user actually gets is already often slightly forked in that it's often not from dmd.

The way to get a stable foundation that can be ported easily to the likes of wasm and so on is not making druntime bigger. You end up with a good result by doing so but it would be like taking a shortcut through a maze rather than the road - fragile.