History and Problem
Now that the Editions DIP is up public comment it is time to have a frank discussion about where the D Runtime (DRT for the remainder) fits into the puzzle that Editions introduces into the D ecosystem. The reason I bring this up is that it appears that, at present, most people treat DRT as a particularly vexing extension of the compiler that nobody wants to touch. As a result, the presumption seems to be that, with the advent of editions, we should just freeze DRT wherever it is when the first edition drops and never touch it again because doing so might break something across editions. While this position is certainly expedient, I would argue that this position is going to severely limit the future of Phobos and D.
Furthermore, I would argue that this position makes a presumption that all DRT should ever be is a compiler extension library. I propose that this view is both incorrect and a significant limitation on what we can do with Phobos 3 and even future editions.
I want to call back to DConf 2023 where I spent much of conference working to undo the deprecation of the ODBC bindings in Phobos because they were being replaced with version (Windows)
bindings in DRT, which is utter nonsense, ODBC can be used on POSIX quite easily (and I do every day). The explanation I was given as to why this was done is that the bindings have nothing to do with system resources, but because the bindings were provided by DRT they should be removed from Phobos because Phobos is not in the business of providing library bindings.
This creates a rule for Phobos. Phobos does not expose system resources or bindings. IMO this is a good rule. Phobos is bespoke, and that is the proper situation. The standard library really should not be in the business of exposing system level constructs.
Then why not use the DRT bindings? Because the bindings in DRT are automatically generated and are fundamentally broken because of it. At the time it was noted to me that those bindings really should not be in DRT either because DRT isn’t supposed to provide access to non-System resources like ODBC and they were only there by accident because the ODBC headers got swept into the auto-generation process.
This creates a rule for DRT. DRT is only to expose system resources, nothing more. And indeed, this is a good rule as well. The purpose of a runtime is to expose system resources in a uniform manner, and to the extent that DRT does this, it does a fantastic job.
Currently we call DRT a runtime, but when you catalog what it does, it’s really a “compiler support library that provides some runtime services.” There is no argument from me that the GC is a runtime service, the same for threads, but the rest of the listed items that DRT provides are very definitely aimed at enabling the compiler to do things like AA’s for example. AA’s are a feature that is only accessible from the language hence “compiler support”.
The most often given reason for keeping DRT as-is is that “we only need to port DRT to a new platform, Phobos is platform independent.” I submit that this is factually incorrect. There are currently 31 files in Phobos with version (WinXXXX)
statements in them. 14 with version (linux)
. And IIRC everybody who has tried a port D to a new platform recently has just not bothered to port Phobos at all. To be sure, some of these version statements are benign, but take a look at std.stdio
, I wish whoever wants to port that much luck in their endeavors. It has 32 instances version (WinXXXX)
alone. And that is our basic I/O module.
Runtimes are not compiler support libraries, they are application-system interfaces. The .NET runtime does much more than threads and the GC. The Java runtime does much more than threads and the GC. The C runtime does much more than memory management. I could go on, but you get the point.
The purpose of a runtime is to provide a system resource access layer for the standard library. Consider the C Runtime and the C Standard Library. The C Runtime provides the system specific implementation, and the Standard Library provides the standardized application interface. No matter what system you are on, calling printf
will print a string to the terminal via the runtime. If you get a FILE*
you will work with a file on disk no matter what filesystem the system is using. These are not compiler supports. They are a universal system-application interface.
An example of why this matters is found in std.stdio
. We currently use the C Runtime to do basic I/O operations. While that works and is expedient, it limits us. For example, doing colored terminal output is so difficult nobody does it, or, we are limited to the C file interface, etc. To gain access to more advanced capabilities, we need to use the system API’s, but these are diverse, and putting them in Phobos would significantly expand the code required and make maintenance a nightmare. What is interesting about this is that, in fact, D has two runtimes. The C Runtime, which we use as a universal system interface, and the DRT which is a bunch of compiler support tools that the CRT doesn’t have.
Proposed Design
My proposal is that we expand DRT into a universal system interface for the standard library. DRT would contain low-level interfaces for Terminal and File I/O, threads, event-loops, sockets, cryptography, anything that would normally access a system API. Why go to all this effort? Simple, so that we can move beyond the limitations imposed by the C Runtime. Using the CRT was certainly expedient in the early days of D, but we’ve grown beyond that and it’s time to upgrade our capabilities.
However, this will significantly expand the scope of DRT and will make porting harder. I think the answer to this problem came out of left field in a conversation I had last Thursday with an ex-Google engineer who has no prior knowledge of D. What he said was: “you need to shard your runtime.” This statement caused several things to “click” and got me started writing this missive.
Before I get into the design of the DRT I want to propose rules that will allow us to continue to evolve DRT in the future without breaking past editions. From an ABI standpoint, there are only three actions that can be taken: Add, Rename, Remove.
- Remove. Banned. We cannot remove symbols from the DRT API. But we can rename them provided we follow the rename rules.
- Rename. Renames include changing the parameters in any way. We can support this by offering either a shim that redirects to the new method or leaving the original method alone and building a new one next to it.
- Add. The is probably the easiest. New symbols can be added at any time.
If we follow these rules, then we can safely use the latest DRT version with all prior editions.
For DRT I propose a sharded design with a split between the compiler support modules and the universal system interface shards. What this means is that there will be a “Core-DRT” that only contains things that the compiler needs to function. Then, for each subsystem in Phobos, there would be a corresponding DRT subsystem that can be built and linked (statically or dynamically). This means that the only mandatory component that would have to be ported is the “Core-DRT” component for the compiler. The porting engineer would then be free to choose to port as much or as little of what remains of DRT and know that the corresponding Phobos subsystem is guaranteed to work once the port is complete. Another to view these components is as an “onion” where each one builds on the one above it. Core must exist for System, System exists for Cryptography, etc. Below is a, likely incomplete, list of potential runtime shards.
Core:
- Compiler Hooks
- GC
- Threads
- Event Loop (in later iterations)
System:
- Console/Terminal I/O
- File I/O
Cryptography:
- System Cryptography Primitives
- Optional OpenSSL extended Cryptography Primitives (to support what the OS does not).
Network:
- Sockets
- SSL/TLS
I’m not married to the above so please destroy. One thing that I think would be beneficial for ease-of-porting is researching if it would be possible to move Threads and the Event Loop out of Core and into the System component then provide a mechanism for notifying the GC of thread creation/deletion without actually requiring the Threads (and thus the Event Loop) implementation to be in the Core DRT so that people porting to systems that don’t have Threads (or simply don’t want to support them) don’t have to port them to get basic D code running.
The goal of Core is to be the most reduced component of the runtime that can still produce a functional program in D. The larger goal of this DRT expansion project is to eventually drop the requirement for the C Runtime. By doing so we free ourselves from the limitations of the CRT, such as being unable to handle Console Colors on systems that support them, and we remove a layer of abstraction by calling directly into the system API’s. This does mean more work for porting in terms of needing to call the system API’s directly as opposed to using the CRT, however, one option could be to provide an opt-in CRT-enabled build of DRT that uses the CRT for a limited set of features.
I want to point out that these are ideas intended to address the limitations of the current situation and the proposal to make changes is a result of looking at the work we’re going to have to do with Phobos 3. I fully understand that I can be wrong on things here, but the present situation with DRT is not going to remain tenable as we move forward with expanding Phobos.
Another angle that might be worth pursuing is building DRT up into a Runtime that can be used by many languages, similar to how the CRT is used. It would also increase the appeal of D to folks looking for a platform to start building a language on, like how an ARM backend will help the hobbyist language community. I understand that LLVM and GCC are there, but they are not simple to work with. If we can provide an ecosystem that is simple to work with for language experimentation and development, we can capture people looking for a tool to start designing languages quickly and still provide them a path to integrate with LLVM or GCC via our existing integrations with those tools.
These are ideas, not directions, so please destroy.