Hello everyone,

As you all know, the official DMD releases got severely delayed. I think you're owed an explanation of why it took so long.

Iain Buclaw, our release manager, was working on streamlining the release process to make it more automated and reduce the 'bus factor'.

Then he got 'hit by a bus'.

Development on dmd and automatic nightly builds continued, but the official releases for 2.110 and beyond got delayed and delayed, as nobody was around with the know-how and access to make a proper release.
Luckily, as the November update alluded to, Iain was going to slowly return and on January 8, he uploaded the release candidate for 2.110. While doing that, he walked me through the process in a Slack huddle / tmate session, so I could take over the task for the time being. I was optimistic that things were going to be back on track now, and updated the release schedule with 2.110.0 due for February 1st 2025. However, once that day passed, it turned out we were not in the clear yet. At first I thought there was just a small wait for the new changelog generator after our migration from Bugzilla to GitHub issues, but no: 2.110.0 still needed to be built and uploaded, so this week I made it my priority to get everything sorted out.

Long story

Feel free to skip to the next paragraph, this is just a bunch of technical details for your amusement / understanding of why it was so hard to complete a release for me.

So like I said, Iain and I scheduled an online meeting to go over the release process on January 8. Right from the start, Slack huddles don't work in Firefox, so I downloaded the client, which crashed really badly when I started sharing my screen through it, requiring a reboot of my computer to fix my audio. That's not relevant, but it set the tone wonderfully for everyone that was about to come.

So the release orchestration is largely handled by one big release.sh bash script with about 20 sequential steps, but it can't just be run on any computer: it has a lot of dependencies on software being installed, configuration, authentication to various servers etc. So Iain started by giving me an account on his server from which he builds releases, that I can ssh into. Then we walked through each step, one by one. Because I had some trouble getting my git configuration/authentication/keys etc. working on the server, I ended up rsync'ing my user folder on the server to my own desktop so I could continue locally. That folder is 18 GB because it contains 4 different virtual machines for building the Linux, Windows, OSX and FreeBSD releases, but once the download completes, I should be able to build the release, right?

Each step uncovered new missing dependencies / setup on my part. I installed vagrant and virtualbox, had to modprobe vboxdrv, but that required installing linux-rt-headers, wait it still doesn't work, oh the right package is linux-headers yadayadayada... Running the VM also requires enabling virtualization in your BIOS, a setting which I couldn't find at first because it's labeled 'SVM mode' on this particular motherboard. The biggest issue was that I have an AMD processor, and OSX refuses to boot on anything but an Intel processor. Apparently it's easy to fool the cpuid on libvirt, but not on virtualbox.
That's how far we came in that huddle: I made progress, but ultimately couldn't build myself yet.

When it became clear this week that I had to figure out a way to build myself, I started to look if I had a different computer laying around. Since my old desktop with an Intel processor has a broken motherboard, I dug up my old 2015 student laptop with Windows 10, and installed linux on a new partition. The USB drive from which I previously installed NixOS on a mini laptop suddenly failed to start the display manager. It looks like some bits got corrupted, because a decompression program reported bad file contents. I always suspected that particular USB stick to be unreliable, but the 2 other drives I tried wouldn't even get picked up by my Laptop's boot loader, so I copied a fresh NixOS installer on the sketchy one and hoped it would work. It did. I can't seem to boot it directly though, I have to boot Windows 10 first and then shift + restart => use device => Linux Boot Loader. Oh well.

Now I had to configure everything on my new installation. Some ChatGPT assistance, Stack Overflow, and an MVP GitHub comment later, I could finally boot the virtual boxes! Then it exceeded my 8GB of RAM and the OOM killer crashed the whole PC.

I rebooted and set up an 8GB swap file. I retried, but suddenly the Virtual Boxes wouldn't boot anymore, now reporting a new error. Huh? When I retried, it failed even before getting to the command to start the VM. Can't write to /tmp/XXX? Oh, I see. I ran out of disk space. I only had room for a 58 GB partition, which I thought was enough for the 20 GB of VMs, but not when ~/VirtualBox VMs and /tmp get filled up by running vagrant so many times. Some disk cleaning and 2 hours of anxiously watching my task manager showing 95% RAM usage later, it actually did it. It built releases for all platforms! That felt real good, but my troubleshooting hadn't come to an end just yet...

Long story short

To run the script that handles the release process, to quote Atila, you need a special Windows Linux box commisioned by the Vatican and blessed by the Pope himself. But after a ton of troubleshooting, my old laptop became such a blessed box. It's probably the third one on the planet, and I'm storing it in a safe in a nuclear bunker. :)

GitHub actions outage

Every release is accompanied with a PGP signature in the form a .sig file, which the install.sh script and the setup-dlang GitHub action uses to gpg --verify the executables with. Since I'm a new signer, I added my public key to the gpg_keys page. However, I wasn't aware that I also had to manually update the binary d-keyring.pgp file, which is the source of public keys installation scripts actually use. So yesterday (March 5) you might have noticed the following error in your CI:

Run dlang-community/setup-dlang@v1
Enabling dmd-latest
Downloading https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz
Verifying the download with GPG
gpg: Signature made Wed Mar  5 12:46:50 2025 UTC
gpg:                using RSA key F3F896F3274BBD9BBBA59058710592E7FB7AF6CA
gpg: Can't check signature: No public key

I fixed that this morning (Mar 6 11:00 UTC) with assistance from Andrei Horodniceanu on Slack (thanks!). I'm sorry for the inconvenience. If you encounter this error while manually installing/verifying, ensure you are using the latest keyring.

What's happening now

My current plans are:

• Fix the changelog (either by trying Robert's new tool or writing something manually)
• Formally announce the 2.110 release on Friday March 7, and publish the download links on the website
• Update the release archive index (currently I hit a snag where d-requests can't find the right libssl.so)
• Update the documentation archives
• Document everything I learned about the release process for future maintainers
• Do the releases for 2.111 onwards until Iain or someone else takes over again

If you want to download 2.110 already, you can do so at:

libphobos2-109_2.110.0-0_i386.deb.sig
libphobos2-109_2.110.0-0_i386.deb
libphobos2-109_2.110.0-0_amd64.deb.sig
libphobos2-109_2.110.0-0_amd64.deb
dmd_2.110.0-0_i386.deb.sig
dmd_2.110.0-0_i386.deb
dmd_2.110.0-0_amd64.deb.sig
dmd_2.110.0-0_amd64.deb
dmd.2.110.0.windows.zip.sig
dmd.2.110.0.windows.zip
dmd.2.110.0.windows.7z.sig
dmd.2.110.0.windows.7z
dmd.2.110.0.osx.zip.sig
dmd.2.110.0.osx.zip
dmd.2.110.0.osx.tar.xz.sig
dmd.2.110.0.osx.tar.xz
dmd.2.110.0.linux.zip.sig
dmd.2.110.0.linux.zip
dmd.2.110.0.linux.tar.xz.sig
dmd.2.110.0.linux.tar.xz
dmd.2.110.0.freebsd-64.zip.sig
dmd.2.110.0.freebsd-64.zip
dmd.2.110.0.freebsd-64.tar.xz.sig
dmd.2.110.0.freebsd-64.tar.xz
dmd.2.110.0.dmg.sig
dmd.2.110.0.dmg
dmd-2.110.0.exe.sig
dmd-2.110.0.exe
dmd-2.110.0-0.openSUSE.x86_64.rpm.sig
dmd-2.110.0-0.openSUSE.x86_64.rpm
dmd-2.110.0-0.openSUSE.i386.rpm.sig
dmd-2.110.0-0.openSUSE.i386.rpm
dmd-2.110.0-0.fedora.x86_64.rpm.sig
dmd-2.110.0-0.fedora.x86_64.rpm
dmd-2.110.0-0.fedora.i386.rpm.sig
dmd-2.110.0-0.fedora.i386.rpm

Future release schedule

Note that 2.110 only contains bug fixes implemented in the last 7 months, not the new features of the last 7 months. Those are scheduled for 2.111, which brings up a question: Iain and I had the idea of skipping 2.110.1 and going straight to 2.111.0, since we're so far behind. There's also been talk of slowing down the amount of releases in general. What do you all think?

Final words

I know the whole situation is very unprofessional. Personally, I'm working part time as Pull Request and Issue Manager for the D Language Foundation, and part time as D Programmer for SARC, a Dutch maritime software company with a roughly similar head count as the DLF 'core team'. Every week, I go to SARC's office where there's a weekly meeting with everyone. You can see what everyone is working on, knowledge is easily shared, and issues can be solved together. While I think it's amazing that DLF operates completely remote, with volunteer contributions coming from all around the world, it is awful for handling situations like this 2.110 release debacle. But we are learning and improving overall I think.

I want to thank Iain for doing the thankless job of being release manager, and helping us out recently when he really shouldn't have had to.

Have a good day everyone.

~Dennis Korpel

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

Thanks Dennis and Iain for your great work!

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

Yes, there's no need to release 2.110.1
I believe that the separation between people which updates their compiler is basically;

1 - Bug Fixes
2 - New features

Since bugfixes entered 2.110.0, there isn't much point into releasing the minor

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

[…]

Thank you for doing this very important work, it is good to see this train being put back on the tracks again!

Regarding preference for release cadence I think this should be mainly up to the person doing the work. I also think that it makes sense to prioritise making the process more robust and easier to operate over a high cadence.
Also, it may be more valuable to have frequent bug fix releases than feature releases.
And, should it take extra effort to prepare installer packages than the zip archives, it could be an option to have a lower cadence for these. People that require the latest and greatest are probably fine with a zip. At SARC we have automated compiler installation as part of the commit history, which relies solely on zip archives. This is probably a common approach in professional teams.

— Bastiaan.

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

Thank you and Iain for all your hard work.

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

Thank you for the update and the details behind. It is much appreciated.

Thank you and Iain for doing this work. I hope you can manage to streamline some of that.

With respect to the frequency of releases, I quite like the bi-monthly cadence, but really anything else that will be more than once or twice a year is great.

Much obliged!

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

It is clear that you enjoyed what you wrote (especially the above), which I cannot quote in its entirety. Most of all, I'm glad you made it.

I read all the lines that you sincerely wrote. This was difficult due to the fact that my native language is not English. However, the difficulties you are experiencing are insignificant compared to the difficulties you are experiencing.

In summary, I congratulate you for what you have done

Updating bugs was important. Maybe we should do a little review instead of skipping the version number. And no matter how difficult it is, version 2.110.1 can be completed with a small update, what do you think.

In summary, it is better not to spoil what we are used to. Because the previous "versioning method" had already changed

Thanks Dennis...

SDB@79

On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:

Thanks for the writeup - and especially the work to do this. Wow, that was quite a campaign to get it working!

Since your box runs NixOS as the host OS, I guess you can pass its configuration.nix to the Foundation, so when it comes a time to set up a fourth box someday it should be at least a bit easier.

Good idea. There are a varying number of patch versions for each major version anyway. It doesn't bother me if that number is sometimes 1.

I don't think the delay itself is that unprofessional. Iain has an incredible load of responsibilities so it would be a surprise if something like this didn't happen sometimes. For a small organisation like the Foundation I don't expect that every role can have a timely back up that would be there in an ideal big company.

The only gripe I have is about keeping the users up to date when the release doesn't happen when it was planned to. Why? What's the new ETA? Or if there's no ETA currently, what's the ETA for the new ETA? But you just remedied that with your post.

+1