D Language Foundation April 2024 Quarterly Meeting Summary

July 12

Posted by Mike Parker

Permalink

Mike Parker

Permalink

The D Language Foundation's quarterly meeting for April 2024 took place on Friday the 5th at 15:00 UTC. It lasted for about 50 minutes.

Our quarterly meetings are where representatives from businesses big and small can bring us their most pressing D issues, status reports on their use of D, and so on.

The Attendees

The following people attended the meeting:

Mathis Beer (Funkwerk)
Walter Bright (DLF)
Johan Engelen (Weka)
Dennis Korpel (DLF/SARC)
Mathias Lang (DLF/Symmetry)
Razvan Nitu (DLF)
Vladimir Panteleev
Mike Parker (DLF)
Robert Schadek (DLF/Symmetry)
Bastiaan Veelo (SARC)

The Summary

Vladimir

Vladimir brought three issues to us that he said were not a priority, but had been a pain in his side. He had now encountered them in the code base at work, so he wanted to ensure they were at least on someone's radar. He stressed they were not super urgent. Two of them were codegen bugs, and one of those two was a regression. The third one he said was just weird.

Bastiaan

Bastiaan began by telling us he was very happy that Dennis had started working with SARC. He then gave us an update on the conversion from Pascal to D.

He reminded us that in our October 2023 meeting, he'd reported on a thread contention problem he'd encountered with allocations (which he subsequently reported had been solved). Since then, he'd had a problem with his implementation of Pascal sets. Those were bit arrays, and the implementation had been a bit sloppy regarding allocations, but that was solved now, too. He said the @nogc attribute had helped track down where the allocations were happening.

He said that overall, prospects were good. But one thing he'd noticed was that when doing function overloads, you could have inout to account for const, immutable, and mutable function parameters vs return types, there was nothing comparable for the presence or absence of @nogc on parameters which were delegates. Especially in opApply.

Dennis said this was a longstanding issue. He thought Mathias Lang had a DIP for attribute polymorphism on delegates, and there had been other proposals. But they were kind of complex and so far hadn't gained traction.

Mathias said that, yes, he had a DIP for it, but he wasn't 100% confident about it due to some corner cases. He hadn't had a reason to make the time to pursue it further. If anyone was willing to pick it up, he'd be happy to discuss it with them, but he didn't see himself getting back to it anytime soon.

Bastiaan said it wasn't a big problem, but especially if you wanted to overload opApply, you had to do it twice to account for @nogc delegates and those that weren't.

Dennis said that sometimes with templates you could have a generic template type for your return type. He didn't know if opApply supported that, but he thought it did. Mathias said templates were nice until you needed to use classes or virtual functions, then things would just break.

Walter asked Bastiaan to file a Bugzilla issue on this, and he said he would.

Mathis Beer

Mathis said he'd been trying to push incremental compilation at work as everyone was switching to 32GB laptops. A problem he'd encountered is that once you pass the compiler a D file along with some other files, you'd then not be able to rebuild that file in isolation. The compiler would insert some symbols into it which would then be missing in another module, or the other way around.

They'd been working around this by tracking which D files they were passing to which command line. Then when they needed to rebuild that file in isolation, they'd go ahead and rebuild the other files it had been built with. So they were kind of taking the files as a set, then breaking it into two different sets and building both of them. The first time around, there was no speedup, but they were seeing an improvement the second time around.

The main problem they'd had with that was memory usage. Even with their laptops as large as they were, it was getting critical. Especially if they broke it up too far and had something like eight instances of DMD running or passing the same modules in parallel. They'd been optimizing stuff to have the right amount of template usage for a single build and take good advantage of the object and module caches. If they were passing the same modules eight times, it may be faster, but it wasn't better for memory usage.

He pasted a Bugzilla issue in the chat and said we should treat it as low priority. He said the reported issue was what happened when you tried to rebuild a single object that was part of a larger build set earlier. He anticipated that once this was fixed, there would be three more issues waiting in the wings behind it. What they had right now did seem to be working.

As an addendum, he noted that the mold linker was absolutely amazing. He said that anyone on Linux needed to try it. He thought it should be part of the distribution, as it was very fast.

Dennis agreed that mold was fast. He added that ryuukk_ had published a forum post saying how you could speed up build times by using mimalloc rather than the glibc malloc. Dennis said he hadn't looked into it yet.

Johan asked if Mathis could share the build system he'd described. Mathis said that if it worked out, he'd do a write-up on the forums about how it worked. Right now, it was just a heap of scripts that he was the only person actively maintaining. He wouldn't want to impose them on the world. He said that if you had an estimate of how much memory you were going to use, it was the sort of thing that could probably included in dub fairly straightforwardly.

Robert asked if they had tried the combination of reggae and ninja. He said ultimately, they'd run into the same problem, but instead of doing it all themselves there might be something there that was good enough for them. Mathis said that the big problem with the build systems he'd considered with this, or at least as far as he was aware, was that when you were using DMD in this way, then you'd have a build rule that reacted to change in one of five source files by rebuilding five object files, only some of which would be outdated. He said that was generally hard to express.

Robert said he was using reggae with ninja on a pretty big project and it was working well. Mathis provided more background and explained that they'd run into an issue when passing -op to DMD to maintain the source paths on generated object files so that they'd end up with a directory tree in the build folder that matched the source tree. This worked pretty well most of the time, and without it, you sometimes ran into name collisions with the object files. The problem came in when using it with dub when you passed files with absolute paths. You might see the object files ending up in a folder where you didn't want them. He intended to file an issue about it but hadn't done so yet. That was why they were doing a lot of stuff manually at the moment.

Johan

This was Johan's first time joining the quarterly, so he'd intended to just come in and listen and hadn't brought anything specific to discuss. He reported that Weka was still at LDC 1.30, which was the equivalent of DMD 2.100. He said it was common that they skipped multiple versions because the updates were too frequent, but they'd recently started upgrading to LDC 1.37 or maybe 1.38. They'd run into the usual problems, but fewer than he'd expected, so that was positive.

As a result of the upgrade process, he said we may have recently noticed a few extra bug reports on Bugzilla. These were for issues for which there were no easy workarounds. Overall, he thought it was working out pretty well. There'd been nothing completely unsolvable or very crazy yet, but so far they were still at the stage of fixing compilation errors and hadn't gotten to the point of running anything. He said that in the past, compiler upgrades would break the binary, but that hadn't happened the past couple of times.

He said that Weka had so much templated introspective code that they seemed to hit every corner case of the language. With every compiler update, a corner case may have changed. They were very sensitive to that.

As an example, he said they had a struct with an opAssign override for const. Then somewhere in Phobos, it broke. There was a range iterator testing for opAssign, then afterward it didn't perform a move, but instead did a memcpy on the object, and then complained that it couldn't copy into constant memory. So it had this check for the assignment, and it passed because this opAssign override on a const object was allowed.

He said that Jonathan had already come up with a fix for it, so he was happy that it had been noticed and acted upon so quickly. But it showed how they had all these funny things in their code base. So he had the idea that he should make a test file that collected all these "structures from hell" where they did strange things. Then if we were looking for an annoying structure to test against, we could look at this file where they had a bunch of them. And the list would link to the bugs that caused each structure to be included in it.

Walter said that would be great, and that's what the test suite was for. Johan said the difficulty was often in figuring out whether it was valid code or not. Should the language support it, or was it just too crazy?

Mathis Beer asked what they were doing with that opAssign overload, and Johan explained it. It was just one place in the code base where they were essentially using a mixin to fix up the structure to basically have tail const. He said that very often in these cases, he didn't even know how the whole thing worked. He'd just update the compiler and then encounter compilation errors across this huge code base with so much template stuff going on that it was usually kind of difficult to figure out what the error actually was.

He said that understanding what the code was doing was usually his smallest concern. Sometimes he had to, but he usually tried to avoid getting into the meaning of the code, because then he might try and fix it and then run into problems with changing program behavior. So he usually tried to reproduce what the old compiler was doing to see if that was correct or if it was actually a bug.

He said one thing that was nice to mention was that the compiler now complained if you had a static assert on just a string, where instead of static assert(false, "message") you were just doing static assert("message"). This had uncovered a bug in their code base where someone had done exactly that, so he was really happy that the compiler complained about it. But even in that case, he had to stop and figure out what was going on, and if he could fix it by just adding false to the assert. Would that break the code? Had people started depending on the current behavior? And with all the introspection going on...

Razvan noted that he'd always found the ability to have const or immutable on opAssign really odd. He'd raised it in the past, but at the time some people had a use case for it. They weren't modifying any data, but just logging calls to it. He still found it odd and thought that we shouldn't allow immutable or const on there. But people did crazy things so now we had to support it.

Razvan then asked Johan if there was anything we could do to make things easier for him. Obviously, we couldn't put the Weka code base into our testing pipeline. But maybe there would be some way that when we had patches we could automatically send them over, then if Johan had any errors he could reduce a test case. Maybe that would take some of the load off of his shoulders.

Johan said that DMD couldn't compile the Weka code base anyway, though he hadn't tried it recently. He'd been trying to minimize the delta between Weka's LDC and vanilla LDC, and now it was very small. He thought it might no longer be relevant for just testing whether compilation worked. So the first step would be to get very close to the upstream compiler being able to compile it. But not to generate the binaries---there were some LDC-specific things in there that DMD couldn't compile, so just being able to do semantic analysis on it would be great.

He thought they were getting there. Then someone like him could just do a compilation check once a month or maybe once every week. We wouldn't want it on every DMD update, for sure. They didn't have that even on LDC. But it would be nice to have it done earlier. Deciding if it was valid code or not would still take time, but it would be better than the current situation. He didn't check LDC betas against Weka just yet. That might be a good first step.

He said recent template emission changes in both DMD and LDC were an improvement for Weka. They used to need to fix or change LDC to match whatever the Weka build system expected with template emission. That would never work with the upstream tests. It was only valid for Weka and was invalid in general. But now because of these upstream changes in template emissions, they had changed their build system significantly. So maybe now upstream LDC could compile Weka's code base. He was more skeptical about testing the binaries.

Razvan said that DMD didn't get many patches to the backend. Most changes were in the frontend. It would be great if we could test against Weka somehow when making changes to the semantic analysis. Johan said it was good to keep reminding them to do these kinds of tests and not delay them as much as they do now.

Johan added that a big incident they'd had with binaries on a new release was when the object layout had changed due to a change in class layout. The monitor field had been shifted toward the end, and that had an unexpectedly big impact. That kind of thing had been so bad in the past that they had tooling to check it. Every class and structure is hashed by introspecting the binary layout. If anything changes, the build system flags it. At a deeper level, it should flag even when the compiler does something like changing the padding. He said that kind of thing can happen even on an LLVM update, or when changing optimization flags, so it wasn't just the frontend they had to think about.

Bastiaan pointed out that the policy now was we didn't want to break things as frequently, and there had been some reversions. So in incrementally updating, Johan might end up making changes that turned out to be unnecessary. Johan agreed. He said that if they increased the amount of testing, then instead of fixing the code, he'd just report it and then fix it upon a real release in case a change of fix got reverted.

But he clarified that in upgrading from LDC 1.30 to 1.37, he wasn't testing every release in between. There were some places where they were still using 1.24, but in general, the code just had to compile with one compiler version, although compiling with two was very good. Ideally, he'd prefer to be able to make the code changes, test with the old compiler to make sure nothing had changed in terms of performance or whatever, and then update to the new compiler without fixing any code. Then they'd have a little separation between code change and compiler change.

He said that Weka had a script that automatically selected the compiler for you based on whatever git hash you were on. This allowed them to use several versions but always ensure that a given version of the code base was compiled with the version of the compiler it was known to work with.

Addendum

Johan's point about compiler versions reminded Bastiaan of an issue with dub dependencies. Sometimes maintainers pulled packages off of the dub registry, or they'd go inactive and code would stop compiling. He wondered if there was an established way to deal with that for commercial apps. He thought it might be best to make sure you had forks of all those packages. It was nice that you could configure dub to find the dependencies elsewhere.

Mathis Beer said that at some point, dub was changed so that you could query the server using an API, and it then became incompatible with just dumping a copy of the registry in a folder. But you could still configure dub to use a registry that was just file and folder-based, and he thought that's what they were doing at Funkwerk. Then you could write a script that just downloaded all the packages and stashed them in a folder somewhere so you'd always have a backup. Anything that Funkwerk used or installed was backed up to a local server.

Mathias Lang said that they had a monorepo at Symmetry and wanted dub to support that better. So there were probably some changes coming that would make that possible. The last release had made dub fetch project aware so that it would fetch all a project's dependencies. And then using --cache=local with it, it would store them in your project. There were more improvements they wanted to make.

The Next Meetings

Our April Monthly meeting took place the following Friday. Our next quarterly was on Friday, July 5th.

Forums