Permalink
Reply
We held the monthly meeting for April 2023, on Friday the 14th at 14:00 UTC. It lasted just short of an hour and a half. We'd had a quarterly meeting the previous week. This was the first time wev'e held a monthly meeting in the month of a quarterly. Previously, we'd combined them, but decided in our March meeting to split them up.
The following people attended:
- Walter Bright
- Martin Kinkelin
- Dennis Korpel
- Mathias Lang
- Átila Neves
- Razvan Nitu
- Mike Parker
- Robert Schadek
The summary
Me
I opened the meeting with an update on DConf preparations. Most of it was boring details. Other than that, the venue contract had been signed and a down payment sent a couple of weeks before, and Symmetry had reached an agreement with our event planner on the terms and conditions of their contract. I was waiting on word that it was signed (I have since received it).
I noted that Sönke Ludwig had contacted me before I left for my trip in late March. He was doing some maintenance work on codemirror.dlang.org and asked if I wanted to redirect the URL to his new VPS or if he should move it onto a DLF server. Since this is one of the services we need to move over eventually anyway, I asked him to go ahead and do it. Vladimir Panteleev had already set up the documentation tester on a DLF server, so I reached out to him to see if we could move codemirror there or if we needed a new one. After discussing it with Sönke, he said we had enough resources for both services. So I set up Sönke with an account, then he and Vladimir got it up and running while I was on my trip.
Dennis
CI failures
Dennis started by saying that the CI was randomly failing again. He didn't have a Mac, so he'd been unable to chase down the problem. Random CI failures are a recurring problem. There are so many checks, and he doesn't know who created them or who knows exactly what the checks are doing. He wishes the tests had someone responsible for them who he could turn to when they fail.
Walter asked who had previously been in charge of the tests. Razvan said he didn't recall if one person was ever in charge of them. At some point, someone decided it was a good idea to have a particular test and it got added to the pipeline.
Dennis asked if we should only keep tests that have a maintainer. Martin and Mathias quickly rejected that. Martin said the tests are good. CI failures are usually caused by CI image bumps or a PR. CI image changes are a PITA for LDC's tests, and PR-related failures may not be easy to resolve, but failures are hardly ever the fault of a test. And there's never been any specific person responsible for any of DMD's CI systems. They just grew organically. Then someone who knew the details moved on and no one else knows them... it's a constant maintenance burden, but it's worth the effort.
There was a bit more discussion about the maintenance burden, after which I noted that this is the story of our ecosystem. We're responsible now for things none of us set up, and we need to get a handle on it all. Dennis agreed and added that the CI is in a special position. When one of them is outdated, it doesn't just sit there out of the way, it becomes an annoyance to development.
(NOTE: This is one of the many aspects of our ecosystem that we'll be working to improve under our new workflow.)
The -J compiler switch
Next, Dennis said that, as he understood it, -J was added as a measure against string imports being used as a malware vector. As he sees it, string imports are no more risky than normal imports. Why can't they be handled the same way? Walter said the concern was that they could be used to read any file from the filesystem and bake it into the binary, and that could be used to steal data from the user's system. With normal imports, the compiler only reads D source files. It can't import arbitrary data. He then talked about some examples. What it comes down to is that there are so many clever ways to implement malware, he just doesn't want to open that door wide to the possibility.
Martin talked a bit about how Symmetry uses -J in their codebase, and Átila told us how he used it a lot with reggae. Dennis proposed that any directory given to -J be treated as a root, and its subdirectories added to the search path. Martin and Átila both said that the compiler already allows string imports from subdirectories of any -J directory (when the full path is included in the import expression).
The DMD backend
Dennis said that because the DMD backend was shared with the Digital Mars C and C++ compiler (DMC), there was a rule in place that changes to the backend couldn't break DMC, but there were no checks for it. Now that ImportC is a thing, do we still need to maintain DMD's backend with an eye toward not breaking DMC, or can we say that DMD's backend is only for DMD and just leave DMC frozen in time?
Walter said he had given up on maintaining DMC. The DMD backend had already diverged, and he saw no further point in maintaining DMC. Dennis said that will make maintaining the backend easier, and we could remove the notes in the source about being careful because of DMC. Walter agreed.
Dennis said that when building DMD, the backend is using some translated header files where it's using extern declarations instead of D imports. Those should be changed to more modern D instead of this archaic header business. Walter agreed. He said there's a lot of old-fashioned stuff in there. He hadn't taken advantage of the separation to do any big refactoring but had stopped backporting changes to the DMC backend a while ago.
Dennis next noted there were a bunch of version(Mars) and version(scpp) blocks, and asked if they could be changed to version(all) and version(none), and the latter then removed. Walter said he's not in a rush to do a big refactoring, but yes, that's fine. He hadn't worked on the backend for a while other than for bug fixes and had gradually changed some things to a D style in the process. He gave a recent example involving some changes he made when he fixed an inliner bug.
Walter said ImportC will probably never replace DMC, mainly because DMC also compiles C++ code. Dennis added that DMC also still has 16-bit support for the niche programmer.
(UPDATE: Dennis has since submitted several PRs removing function prototypes from the backend, starting with pt1.)
Mathias
Mathias said he had nothing to report. Walter asked what he had been working on. He said he hadn't been working on any D stuff lately. He still had an open PR that fixes a CTFE bug related to -preiview=in. In the PR thread, Martin had pointed out an issue with it regarding the evaluation order of argument expressions, and then the two had discussed it further outside of the PR thread, where Martin explained how the order of destruction in the presence of exceptions needs to be taken care of. That led Mathias down a rabbit hole, as it seems there are cases where destructors aren't called in CTFE, so the PR was stalled (and it's still open as I write this). He plans to finish it up, as his goal is to make it possible for -preview=in to be enabled by default.
Robert
Robert said he was going to ping me the following week about finding some time to get the Bugzilla to GitHub migration. Other than that, the compiler was still too slow.
(NOTE: He eventually emailed me everything I need to get the Bugzilla to GitHub migration done. Now I'm the bottleneck. I'm behind on everything right now, but I'm slowly catching up. When I work my way to this task, I'm first going to test it out on a private repo to make sure I've got everything in hand. Then I'll make an announcement here in the forums with a date on which I plan to start. When that date comes around, I'll open a forum thread with periodic updates, e.g., "starting the migration of tools issues now", and so on. We're almost there!)
Razvan
Fixing issues
Razvan reported that he had mostly been fixing segfaults in the compiler and had found that there were some ICEs in the C++ mangler. He'd submitted some PRs to fix those. There was one in particular for which he wasn't sure if the fix was the right one. He asked Mathias to take a look at it as the original author.
Razvan had also been fixing some nosharedaccess issues. He had seen a regression that Átila had reported, but other than that there were no other open nosharedaccess issues. He expected there would be more bugs out there, and Átila expected he'd find some.
BetterC linker errors
Next, Razvan said he'd been looking into BetterC errors. He then described an issue related to template instantiations in CTFE. When a template includes runtime hooks, it's no problem in BetterC if it's only used for CTFE and there's no code generation, but sometimes the same instantiation gets used at runtime, the front end doesn't catch it, and it results in linker errors. The problem was uncovered by a fix from Walter for an issue related to @nogc functions in BetterC, and that PR also introduced a regression.
Razvan then described a related BetterC problem. There are some cases where the compiler tries to semantically analyze the body of a generated function and marks the function as disabled if something fails. He then described a chain of consequences that can arise in BetterC in which the generated function ends up instantiating a template that uses a DRuntime hook, and that instantiation ends up causing linker errors even though the generated function that instantiated it was disabled.
He said the fundamental problem is that the compiler currently has no way to reanalyze template instances once they've been cached. He's not sure how best to solve this. He said he doesn't like the way the compiler is going with special cases: if you're in ImportC do this, if you're in BetterC do that, if you're in CTFE do this other thing. Átila and Martin nodded in agreement. Martin said that's his main problem with BetterC. It's full of hacks in the front end and the glue layer.
Martin said he'd seen many missing symbol errors in BetterC. The majority of them were related to template culling, which is a compiler optimization. He described how it works, and noted that it doesn't happen when using the -allinst switch. That's the first thing he suggests trying when linker errors arise in BetterC. When -allinst works, then the problem is probably just template culling.
Regarding reanalyzing template instances, Martin said the compiler does try to do it. That's the cause for a funny issue where you have static if(!__traits(compiles, something) { // do the something that failed the compiles test } and it only works with the static if(!_traits(compiles, ...). When analysis of template instantiation fails and the compiler comes across a second instantiation, it tries again and again, and sometimes it works due to forward referencing issues (which he calls the root of all evil and says is the main problem we have with the frontend).
Átila agreed that the special case stuff is something we shouldn't be doing. He thinks we should get to the point where BetterC is a no-op. If your code doesn't require the runtime, then it just shouldn't be linked. You shouldn't have to do anything special to get that behavior. The only thing the -betterC switch should do is tell the compiler to analyze your code and tell you if something won't work in BetterC. Nothing else would change.
Walter said the root of the problem is trying to use CTFE in BetterC. CTFE uses the GC to allocate and is crippled without it. Átila said that should be possible in BetterC, and Razvan agreed. Walter said the problem there is: how can the compiler know that the function is only used in CTFE?
This took the discussion down into the weeds, and I'm not going to attempt to summarize all of it. Eventually, Martin said the root problem is that the BetterC checks (for TypeInfo, ModuleInfo, etc.) are done in the front end rather than in the glue layer, and that's a problem for CTFE. If they were done in the glue layer, it would be a PITA in that each compiler implementation would have to do it then, but at least then you'd only get issues if that thing is going to be codegenned.
Dennis asked if we should have a separate BetterC pass that does the checks between Semantic3 and codegen. Martin said that's an option at least. Átila reiterated BetterC should be a no-op. Martin agreed that's a great long-term goal, but before we get there we need to get rid of the module and type infos.
Átila took us back to the idea of indicating a function is CTFE-only. Rather than add a new UDA, why not just use an in(__ctfe) contract? Then it's easier for the compiler to analyze, as there's no need to look at the body of the function. Walter said he'd used if(__ctfe} { // code } else assert(0). Átila thinks the contract is a lot prettier than that, and Walter agreed, and it does avoid the need for new syntax.
Átila said it also makes sense in that it declares the function contract as being "I only work in CTFE, so don't bother calling me otherwise". It's not even a hack. Dennis thought it is a bit of a hack because you can still take the address of it. You have to treat it as a static condition, but __ctfe is a runtime variable. Regardless, Walter thought it was a very good idea worth exploring.
Martin suggested using if(__ctfe) as a template constraint instead of an in contract. Dennis liked that idea, but Walter said he didn't want to start applying template constraints to ordinary functions. That would be new syntax, and that's what the in(__ctfe) was avoiding. Átila said it implies a template function and probably isn't what we want.
More on BetterC
Razvan next brought up Teodor Dutu's work on converting DRuntime hooks to templates. An intended side effect of this effort is that it should make things more usable in BetterC, but sometimes, getting real work done in one of these templates means calling into the C library (memset, memcpy, etc). This is an issue when there's no standard C library.
Walter said BetterC was never intended to be used without a standard C library, so relying on the presence of those functions is fine.
This prompted more discussion about linker issues with BetterC and how to alleviate them. At the end of which Martin said BetterC is a sledgehammer, and that's exactly why it has the special case code Razvan mentioned before. We need to be working to cut that down.
Walter said we should look into redesigning features so that they don't need TypeInfo. Átila thinks TypeInfo should be opt-in anyway. There are ways to use compile-time reflection to generate all of that for us. Martin said the new templated hooks help in that regard (though he later admitted that they were a bit of a PITA causing some extra work for the GC-to-stack optimization in LDC, which he went into some detail about). The big thing hindering it is associative arrays. They heavily use TypeInfo. He mentioned Steven Schveighoffer's implementation which, as far as he knows, is free of TypeInfo. Maybe eventually we can get to use that.
Martin
LDC
Martin said he was swamped with LDC-related work, with no time to work on the front end. He still had a lot of catching up to do with LLVM versions and the front-end version. LDC wasn't yet at D 2.103. The tests were looking quite good, but the previous release had some problems. LLVM 15 brought some nasty surprises that had only been caught after the final release of LDC 1.32, so he needed to put out a point release (which he did a few days later).
He had noticed that the LDC beta releases didn't seem to be used as much as they used to be. They used to see 1000 downloads of beta releases in the first week, but with the most recent they'd had only 300 or 400 in the first two weeks. It could be because we have more frequent LDC releases now, so people just don't bother with the beta anymore. He was thinking about doing some announcements about how maintainers of dub packages can contribute to compiler stability. For example, in GitHub actions, there's the "latest beta" tag for LDC (he says it's a misnomer, as it always refers to the most recent release, whether it's a final or a beta). If people set up scheduled builds using that tag every week or so and report issues when they happen, that could improve the situation.
DIP1000 issues
Martin said they were still using D 2.100 at Symmetry because the 2.101 release came with lots of scope-related deprecations, which are enabled even if you aren't compiling with -preview=dip1000 (they're errors with the preview, and without it they're deprecations). In the beginning, there were about 1000 deprecations in their main project, which consists of around 150 dub packages overall. At the time of the meeting, they reduced the count to 200 by hunting them down and changing the code. Mostly, the deprecations were in third-party packages.
They had also encountered an issue related to using std.algorithm.all with something else (he couldn't remember). It prevents them from looping over an array of structs that have opEquals in @safe code, and there's nothing they can do about it on the user side. He said it might be a Phobos bug, but he suspects it's a compiler issue regarding attribute inference. He'd not yet had time to reduce it and file an issue.
Also, he had worked on fixing deprecations in the concurrency dub package. He got those fixed for a library using that package, but then trying to compile the concurrency package's unit tests segfaulted the compiler. This happened in both the D 2.101 and 2.102 versions of DMD and LDC. Again, he'd not yet had time to reduce it to file an issue. Dennis offered to take a look at it.
Átila said the main takeaway here is that dealing with deprecations is time-consuming, especially when they're in third-party code. And in this case, trying to find out why the deprecated code would be inferred as @system instead of @safe isn't always trivial. Maybe we should rethink DIP1000. Symmetry, at least, is devoting resources to it, but most people would just give up.
Dennis said he preferred using return scope as little as possible and just letting the compiler infer it. It's annoying opening BuildKite projects to fix related issues when someone got return scope wrong because the compiler was wrong.
(There was more discussion about the value of @safe, and how the GC and tools like ASAN can mitigate the need for it.)
DIP1008 issue
Next, one of Symmetry's projects uses DIP1008. It makes sense there, as it's a project that interacts with Dot Net code, which also has a GC, and you're not able to reliably suspend one GC when the other one is active, so they're trying to avoid the D GC in most cases. They still need to throw exceptions, so that's why they're using -preview=dip1008. Now they want to statically link that project into the main project. The problem then is that all of the dub projects pulled in (around 300) will be compiled with it. That leads to garbled exception messages, as non-DIP1008-aware code doesn't prevent the exceptions from escaping their scope.
Martin proposed coming up with an alternative DIP1008 implementation where the only change is that nogc checks are disabled for throw expressions. He said the team told him they'd be okay with leaving garbage lying around resulting from exceptions. He thought it would be pretty easy to get that done. He suggested implementing it as an experiment, then upstreaming it if it works with their codebase. It would mean not relying on malloc and reference counting, just allocating from the GC and never freeing it. The GC is disabled in that project anyway.
Walter said he's kind of sorry that D has exceptions at all. They're an unending source of problems. Using option types would have been a better design choice in hindsight, but at the time he was just following what C++ and Java did. One thing we might do is to remove exceptions from Phobos so that you can reliably use it without exceptions at all. Another thing is to continue minimizing the use of the GC in Phobos.
Dennis said that there are places in Phobos where exceptions are just ingrained in the API. He cited ranges specifically, where you can just decode text and get a UTFException, or parsing numbers could throw a ConvException. To change that would require all users to unwrap the integers, unwrap the codepoints, etc. Átila said we could use an alias this that behaves as it does now and if the user forgets to unwrap, you throw an exception then. Dennis said that was an interesting idea.
Walter gave the example of using std.algorithm.chain in std.path to avoid allocating memory when concatenating paths and returning the chain thing. He's wondering if there are other opportunities like that in Phobos that can help us avoid memory allocations and exceptions, but he hasn't gotten that far in looking into how to eliminate exceptions. Átila said he'd been thinking about copying Herb Sutter's idea for zero-overhead C++ exceptions. He said the basic idea is that you get rid of exceptions while keeping the syntax. (Here's a link to the PDF).
Átila
Átila had to leave right after the discussion above on exceptions.
Walter
Walter had been continuing his pause on implementing new features and had been going through the ImportC bug list. The biggest problem he'd run into is the pervasiveness of compiler extensions in system header files, and it's almost always unnecessary. He then went on an entertaining rant about it (a header will compile on FreeBSD but not macOS, then a different header succeeds on macOS and not FreeBSD; Microsoft is the worst offender; etc.). ImportC was intended to be a standard C compiler, but you can't compile any system headers with a standard C compiler. He ended up implementing a lot of extensions to get things to work.
On the plus side, he'd made a lot of headway with it. For example, he'd figured out how to implement statement expressions with just a few lines of code. He was shocked it was possible with only a few lines and was happy with the results. So he's making good progress with getting ImportC to compile all these random .h files. That and documenting this stuff was mostly what he'd been working on. He hoped to get the ImportC bug list down soon and turn to other bugs. (He posted about this in the forums a couple of weeks later with a call for help finding problematic C headers.)
He then reiterated he'd like to reduce GC and exceptions in Phobos and eliminate dependencies such as TypeInfo. The further we can go with that the better.
Next, he said he'd stepped up his use of Twitter. He encouraged us, and anyone with a Twitter account, to tweet more about what we're doing with D. He'd looked into common hashtags with programming posts and started using them more, and he also signed up for the blue checkmark. He has since noticed an increase in followers. If Twitter's not your thing, then consider posting on whatever social media platform you prefer. This is our most effective means of marketing. He and Andrei had tried various approaches to publishing ads via Google, Facebook, and other platforms in the past with no meaningful results whatsoever. What he had found effective was writing articles and just posting on social media about what you're doing. Just talk about the work you do with D.
I noted we had discussed before the idea of Walter either doing an occasional live-coding session or just recording himself working throughout the week and editing together a video summary of the work he'd done. Walter brought up what John Carmack did back in the day with his .plan files to keep gamers informed of the work he was doing. As Walter sees it, that was an early version of what Twitter has become. It's a great way to let your customers know what you're doing, and because they make a conscious decision to follow you, that's not considered spam. He decided to keep his tweets strictly related to programming topics. He isn't going to tweet about politics or what he had for breakfast. He'd followed several programmers because he wanted their opinions about programming, only to be put off when they tweet about everything else. He cited John Carmack again as an example of someone who tweets mostly about programming topics. He said that if you wanted to use Twitter for outreach, just keep it on topics related to programming.
I mentioned that I'd like to record some more conversations like the two I did with Walter last year (Part One and Part Two). I have a list of people I plan to contact for that.
Razvan doesn't use Twitter, but he's interested in what Walter has to say about what he's working on. He suggested that Walter post updates on the forums for people like him. Walter said he could do more of that and that's a good idea.
Walter finished by saying that he has so much work to do and it keeps growing all the time. He said he appreciated Razvan and Dennis helping out with his pull requests. They'd been a big help.
The next meeting and more
The next meeting was a monthly that we held on May 5 at 14:00 UTC just before our final IVY program session with UCora. I'll try to have the summary for that ready before the end of this month. Then on May 12, we held the first of what will be weekly planning meetings. I'll write more about that in a separate post.
