Permalink
Reply
The D Language Foundation's October 2022 meeting was a quarterly, meaning that several industry representatives attended. It took place via Jitsi Meet on October 7, 2022, at 14:00 UTC. The following people attended (those with DLF next to their names are either D Language Foundation board members, paid employees, or affiliated volunteers):
- Andrei Alexandrescu
- Mathis Beer (Funkwerk)
- Walter Bright (DLF)
- Iain Buclaw (DLF/GDC)
- Ali Çehreli (DLF/Mercedes Benz R & D North America)
- Max Haughton (DLF/Symmetry)
- Martin Kinkelin (DLF/LDC)
- Dennis Korpel (DLF)
- Mario Kröplin (Funkwerk)
- Mathias Lang (DLF/Symmetry)
- Robert Schadek (DLF/Symmetry)
- Bastiaan Veelo (SARC)
Robert
Robert had nothing to report.
Martin
Martin said he was very happy to see the recent work going on with dub (mostly courtesy of Mathias Lang and Jan Jurzitza, a.k.a. Webfreak) and thought it was looking great.
On the LDC side, he was currently working on merging an intermediate step of DRuntime and the D frontend from just before the dmd and druntime repositories were merged. That was the first step toward changing how they fold in DRuntime, D frontend, and DMD test suite changes into LDC.
He reported that Nicholas Wilson had received some outside help for catching up with accumulated changes in LLVM. He summarized some of the changes for us and noted that this was important as it lays the foundation for supporting LLVM 15 and 16, which come with breaking changes. He was not looking forward to doing it himself and was glad that these two had taken care of it.
Regarding ImportC, supporting it wasn't too bad at the moment, but he hoped that it didn't end up requiring too many special cases in the glue layer. Bitfield support, which he had brought up in the previous meeting, was done. Now he had seen some test cases related to taking the address of a struct literal, which in D is allocated as a constant and in C is allocated on the stack. In DMD that was fixed by a glue layer change. It's this sort of thing he hopes remains minimal.
ImportC initializer lists are still in the AST, but another glue layer issue is with initializers of multi-dimensional static arrays in C. The initializer still has some C initializer lists that LDC doesn't support. So he thought those could be translated to D initializers so that we don't have to worry about C initializer lists. Another example he gave involves initializing static arrays with string literals when the literal is shorter in length than the array. This is something else that needs to be accounted for in the glue layer.
Martin is happy as long as the number of corner cases doesn't increase. He hopes we don't see the need for more with each release as ImportC issues are fixed. LDC is a D compiler. Being able to compile C is fine, but he sees the main use case as being able to import C headers to interoperate with C code compiled externally.
I asked him if there was an update about the DMD issue he has often brought up in these meetings in which switching the order of root imports can cause compilation errors. He said he had done a micro adaptation in DMD to fix a specific test case, but it opened a pandora's box of forward referencing issues. The main problem he found was that attribute inference is skipped for aggregate methods when the aggregate itself hasn't been analyzed yet. We need to figure out why. Attribute inference has other problems, so if we want to move to a scenario where attribute inference is enabled for everything, we'll need to work through these bugs.
Bastiaan
Bastiaan reported that SARC has had problems with DMD crashing more often as their code base grows. He so far has been unable to get a reduced case demonstrating why and was unsure if he'll be able to. The good news is that everything compiles fine with LDC, so that's the route they're currently taking. He suggested that it could be that DMD is running out of stack space and that LDC simply allocates more of it. Martin said that he had recently increased the stack limit from 8 MB to 16 MB and that could indeed be the issue. (The forthcoming 2.101.0 release of DMD includes the same stack limit increase.)
Mathias Lang
In the previous meeting, Mathias received the green light from Walter to move forward with a breaking change to Throwable's internal TraceInfo interface. Following that, he had begun adding colors in the stack trace because he thought they were unreadable, and adding colors was not that much work. He had a proof-of-concept but still had a few things to work out.
He also updated us about Buildkite issues. It had been completely broken for a few days before the meeting. DLang Bot had gone down around October 1st, so nothing was being scheduled and Buildkite just hadn't been working. He had gotten it back on track the morning of the meeting. It was running off of his server, so it was a bit slower than before, but it works. And now we can update it. The problem before was that the list of dependencies was static: we had no control over it. For example, we had been stuck with LDC 1.29 because we couldn't update LLVM.
He thinks that Buildkite is one of the best CIs we have because it tests other people's code in practice and catches a lot of regressions. The direction forward is to have something that integrates with GitHub Actions. These days, pretty much everyone has Actions in their GitHub repositories, so if we could use their action definitions as the base for testing, that would be amazing. It's not trivial, but he thinks it's doable.
Dennis
Dennis had begun implementing DIP1035, "@system Variables" and requested that Walter look it over. Walter has since merged the PR, and Dennis has moved on to implementing the named parameters DIP (see PR #14475 and PR #14575).
He had also stumbled upon the attribute inference cycle problem, where checking if a function call is safe might result in a pessimistic assumption that it's unsafe. He's looked into fixing it and he thinks it's possible to, in those cases, optimistically assume the function is safe. He hadn't yet seen a case where that breaks, but he wants to ensure that switching from the pessimistic approach to the optimistic approach doesn't create a big safety hole.
He then brought up pragma startAddress. Iain had found a handful of uses of it in D GitHub projects, and it's only supported for the 32-bit Windows OMF target (see the discussion in this PR thread). It's not a burden to maintain, but given that one of the goals in the Vision Document is to simplify the language, this seems like a good candidate for deprecation and removal. Walter recalled that people often used it in 32-bit builds, and it's a supported extension in all the x86 C compilers. Mathias didn't think getting rid of it would gain us anything, and Walter agreed.
So then Dennis asked if we should spend time fixing bugs with this feature. Martin and Mathias said no. Walter suggested making sure there are issues for them in Bugzilla, but just marking them as low priority. That way they're on record if they become a priority for someone in the future.
Iain
For the benefit of the industry folks who only join us in the quarterlies, Iain recapped some of the things he'd covered in the two previous monthly meetings, such as the move to Backblaze for downloads.dlang.org and docharchives.dlang.io, the transition to Cloudflare that gives us the benefit of free data transfer with Backblaze, and the general tidying up of the dlang.io namespace.
He then said that Martin Nowak will not be doing any more releases of the D language. Iain had been going through the build scripts that we currently have and figuring out what needs to be done to tailor them to run in GitHub Actions. It was taking much longer than he had anticipated. In the interim, he thought he could at least merge master into stable and get some 2.101.0 alpha builds set up by hand. (He then announced the first beta on October 17.)
Next, he gave us a summary of his experience at the GNU Cauldron where he attended a meetup of GCC/GDB maintainers. He reported that there are some really interesting things going on with GCC internals regarding the direction in which they're taking the compiler, including several things we're doing already. For example, they're adding options to automatically initialize all static variables to 0 or a bitmask. This sort of thing is good news for him, as he currently has to do all the memsets by hand in GDC. It's a win if the middle-end can do this for him. If you're interested in all the GCC internal changes that Iain was gushing about and how he can benefit from them in GDC, please ask him :-)
Mario
Mario said he has no problems with D because he can hand them all over to Mathis. So he deferred to let Mathis talk.
Mathis Beer
Mathis said the state of the compiler for Funkwerk was pretty good right now. He was unaware of any crippling compiler issues, though as always they'd been pushing into the dark corners to look for them via his experimental work.
Now that they have librebindable as a backstop (see his DConf presentation about it if you haven't already), they'd been trying to do a lot of immutable stuff. Before, there were things they couldn't do with immutable that librebindable now enables. So now they're doing things like working with internal data types that had previously been "pseudo rvalue structs" and making them into fully immutable structs just to see where that breaks.
He's found it's broken with associative arrays. Issue #22244 summarizes the problem. This causes them to have to use a lot of casts, which he sees as a code smell.
If you're using associative arrays with ranges, you need assocArray to work, and it doesn't work with immutable values. He'd been trying to think of how to make it work so he could submit a PR to Phobos, but he had been unable to see a way forward short of pushing librebdindable into Phobos. In other words, how to allow assocArray to mutate the array, but prevent the array from ever being mutated again and disallow mutable views of it? He was looking for feedback.
Robert suggested this should be a "WONTFIX" because fixing it would create a hole. He doesn't think it's technically possible without having some feature that tracks the variable. Walter agrees.
Mathis said this raises the question of how immutable and associative arrays are ever supposed to work together. Robert thinks they won't, and suggested the best bet is to roll your own hashmap and move on with your life. Mathias Lang noted the same problem exists with const.
Mathis said Funkwerk have their own internal hashmap type, and they can work around the problem fine. He just feels like it's fighting the language a bit and brought it up because he wondered what would have to be relaxed or changed to make that less painful.
Robert mentioned head const, but insisted that we shouldn't go there. Andrei suggested that we need head const for C and C++ compatibility anyway. Walter said that ImportC doesn't support head const. His experience with C is that people generally mean const in C to be transitive. So in ImportC, he turned the head const cases into transitive const and it appears to work well.
Mathis said what we really need is head mutable. Walter said that's fine, it's head immutable that's not going to work. He's thought many times about adding head const, but it would be disruptive to D's type system, and that would likely introduce a long list of compiler bugs. I saw several heads nodding in agreement.
There was more discussion about the topic, but Walter didn't have an answer right now.
Max
Max was having mic issues throughout the meeting, so he didn't make a report.
Ali
Ali had nothing work-related, but he gave us a summary of the talk he gave at Northeastern University and the meetup they had after.
Other than that he said he had been working on his cached algorithm.
Me
I gave everyone an update on my progress editing the DConf videos (I was a little over halfway through Mike Shah's talk at the time), and that I would soon be publishing a conversation Walter and I had recorded.
I reported that we had raised a little over $26.00 at that point via the YouTube partner program. (Now I can report that in the month since we are closer to an estimated $40.00. Ads are currently disabled until we sort out issues with identity verification, but once they're enabled again I look forward to seeing that dollar value climb as we push out more content. I don't think there's any easier way to support D financially than by sitting through a few YouTube ads when watching our videos, so please help us out!).
I said that I'd been so focused on YouTube these days that I'd practically forgotten we have a blog. I was working with Ate Eskola to prepare his subsequently published second blog post on DIP1000, but I had no time to create blog content myself for these.
I then reported that Dennis had been preparing some YouTube tutorials for our channel and that I hoped to start publishing them by the end of the month. (The tutorial series is focused on contributing to DMD. Each video will be short and focused on a narrow topic. I reviewed the first video in the series a few days ago.)
Next, I informed everyone that I'd spoken with Dennis and Razvan about implementing DIPs that have been approved but not yet implemented (one result of which you can see above in my summary of what Dennis said, as well as the project board they started for it). A loose, though likely unrealistic goal, is to get them all done by the end of the year.
Finally, I reported that Max was focused on finishing up floating point emulation. (I heard from him last week that a PR was imminent.)
Andrei
Andrei opened with a request for financial support. It became clear recently how dependent we are upon Symmetry for funding. There's a short tail of small contributions from individuals, and there have been occasional donations from other companies, but without Symmetry we're stuck. So Andrei said he was coming "with his treasury hat in hand" to ask the industry reps to go back to their employers and ask about some kind of sponsorship. (I intend to do a write-up here in the forums about donations and funding sometime in the next couple of months.)
Next, he brought up the work that Walter and Dennis had been doing to squash DIP1000 bugs. He was curious if the industry folks were looking for a safer D language. He also thought we should consider a rebranding of "DIP1000" for the people outside of the small part of the D community who understands what that means. This prompted a discussion about the relationship between DIP1000 and @live, the number of Walter's DIP1000 PRs that are hung up on Buildkite failures, head const, C and C++ compatibility, Rust's marketing success, and more.
Some highlights:
- Mathis Beer's "completely personal opinion" as to why he isn't pushing to use
@liveat Funkwerk: he feels Rust has created a false impression that only liveness and borrow checking can guarantee safe code, but he believes that safe code is also guaranteed by copying, or trusting in a reference counter or a garbage collector. He says the borrow checker enables a tradeoff between performance and safety. That's important for people who chase new languages for performance, but all of Funkwerk's performance gains are algorithmic. They run on servers with 80GB of memory, so it's not the sort of micro-optimization where they need safety at the moment. That said, he thinks it's a good idea from a language perspective, a user perspective, and a marketing perspective - Walter said that with a GC, you don't need a borrow checker, but having one allows you to guarantee no leaks when you aren't using the GC. So although these problems should be rare in D, they can happen. And he'd like to close that hole. Rust has successfully solved the idea of 100% memory safety, and to compete we should do that, too.
- Andrei talked about how Rust has very effectively managed to sell memory safety as the be-all-end-all of good system software, citing as one example a tweet by Microsoft Azure's CTO Mark Russinovich saying that it's time to give up C and C++ in favor of Rust. Since we're working on
@livealready, why not capitalize on their marketing? - Bastiaan said that SARC is interested in safety in principle, but in practice not for a long while. With their code base being transcompiled to D from Pascal, they rely on a lot of unsafe hacks. Removing those hacks and implementing
@safeis many milestones away if they ever get to it.
Among all of this, there was also a discussion about D's opt-in approach with @live vs. a full program borrow checker. Dennis paraphrased previous forum comments from Adam Ruppe and Timon Gehr making the argument that you can't have 100% memory safety and an opt-in @live. Walter said Rust has unsafe code, too, and if you really want to you can write an entire program in Rust with unsafe code and disable the borrow checker. He thinks it's possible to have the borrow checker guarantees in @live functions and be fine with that.
Martin thinks we should be cautious in talking about 100% memory safety because of compiler issues that keep surfacing. Walter stressed that this is why the test suite is important. Every bug that is fixed goes into the test suite, and that's like a ratchet that only moves forward. Mathis Beer noted that the ratchet only goes one way if the compiler stands still; every new feature brings a multiplicative increase in the potential for corner cases. Walter agrees that's a potential problem.
Walter
Walter asked me to ensure that we have our own copies of all of the videos we upload to YouTube. I said that I'd been using Google Drive through our dlang.org Workspace account to store the DConf videos, but I hadn't copied other things there yet. I'm going to look into whether our Backblaze account will end up being cheaper.
Next, he said he had been going through the bug list for ImportC. He started with the worst bugs and then worked his way down. There were still some PRs stuck because of Buildkite. He thinks we're in good shape with ImportC, is pleased with it, and believes it gives us a competitive advantage. Once those are down to a "dull roar", he'll go back to DIP1000 bugs. When those are at a dull roar, he plans to get back on @live.
One thing Walter would like to see from the community is people doing presentations about D outside of DConf. Everyone who has presented at DConf has something in the can for other conferences. Walter has presented at three programming conferences in the last year. Most of them will pay for your travel if you're speaking there. You've already got the presentation ready to go, so why not submit it?
He's thankful for the people who have joined him in defending D on Hackernews. He encourages those who read sites like HN and Reddit to politely and non-confrontationally point out D's strengths when the opportunity arises. Basing other languages turns people off, but you don't need to compare to other languages. Just say, "this is how we do it in D" without mentioning the other language. That's an effective form of evangelism that we need more of.
Ali said that he had talked to someone recently who, upon learning that Ali uses D, said he had become interested in D because of Walter's comments on Hacker News. He was impressed with Walter's demeanor and always followed his posts. Ali admitted that he has a hard time keeping his cool like Walter does when reading some of the comments he sees out there.
Walter talked about an embarrassing situation that had recently cropped up. Someone was having an issue with DMD crashing in random ways when it allocated. Walter was sure he was always checking for overflow when calling malloc, then he and Dennis grepped the code base and found a lot of allocations in DMD that did not check for overflow. He had just put out several PRs to fix that, so now when DMD runs out of memory it will always print an error message saying so rather than producing some other kind of random failure.
The next meeting
The meeting closed out with some last-minute discussions about DConf Online (I'll post the schedule later this week) and Iain's experience with the DMD release process. We wrapped up at almost the one-hour-and-twenty-minute mark.
The next meeting is happening on Friday, November 4, at 14:00 UTC. This is a regular monthly meeting. If you have anything you'd like to bring to us, please let me know and I'll see about getting you into either this meeting or the next one, depending on your schedule.
