This summary is quite a bit overdue. Sorry for the delay.
The July 8 D Language Foundation meeting was one of our quarterly meetings. In the first part of these meetings, representatives from industry join us to provide us with updates, notify us of issues they're experiencing, and provide us with feedback and ideas. The second part is the typical monthly meeting addressing foundation business. The industry reps are always invited to stick around for it and usually do.
For this meeting, we had two new faces join us: Paul and Robert Toth of Ucora. They let us know back in June that the company has been using D for years. We invited them to join us for our quarterlies, and they accepted. We hope we can help resolve any D issues they may face, and we welcome their feedback on the D user experience.
The meeting took place on July 8 at 14:00 UTC. The following people attended:
Walter Bright (DLF)
Iain Buclaw (DLF/GDC)
Ali Çehreli (DLF/Mercedes Benz R & D North America)
Max Haughton (DLF/Symmetry)
Martin Kinkelin (DLF/LDC)
Dennis Korpel (DLF)
Mario Kröplin (Funkwerk)
Mathias Lang (DLF/Symmetry)
Razvan Nitu (DLF)
Robert Schadek (Symmetry)
Paul Toth (Ucora)
Robert Toth (Ucora)
Bastiaan Veelo (SARC)
Joseph Rushton Wakeling (Frequenz)
Funkwerk had a problem with Issue #23234. There was a little discussion about the issue, and that Funkwerk has a workaround for now, but Razvan put it on his list of priority issues and subsequently resolved it.
Mario said they sometimes run into issues like this when they update the compiler after a long time, or when Mathis when they try something weird (i.e., when Mathis has ideas and they find new areas in the compiler that don't work as expected). On the other hand, they've been using D for well over a decade. They have their own set of libraries and the solutions to their problems are usually straightforward. D has been a big help in their work.
SARC has marked a major milestone in that their 500KLOC Extended Pascal codebase has been completely transcompiled to D (if you aren't aware of this project, you might enjoy Bastiaan's talks from DConf 2017, Extending Pegged to Parse Another Programming Language, and DConf 2019, Transcompilation into D). Now they are increasing their testing and working out other steps they need to take before going into production.
One of those steps is internationalization. For that, they're using GNU getext, and a couple of weeks before the meeting had released their D package for interfacing it on dub. He said they would soon be releasing a package for string extraction, and that with that we'd have comprehensive support for gettext in D. He said everything is looking good, though they do have a couple of nasty bugs for which he hopes to find a solution.
Here I can quote Robert in full:>
Not much to report. The compiler's too slow. Other than that, D is the best language.
And I can quote Joe in full following Robert:>
What he said I think. Nothing to add from my side.
Paul & Robert T.
Robert noted that one of their new programmers has had a hard time getting up to speed with D via the existing documentation.
Ucora have been working with D for about 7 years, and their primary product is written in D, so they are heavily invested. Paul's interest is in the accessibility of D to the wider programming community. He was pleased with the response to the post he made in June with a suggestion for the D documentation. Being new to the community, for now he's just observing, but they are interested in helping out in any way they can.
Bastiaan noted that in Paul's forum thread he requested that Ucora be added to the Orgs Using D page. Max had opened a PR, but then got busy with other things (and has since been unable to build the website). The PR hasn't yet been merged, but I'll see about making that happen as soon as I publish this summary.
Paul said that for his own purposes, the documentation is well-written and he can find what he needs. But there is an initial hurdle to get over to be able to use the documentation effectively. New programmers rely on a programming language's manual when learning the language, so he's interested in ways of making D's documentation more accessible for new users. Enabling comments on documentation pages was the only concrete suggestion he had now. Walter suggested this could be done by creating a forum thread for each page in the documentation where people can post comments.
I noted that we have previously discussed overhauling the website and that a complete evaluation of the UX should be part of that. This is something that will fall under the purview of the ecosystem management team when it's in place.
We then discussed the topic of the user experience of documentation which touched on the experience with programmers at Ucora (in using the documentation and why they aren't using some of the more powerful features of D), comments on PHP doc pages, the D Tour, different kinds of documentation (reference vs. tutorial vs. how-to guides), Ali's book, ways to delineate them, and more.
Some good ideas were brought forward. Overhauling dlang.org and the D UX as a whole is something we want to get done as soon as we can after we get our ecosystem services sorted.
Mathias had a few things to bring up.
The first was Issue 23164, which he encountered when preparing his DConf talk. In short, it looks like dmd is moving structs that have copy constructors. This has resulted in portions of our
core.stdcpp being disabled on different platforms. But LDC and GDC work as expected. Walter said the first problem is that the example has an internal pointer, which is noted in the D spec as something that shouldn't be done. He said the second thing is he doesn't think dmd is moving structs, even though the spec says it can. He thinks what's happening is that one of the copy constructors is passing an rvalue, causing a copy onto the stack and not updating the internal pointers. But Mathias said it's working as expected with LDC and GDC, and Andrei had said in the past prohibiting internal struct pointers is untenable. (This has all since been noted in the Bugzilla issue).
Ultimately, Walter suggested the example be further reduced to determine exactly where the problem is. There's too much going on with it right now. Mathias said he would try to do that.
Iain noted that GDC's behavior can be attributed to an internal flag attached to non-trivial structs that prevent it from being copied around, the compiler will crash instead. That means this kind of struct is always passed and returned by reference.
The second thing Mathias brought up is that he had been working on improvements to dub to support more use cases and output better error messages. He encountered a blocker when he had to enable DIP 1000 on some code and ended up with memory corruption. He hadn't yet submitted an issue for it at the time (and I don't know if he has submitted one since).
The third thing was the merger of the DMD and DRuntime repositories into a monorepo, which was something he and others had scheduled for the next day. Mathias warned that there might be some disruptions in CI and such as a result. He asked if anyone had any objections to the move. No one did. The merger was subsequently carried out. It did cause a few hiccups, but Mathias, Iain, and whomever else was involved handled them all (as far as I know).
Next, he said that we have a problem with some preview/revert switches (and it arose specifically with
-preview=in). DRuntime and Phobos need to be compiled with a certain set of flags. If the flags used when compiling user code don't match those flags, we have a problem. We want a period where the user may or may not be using a certain preview, but if that preview changes the ABI, it doesn't work. One idea he had for this is a new
pragma that could be put at the top of a module to specify which flags it needs to be compiled with, but it's not very elegant. He was hoping multiple brains could come up with something better. This led to some discussion about whether the pragma would work (Walter doesn't think so).
Walter said the only way he sees around this is that preview features shouldn't change the ABI, but one thing that could help is moving to header-only libraries. Martin suggested this happens with DRuntime and Phobos because they're the only pre-compiled libraries we have in the ecosystem. If they were dub projects, the problem goes away. In the end, no solution presented itself in this meeting, so we deferred this to a future monthly meeting.
Next, Mathias brought up the two ways to define C++ namespaces in D, one using an identifier and the other using a string. The former came first, with the latter being added later to address deficiencies in that approach. Mathias had encountered problems in his work with C++ interop using the identifier version and has since ignored it completely, relying on the string version. Plus, having two alternatives is confusing to people with no experience in D and C++ interop. He would like to deprecate the identifier option. Walter recalls past discussions about this, but can't remember the details about the tradeoffs. So he had no answer at the time and suggested they talk about this at DConf (I don't know if they did).
Finally, he talked about how
core.stdcpp is compiled into DRuntime and is dependent on a specific C++ runtime. Initially, it was added to DRuntime on the premise that it's equivalent to the C bindings already there. The problem is that the C bindings don't cause any symbols to be generated, but the C++ bindings do. This can result in an ABI mismatch. Mathias thinks they should be out of the runtime. Walter agreed.
Martin still has some annoying issues with semantic ordering and cycles. Apart from that, he had been waiting for the 2.100.1 release of DMD, and if it didn't happen in the next week or so he would go ahead and push out a new release of LDC anyway (which he did on July 20).
He said the move to a monorepo for DMD and DRuntime provided the opportunity to undo some mistakes made with LDC's repo in the past by moving it also to a monorepo. It also presented some challenges, but most of them had been cleared up. He expected everything to be fine. It would simplify things for LDC in the future and he was looking forward to the merger.
Iain has done a lot of internal GDC work which mostly benefits him and anyone using GDC as their main compiler. He had been doing a bit of tech support for Bruce Carneal related to SIMD. That has resulted in some new SIMD intrinsics. He had also moved about 600 lines of code out of the compiler and into the library.
Upstream, he had added float and double precision implementations of the log family functions in
std.math. This has the potential to break compilation for projects passing integrals to these functions. Additionally,
logb is now pure.
On infrastructure, he had seen no pushback on the monorepo from anyone heavily involved with the DMD and DRruntime repos. The only thing left to test was whether the pipelines would be okay after the change. There were no issues running the test suites locally.
On DMD releases, Martin Nowak has less and less time to do them. Iain was in the process of taking over. The intent is to take move the specifics away from one person (e.g., having secret keys on one person's laptop) and set things up on GitHub runners so that anyone on the DMD core team can trigger a release. He still had blockers from making that happen. The first was that he did not have access to the downloads.dlang.org server. The two people with the AWS credentials had been unresponsive (though I believe that has since changed). The second is that he has no certificates with which to do code signing for the Windows binaries.
Since the meeting, Iain and I have set up a new server for the archived downloads. We've moved away from AWS and are now using Backblaze (with free bandwidth, thanks to the Bandwith Alliance and our use of Cloudflare). All 235.2 GB of DMD downloads have been moved. Iain can correct me if I'm wrong, but I believe this is active right now, so anything you download from downloads.dlang.org will come from there. As for the code-signing certificate, we're trying to decide on an option that's best for us. In the meantime, the last I heard, Martin Nowak was going to put out a release of 2.100.1 without signing the Windows executable.
I told everyone I'd not had an update from Vladimir Panteleev or Petar Kirov regarding our migration of ecosystem services to foundation control. I'd created accounts with Hertzner, Netlify, and Cloudflare, and granted Petar admin access to them. Petar had since become busy, and I anticipated we'd have to wait until after DConf for any progress. That turned out to be the case.
We've since migrated our DNS to Cloudflare (which some of you may have noticed a couple of weeks ago when we had outages for some subdomains) and I've granted Cloudflare access to more administrators (Iain, Vladimir, and Mathias). We hadn't originally intended for the downloads to be the first service migrated---that was dictated by circumstances---but now that it's done we can look at migrating the next thing. I'm not sure at the moment what that will be, but I'll announce it here when I know.
Razvan had nothing to report.
Dennis brought up a discussion about reducing the size of
object.d that came up in a PR thread. Specifically, moving things into separate modules, then making
object.d a list of public imports would make it easier to maintain a custom
Walter said that the bigger
object.d gets, the slower it compiles, but splitting it into multiple modules would make it slower. Compilation speed is less dependent on file size and more dependent on the number of files. We may be expecting too much from
object.d, but he had no answer for now. We need to be more parsimonious about adding things to it.
Martin had no magic answer either but said he's always seen
object.d as the foundation of DRuntime. Any pay-as-you-go runtime would need to start from there. But he noted that most of the recent size increase is because of the project converting DRuntime hooks to templates. He suggested those imports could be removed from
object.d, and the compiler can insert imports explicitly where they are used so that they aren't part of the compilation for modules that don't use those hooks.
Next, Dennis talked about the focus of the PR that brought about the discussion: that symbols like
destroy are part of the global namespace (the PR added the ability to disable
destroy via a special version identifier). Should we be concerned about this? The consensus was that with D's built-in ability to differentiate symbols (FQN, static imports), this isn't a problem. Iain closed the PR after the meeting, noting that we agreed something needs to be done about
object.d, "but not this".
The third thing Dennis brought up was a recent thread about private-to-the-module, and how some people would prefer that it be private-to-the-class instead. He doesn't think the feature should change, but he did put together an implementation to try it out. He asked if this was going to be dead on arrival. The consensus was "yes", with Martin noting that what he liked most was Dennis's "thumbs down" for his own pull request. We did have a short discussion about this, with Martin noting that this isn't a problem if you're properly encapsulating your types. It's only if you're used to older codebases or other languages that it may seem to be a problem, but in D we don't need to worry about it. Dennis subsequently closed the PR.
As part of that discussion, I raised the issue that came up in the aforementioned forum thread: we treat the private API of the module as part of the class/struct's private API, but we don't do the same for the public API. Meaning, calls through the public API of a class always trigger its invariant, but calls through a module's public API do not. This means it's safe for the class to modify its internal state through its private API, but not for the module to do so (similarly, a module's public API does not obtain the lock of a
synchronized class). I think it's something most programmers don't consider, and we should think of some way to address it. Though, we didn't do that in this meeting :-)
Max said he had several PRs he needed to finish. For one of them, he had implemented
with expressions to play with in a branch, something he and John Colvin are fond of. He had also opened a newCTFE branch from Stefan Koch's old work. He and Nicholas Wilson had been going through to get rid of problematic code. His opinion was that we should get rid of its warts, refactor it, and upstream it. The preview flag was enabled so people could play with it, but it will revert to the old engine in cases where it fails.
Martin wanted to know more about the revival, as he was under the impression that Stefan had given up on it because it didn't provide the performance gains he had expected. Max estimated it should generally be between 5 and 10 times faster, but that it can be much faster in situations where the old engine gets bogged down. In contrast, the JIT in SDC (which uses the LLVM JIT API) can be up to 50 times faster. But for Max, performance isn't as important for newCTFE considering that the code is easier to work with and it uses less memory.
Martin then brought up the fact that newCTFE had been restricted to using 64-bit doubles and asked if that's still the case. Max affirmed that's so, and Martin said that won't do. Max suggested that since we now have ImportC, we could just use it to compile x87 soft floats, assuming the license is compatible. Martin said we should focus on x87 too much. It's interesting for DMD as it's x86 only. GCC/GDC uses soft float emulation across platforms, and LDC uses the host platform's
real. We have to decide if we want an abstract precision across all platforms like GDC is using, or try to emulate the target precision on any host. What's important is that newCTFE must be able to cope with arbitrary
real_t types as the frontend currently can. Max thinks it's doable with some refactoring.
Walter said he had lost a lot of time in the preceding few weeks trying to debug failures in the test suite that can't be reproduced. He had been complaining about the lack of information in the test suite logs. He cited a specific example where running the test online showed a failure in a file, but gave no other error messages about that file, but then running the test suite locally resulted in a useful error message in the log. He suspected it might be a
stderr issue. It's a general problem he has. Further, the tests put out log files that are thousands of lines long, making it difficult to pinpoint the actual problem. Sometimes, an error just disappears on a subsequent run and he doesn't always understand why. He had noticed that it sometimes happens after he rebases a PR. He noted that the autotester rebases automatically, so the other tests should, too.
Iain noted that the need to rebase should disappear once DMD and DRuntime were merged, as those two repos being out of sync was usually the culprit.
Another issue driving him nuts was running
cpp on Mac for ImportC. The problem is that
cpp on the autotester fails on some of the Mac system header files. So he was wondering if anyone knew which C preprocessor he should be invoking on Mac. Mathias noted that the autotester's Mac version is probably ancient, but suggested he invoke clang with the switch to use the C preprocessor rather than invoking the preprocessor directly. Walter said that was a good idea.
Next, he said he had made a lot of progress on getting ImportC to compile the C files in DRuntime. It was working Phobos except for the Posix files. Every Posix platform was failing differently, but he hadn't figured it out yet.
He said that other than that, things seem to be going rather well.
The next meeting
Given the lateness of this summary, the next meeting has already happened. It was a monthly meeting that took place on August 4 during the DConf Hackathon. I'll publish a summary of that in the next couple of days.
The next, next meeting is happening Friday, September 2, at 14:00 UTC. If you'd like to bring something to us, please let me know and I'll see about putting you on the agenda.