Permalink
Reply
The D Language Foundation's second quarterly meeting for 2023 took place on April 7 at 14:00 UTC. It lasted just over an hour. This was the first "standalone" quarterly meeting, where the foundation's monthly discussion was split into a separate meeting the following week.
Two new members joined us this time, and we look forward to hearing more from them at future meetings.
I was in transit at the time the meeting took place, so I want to thank Razvan for conducting it in my stead and Dennis for recording it so that I can write up this summary.
The attendees
- Mathis Beer (Funkwerk)
- Walter Bright (DLF)
- Andrea Fontana (2night)
- Mathias Lang (DLF/Symmetry)
- Martin Kinkelin (DLF/Symmetry/LDC)
- Dennis Korpel (DLF)
- Mario Kröplin (Funkwerk)
- Vijay Nayar (Funnel-Labs.io)
- Átila Neves (DLF/Symmetry)
- Razvan Nitu (DLF)
- Robert Toth (Ucora)
- Bastiaan Veelo (SARC)
The summary
Mario
Mario started by asking about the future of formatters in D. Funkwerk has put a lot of effort into dfmt, and they run it over all of their code, but pull requests for it have long waiting times and new releases are infrequent. Typically, when the Funkwerk programmer who contributes to dfmt makes a PR, it goes for weeks with no progress. There's also sdfmt (part of the Snazzy D Compiler project maintained by Amaury Séchet) that seems to get more use. The question he was looking to answer: should Funkwerk keep putting effort into dfmt, or switch to sdfmt?
dfmt was created and originally maintained by Brian Schott, but for some time now has been under the dlang-community group at GitHub, and no one at the meeting could say who was responsible for it now. Mathis Beer noted that Jan Jurzitza had mostly been the one merging things.
This was followed by a 15-minute discussion that went off on a couple of related tangents, but ultimately came down to two conclusions:
- Funkwerk should get increased access to the dfmt repository
- Razvan will have a new crop of students next fall, and he can find one to work on integrating dmd as a library in dfmt
The intervening discussion flowed as follows.
Mathias Lang said that Symmetry uses sdfmt since the maintainer is also part of Symmetry's team. The perceived advantage it has over dfmt is that it's based on the compiler as a library. He posed two questions: do we want to have one formatter we endorse as the D Foundation and, if so, which one should it be? If we do endorse one, then we should distribute it with the compiler. Walter agreed that a formatter should be part of the compiler release.
Mathias added that it would be much better if we have the compiler as a library. Razvan agreed and said that if we could convert dfmt to use dmd as a library rather than libdparse, that would be a step forward. Dennis questioned that point. The formatter only needs the parser. libdparse takes everything into account for syntax handling, but the dmd parser skips over some things (such as comments and unit tests). And even if the dmd parser is changed to account for more, how is it a better option than libdparse? This prompted a brief side discussion that resulted in general agreement that the main problem with a third-party parser is that it falls behind (some recent examples were cited) and will rarely be in sync with the latest compiler release. Átila added that even when it is current with the latest language features, the fact that it's a separate implementation means there are bound to be differences that manifest as bugs.
Razvan said that another issue with libdparse is anything using it that needs semantics, like dsymbol, has to come up with something mimicking the compiler's behavior. He has a student currently working on replacing the use of libdparse and dsymbol in D-Scanner (see Lucian Danescu's DConf '22 talk, Integrating the D Compiler as a Library in D-Scanner.
Walter noted that he had spent some time converting dmd's lexer and parser into standalone components. At the time of the meeting, the lexer was fully independent and the parser was mostly independent. There was still some work to be done to remove the parser's dependencies. He said he could get back on that and then it would be better suited to replace libdparse. That won't solve the semantic issues, but at least then the lexer and parser could be the same as what the compiler uses. (He posted in the forums a couple of days later that the lexer, parser, and ImportC parser were now all independent.)
Walter then suggested that, down the road, we would ideally have only one formatter. We can continue to update dfmt, but if sdfmt is getting more use, then maybe it's the better candidate to be the sole formatter in the future. Mathias noted that sdfmt is implemented with the SDC frontend, and since that's a parser completely independent from the D frontend used by DMD, LDC, and GDC, then sdfmt has the same problem as dfmt does using libdparse.
This was followed by some down-in-the-weeds discussion about specific semantic problems and how to adapt DMD as a library before coming back to Mario's original question and the conclusions I cited above. (One of Funkwerk's employees has since been granted increased privileges for the dfmt repository.)
Mathis Beer
libRebindable in Phobos
Mathis said he was working on PRs to push libRebindable into Phobos (see his DConf '22 presentation for more on libRebindable). He's thrown out the parts that are useful only in corner cases and is keeping the parts that are independently useful in the general case. He was aiming for a version of libRedindable into which you can push an arbitrary data type, pull it out again, and have all the reference counting and so on accounted for correctly.
He's noticed by pushing more immutable data types that something like half of Phobos is unprepared for it. For example, maxElements seems like a simple function, but its current implementation cannot handle immutable data types. He had a PR to work around it that was getting reasonably scrutinized as terrible, but if he could throw libRebindable at it and other such issues he's hopeful something can be done about it. So if the libRebindable stuff gets merged, things should be looking better.
Profiling compiler performance
He next brought up the persistent issue of slower compile times. He noted that LDC now has a "very, very snazzy" command-line option, -ftime-trace, that outputs profiles of LDC compile times in the Chrome Trace Event format, which can then be seen as a flame graph in a viewer. It's amazing and he loves it. Running it on Funkwerk's internal codebase, he found that one module which generates enums for HTTP error codes (with a CTFE loop and a bit of string processing) takes 300ms of compile time on its own. If someone is looking for a toy to play with to understand what the problem is with pretty straightforward CTFE that has a lot of hidden costs, this would be a good example. He was considering posting it in the forums.
Dennis agreed that it's a very useful tool. He had employed it to shave 25% off of the front-end time. He added that Johan Engelen had recently enhanced it to track CTFE (previously, it only tracked semantics) and also to output to a text format that's easier to read in a text editor. Dennis was thinking about making a tutorial on how to use it for the foundation's YouTube channel.
Using Phobos in the compiler
Finally, out of personal interest and unrelated to anything. Phobos had been prohibited from use in the DMD codebase because of, as he understood it, the need for C++ bootstrapping. He asked if that was a permanent situation, or if there could ever be a point where D's presence in GCC is widely spread enough that we can rely on that being present and dogfood Phobos more in the compiler itself.
Walter said he'd rather not use Phobos in the compiler. If it relies on Phobos, it makes the compiler more difficult to build and test because of a circular dependency: the compiler relies on a working Phobos, and Phobos relies on a working compiler. He doesn't think that using Phobos in the compiler would be enough of an improvement to outweigh that.
Martin noted that LDC does use Phobos for a few LDC-specific parts, but it is kind of a problem. It means you have to be compatible with all Phobos versions from the oldest version of the compiler that can be used to compile the compiler up to the current release. He agreed with Walter that we should minimize the use of Phobos.
Walter added that we've had problems now and then where we can't test the compiler because it fails somewhere in compiling DRuntime, and then you can't build the compiler to test it and find out why it's failing. That's a large time sink to resolve every time it crops up. Trying to add Phobos into that would be a large nuisance. The compiler should have minimal dependencies so that they don't get in the way of testing it.
Robert
All the talk about code formatters brought to mind something Robert hadn't intended to bring up, but it's something he'd wondered about before. He asked if we had put any thought into automatically upgrading old code after deprecations and such on new compiler releases. It's no problem when you only need to update a function or two, but when a deprecated feature permeates your codebase, fixing it is tedious.
Mathias Lang said it had come up in the past and mentioned Go's gofix as an oft-cited example, but he wasn't aware of any work that had been done on that front.
Dennis said this depends on what the deprecation does. He brought up the deprecation of alias this in classes as an example that has no trivial fix to make everything work again. It may require restructuring code. There was a brief bit of discussion of what could be done in that specific case, but Razvan quickly brought things back on topic. He said that we could surely implement some automatic fixes for trivial cases. Atila said "yes and no". Because of templates and mixins, it's not always possible.
Bastiaan linked dfix in the chat. It hadn't been updated in years, but it's the sort of thing Robert asked about.
Razvan suggested that if we move forward with integrating dfmt into the compiler releases, then anytime someone implements a deprecation, they should also implement a compiler switch for the formatter to update code when it's possible. That seems like the best solution.
Next, Robert said he had begun working more with dub. He noticed there once was a bounty for modifying dub so that it detected file changes via hashes rather than timestamps. He asked if that had been implemented. Mathias Lang said we're still using timestamps. (He was referencing a bounty that Symmetry had posted for this. I know at one point they had selected someone to work on it, but as far as I know, the bounty is still open at $2,000. We no longer have our Flipcause account where we were posting bounties, but I saved all the relevant details for each of the open ones before our account was closed. I'll post them again once we decide on a medium for them.)
Bastiaan
Bastiaan said he'd been maintaining the Pegged project for the last few years. Phillipe Sigaud, the repository's owner, granted him a maintainer role for the repository a while back. He thinks it's a good candidate for addition to the dlang-community organization, and that's something we should look into making happen. Phillipe is willing to hand it over.
Mathias Lang said that if Phillipe could invite him to the project with full privileges, then he could transfer it to dlang-community. He'd recently done the same thing to transfer ddox to dlang.
Next, Bastiaan said he's been a frequent user of the Windows installer. There's a regular issue with it that when the installer is finished, there's nothing there. Trying it once more works. He thinks it's worth a bug report, but he doesn't know where to look for the cause.
Finally, he's noticed with debug builds on Windows that sometimes the linker complains that the debugging information is corrupt and he should recompile, but then after a recompile, it persists. He found that removing several instances of __FILE__ and __LINE__ causes it to go away. He didn't know if that was part of the problem or not. He noted that dub still has the problem that file paths can sometimes be very long, and suggested this issue might be related to that. The dub problem had been addressed in a recent PR, but not yet released.
There was some discussion about whether it was reproducible. Dennis suggested the next release of dub should solve the problem, and to report it if it doesn't. Martin agreed and reminded everyone that Symmetry had encountered that issue frequently. He wasn't sure if the fix will solve all cases, but it should solve most of them. He talked about a workaround involving the Windows registry and a manifest file for the executable to enable the C API functions to handle long paths. If the prebuilt dub had the manifest, then users could do the registry hack themselves and all of these problems would be gone.
Mathias Lang said the reason that the bug started triggering was because of a fix for another issue. Before, dub was using relative paths in quite a few places, all relative to the current directory where things were being built. That caused issues with system-wide packages. For example, you couldn't install a package in /usr/lib, as doing so would cause dub to try to write there. He fixed that by adding a central build cache next to the package cache so that all of the paths are absolute. That has resulted in longer file paths.
Andrea
Andrea introduced his company, 2night, based in Venice, Italy. They create interactive experiences for bigger international and Italian brands. They usually work with Raspberry Pi and other embedded platforms and with a lot of different technologies on the software side. In the past year, 2night had increased their use of the D language. They had two main issues.
The first was that D support for 32-bit Raspberry Pi and other ARM platforms (via LDC) was a couple of years out of date. Martin said that currently, LDC only supports 64-bit ARM and supports it well, and that includes all of the newer Raspberry Pi releases. He's not interested in maintaining support for older 32-bit ARM. He was going to mention that someone had ported a recent build of LDC to 32-bit ARM, but as he was saying it he realized it was Andrea who had done so. Andrea said they'd needed it for a project in Venice. He'd managed to get it working but hadn't tested it deeply.
Martin thanked him for his contribution. He said it didn't work in 100% of cases, but 32-bit ARM support wasn't CI tested anyway. And there were unit tests that had never worked on 32-bit ARM. 64-bit ARM is much better, much more mature, and is CI tested. Moreover, the 64-bit ARM ecosystem is more homogeneous than that of 32-bit ARM, where there are many ABI differences and other ugly problems. That's why he's only focusing on 64-bit. Andrea said he would look into switching to 64-bit.
The second issue was the integration of ImportC in the D ecosystem. They use ImportC a lot, but it doesn't work well with dub and other tools. It was a bit rough to use ImportC inside a project, with many steps that were difficult to automate. Walter asked him to write up a more detailed account of their issues with ImportC and post it to Bugzilla. Andrea agreed to do so. Dennis asked if Andrea had added the .c files to dub's source list, and Andrea said the last time he'd tried it didn't work.
Martin noted that you don't get all of the latest ImportC features with LDC. One important missing feature is the implicit invocation of the C preprocessor which has been part of DMD for a couple of releases. Using DMD might be better in that regard. Andrea said they're using that with a different project on an Intel platform, and the last time they'd used it was last summer. He doesn't recall that implicit preprocessor invocation was implemented at that point. He was only able to get it to work with the Tiny C Compiler (TCC). He would have to try again to see what has changed, then open bug reports on anything he finds wrong. Walter asked him to report even small issues, as he wants to fix them all.
Vijay
Vijay started by saying he's very happy with the D language and the build tools. He develops with emacs in a Linux environment, and he's found that the language and tooling have been very good.
He gave some background on his past work helping businesses where he'd worked find massive savings on their cloud computational costs by making their services more efficient (his DConf '22 talk, Efficient Microservices using D, was related to this). He thought he'd be able to build a business out of that with D, but after a year, he'd learned that investors tend to prefer cutting costs by laying off employees first, so the demand isn't there.
The second thing he'd observed is that D has very low library coverage, so he'd been porting over several libraries from Python, Java, and C++. It's easy to do for anyone familiar with those languages, and even individuals can move quickly to get things done that most teams would struggle with. However, he sees this as a symptom of a bigger problem: the D community is quite small, so the number of people who encounter random problems and solve them is also small. This has a feedback effect in making it harder for people to adopt D because the libraries they need are missing. He doesn't have an answer for how to solve it.
Walter said the point of ImportC was to expand D's usability with existing libraries. Vijay said that one of the things younger developers rely on when using libraries is popup documentation in their IDEs. While ImportC provides binary compatibility, it doesn't make documentation or autocompletion available through, e.g., the Language Server Protocol (LSP). The ability for ImportC to do that in combination with autocompletion might further accelerate integration, though he knows that's separate from Walter's efforts.
Dennis said he could relate to that. He had recently used ffmpeg with D. Instead of using the existing bindings, he tried using it with ImportC. The lack of autocompletion was notable, and he sees that as something that will hold it back. serve-d gives us all of that for D code, but we have nothing for ImportC.
Andrea said he'd had the same problem, also using ffmpeg. This is the kind of thing he meant when he said ImportC doesn't integrate well with the D ecosystem.
Walter noted that there'd been talk at past meetings about getting LSP support into the compiler. He hadn't yet looked into it, so he didn't know what it would take to get there. (And for the record, Walter was receptive to the idea in those past discussions.)
Everyone else
At this point, Razvan noted that the attendees associated with the DLF would normally take their turns now that the industry reps had all had a chance to speak. Instead, since we're splitting the monthly meetings out of the quarterlies going forward, he asked if any of them had anything to say that couldn't wait a week. None of them did. He asked for any other comments or questions.
Walter took the opportunity to thank all the industry reps for joining us and telling us about their experiences. Bastiaan said it's very nice that we're listening. It's great to have a place to come and discuss issues in person.
Robert was curious about the monthly meetings and wondered if he could or should join. Razvan and Dennis described how they usually work and noted that I post the summaries here in the forums. They said I had mentioned that I'd like to expand them by regularly bringing in more contributors, but they weren't sure what my plans were. They suggested he ask me about the details. (Yes, I do want to expand them, and I did provide more info for Robert.)
Vijay said there are two real-time chat channels listed on the D homepage (Community Discord and IRC) and wondered which was the primary channel. Razvan said that Slack is the "official" place where contributors reach out to each other when they need to, but it's rather quiet there. Discord is busy enough that he has to keep notifications muted. He noted that I've often said the Community Discord is flourishing and is the entry point to the D community for more people than the forums these days.
Walter said he finds the real-time chat formats disconcerting. He prefers newsgroup-style communication. He said that he tries to maintain a D presence at Hacker News and welcomes anyone else to help out. Click on his name and you can see where the action is. He encourages everyone to monitor sites like Hacker News and Reddit to participate in conversations about D and initiate posts about D when they can.
The next meeting
Thanks again to Razvan for running the show. He did such a good job that I'm tempted to let him keep it.
If your company is using D in production and you'd like a seat at the quarterly table, let me know! Now that we've split the monthlies out, we've got plenty of room to add more people. We also welcome anyone to join us for our monthly meetings if they have a specific issue they'd like to discuss with us. Again, let me know, and I'll set it up.
The next meeting was a monthly meeting that took place on April 14 at 14:00 UTC. I'll have the summary for that in a few days.
