Permalink
Reply
The D Language Foundation's quarterly meeting for January, 2024 took place on Friday the 5th at 15:00 UTC. It lasted for about 45 minutes.
Our quarterly meetings are where representatives from businesses big and small can come to bring us their most pressing D issues, status reports on their use of D, and so on.
The Attendees
The following people attended the meeting:
- Walter Bright (DLF)
- Iain Buclaw (GDC)
- Luís Ferreira (Weka)
- Max Haughton (Symmetry)
- Dennis Korpel (DLF)
- Mario Kröplin (Funkwerk)
- Mathias Lang (DLF/Symmetry)
- Vijay Nayar (Funnel-Labs.io)
- Átila Neves (DLF/Symmetry)
- Luna Nielsen (Inochi2D)
- Razvan Nitu (DLF)
- Mike Parker (DLF)
- Guillaume Piolat (Auburn Sounds)
- Carsten Rasmussen (Decard)
- Robert Schadek (DLF/Symmetry)
- Bastiaan Veelo (SARC)
The Summary
Luís
Undefined references
Luís said that Weka had encountered a lot of undefined references when attribute inference infers the wrong attributes. For example, sometimes @nogc was inferred when really the GC was being used, or vice versa. Then when linking with other modules where the attributes were inferred differently, they ended up with undefined references.
They had a workaround---creating an alias to another template with the mangled name that is supposed to be generated---but it was very cumbersome. Right now, this was their biggest issue.
He said this didn't manifest when compiling with one compiler invocation, but when multiple invocations were involved. They would like to be able shift to building with LTO and compiling the objects file by file rather than in one compiler invocation.
He also noted that one of the issues they had with compiling as a single invocation was the amount of RAM used.
He said they also saw errors with attribute inference that were dependent on how the compiler was invoked. Each of the files should compile independently, but depending on the order they're passed to the compiler or which ones are passed, the errors they get are different. He said that basically, in a specific semantic pass, the compiler was just giving up on attribute inference.
He also tangentially brought up a problem with import order causing forward reference errors.
Walter said he needed to see Bugzilla issues demonstrating the problems before he could attempt to fix it. Luís said there were some similar issues reported. He was trying to Dustmite down to some minimal examples. Átila noted that it was very difficult to isolate template issues like this. He'd run into similar issues over and over and hadn't yet filed anything on Bugzilla because it's was never a trivial example.
Walter said he understood and was aware of Átila's longstanding problems with this. Robert said he'd never encountered these problems at all, but that could be because of the way he used templates. Atila thoought it came down to whether you compile all at once or not.
Max said that Symmetry had gotten a reduced case at one point. He thought it had made it into Bugzilla, saying Martin would have filed it. That particular case may even have been fixed.
Walter reiterated that he needed a test case to reproduce it. Talking about it wouldn't move anything forward. Luís said he had a test case in the PR he had submitted in an attempt to fix it. He noted that Walter had also made an effort, but that the PR got stuck and then sank because it didn't solve the problem.
(NOTE: Bugzilla Issue #17541 describes the problem. The failed attempts Walter and Luís made to fix it are in dmd PR #10959 and dmd PR #15534.)
Compile times
Weka's biggest issues with compile times were with the performance of template instantiations and CTFE. They use the time trace flag with LDC and it shows these are the biggest problems.
Luís asked if there were any plans to implement caching of template instantiations so that they wouldn't need to be instantiated on subsequent runs of the compiler. He also asked if it would be possible for us to have a JIT to speed up CTFE.
Razvan said he was unaware of any plans to implement template caching. Luís said the idea would be to do something like LDC does with its codegen cache, where it caches the LLVM IR. Razvan asked if he was suggesting caching the actual template instance or the result after the template has been interpreted.
Luís suggested that the AST should be pure. Some work could be done to separate out the member functions that mutate AST nodes so that they could be made immutable. With such a pure AST, then we could make it hashable and cache some of the work after the semantics, after some code gen.
Razvan said there had been some discussion of this in the past, but he was unaware of anyone working on it.
Linter update
Luís said that after many attempts of implementing a linter in the compiler, he had been succssful running LDC-lint over the Weka code base. They only had two rules enabled for the moment. He planned to enable broader rules eventually, but there were too many false positives now. But down the road, he hoped this would be a great thing to show off, having a linter built in to the compiler as opposed to being an external tool.
Huge stack allocation
Luís said that Weka's code used a lot of fibers. They'd found that opAssign on huge structs generated huge stack allocations when it shouldn't. On their LDC, they had an attribute to see big stack allocations at compile time so they could detect those, but they ended up either disabling opAssign or overriding it to not use stack allocations.
Mathias asked if it was because the parameter wasn't passed by reference. Luís said he wasn't sure. This was the opAssign generated by the compiler. Maybe it wasn't being passed by reference and should be.
(UDPATE: I asked Luís to email me later with the Bugzilla links related to the problems he raised along with any other relevant details. He did, and included a couple of other things he'd forgotten to mention in the meeting, one of which was another problem that prompted Walter to open a Bugzilla issue. The issues with undefined references and compile times are still unresolved, but Walter closed the stack allocation issue as WONTFIX, saying the existing algorithm was the pedantically correct one. Generation of opAssign could be turned off with @disable, and changing the algorithm could be done by overriding opAssign.)
Carsten
Carsten said everything was working okay. They had been optimizing some stuff and had some problems, so they had moved away from libp2p (they were using the Go implementation) to another library written in C. They wanted to have everything in clear code.
Now all their code was in D and C, and that was nice. At the moment, they were precompiling all the C code and then linking it in, and they'd had some issues with memory leaks. What they wanted to do was to use the C code via ImportC, so getting that working was their next step.
Other than that, they were profling to find where their time was going.
Walter asked him to keep us posted on their experience with ImportC. Carsten said they only had three C dependencies, and having everything in D and C was much nicer to work with.
Carsten then reported that they'd had their server running for a month and were working on improving it. They'd had one crash that came from someone sending in malformed data that they hadn't been prepared to handle. They were currently working on being able to seamlessly swap out nodes. He said they had a lot of work to do, but at least now they were in a position to have fun doing it.
Basiaan asked which compiler they were using. Carsten said they were using DMD for development because of its faster compilation and LDC for deployment. Bastiaan said he'd had trouble using ImportC with LDC, so he was using only DMD for now. It was probably a preprocessor issue. Carsten said they hadn't fully incorporated ImportC yet and were only experimenting with it. He had considered doing the preprocessing first. It was a learning process.
Bastiaan
Bastiaan reminded us of the big problem regarding multi-threaded performance that he'd brought up in the October quarterly meeting. He noted that he'd replied to my summary of that meeting with a report that they'd since solved the problem. He said they were very pleased with that.
He said it had been frustrating because there were multiple causes. Every time they'd thought they'd found the cause, it didn't have any effect at all until they'd fixed all the causes. The three major problems are outlined in his forum post. Once they solved those, they saw good improvement.
Mario
Mario said there was nothing new from Funkwerk. Mathis Beer was still on holiday, but would probably have something for the next quaterly meeting.
Guillaume
Guillaume said he didn't have many issues to talk about because everything worked.
One problem he'd had was with newer releases of LDC. He does consumer software, but with LDC he couldn't target old versions of macOS. It wasn't a big deal as he could just mix and match the binaries.
He also wanted to reiterate that better support for Mac silicon and WebAssembly would eventually be a good idea, but there was nothing really hardcore urgent he could think of.
He also wanted to thank everyone. He had filed 64 bug reports over the course of his involvement with D. 45 of them had been solved and the others weren't that relevant. He thought that was excellent bug coverage that he could enjoy.
Max
Max had to leave for a work meeting shortly after I called on him and said he would try to make it back if we were still going later. He didn't, as the meeting didn't last much longer beyond this point.
Vijay
Vijay said he was very happy with D and how it functioned, but he had a couple of questions that had come up during a recent small project.
Question about message passing
First up, he'd noticed that when you were doing message passing between threads using the send and receive functions, you almost invariably had to duplicate the data because you could only pass immutable objects. But when you were no longer using the object in that thread, he wondered if this would result in a copy or if it was being optimized away.
Átila said the options were to either pass immutable data, which he knew could sometimes be a pain, or shared data, which was a bigger pain. But the only other way he knew to get around that was some form of ownership in which you could pass ownership from this thread to the other. Even though the data would be mutable, you wouldn't be able to use it in this thread anymore, so it would be safe. He said the Pony language had something like that. But that would mean some form of isolation in the D language.
Question about Ddoc
Second, when documenting template parameters vs. regular paramters, was there a separate segment for template parameters? In the language and the code, they were distinct, but in Ddoc there was only Params:. Mathias said that putting them all in Params: handled it.
Vijay asked if grouping them together was the right choice given that they were distinct in the language and the code. Mathias thought so. He said that at Sociomantic (where they had maintained their own fork of the compiler), they'd implemented Template_Params: at one point. He said from his experience with that, there wasn't much value in it. If we really wanted to separate them, the compiler could figure it out and we could just display them separately. But he didn't see the value in it.
Átila agreed. He said that you knew the names, the names weren't going to repeat, so you could probably tell them apart.
Luna
Luna said her current blocker was the lack of ObjectiveC support in LDC. She was considering making a pull request for it.
Another issue was that DRuntime on macOS was using a function that Apple didn't allow to be present in executables released on the App store. She was also planning to submit a PR to LDC to fix that. Carsten said they'd had the exact same problem on iOS. He didn't know which function it was or what the fix was, as his colleague had handled it.
Walter asked what function it was. Luna said it was something in the dylib API. Apple didn't allow anything using private APIs in the App store. She was trying to figure out how to make that work. It was related to how the GC scanned executables and shared libraries for spots to search. You had to use a different API for that.
Walter said he hadn't been aware of that at all, and he'd appreciate a write up on what was wrong or what the correct API was. Luna said she'd filed an issue on the LDC repo. She said this was an LDC-only issue. She hadn't seen that the API was called on the DMD version of DRuntime. Walter said that was something for Martin to handle, then.
(UPDATE: Luna subsequently fixed the dylib issue.)
DLF Team
Robert, Mathias, and Iain all had nothing to report for this meeting, though Mathias said he was happy with the language.
Walter said he'd been working on improving ImportC and addressing bug reports. He was always interested in hearing if people were finding success with ImportC and if it was fulfilling its mission to save them time. He had nothing else for this meeting.
Luís asked if Lua was compilable with ImportC. Walter said he had no idea. Luís said he had a use case for it so would give it a try. I speculated it probably would be because it's a clean codebase with wide support across compilers and platforms. Luís said it didn't have a lot of macros.
Carsten said in their experiements with ImportC, they'd run into the problem where module names clashed when two C files had the same name. But they were able to work around it by using soft links that combined the directory name and file name. Walter asked if he could do a write up about their approach and then we could add it to the documentation for ImportC. He said this was what he called an impedence mismatch between D and C, so he'd been collecting the workarounds people used because other people were going to run into the same things. Carsten said he could show Walter his script.
Dennis said he'd been working on named arguments and had made a forum post about some issues he'd encountered that could use some feedback. He then said he'd had more success with ImportC. Most of his bugs had been fixed for TreeSitter to compile. There was just one more designator issue that was open, but apart from that he could now use ImportC to compile it.
The Next Meetings
Our January Monthly meeting took place the following Friday. We also had a planning session on January 19th. The next quarterly meeting is happening on April 5th.
