Permalink
Reply
The D Language Foundation's quarterly meeting for July 2024 took place on Friday the 5th at 15:00 UTC. It lasted for about an hour.
Our quarterly meetings are where representatives from businesses big and small can bring us their most pressing D issues, status reports on their use of D, and so on.
The Attendees
The following people attended the meeting:
- Mathis Beer (Funkwerk)
- Walter Bright (DLF)
- Johan Engelen (Weka)
- Luís Ferreira (Weka)
- Martin Kinkelin (DLF/Symmetry)
- Dennis Korpel (DLF/SARC)
- Mario Kröplin (Funkwerk)
- Mathias Lang (DLF/Symmetry)
- Mike Parker (DLF)
- Carsten Rasmussen (Decard)
- Robert Schadek (DLF/Symmetry)
- Bastiaan Veelo (SARC)
The Summary
Carsten
Carsten said Decard had been experimenting a lot with WebAssembly. They'd stubbed out DRuntime until it worked. They had some of their library functions running in a browser and had tried different virtual machines. They wanted to do more work on that. They had a lot of issues with mobile because different chips worked in various ways. They wanted to get their entire SDK working on WASM, but it would probably take a year to complete because there was a lot of stuff moving around in WASM.
Robert asked if they'd gotten the GC working. Carsten said they hadn't focused on that. They were just calling functions. They didn't know how to write a GC, so they were hoping to use the WASM GC eventually.
He said one thing that had been annoying was that the WASM interface was basically POSIX, but the compiler didn't treat it as such for versioning. He'd had to implement a lot of versioned dummy code to get it to work. He would like to see the compiler recognize WASM as POSIX.
Robert said he'd seen that the standardizing body had accepted a proposal for GC. Carsten said they'd also accepted multithreading. He thought WASM was moving into virtual machines and such, so it would eventually be a VM for both in and out of the browser. Robert said it was a really nice interface.
Bastiaan
Bastiaan said he was still working on SARC's D translation. Now and then he encountered a problem that was difficult to solve or difficult to understand. That was his life, but there was nothing major to bring up here.
Mathis Beer
Mathis said that for the first time ever Funkwerk was using a patched version of the compiler because the rebindable range destroy bug was so horrible. He said that wasn't a new thing, though, because it was mostly his fault. He thought the destroy function should just be deprecated. It just did so much harm, but he didn't feel it was debate-worthy.
Dennis asked if he meant the general core destroy function. Mathis clarified that he meant the Object version. He said it had this issue where when it was dealing with a value type it would do something to the value, but when dealing with a class it actually destroyed the class value, not the reference but the actual object itself, in a way that was very hard to notice. Sometime later you'd notice that your vtable pointer had become zero, so you had to know what that meant and find out where it happened.
He thought it was almost never the function you wanted to call in the generic case. He couldn't think of a single case for it. He found it bewildering.
Walter thought the purpose of destroy was so you didn't have to wait for the GC to collect an object. You could just say you were done with it. Mathis said no, it didn't even deallocate.
Walter asked why it mattered if it set the vtable to zero if you weren't going to use it anymore. Mathis said the problem was that he was using it in a generic context and wanted to call the semantically appropriate destruction operation for a value. For a struct, that meant calling the destructor because the value's lifetime was ending. But when the lifetime of a class reference ended, nothing should happen because there might be other references to that class instance. But what actually happened was that the class instance was destroyed no matter how many references remained.
Walter said that was the point of destroy. Mathis said it didn't do that for any other type. If he had a struct lying around somewhere and passed a copy to destroy, the original version wouldn't be affected. Dennis said that was true, but if you had a pointer to that struct, then the pointer would be dangling. Mathis said he didn't actually know what destroy did with pointers. He thought it did nothing.
Walter said if you were to destroy something, then there should be no more references to it. It was up to you as the programmer to ensure there were no more references to it. Mathis said in this case he was using it to compensate for lifetime management of a data object in rebindable where he had to take charge of the lifetime. He thought he'd call destroy to end its lifetime.
Walter said, no, you called destroy when you as the programmer know that was the end of the targeted value, which was dependent on the specific type. So if you were using it in a generic function, if it was a struct, then it was going to be a copied value so it made sense to say you wanted to end the lifetime of that value. But if it was a class, the copy constructor was literally just making a copy of the reference. If you called destroy on that, you were saying that there were no more references to it.
Mathis said that was his point. The behavior was different semantically. He wasn't calling it on a class specifically. He was calling it on a T, an arbitrary template parameter. Walter said that by doing that he was saying there were no more references to that T instance. But here in the meeting he was saying he had more references and they were going to go bad, but that was not the point of destroy. The point was that the programmer was saying there were no more references.
Mathis said the behavior made sense for classes and made sense for structs, but it didn't make sense for both classes and structs. It was different. Walter said that was the way class objects worked. Calling destroy was unsafe.
Mathis said that basically, it didn't make sense to be overloaded. There should be a destroy and a destroyClass. Walter again repeated that this was the point of the function: calling it meant you were indicating there were no more references.
Martin said the problem with classes was that you could never be sure that the reference you were destroying was actually the final reference. He suggested we should have a separate finalizer for classes. It would mean breaking changes, but we could add a finalizer for class objects and then in an edition, we could change the semantics of destroy.
Walter thought it was a good idea, but said the trouble with having a separate finalize and destroy was that people would mix them up, wouldn't understand the difference, would use the wrong one, and then be unhappy. It was inevitable.
He said one thing Mathis could do was to wrap the call to destroy with "is this a class". Mathis said he'd been doing that. He couldn't ever remember calling destroy without wrapping it like that. Walter said okay, it was working the way it was designed to work, and he didn't think we could change that behavior.
Luís thought one of the problems here was special cases for classes. Not only destroy, but a lot of stuff was special-cased for classes because they were just pointers to hidden objects plus a vtable. Another thing was that classes implied the presence of the GC, but you could also allocate them without the GC. The existence of the special cases was why Weka didn't use classes. Most of the cases could be solved with structs.
Walter said Luís was right that classes and structs were different from each other. Algorithms that ignored those differences were going to lead to trouble. Luís understood that but felt the design could have been more similar to structs without the differences that always need special cases in generic programming.
Walter said that in C++ you had to special-case for references. When you had value types and reference types, they were going to be different. He didn't think there was any magic way to fix that. Luís said at this point he didn't think so either.
Mathis agreed. He said he really liked the class design in D. The standard range algorithms worked fine with classes as well as with structs when written in a straightforward way. The only function he had a problem with was destroy. He felt that if he had a generic function that required him to use a static if to call a function, then in fact it was two functions that happened to be named the same.
Walter said that was a good point, but if you were using value types, you should be relying on RAII to do the destruction, not an actual destroy function. And RAII didn't happen with classes.
Mathis noted that his case was a special case. He had this magic rebindable thing going on where he had to take manual control of lifetime. He supposed it was something that didn't come up often. Walter said when you were manually doing lifetime as a special case, then it was a special case and you had to put an if in there.
I noted that Object.finalize in Java had been deprecated because it was so easy to misuse. I remembered that Sun used to always tell people not to do this or that in finalize, because it wasn't a destructor, yet people would still do this and that. We had kind of the opposite problem in D in that our destructors were also finalizers.
Walter said you were never going to get fire and water to mix. Reference types and value types were just intractably different from each other.
Mathis said this had come up because he had to add that rebindable alternative to support immutable in std.algorithm, which all went back to his DConf '22 talk.
He then wrapped up by reporting that Funkwerk were on DMD 2.108 and LDC 1.38 everywhere and it was working pretty well. Their code was internally managed with no major external dependencies. They were happy to be off of vibe.d and no longer dealing with its issues from compiler updates. And they were happy with 2.108.
Luís
Luís said he had nothing new to bring, but he wanted to understand the status of the AST refactoring project. Weka still had linkage issues resulting from the incorrect inference of templates and attributes. He thought it was mostly because the way we queried the AST was kind of a mess. If we had a way to make that better, then it would make other things better.
I said that Razvan was the one who knew all about that. I knew that it was mostly Razvan working on it right now, though others, including Bastiaan, had contributed. It was also an officially sanctioned idea for a SAOC project. So it was in progress, and Luís should ping Razvan if he wanted an accurate picture.
Luís closed by saying he'd seen some good features come into the language lately that he'd been waiting a long time to have, like string interpolation and named arguments. He thought that was really cool to see.
(UPDATE: One of the SAOC 2024 participants is working on the AST refactoring project right now, and Razvan intends to enlist some of his students at the Politechnica University of Bucharest to contribute. He could always use more help on this. It's not difficult work, there's just a lot of it. But it can be done in small chunks which makes it really amenable to contributions from multiple people. This is a high-impact project. If you'd like to find out how to contribute, please read Razvan's call to arms on the D blog.)
Mario
Mario had no programming issues to report but said that three Funkwerk staff would be attending DConf this year.
Mathis said that Luís mentioning named arguments had just reminded him that Funkwerk wanted to use them, but currently couldn't because they weren't supported in dfmt. They used to allow one of their employees some time to spend on dfmt, but that employee was no longer working for them and they were now in a gap. He wasn't sure what to do about that. Maybe he could take a few days to look at it, but they used dfmt in all their projects, so that made named arguments a non-starter for them.
Dennis said he had started using dfmt recently for his own code and noticed a few bugs. He'd opened some issues, but it seemed no one was actively monitoring it. He said he might take some time to figure out how it worked so he could fix these issues. It was really annoying when the formatter squashed things in attributes, got confused by the colon in named arguments, and completely ruined the argument list.
Mario said that he hadn't assigned Mathis to fix dfmt because he wasn't sure if it would be a waste of time and money. He knew that there was also sdfmt from SDC, and he also knew that something had been done with dfmt with regards to integrating DMD-as-a-library, but he didn't know what the progress was. He didn't want to task Mathis to fix dfmt if the fixes would become irrelevant when it was replaced with something else.
Dennis said one of Razvan's students had worked on the project to replace dfmt's use of libdparse with DMD-as-a-library, but he didn't think it was enabled by default. The last he'd heard there was just a switch to enable it and it was still kind of early. He didn't think the change would affect the formatting logic, but we'd have to see.
I noted that in the past we had discussed shipping a formatter with the compiler. We had agreed it would be dfmt because it was going to use DMD-as-a-library, which made sense if we were going to ship it with DMD. As such, any fixes to the formatting logic shouldn't be wasted time.
Luís said that Weka hadn't been able to test the dfmt changes yet because they were a few compiler releases behind, but he hoped they'd be able to soon. He'd report any issues they found with it.
Johan
Johan said there were probably a few issues from Weka to talk about, but he hadn't collected them before the meeting, so he'd bring them up next time.
He did have some news for us. He said that Weka had been interested in linking executables with musl libc. He'd fixed the last parts that were required to make it happen. He'd upstreamed almost all of it to DRuntime. He said that was a nice success story. You could now create statically linked executables that didn't depend on any libc implementation, and that had been the goal.
Next, he requested that we include the language spec in the compiler repository to prevent the kind of parallel pull requests where you ended up having to revert one if you had to revert the other. He was sure it had been debated before, but he would like to see it.
Robert said it would be even more awesome if the spec were unit tests, i.e., docs that were actually testable. That would probably be more work, but it would be really cool. Even with the spec in the compiler repository, you'd still have to remember to update it. But if you had a spec that failed to compile and the documentation was generated from that, it would be a bigger win.
Walter said that was a great idea.
Dennis noted that examples in the spec were currently run if they had the proper macro. He said that Nick had been working on transforming raw, untested code into runnable examples.
Robert said that was awesome and asked if it included other test cases for the language. Sometimes when he read the spec and something wasn't clear, he'd go to the tests and see what actually came out at the end. If they were included, that would tighten up the spec quite considerably.
Dennis asked if he meant linking the DMD test suite with the specification, and Robert said yes.
Walter said the test suite was really not meant for public consumption. It was not designed to be that way. It was designed to make the compiler easy to debug.
Robert agreed but reiterated that sometimes when he read the spec, there was some nuance missing, or the wording was such that he misunderstood. Reading some obscure source code that might be hitting his specific need helped him in that regard. He understood where Walter was coming from, but it was just an idea to float.
Walter said Robert was right, but he pointed out that every time he read a computer software specification, including language specifications, the specs on how the function call ABI worked always confused him. So he would write little test cases, compile them, and see what happened. Robert said he was a lot lazier than Walter and was looking for somebody who had already written the thing he was interested in.
Walter said that once he had written the code, compiled it, and seen what happened, then he'd go back and read the spec again and it would make sense. He wished he knew a way to make a better correspondence between the specification and understanding it.
Luís said his experience in relation to Walter's had been the opposite. For example, reading the D spec, assuming the ABI worked as described, and then experimenting with DMD and finding the ABI didn't work as specified. For example, arguments were getting reversed. In the calling convention, the arguments were supposed to be one way, but in the compiler, they were the other.
He said we had two different kinds of tests on the compiler. We had all sorts of weird tests for specific compiler bugs. He thought Robert was describing tests that served as examples but that also confirmed the spec was conformant with what we wanted to say.
Walter said if the compiler and the spec didn't match, that was a bug and should be reported. They happened, but if they were reported we could knock them down one by one. Luís said he had tried to fix the calling convention bug, but it was a very hard problem because it was everywhere in the DMD backend.
Walter agreed that the calling convention was a large source of complexity, especially in the backend because it supported multiple calling conventions. It was a large tangle of code.
Luís said one of the problems he saw was with DMD's math intrinsics. They were implemented based on the calling convention, but because the argument order was actually reversed, we now needed to change all of those intrinsics. He'd submitted some PRs to try to fix them, but the tests failed. From talking to Iain, he thought GDC was the most ABI-compliant D compiler.
Johan brought us back to his original point: if we had the compiler and the spec in the same repository, it would be easier to know which language spec matched the version of the compiler you had.
As for testing the spec, he said there were companies out there that made test suites for C. This was tough work. He thought it was a completely separate issue from having the spec and the compiler in the same repository. It would be a huge amount of work to write all the lines of test code that followed from a single line of the spec. It would be great, but it was a huge project for sure.
As for the ABI, he felt that was outside of the language spec. To him, the spec was at a higher level. We implemented a C ABI because we had to, but otherwise, the ABI was quite open. That was on a lower level. The more interesting thing for programmers was on the higher spec level, and that was linked with your specific compiler version. When they were in separate repositories, then you had to figure out which dates matched which and it got quite tricky quite quickly.
Walter agreed that the language spec and the ABI spec were two distinct things. He noted that after the C language was standardized, a company made a test suite out of every piece of code in the C spec and then released it. Every C compiler fell flat on its face. It took years for all the C compilers to successfully run the tests extracted from the spec. This was the whole problem.
Mathis suggested that when you're looking at the language spec and something doesn't make sense to you, but then looking at a test causes it to make sense, then that was a test that should go into the spec. You didn't have to put it in at that moment, but you could file an issue for it. Walter agreed.
Luís gave an example of a situation Weka had encountered where having the spec and the compiler in the same repository would have been a help.
Walter said he had no objections to it. He didn't think it would be a big problem to merge them.
Finally, Johan reported that Weka were still on LDC 1.30, but they were close to upgrading to 1.38, which was D 2.108. He had already seen some changes in 2.109 to which he thought it would take them some time to adjust, so the plan was to upgrade first to 2.108 and continue from there.
Luís said he was very excited to use some DIP 1000 features. The DIP 1000 changes were what had been keeping them on an older version because of breakage, but that had been fixed.
Mathias Lang
Mathias wanted to reiterate the importance of tooling, and he thanked Dennis in advance if he would take a look at dfmt. Symmetry were using a formatter as well, though they were using sdfmt since Amaury maintained it. But a lot more could be done with the tooling and he thought it would integrate with their projects quite well. DMD-as-a-library would enable a lot of good things, and that would be enabled by the AST refactoring project. If we had to choose between an ARM backend for DMD or an immutable AST, his vote would go for the immutable AST.
Dennis
Dennis had been working with COM lately. It had a special calling convention with named arguments, and D had named arguments, so he thought it would be nice to try to combine them. But opDispatch and opCall didn't support named arguments, and the DIP didn't provide any mechanism for it.
He had an idea to add something like opNamedDispatch that would push the named arguments as a template argument as a string array. That would make it work. Walter thought that sounded fun.
Walter
Walter said that he'd been posting updates to X about his daily progress on the AAarch64 code generator. He was currently working on the function call return sequence. Having spent some time with it, he had slowly begun to understand the ARM instruction set. Some parts of it were still baffling. For a supposedly simple instruction set, it was extremely complicated. He had to write little snippets of code and try them out on godbolt to see what they actually did.
He'd gotten to the point where he thought it was going to work, but getting the details done would take a bit of time. He expected to have the basics running fairly soon.
Johan asked if Walter had a machine to run it on. Walter said he'd just been using godbolt. He'd write a function that was two lines or so and wanted to see what instructions were emitted for it. How were they doing pushes and pops? How were they doing function calls? Godbolt was perfect for that.
He said he also had a Raspberry Pi. He expected that Raspberry Pi users would be a potential user base for a simple AArch64 DMD backend. That's what he planned to use as a target machine. The code the compiler currently emitted didn't handle fixups in the elf object files, so actually trying to run the generated code wouldn't work.
Unfortunately, the elf generator code was a rat's nest. He'd been refactoring it to make it more tractable. With the refactoring, it should be easier to implement the AArch64 fixups, which were completely different from the x86 fixups.
He'd already written an object file dumper and a disassembler for it, so the pieces were slowly coming together. Once he had it working on the Raspberry Pi, he'd get a recent Mac with an ARM chip in it and make it work there.
Johan said he'd thought Walter was a Windows user. He'd learned there were some good Windows ARM laptops these days. But at some point, it would be good if Walter had a machine to run it on.
Walter said the Windows backend would probably be the last step, primarily because he found the Windows ABI poorly documented and much more high-effort to get things working. He also didn't like working with the Microsoft SDK. It was excessively massive and complicated, and it was hard to find anything in it. It had too many directories and files, and he just didn't like messing around with it. That was also the reason Windows support was the last to get implemented for ImportC.
He said the Linux development tools were just so much better. They were easier to understand and oriented toward a command line, not a GUI. He found them much simpler to develop code on and much easier to understand what was happening. So he was going to defer the Microsoft problems to last.
Martin noted that DRuntime most likely needed work for Windows AArch64. Johan said that, indeed, LDC probably would not run on a Windows ARM laptop. It would be nice if we had some developers that had one to work on it. It wouldn't necessarily have to be Walter. Maybe one of the core contributors, or someone with skin in the game and the motivation to fix the final bits and pieces.
Dennis asked what Walter planned to do with the PR. Was he going to keep adding to it and merge the whole thing once the test suite passed, or was he going to merge a minimal working version and build on it?
Walter was aware that it was getting larger and larger, and there was no end to that in sight. He was constantly rebasing it so it didn't become hopelessly divergent. It was so large now that it was frankly impossible to review. He asked if Dennis thought it reasonable to simply mainline it even if it wasn't a fully functional backend yet. That would make it easier to manage and avoid it becoming impossible to merge while making future additions easier to review.
Dennis thought in any case it would be nice to have the flag and the skeleton that the actual code generation could build off of. Walter said in that case, he'd be happy to merge it if it passed the test suite. Dennis asked if there were any ARM tests in the test suite that actually ran the generated code. Walter said there were not.
Dennis asked if it was just testing against expected assembly, and Walter said that's what he was doing. He was doing visual checks of the assembly. The -vasm switch had turned out to be really useful for that. He compiled with -vasm, looked at the disassembled code, and compared it with what godbolt put out. He'd even take what godbolt emitted and put it into the disassembler as part of the test suite.
Dennis said he could take a look at it some time. He wanted to look at the code, but it wasn't something he'd be able to do in a single evening. Walter said that was exactly the problem, so he was all for merging it and continuing on from there.
The Next Meetings
Our October Monthly meeting took place the following Friday, July 12th. Our next quarterly was on Friday, October 5th.
