Permalink
Reply
The D Language Foundation's monthly meeting for July 2024 took place on Friday the 12th. It lasted a little over an hour.
The Attendees
The following people attended:
- Aya
- Walter Bright
- Rikki Cattermole
- Timon Gehr
- Dennis Korpel
- Mathias Lang
- Mike Parker
- Steven Schveighoffer
- Adam Wilson
The Summary
Bitfields
The meeting got off to an impromptu start. Walter, Rikki, and I arrived several minutes early and started chatting. At some point, the conversation turned to Walter's bitfields proposal. It continued as I started recording just before the scheduled start time and the others started coming in. When Timon raised his hand to comment, I said that this may as well be the first agenda item.
The recording begins with Walter and Rikki debating something about crossing storage unit boundaries, with Rikki saying it was needed and Walter saying it wasn't. I don't have the full context on that, though. Just after that, I said that I was curious about how often we'd really need to interface with C bitfields from D. In the popular C libraries I'd maintained bindings for over the years, there was only one that used bitfields. I just ignored them, but they were using uint so, according to Walter, that meant they should work out of the box. Even so, how often would we need to worry about it?
Walter said they were needed in the D compiler. GDC and LDC were hybrid C++ and D programs that couldn't use bitfields now. I said that was fine. We could do whatever needed to be done in the compiler. What I was wondering was how often we would need to use bitfields in the wild. If it wasn't very often, then requiring extern(C) to use the C compiler layout for mixed language code and defining our own layout otherwise might be a worthwhile compromise.
Walter said a more interesting question would be how often bitfields must conform to an externally imposed layout. He said that was almost never. The classic case was a hardware register. But then the hardware register was on your computer and the C compiler was on your computer, so it would do what the C compiler did.
Rikki said he'd dealt with it far more often with file formats and networking than when interfacing with C. It was just normal in those contexts. Walter asked if Rikki used file formats with bitfields in them. Rikki said yes, they were pretty common. Walter asked how C interfaced with that. Rikki said by doing it manually. Walter said, "Exactly!"
Timon said Walter was basically advocating for not using bitfields. If he didn't want people to use the feature, he shouldn't give it a nice syntax. That was just a bad design. Walter said they would have to disagree on that. Timon asked if it was a good design to create something that shouldn't be used but had a better syntax than the thing that should be used. In that case, yes, he would agree to disagree.
Walter said he wanted to be able to use bitfields in the D frontend. Timon said he should be allowed to and it should have nice syntax. He agreed with those constraints. He didn't agree that we should just copy the C thing into D. He also thought that if you defined it on the D side, and you defined the C side and it looked the same, then they should match. He wasn't against this.
Walter said they agreed on that point at least. If it looked the same in D and C but did something different, that was going to be completely unexpected and would potentially be a memory corruption error. However, it was documented that D bitfields would match the layout of the associated C compiler.
If you were using another C compiler on the same platform that didn't match the associated C compiler, then there could be no guarantees on the C side that the layouts would match. On POSIX and Mac, he'd never seen a C compiler that didn't match what all the other compilers did on those platforms, because that would be madness.
The only place he'd seen differences was on Windows. The Microsoft compiler did it one way and the GCC ports did it the GCC way. He thought that was an unfortunate decision, but there was nothing he could do about it. The Digital Mars C compiler layout was designed to match Microsoft's. And Microsoft's layout was designed to match what the other C compilers did in the olden days on x86.
He said the people writing C compilers weren't crazy, and they didn't do perverse things just because the standard let them. They followed the herd because they wanted people to use their compilers and didn't want to upset their users. So the only place this had been a problem was between Microsoft and GCC on Windows, where he'd seen people complain about bitfields and other subtle differences between them.
Rikki said he had proposed another solution: an attribute. The default would still match the C compiler, and that was fine, but then people who wanted an out would have one with a UDA.
Walter said the UDA was a good idea, but he thought doing manual alignment as he had proposed in the forums was a better one. It was simple and instantly understandable when you looked at it. There was no confusion about what was happening and it worked. He didn't know of a case where he couldn't make it work. You just lined things up before the bitfield. You could match any layout you wanted by changing how you laid things out. He did that to avoid normal member alignment problems. He'd either add or remove padding fields to make it work so he could avoid having to fiddle with alignment attributes.
I said that my one argument against this was that a lot of our user base, younger people coming into D these days, just generally weren't aware of this kind of thing. They weren't coming from a background where they needed to know these things. They were coming from dynamic languages, or Java and C#, or something else that wasn't C. Experience with that kind of knowledge about layouts just wasn't there. That would be a big hurdle for them, and I thought we should keep that in mind.
Walter said that was a great point. But as he'd mentioned on the forums, that could happen now even without bitfields. Structs had alignment differences between C compilers, and you had to deal with that at some point. I said that very rarely came up in my experience. Walter said that came up for him all the time.
He said there was also byte ordering. Was it Big Endian or Little Endian? He didn't think anyone was making a Big Endian machine anymore, but that was a big deal. That was what the bswap hardware instruction was for. Rikki said Big Endian was still used in networking and wasn't going away anytime soon. Walter agreed.
Steve said Walter was kind of correct in that if you had to deal with it, then you would just have to learn how to deal with it. If the layout were important to you, then you'd learn how it worked. And if you didn't care, you'd be able to use bitfields just fine. If you weren't trying to match a specific layout, then it didn't matter.
His problem was that it was just confusing. If you did want to know how to do the layout, then you needed to understand all these arcane rules and which C compiler was being used. It wasn't obvious why something was laid out in a specific way.
Walter asked what kind of naive users would add attributes to their bitfields to match a certain layout. It wouldn't occur to them that they needed to do it. Steve said the naive user wasn't going to care. They weren't going to be doing layout things. If they did want to match what a C header did, then they'd type in whatever was there and it should just work. Walter said it would work.
Steve said yes, that was the whole point. D was matching the C layout, so it would work. But if you were trying to match a layout that was in a protocol spec or something, the spec wouldn't say "use an int here" or "use a long there". It would just say "use this many bits". So now you had to figure out how to make the bits line up.
Walter said the beauty of that was that if it ended up being misaligned, you'd find out immediately. It wouldn't work. Steve said then you wouldn't know why. Walter said you would. He'd conformed to external layouts before, and the first thing he did was test to see that it worked.
Steve said you wouldn't know how to fix it. Say you needed to align something to 64 bits, so you used a long and it didn't work. What then?
Walter said he'd be happy to add a description of how to do manual alignment to the bitfields documentation. Then anyone reading would see a section that said, "If you need to conform to an external layout, here's how".
Timon said he shouldn't need to figure out how to do the layouts on the platforms he cared about. It was unnecessary work. He worried that people who saw the shiny bitfields would assume they behaved the same way everywhere, would do something that wasn't sane, and then a problem would crop up in one of his dependencies. He would have to fork the dependency to make it work on the platforms he cared about because they differed from the platforms the other person cared about.
Walter said if you were using code from somebody else who made it work by using alignments or manually adding padding, then it would work on your system, too. Timon said he understood that. He was talking about two separate people working on two separate dependencies, and one of them screwed up on Windows because they only cared about Linux. He had Windows users, and now it was his problem.
Walter said that would be true even with a struct layout that wasn't anticipated for the system you were building the same code on. Timon agreed but said that was much less common. Walter said if you switched between -m32 and -m64, your struct layouts would differ. That was true on every compiler he'd ever tried it on.
Timon agreed, but by doing things in a sane way, you'd never run into this. We should just enforce sanity. He didn't see why this was so controversial.
Walter said it was controversial because if you wanted the default in D to do something different than what the C compiler did, it would be surprising to the user. A few people shook their heads, including Timon and Steve. Walter said he saw they disagreed with him, but he would be surprised if it happened to him.
Steve said he disagreed that this was their position. Walter said he didn't see how it wasn't their position. The default behavior should match the associated C compiler and do the same thing. It was the same with structs now. If you had a struct in C and wanted to use it in D, you could do that because it matched the associated C compiler. Steve said that was his position as well.
Adam said he wanted to back up the point I'd made about new users based on his experience with C#. They had bitfields there that they called "flags". It was just an enum that you'd stick what we'd call a UDA onto, and then you could do bitfield operations on it. People generally didn't have a problem with it. There was a little bit of documentation on it, but requiring an attribute to change the layout of an enum wasn't asking a whole lot. Even people coming to D would understand that a bitfield was something special to which they'd have to attach an attribute.
Rikki said Nim had three modes for bitfields. One matched the C compiler, one was the packed thing that we wanted to do, and the third was it could do whatever it wanted. That was working for them. He said Rust was getting bitfields as well. He thought they weren't always going to use the C layout for them.
Walter said the original design for D was that the compiler could reorder struct members any way it wanted. He'd been specifically thinking about reordering to eliminate packing holes and things like that. It soon became obvious that it was just a bad idea. So he'd implemented it the way C did it and he was done. He'd never had a single complaint about field alignment in the D compiler. Not one. And the alignment did change things.
Another reason he'd abandoned reordering was that he'd realized that people sometimes ordered fields in certain ways for caching. They wanted their most used field in the hottest part of the cache. You wanted to group together the stuff you used regularly and put the rest somewhere else. In that case, you didn't want the compiler reordering it. If somebody carefully laid out a struct in C, and then the D compiler came along and reordered it, that wouldn't go over very well.
At this point, I said we should table this discussion. We had two scheduled agenda items to get to, and this one had taken up a significant chunk of time.
(NOTE: In a planning session in May, we had agreed that Walter should make some changes to the bitfields DIP, after which it would be okay to move forward for a verdict from Átila. Walter did make some changes to it, but has put it on the agenda for the December meeting for further discussion.)
Sumtypes
Rikki said he had a DIP in development which covered the basics of matching a type. It was three weeks in and was ready for a pull request. He said he'd give it another week if anyone wanted to have a look and comment. He said there was something iffy about how he was supporting static if, but it was ready to go.
Second, he talked about his proposal for a "memberof" operator. It had a problem regarding argument-to-parameter matching, something that Walter had voiced strong opinions about in the past. That had come up again in the form of type inference in the DIP Ideas forum. Rikki said he was blocked on that and wanted Walter to make a decision about a way forward if there was one.
Walter asked Rikki to write it up so he could fully understand it. Rikki said he'd simplify it: it was a simple rewrite to context.identifier. Walter said he would have to look into it and study it. He wasn't that good at thinking on his feet about something he wasn't using. Rikki said it was already implemented. Walter asked him to share the PR link and asked if there was any documentation for it. Rikki said there were comments. Walter asked if he could also do a changelog entry. Rikki said he would.
Rikki said the main problem with parameter/argument matching was that it had to call into semantic for verification because he didn't see another way to do it. He needed to know if it would have to be rewritten to avoid calling into semantic. Walter said he understood the question but he didn't understand the problem.
Rikki said that as a feature, it could be extended out to identity types and a bunch of cool stuff. It would be nice to have. Over the years, a frequent request was to be able to say, "Here's my identifier, here's an enum, get the member for me so that I don't have to write out the enum".
Walter said he would need an explanation of the feature and what it was for because he didn't know. Since it was something he wasn't familiar with, he couldn't have an opinion on it right now.
Rikki said that was good. He added that it wasn't just blocking sumtypes, it was also blocking value type exceptions because they needed a zero-size value type to be returned. It was in a register. That was how you got zero-size exception support. Walter said he didn't know what a "zero-size exception" was. Rikki said it was from one of the C++ proposals.
Walter said that C++ had gone so far off the deep end that he was very wary of doing anything just because C++ was doing it. Rikki said it was their family of sumtype-based exception handling. It was all kind of the same thing, just different names. He was pretty sure that for D we would only need to do what he had designed.
Steve asked if there was an article or video that explained sumtype exceptions so that we could understand them. Rikki said he'd written a DIP. Steve said he didn't need a DIP. He was asking about something for the layman user. How would they use it and how would it work? Rikki started to explain what it was about, but Steve said he was asking for something to look at outside of the meeting, not to talk about right now.
Walter asked if Rikki was talking about throwing a sumtype value. Rikki said it was a struct, not a sumtype. It was a sumtype under the hood. Walter said D didn't throw structs. Rikki said under a new mechanism it could. It wouldn't be calling out to the runtime exception mechanism. It would all be through the return of the function. It would just do it very nicely and hook into try/catch and throw.
Walter said he wanted to reduce the amount of complexity in try/catch and throw, not add value types to it. Rikki said it was a completely different mechanism, something we'd been talking about for ages. Walter said he was going to have to see a document on this. C++'s ability to throw values had been a gigantic design mistake.
Rikki said that was completely different. Mathias suggested that the value type was just an implementation detail. Rikki said it sort of was. It would throw your struct. The implementation detail was that sometimes it would actually be returned from the function in addition to the return type that you would specify.
Walter said he didn't understand how this had anything to do with exceptions. Rikki said it used the same syntax and was called an exception, but was a completely different mechanism.
Walter said this was obviously a complicated thing and he could not say it was a great idea or a bad idea or anything in between without taking the time to study its design. He said we could argue about bitfields all day because he knew it well. He didn't know anything about this.
Timon said he was pretty sure that Walter had suggested something like this before in the newsgroups. He expected Walter just didn't understand what Rikki was referring to. Walter said that might be entirely true.
Dennis asked if the idea was that you'd be returning a traditional error code, but in this case, the compiler would rewrite throw and catch to check the error code. Mathias said that was his understanding, too. Rikki said you could pass whatever user data you wanted.
Walter asked why you couldn't just use return, then. Dennis said you could forget to check it. It was also error-prone. Walter said okay, he just needed more information. He couldn't figure it out and think about it with just a conversation. He asked Rikki to please write a document, then he would study it and get back to him.
Mathias asked if we did that without using the exception ABI, wouldn't it break the ability to use exceptions between D and C++? Rikki said it wouldn't. One was a class which used the existing mechanism, one was a struct which didn't.
Walter said he'd given up on catching C++ exceptions in D on Windows. It was undocumented by Microsoft and too complicated, so he'd just abandoned it. It worked on 32-bit Windows because it was documented. Mircrosoft claimed they had documented it for 64-bit, too, but they hadn't. It was a giant mess. He would have to spend a great deal of time reverse engineering it, and he didn't want to spend the time on it. So D had its own exception handling mechanism on 64-bit Windows.
Returning variable-sized stack allocations from a function
Aya said she'd been wondering if it were possible for the language to return a variable-sized stack allocation from a function right now. Walter said it was not. Aya thought we should have some syntactic sugar for it. There was a way to do it, but it was clunky.
Walter said that normally when you returned variable-sized things, you did it by reference. Aya said you would have to allocate at the call site and pass the function a pointer to the memory for it to populate. That would effectively be a stack return.
Walter said that would work, but another thing was that we didn't want to step away from the C ABI. He'd tried that before with D and it had been a disaster. We had to conform to the C ABI because not only did everybody follow it, the debugger wouldn't work with anything but the C standard ABI. Even though the documentation said it would, it would not. Apparently, no one ever tested the debugger with a non-C calling convention. Inventing our own would not work.
However, the C ABI did say that if you were passing a large struct, a pointer to it would be passed instead and then the code generator would fill it in. That was just a variation on the idea of return by reference.
Aya said she was thinking about smaller allocations in general. It was possible to call a function that returned the size of the allocation. Then maybe you'd need to store what you were going to allocate in intermediate storage, which then would need to be allocated. So you would either need to allocate twice anyway or call twice all of the logic that determined the size of the return value. Her main thought was that she didn't know if there was a nice way to simplify it.
Walter said it would have to be done by reference one way or another. Aya agreed.
Timon said this was just the named return value optimization for a type with sizeof that was a dynamic value. He thought it would be a nice thing to have, but a bit tricky to build.
Aya said she'd also been thinking about struct interfaces. I mentioned Atila's library. Mathias asked why Aya couldn't use classes. Walter connected that back to the variable-sized allocations, saying that classes were variable-sized objects passed by reference. Maybe Aya could think about using classes to implement the idea.
Aya said the problem was when you had a very small struct you wanted to have an interface for and a lot of small structs. You'd just have a lot of small classes implementing the same interface with lots of heap allocations and fragmentation for no reason.
Walter suggested using COM classes. They were simpler and smaller. Aya asked if they were portable. Walter said they were. They were meant to match the COM interface in Windows. The compiler support was there. The smallest COM object only consisted of a pointer. Googling would turn up an explanation of them. They were pretty clever and a nice feature. Aya said she would look into it.
While the rest of us discussed the next agenda item, Aya did some searching about COM, so we came back to her.
She asked if IUnknown was required to work with COM interfaces. Walter said it was. Aya said that was behind a version(Windows). Walter said we could remove it. Rikki said it could be used without IUknown. It could work with anything.
Adam said several things in D were improperly versioned to Windows, so it shouldn't be taken as gospel. He'd encountered it with ODBC support, something he'd been using on Linux. In those cases, it should be okay to just PR it away. He could guarantee that COM worked on Linux and on Mac. It was just a C ABI thing.
Walter noted that Microsoft's COM interface relied on the existance of the functions QueryInterface, Add, and Release. If you didn't inherit from IUnknown, you wouldn't get those. Adam said that was correct. You could do it without those, but you'd have to recreate the functionality.
Dennis said that Vladimir had wanted to expose some Windows bitmap-related structs, and as he recalled Walter and Jonathan had been opposed to it because it would be a bug to import Windows stuff on non-Windows. Adam said in this case COM itself was not Windows-specific. Walter brought up the GUID thing in COM and thought that was Windows-specific.
Adam said that was true. The struct itself could probably be exposed because there was nothing specific to Windows in that. Walter said you would need a way to get a GUID. Adam asked if we could use UUID and use Microsoft's methodology. Walter said yes, but something would have to be done to make that work.
Adam said his general point was that COM was just a C ABI trick. Windows had a bunch of support for it that probably wasn't available elsewhere. If we wanted to make the effort to recreate it in our own environment, that might work. He thought we didn't need the interface query part of it.
After this, there was a bit of discussion to clarify some confusion Aya had about the spec. In the end, she said she was good.
An update from Walter
Walter said he'd been working on two things.
First, we'd had two major requests to get move constructors working, so it was a priority. He would divide his time between move constructors and ARM code generation. Weka had wanted the feature for a long time. They were a major user, so we needed to work on it.
Second, the PR for the ARM backend was a diff of over 5000 LOC and was practically unreviewable. He'd talked with Dennis about merging it even though it wasn't yet a functioning backend. He continually rebased it, but it was hard for anyone to follow his progress because he kept adding to this gigantic PR. He thought it would be better if it were mainlined.
Walter said that Dennis had brought up testing it. Running it through the test suite would not work, because the test suite required a fully functioning compiler. Adding an ARM target to the test suite would probably be necessary in the future. He didn't know how that might work. He didn't know much about how the test suite was set up.
Rikki said as long as the ARM backend wasn't exposed and none of the pathways were called into, then merge it. Walter said he'd put it behind a switch and he didn't think it would affect anyone's code. Rikki said he should document the switch, but otherwise it sounded fine to merge.
Conclusion
Before we left, I gave an update on the planning for DConf '24 and some potential hiccups (which thankfully didn't come to pass), and Rikki said that he and Steve had been looking into how other languages handled coroutines and had concluded that stackless was the way to go. Then we called it.
Our next monthly meeting took place on August 9th at 15:00 UTC.
If you have something you'd like to discuss with us in one of our monthly meetings, feel free to reach out to me and let me know.
