Permalink
Reply
The monthly meeting for June 2023 took place on Friday the 2nd at 15:00 UTC. It lasted about an hour. We had three new faces for this one: Adam D. Ruppe, Steven Schveighoffer, and Nick Treleaven. I'll be sending them invites to all of our future monthlies, but we've left it up to them to decide if they want to become regular or part-time members.
The following people attended:
- Walter Bright
- Ali Çehreli
- Martin Kinkelin
- Dennis Korpel
- Mathias Lang
- Átila Neves
- Razvan Nitu
- Mike Parker
- Adam D. Ruppe
- Robert Schadek
- Steven Schveighoffer
- Nick Treleaven
The summary
Me
I opened with a reminder that Razvan had sent most of us an email reporting that he had set up an initial set of projects on our GitHub Projects page as we'd decided in our planning sessions. The projects were all private. I emphasized that once we made them public, we needed to start focusing on them. The project list should become part of our regular workflow. I asked that everyone look over the project list and make sure they're okay with it, after which we could make it public. (We have since done so.)
Razvan said that when any of us submit a PR related to one of the projects, we should add it to the project so that anyone who wants to can follow along. Dennis said he was excited for us all to start using organized task lists, and Walter thanked Razvan for setting it up.
Next, I gave an update on the DConf submissions. We'd received 39 submissions from 20 people and 1 team. We had already begun the selection process. I and one other person had made our selections and I was still waiting for two more. I reminded them that I wanted to publish the schedule as soon as possible, so I asked that they get their choices made over the weekend. (The four people making the selections were Walter, Átila, Ali, and me. I had all of the selections by Saturday night my time and subsequently informed all of the submission authors of their status.)
Finally, I told everyone about a conversation I'd had with someone who was planning to submit a proposal to add slices to C. Walter was happy to hear about that, as he had been informally pushing for that for years. He talked about his article on the topic (C's Biggest Mistake) and how C++ had copied that mistake. He says it can be backward compatible in C and analogous to D's slices, it just wouldn't allocate memory for you.
Robert
Robert said his biggest gripes had been added to the project list, so he was a happy camper.
Other than that, the only thing he had to report was an update on the Bugzilla to GitHub migration. At the time, I had done some testing with the program he wrote for it and found some problems with it, so he was going to work on fixing those and improving the usability. I had set up a dummy repository to test it out on and he would use that for further testing. (He got it back to me a couple of weeks later. I have since submitted two forum posts (post one and post two) soliciting feedback and providing updates. You can track the progress on this task in our Enhance the Ecosystem project.)
Mathias
Mathias had nothing to report.
Martin
Martin had been on vacation, so had nothing completed to report on. His immediate plans were to finalize the LDC 1.33 release, which was based on D 2.103, then get to work on the 2.104 merge. (He announced the first 1.33 beta on May 14 and the second on June 11).
Dennis
Dennis was in a car, so was only listening.
Ali
Ali had nothing to report. He was still using D at work. He was happily producing successful stuff with it.
Razvan
An issue with runtime hooks
Razvan opened by reminding everyone that Teodor Dutu was still working on converting DRuntime hooks to templates and that the project was part of a major goal we had set to get to a pay-as-you-go DRuntime (which you can follow on our project tracker). He had noticed that some of those hooks are implemented in rt/lifetime.d. Nothing in the rt package is supposed to be imported outside of the rt package itself (see the DRuntime README). However, object.d needs to use some of those hooks, so it declares extern(C) function prototypes to make that possible. He asked if anyone knew why we were doing it that way.
Mathias said it's just for historical reasons. Many of those hooks had been moved to core.internal. Razvan was welcome to move more of them there if he wanted to. Razvan said he'd like to move everything from lifetime.d to core.internal. Some of those hooks also use private functions in that module which then use the GC. He had tried to lower one hook and had to resort to all sorts of trickery to make it compile. Mathias said to just move it. There's precedent for that.
Steve said he would caution against moving everything into a module that object.d imports because then everything is going to import everything. There's always some reason for hiding an implementation behind an extern(C) function, even if it's not always an exceptional one, but not importing the whole world in every module would be a good goal to keep in mind. Walter agreed. object.d is special in that every program is going to import it, so anything added to a module that object.d imports needs to be evaluated to understand if it's going to increase compile times for user programs.
Martin said there's more to it than that. It's the whole import graph with all of the ModuleInfo dependencies. If object.d is just importing stuff from internal packages, the compiler needs to load it, parse it, and check it, but if one of the ModuleInfo dependencies has a module constructor that depends on other modules, then as soon as you need one little thing from object.d, you're going to pull in half or more of DRuntime just due to the interdependencies. It's a tricky business.
Adam noted that object.d is responsible for the largest compile-time slowdown compared to D1. import object is more expensive than compiling his entire old D1 game that has fancy art and music.
Razvan said that he hadn't done any experiments, but object.d is already importing a large part of DRuntime. It's not importing rt.lifetime, but it's using the extern(C) functions from there. He doesn't know if those are hiding anything, but it seems they're extern(C) only to make them available at run time. Martin said we might do something like lazily importing the module during template lowering instead of importing everything into object.d.
After the sidetrack we got onto (summarized below), Razvan said he would experiment with moving some of the rt hooks into core for import to see what sort of impact it has.
Sidetrack: static constructors
Steve said this reminds him that we have this longstanding problem of import cycles when static constructors are present, and it can be hard to get rid of some of those. Yes, you can refactor your module system when that happens, but it's not always easy. That's one of the reasons why we have all these extern(C) prototypes. He wondered if there's any room for a feature that allows setting up a static constructor to indicate that it doesn't have any dependencies, i.e., it runs independently so shouldn't figure into the cycle detection.
Walter said the pragma(crt_constructor) will do that. Martin disagreed. You can't use the GC at that point, because it's initialized after CRT constructors are run. So it's not a direct one-to-one alternative. Walter said that, yes, it has to be independent. And that means being independent of the GC, too. Adam said that's not technically true. CRT constructors also have priority lists, and you can have them run before or after malloc is even run. They're not a solution to much of anything.
Walter said he'd used them to break cycles many times. He said CRT constructors are added to C's static initialization lists, which happens before D is even set up to run. So you have to be careful about what you're using it for, but it's very handy to have that. Mathias noted that's breaking the cycle by just removing the cycle detection. Walter said that's true, but they also have to be completely independent because the order in which CRT constructors are run is not defined. It's a system thing, so it requires more care than using D's static construction. But it exists. It's a way to run an initializer before C's main gets called.
Adam said that works fine until someone puts it in a DLL. Walter agreed, and that's why you have to be careful with it. Adam then said that this is a general problem with D: someone points out an issue, and the answer is a workaround hack that works in some cases. We should be looking at fixing the actual D feature instead of punting it to a C thing that doesn't solve the problem for most people. Walter said the static constructors he'd looked at all worked rather nicely. They had no dependencies, although they were circular, so it worked fine to make CRT constructors out of them. It doesn't add any more problems that you don't already have in C when you're using static constructors.
Walter said the reason circular constructors are a problem is because there's no way to tell which one should come first. Adam said we should fix that. Walter asked if it's something that really needs fixing. If you have modules importing each other, there isn't any obvious way to determine which has priority. If you import more modules in there, you create more problems. Adding keywords to D to say which one should come first doesn't make a whole lot of sense. That's why he's never done it.
Mathias said to some degree he agrees with Adam. The compiler should be smarter about detecting module dependencies. Walter didn't see how. Mathias said that currently, it assumes there are dependencies on everything. If it were a bit smarter, most of those cases could probably be handled. Walter said he didn't know how it could reliably determine which one should come first. However, if Mathias or Adam (or anyone else) came up with a proposal for that, he'd be interested to see if someone could solve it cleanly.
Steve said his point in bringing it up was to see if we can think about that as an enhancement. Walter said we can. Steve detailed a case that had come up in Discord where an exception was thrown during static construction, but there was no stack trace because a particular DRuntime static constructor hadn't yet run. These kinds of problems aren't uncommon.
Átila said he found out the hard way that cycle detection can vary depending on if you compile everything all at once, or separately by module or by package. One of them might say there are no cyclic dependencies, but another says there are (Steve said in the chat window that this is a straight-up bug). Átila had also learned that static constructors should always be nothrow. Walter said he couldn't see why anyone would write a static constructor that throws. Átila said the point is, it's not muscle memory. It's easy to forget to add nothrow. Walter asked him to file an enhancement request on Bugzilla to make static construction nothrow.
Adam proposed one possible way to solve the problem: you can see which symbols are referenced inside a function, so if you can see that it doesn't reference anything outside of its namespace, you can then infer that it is independent. Another: you could do a pragma(nocycle) or something. There are a few things we could do if we're willing to change the language.
I suggested we revisit this in the future. That took us back to Razvan.
The project tracker
Razvan said he had been looking at updating the project tracker with the tasks we had discussed at our planning sessions. Before this meeting, he'd had a meeting with Mathias to discuss items in the ecosystem category. They concluded that there are two projects each of them could own, but there are other projects for which we need to find owners. For example, WebASM support. He didn't know who proposed it, but we didn't have a description yet of what it entails. We needed to iron out the specific tasks. He then noted that two projects Robert had proposed (an incremental compiler daemon and built-in Language Server Protocol support), though independent projects, also fit under the 'dmd as a library' concept.
He said that in a few weeks, he would be hosting the D Summer School (which he usually runs with Eduard Staniloiu) at the Politechnica University of Bucharest. He hoped to identify some students whom he could nudge toward some of the projects on our list.
Nick
Nick said he had been looking at the Gripes and Wishes feedback. He wanted to go through it and think about it more but would like to discuss it at some point. I told him that the plan was to ultimately resolve all the items on that list in some form (implement it, reject it, push it off until later, etc.). We had already selected a few items with which we seeded our task lists. I shared with him the notes I'd made from the planning session where we'd discussed it, which includes items from the Gripes and Wishes feedback and others proposed in the meeting.
I told Nick that Ali and I were managing the community-focused projects, one of which was going through old articles and other documentation on dlang.org and bringing them up to date. Given all of the work he'd put into fixing up the documentation, I thought it would be great if he could help us with that. He said that sounded good.
(Nick has put out a steady stream of PRs since then, but I haven't yet arranged a time for us to discuss the documentation update project. I do intend to schedule a meeting with Nick and Ali to coordinate an approach to it. I've recently set up a dedicated project in the project tracker and seeded it with a couple of tasks that came up in our recent quarterly meeting.)
Steve
Steve said that after a discussion at one of the BeerConfs, he had built a library that's an exact copy of how we lay out associative arrays but can be built at compile time. Then at run time, you can cast it to a normal associative array and it just works. We should be able to have a hook that does this. Since he already had the run-time part of it working, it just needed someone to handle the compiler side of it. He said that would be a huge win if we could get it working, as people complain all the time that they can't statically initialize associative arrays.
Átila said it sounded awesome, and Walter thanked him for working on it. I noted that we had discussed it at a planning session and marked it as a future task (as per my previously linked notes). However, this does seem to fall under our current goal of stabilizing the language, so if there was a path for us to get it implemented sooner, that would be great.
Steve gave a brief description of what needed to be done, but he said he didn't know enough about the compiler internals to implement it. But if someone could give him the necessary hook, then he could write the hook. Walter said we should put it on the list.
(Steve and Walter had a call about this a couple of weeks later. The outcome: Walter thinks it's feasible, probably with some modifications to the initializer code in the compiler, but he doesn't have the time for it right now. He suggested Steve do a writeup of what needs to be done and post it to the forums to see if he can find someone to help with the compiler part of it.)
Adam
Adam said it was hard for him to think of issues to bring up specifically, as when he encounters one he usually just finds a workaround and moves on. The static constructor cycles we discussed earlier were an example of that. Even if we do fix the problem, it won't affect him directly, but it would be great if the language worked as advertised. Being told to "just use C" isn't helpful, and he would like to see that attitude change.
I asked him to flesh out one of the possible solutions he had mentioned earlier, then at a future meeting, we can decide if it's something we should pursue. I told him I'd follow up with him in a week or so.
(When I reached out to him later, Adam proposed an attribute that causes a static constructor to be added to the order-independent list rather than the order-dependent list, e.g.,
@core.attribute.order_independent
shared static this() {}
)
He then brought up object.factory, which he had written a blog post about (he had posted the link in the chat during our discussion about static constructor cycles, as the article touches on that). He said he had written it in response to a pull request that was going to add a new trait that Walter had said could replace object.factory, but Adam said it wouldn't actually do that. It would help if Walter would try to use things like this himself so he could see when something is going to be a dead end. Then we wouldn't end up with cruft just lingering in the compiler.
Átila
Átila said he was tired of shared and he wished someone would take it off his plate. That's what he'd been working on and he just wanted it to go away. He'd been trying to get DRuntime to compile with -preview=nosharedaccess. He'd found some bugs along the way, and some other things were simply annoying. But if that goes well, then he'll move on to Phobos. He didn't anticipate any big problems getting it to compile with the switch. And then after that, he'll start trying it with different dub packages.
As an example of the kinds of issues he'd encountered, he had figured out that his changes to DRuntime for using fprintf would be a pain to anyone ever using that function. If you're passing in stdout or stderr, you're going to have to atomic load every single time. Is that what we want? He wasn't sure. But he couldn't see any other way to do it since those are shared pointers.
Walter said that in that case, C does all the locking and takes care of the synchronization for us. Átila said that had crossed his mind. But, though implementations probably do it that way in real life, he didn't think there was any guarantee from the C standard. Walter said that, pragmatically, it's all taken care of. We should make stdout and friends unshared and let the C standard library deal with it. Átila said that, again, it had crossed his mind, but there were potential problems because of how DRuntime uses them internally.
Steve told Átila he was interested in getting the shared stuff working, and asked him to ping if he needed any help.
Walter
Tweeting about D
Walter started by telling us his evil plan to increase his follower count on Twitter had borne some fruit. He'd tried various things and found that what worked was thinking of a feature of D that could fit into a single tweet and just write about it. That had generated a fair amount of interest from other people. He'd also been tweeting about whatever he was working on and had added some hashtags for programming communities. It seemed to have worked out positively so far. He encourages everyone who uses Twitter to do something similar. Tweet about what you're working on in D or your favorite D features.
ImportC
He had fixed enough problems with ImportC that he thought it was ready for use. It wasn't perfect. There were still some missing things, like variable-length arrays, a C feature he dislikes intensely. It's something not commonly encountered. However, someone had posted to Bugzilla about it, so though he was hoping to avoid it he may end up having to do something about it. Other people had pointed out that all of Microsoft's intrinsics that D didn't support. He was also hoping we wouldn't have to deal with that. But in general, ImportC seemed to be working. The closer we can get to not having to do any manual C bindings, the better. Even if you can import a C header file and it mostly works, then you only have to write a few bindings by hand, it's still a big win.
Steve said it was not exactly clear what the result with ImportC was going to be. He remembered at one of the BeerConfs he had mentioned the issues he had porting Raylib to D because of all the crazy preprocessor stuff it does in its internal headers, and Walter had suggested trying ImportC. He had tried it, but it had broken in some really deep vector stuff, and he wasn't sure what was happening. If the goal is to be able to import header files to link against C libraries, that's one thing, but if you want to build dependencies with ImportC, that's a different thing. Was the goal to import header files, import header files with a few tweaks, or build C projects? It would be nice to have that well-defined.
Walter said the Number One goal for ImportC was to obviate the need to write C bindings by hand. The next goal is to make it easier for your C code to call D code. The interface works both ways. The third goal is to compile your C code with ImportC. That's probably not achievable because it's unlikely ImportC is ever going to support all of the compiler extensions out there. It's like Microsoft's blizzard of intrinsic functions. It's kind of an impractical goal to try to support all of that. He likes to try to get ImportC to compile as much as he can because that makes it more usable. But some things it likely will never be able to.
Martin said that from his perspective it's only goal Number One that counts. If you look at how libraries are built these days with all the CMake configurations and such, if we want to have all of that in the D compiler, too, or support in CMake for D compilers to compile C files, it's not worthwhile. On Linux, we all link against the prebuilt distro packages anyway, or at least those which are available. So for him, the absolute single goal of ImportC is the import of header files.
Walter said Martin was right. We'll never succeed in making ImportC a general-purpose C compiler. But, for example, Phobos has the zlib package which is a bunch of C code. There's no reason we can't compile that with ImportC and dogfood it. That's a worthwhile goal. It should be able to compile standard-conforming C code. It's unreasonable to try to implement all of the GNU C and Clang extensions.
A lot of ImportC contains hacks to get it to compile the various header files, and that's going to continue. For example, it recognizes the GNU inline assembler code, except it just ignores it because it can't compile it. But some of the C header files require you to benignly parse it to be able to import the header file. ImportC is never going to compile GNU assembly syntax, because it's all wrapped up in how their assembler works. It's too big a project and a waste of time to compile that. So it's always going to have limitations like that, and that's okay.
Adam thought there was nothing to gain by compiling zlib with ImportC since you can just use a C compiler. Martin said it was kind of nice to compile it with ImportC because then you don't need a C compiler to compile DMD/DRuntime/Phobos.
Dennis noted that you currently can't import .h files directly, only .c and .i files. Walter said that's because of a disagreement he and Iain had about the implementation. Since Iain wasn't present, he felt it wouldn't be fair to debate the issue in this meeting.
Windows DLL support
Walter said he had been looking into the longstanding problems with Windows DLL support. In poking around, he'd found that documentation for it was practically non-existent. What little does exist is spread out among random source files and various areas in the documentation. So as his first step in trying to solve the problems, he was writing an article that lays out how Windows DLLs work and how that fits in with D. Once he finishes that, he should have a good understanding of what the problems are and how to fix them. He knows it's been a sore in the side of D developers on Windows for a long time, and he was ready to put his focus on it now.
Martin noted that LDC has some support for DLLs that he'd come up with. Rainer Schütze had opened a PR that tried to mimic LDC's approach, but he hadn't looked at it and didn't know how far Rainer had gotten with it. It's a complicated topic. He had often thought of writing an article about it but hadn't made the time for it. He was open to discussion to elaborate on how LDC does things, though it would be a long and heavily technical discussion, so he wasn't looking forward to it. But we'll need it at some point.
Walter said they do need to discuss it, as it appears he might be reinventing the wheel on what Martin had done, but he didn't know what Martin had done or how LDC handles it. Martin said it involves many things: DRuntime parts, and compiler parts. The main problem is importing data symbols from other binaries. Then there's the Windows limitation of restricting the number of symbols to 64K per DLL, which means large DLLs that export everything like on POSIX systems aren't an option. It also involves dub and what he thinks would be desirable behavior to make this work in a very simplistic fashion so that people don't need to change their code. It's a big topic.
Walter said he'd keep working on the article and make it available so that Martin could poke holes in it. He wanted to get all of this organized.
The next meetings
We had a couple of planning sessions after this, on which I've already reported. Our next big meeting was a quarterly on July 7. I'll have the summary for that in a couple of weeks. Our next monthly meeting is scheduled for July 14 at 15:00 UTC.
If you have something you'd like to discuss with us at one of our monthlies, please let me know and I'll arrange to bring you into one. Additionally, if you're interested in contributing in some way but aren't sure where to start, please get in touch with me. I'll be happy to chat with you to determine where your interests align with our goals. Then maybe we can find ways to help each other.
