Thread overview
D Language Foundation August 2023 Monthly Meeting Summary
Aug 22
An Pham
Aug 25
matheus
Aug 25
harakim
Sep 02
harakim
Sep 02
Bonarc
August 22

The D Language Foundation's monthly meeting for August 2023 took place on the 11th. At just under 35 minutes, it was our shortest monthly meeting to date.

The attendees

The following people attended the meeting:

  • Walter Bright
  • Ali Çehreli
  • Martin Kinkelin
  • Dennis Korpel
  • Mathias Lang
  • Átila Neves
  • Mike Parker
  • Adam D. Ruppe
  • Robert Schadek
  • Steven Schveighoffer

The summary

Me

I had nothing new, other than to say that DConf planning was on track.

Adam

Adam had nothing for us this time.

Robert

Robert said he was happy. Dennis asked if the compiler was fast enough (as regular readers of these summaries may know, Robert often says the compiler is too slow). Robert said it's still slow. He just wanted to see LSP support in the compiler and take all the good things.

Walter noted he'd been working on improving compilation speeds. He'd realized that the compiler was being compiled with exceptions, so he'd been working on making it compile without exceptions to see what effect that had on speed. It might be a small difference, but every little difference helps.

Adam said his library compiled in half a second. Robert said that was still too slow. His work project didn't compile in half a second. He said his goal was that by the time of the key-up event after pressing 'Enter' on Ninja or whatever build tool/compiler he's using, it should have already restarted the application. Átila agreed that half a second was too long.

Adam brought up D1. An old game he wrote back in 2007 compiles and links in 10 milliseconds. If you looked at a big chunk of the difference between D1 and D2, object.d back then was around 600 lines. Now it was much bigger, and that adds up. Walter said that he was a bit upset with the state of object.d. It was supposed to be a minimal thing and had turned into a kind of garbage dump somewhere along the way.

Steve

Steve said that Dennis had been doing a fantastic job with his PR on the compiler part of static associative array initialization. Steve thought it was about ready to go if someone could push the button. He didn't want to be the one to push it, because he didn't know the compiler code very well and couldn't vouch for its correctness. (The PR is still open as I write.)

Dennis

Dennis said he'd been working on that AA PR, but had nothing else to report. He still had some other things to pursue, like looking at pattern support for the deprecation (-d/-de/-dw) and obsolete warning (-wo) switches, as well as the standalone attribute or pragma for static constructors.

(In our July monthly, Adam proposed a means to annotate static constructors that could make them independent of cycle detection. In a subsequent planning session, we agreed to implement it. Since the August meeting, Dennis has submitted a PR for the @standalone attribute, but in the PR discussion thread, Walter thinks it should be a pragma.)

Martin

Martin said the bump to version 2.104 of the frontend went well on the LDC side and was quite painless.

Next, he had an interesting side note inspired by a forum post he'd seen about getting LDC to work on a 8-bit AVR platform. He'd looked into it and found it an interesting question in terms of whether and how we want to support architectures with pointer sizes less than 32 bits. Currently, the frontend has a hard-coded limit: size_t and ptrdiff_t are either 32 or 64 bits, nothing lower than that. It's a frontend check, and the definitions in DRuntime directly depend on the compiler's internal type because they use the typeof expression.

He had tried to make the frontend accept 16-bit and 8-bit size_t and ptrdiff_t, but doing that bumps up against the integer promotion rules. The promotion rules can work in some specific cases when using a smaller int, but when you have something like the length of a slice and do a little plus one or minus one, that's a problem. You need to cast it. So either we need to change dealing with the promotion rules in this regard, try to mimic C++ in this regard, or hopefully do something better. He said Adam had proposed something in a past discussion that would handle it generically so it's still target independent.

Walter wondered if Rust supported less than 32-bit compilation and what compromise they made for that. Martin said he wasn't sure. He'd only checked C and C++. Robert looked it up and found that Rust has a tiny bit of support for 16-bit MSP340, but it's a big asterisk and everything else is 32 or 64.

Martin wasn't sure if it was even worth looking into these architectures, because he had no idea how long they'd be around. For low power consumption, even if you're playing a few years ahead, he thinks it's going to be 32-bits even for those very small and power-efficient sensors, or whatever. But he couldn't look into the future, so he had no real idea.

Walter said when he'd designed D, he thought 16-bit architectures were dead and everybody would be using embedded 32-bit processors. But that was a huge simplification. There was so much trouble in C from the ints not being a predictable size. Just endless problems with it and suffering because of it.

Mathias

Mathias had nothing to report.

Ali

Ali had mentioned in our July quarterly meeting that his company (Mercedes Benz Research & Development North America) was looking for a D programming intern. He'd since contacted Mike Shah at Northeastern University. As an example of industry/academia collaboration, Mike had recommended a strong candidate, and the company had already hired him. They planned to bring on another intern who comes with a good recommendation. So the D team is increasing to three people.

Ali wanted to expand his team to be more like a tools team, like a wildcard team working on multiple different things inside the company. The new candidate agrees. So they were expanding and they were working on something he couldn't yet provide details about.

Átila

Átila had been busy with Symmetry stuff, so had nothing to talk about this time.

Walter

Walter said he'd thought about Robert's observations that until we have the compiler as a language server or a library, we're going to be in trouble. So he'd been going through the compiler frontend and fixing it so that it's much more amenable to having DMD as a daemon.

He'd made quite a number of changes for that. Some of the PRs were still sitting there, waiting to be pulled. The idea was to look at the subsystems of D and remove their dependencies on global state, and dependencies on the environment and the operating system, so that they have a simple entry point where one function takes all the parameters it needs, does its operation, and returns data. Then the caller can decide whether to write a file or what to do with error messages, and so on.

He said that work was kind of satisfying when he got a piece set up like that. The error sink had been his first attempt in that direction. The lexer and parser were now capable of being divorced from the rest of the compiler and are restartable. It also means the LSP daemon or whatever will be able to be multi-threaded, because you can be doing multiple compiles at the same time with this setup. He's still got a ways to go, but it's proceeding.

He was also looking at redefining the interface to the glue code, so that should make Iain's and Martin's lives a little easier. Martin said he wasn't so sure. He thought the interface in its current state was absolutely fine.

Walter said he'd submitted numerous PRs to remove all of the external dependencies from the documentation generator, and that was getting close to being done. He had separated out the macro processor so that it was now standalone. That's the sort of thing he'd been working on.

He saw this as a sort of existential thing for D. He had come around to Robert's point of view and felt it was important to get this moving. Razvan wasn't in the meeting, but Walter was curious about his thoughts about all of this since he'd been working on DMD as a library. He asked Robert if the work needed for both DMD as a library and the LSP server was the same.

Robert said DMD as a library is a big building block to build the LSP, but there was more to it in terms of building a server that takes commands, runs the compiler, and returns the information as JSON. The compiler library is a big part of that. His fingers were crossed that the rest was just glue. He thanked Walter for working on this. Walter said Robert should be thanked for making the observation that gave him the impetus to get this done and realize its importance.

We then fell into a bit of side chat for a few minutes about GitHub handles vs. real names and how confusing it can be when they differ, and how the programming world had changed in terms of new programmer expectations.

Then Walter thanked Steve and Dennis for working on the static AA initialization. For a long time, he'd thought it was completely impractical. Steve had convinced him it could be done reasonably. It was a nice improvement that didn't need anything different because it was just turning on something that hadn't worked.

Steve said there was a note in the spec saying this was an intended feature that wasn't implemented, so we could finally remove that. He said that Dennis was doing a great job. He sometimes felt that every time he brings up something that's wrong, Dennis has a PR to fix it a day later.

Finally, Walter had a question for Mathias about -preview=in. He remembered that we had discussed in a meeting that the feature should always pass by reference, but when he looked at the implementation, he found that it hadn't been changed. It was still taking the original approach of passing by value or by reference depending on the case. Mathias remembered this conversation and that we'd agreed to change it, but he had just never gotten it done. He said he remembered Martin saying he'd keep the current behavior for LDC. Walter asked if Mathias wanted him to make the change. Mathias said he'd do it, and he'd do it in a way that didn't prevent LDC from doing what they want.

Walter recommended Martin make the change for LDC, as there was a solid reason for the change. Martin said they'd have to disagree on this point. It was a feature he would have loved to see 10 years ago, and he was very happy Mathias had implemented it. He wasn't going to dump it, but he would probably put it behind a switch to enable or disable it. It wasn't ideal, but he liked this feature very, very much.

Walter said it was a reliability issue. If it's always by reference, it's predictable behavior. He thought it was extremely important to have predictable and portable behavior. Martin reminded him they'd had this discussion quite some time ago. He was still of the strong opinion that once you regard every such occurrence as a reference, or treated it like it should work for a reference, too, but don't depend on its identity, you're fine. And that's 99.99% of all code. And it was a very nice performance thing. He didn't know of any other language having such a feature. For LDC, it was going to be well-defined. Not implementation-defined, but target-dependent.

Mathias said that he agreed with Martin. However, it was fine to just make DMD always pass in parameters by reference. That was the best compromise we could come up with when we last discussed it.

Ali wanted to confirm that this feature solved the problem with rvalue references. If you want to accept rvalues, you just use in. Mathias said it did.

Mathias said he'd seen that the only thing preventing preview in from being finished was a bug in the frontend that is tied to another bug that is unrelated to preview in. It wasn't an easy one. It was something to do with CTFE, but he hadn't had the time to dig into it. So it was still blocked on that. He said it was well-defined, he just needed someone to unwind the issues. Dennis said he would look into it.

Martin thought that issue was blocking one of his old PRs. He couldn't remember which PR, but he was sure it was the same CTFE issue. The problem was that temporaries in statements in CTFE don't get destructed at all. There was a chorus of "That's bad". Mathias said that getting the same destruction behavior as runtime is not trivial.

Walter said he could see that was going to be a tricky one. The CTFE engine was a horrible mess.

Dennis wondered if this would break code once destructors started running in CTFE. A lot of destructors just call free. Martin noted that any destructor calling free would be calling it on something that had to be malloced first. And that wasn't doable at CTFE either. The current behavior of just silently skipping all of the destructors was as wrong as it could get. It couldn't be any more wrong.

Steve said that if you were going to run free in a destructor in a type that's going to be used at CTFE, you'd already have an if(__ctfe) branch to account for that where you call malloc. You'd have the same in the destructor. This is nothing new.

The next meeting

As we wrapped up, I asked if we should have a planning session on the 18th. It was our only real opportunity to have one before DConf. But given that most of us had stuff going on preparing for DConf, or were already traveling, I suggested we not do it. But I left it to the group to decide, and we decided not to do it. In a previous planning session, we'd decided to hold a session at DConf to start talking about long-term plans, so that's our next one.

Our next monthly meeting is going to be on either September 8th or 15th. We'll figure that out at DConf next week.

August 22

On Tuesday, 22 August 2023 at 12:59:12 UTC, Mike Parker wrote:

>

Robert

Robert said he was happy. Dennis asked if the compiler was fast enough (as regular readers of these summaries may know, Robert often says the compiler is too slow). Robert said it's still slow. He just wanted to see LSP support in the compiler and take all the good things.

Walter noted he'd been working on improving compilation speeds. He'd realized that the compiler was being compiled with exceptions, so he'd been working on making it compile without exceptions to see what effect that had on speed. It might be a small difference, but every little difference helps.

Follow the memory. I don't think the exceptions will make any difference at all. The compiler allocates soooooo much unnecessary memory and lays out what it does use very badly. This, and doing less work, is to way to make an algorithm whose complexity you can't change faster. I have recently started using a tool called bytehound which can show you some nice metrics from allocations via LD_PRELOAD-ing a magic .so rather than emulation or recompilation, we should generate (I will do this during dconf perhaps) reports at CI time to find hotspots where possible.

Using some kind of copy-on-write would be interesting because the compiler preemptively allocates a lot during semantic analysis.

Also please check that using the new -nothrow doesn't break the ICE error messages and so on when its enabled for the compiler.

>

Adam said his library compiled in half a second. Robert said that was still too slow. His work project didn't compile in half a second. He said his goal was that by the time of the key-up event after pressing 'Enter' on Ninja or whatever build tool/compiler he's using, it should have already restarted the application. Átila agreed that half a second was too long.

Adam brought up D1. An old game he wrote back in 2007 compiles and links in 10 milliseconds. If you looked at a big chunk of the difference between D1 and D2, object.d back then was around 600 lines. Now it was much bigger, and that adds up. Walter said that he was a bit upset with the state of object.d. It was supposed to be a minimal thing and had turned into a kind of garbage dump somewhere along the way.

A lot of this time is actually just linking druntime (approx 2/3 of the ~.3s on my machine), for the record, rather than literally object.d being compiled.

>

Steve

Steve said that Dennis had been doing a fantastic job with his PR on the compiler part of static associative array initialization. Steve thought it was about ready to go if someone could push the button. He didn't want to be the one to push it, because he didn't know the compiler code very well and couldn't vouch for its correctness. (The PR is still open as I write.)

Dennis

Dennis said he'd been working on that AA PR, but had nothing else to report. He still had some other things to pursue, like looking at pattern support for the deprecation (-d/-de/-dw) and obsolete warning (-wo) switches, as well as the standalone attribute or pragma for static constructors.

(In our July monthly, Adam proposed a means to annotate static constructors that could make them independent of cycle detection. In a subsequent planning session, we agreed to implement it. Since the August meeting, Dennis has submitted a PR for the @standalone attribute, but in the PR discussion thread, Walter thinks it should be a pragma.)

Martin

Martin said the bump to version 2.104 of the frontend went well on the LDC side and was quite painless.

Next, he had an interesting side note inspired by a forum post he'd seen about getting LDC to work on a 8-bit AVR platform. He'd looked into it and found it an interesting question in terms of whether and how we want to support architectures with pointer sizes less than 32 bits. Currently, the frontend has a hard-coded limit: size_t and ptrdiff_t are either 32 or 64 bits, nothing lower than that. It's a frontend check, and the definitions in DRuntime directly depend on the compiler's internal type because they use the typeof expression.

He had tried to make the frontend accept 16-bit and 8-bit size_t and ptrdiff_t, but doing that bumps up against the integer promotion rules. The promotion rules can work in some specific cases when using a smaller int, but when you have something like the length of a slice and do a little plus one or minus one, that's a problem. You need to cast it. So either we need to change dealing with the promotion rules in this regard, try to mimic C++ in this regard, or hopefully do something better. He said Adam had proposed something in a past discussion that would handle it generically so it's still target independent.

Walter wondered if Rust supported less than 32-bit compilation and what compromise they made for that. Martin said he wasn't sure. He'd only checked C and C++. Robert looked it up and found that Rust has a tiny bit of support for 16-bit MSP340, but it's a big asterisk and everything else is 32 or 64.

Martin wasn't sure if it was even worth looking into these architectures, because he had no idea how long they'd be around. For low power consumption, even if you're playing a few years ahead, he thinks it's going to be 32-bits even for those very small and power-efficient sensors, or whatever. But he couldn't look into the future, so he had no real idea.

There are some 8 and 16 bit parts still floating around (ultra cheap will probably never go 32) but its definitely a trend.

>

Ali

Ali had mentioned in our July quarterly meeting that his company (Mercedes Benz Research & Development North America) was looking for a D programming intern. He'd since contacted Mike Shah at Northeastern University. As an example of industry/academia collaboration, Mike had recommended a strong candidate, and the company had already hired him. They planned to bring on another intern who comes with a good recommendation. So the D team is increasing to three people.

Very nice to hear.

>

Walter said he'd thought about Robert's observations that until we have the compiler as a language server or a library, we're going to be in trouble. So he'd been going through the compiler frontend and fixing it so that it's much more amenable to having DMD as a daemon.

Amenable yes, but this is all very easy to work around if the compiler itself was fundamentally suitable to working as a library.

>

He'd made quite a number of changes for that. Some of the PRs were still sitting there, waiting to be pulled. The idea was to look at the subsystems of D and remove their dependencies on global state, and dependencies on the environment and the operating system, so that they have a simple entry point where one function takes all the parameters it needs, does its operation, and returns data. Then the caller can decide whether to write a file or what to do with error messages, and so on.

>

He said that work was kind of satisfying when he got a piece set up like that. The error sink had been his first attempt in that direction. The lexer and parser were now capable of being divorced from the rest of the compiler and are restartable. It also means the LSP daemon or whatever will be able to be multi-threaded, because you can be doing multiple compiles at the same time with this setup. He's still got a ways to go, but it's proceeding.

I think you could already do this, but mostly good changes nonetheless. The lexer I think is actually not safe to run in parallel because of the timestamps, but I think Stefan had a PR to fix that a while ago that got stalled for xyz reasons.

>

He was also looking at redefining the interface to the glue code, so that should make Iain's and Martin's lives a little easier. Martin said he wasn't so sure. He thought the interface in its current state was absolutely fine.

Its not great.

>

He saw this as a sort of existential thing for D. He had come around to Robert's point of view and felt it was important to get this moving. Razvan wasn't in the meeting, but Walter was curious about his thoughts about all of this since he'd been working on DMD as a library. He asked Robert if the work needed for both DMD as a library and the LSP server was the same.

Robert said DMD as a library is a big building block to build the LSP, but there was more to it in terms of building a server that takes commands, runs the compiler, and returns the information as JSON. The compiler library is a big part of that. His fingers were crossed that the rest was just glue. He thanked Walter for working on this. Walter said Robert should be thanked for making the observation that gave him the impetus to get this done and realize its importance.

The compiler needs to be able to reuse its previous work to have any chance of success here. If you have a large codebase its not really practical to have two dmds running at the same time after you save your file. This is more achievable than you might think but I think it really needs to be contemplated what these tools actually do rather than just some vague of dmd-as-a-library + some magic - its not just an exercise to the reader.

Also ergonomically you still want to be using the same "pipeline" the regular compiler is using rather than gluing the bits together yourself when you use the library, there will always be minute details. I think the plugin system ldc is heading towards (upstream this or things will fragment!) is probably uglier but more tractable than building discrete tools using dmd as a library.

>

Then Walter thanked Steve and Dennis for working on the static AA initialization. For a long time, he'd thought it was completely impractical. Steve had convinced him it could be done reasonably. It was a nice improvement that didn't need anything different because it was just turning on something that hadn't worked.

Steve said there was a note in the spec saying this was an intended feature that wasn't implemented, so we could finally remove that. He said that Dennis was doing a great job. He sometimes felt that every time he brings up something that's wrong, Dennis has a PR to fix it a day later.

Should be merged - then use newaa to implement the runtime AAs too, two implementations is horrible.

August 22
>

Dennis

Dennis said he'd been working on that AA PR, but had nothing else to report. He still had some other things to pursue, like looking at pattern support for the deprecation (-d/-de/-dw) and obsolete warning (-wo) switches, as well as the standalone attribute or pragma for static constructors.

Please use "pragma" to be consistent with other static constructors. There are few reasons to do so

  1. DMD does not have consistent way of defining system attribute which can cause conflict with user attribute (DMD system attribute should start with underscore character, "_")
  2. If old DMD version can not handle new pragma identifier, it is because the list is hardcoded. Should have/add ignore pragma identifier list defined in sc.ini so that it can be updated at will without rebuild
  3. DMD has attribute soup problem because it does not have negate setting. If top module has
    "@nogc:" and later a function that supports garbage collect should just use its negate form "@!nogc"
  4. DMD attribute should/must not use negate word form so @nogc -> @!gc ....

Cheers

August 23
On Tuesday, 22 August 2023 at 21:36:49 UTC, An Pham wrote:
> 1. DMD does not have consistent way of defining system attribute which can cause conflict with user attribute (DMD system attribute should start with underscore character, "_")

Not true, these attributes follow module namespacing rules. This is how ldc and gdc do all their special extensions too.

> 2. If old DMD version can not handle new pragma identifier, it is because the list is hardcoded. Should have/add ignore pragma identifier list defined in sc.ini so that it can be updated at will without rebuild

Even if you changed this, it wouldn't matter, since the change wouldn't be present in the old compilers you want to support!

> 3. DMD has attribute soup problem because it does not have negate setting.

This is true but doesn't apply here.
August 25
On Tuesday, 22 August 2023 at 12:59:12 UTC, Mike Parker wrote:
> ...
> Adam said his library compiled in half a second. Robert said that was still too slow. His work project didn't compile in half a second. He said his goal was that by the time of the key-up event after pressing 'Enter' on Ninja or whatever build tool/compiler he's using, it should have already restarted the application. Átila agreed that half a second was too long.
> ...

Reading over about other languages CT problems I wonder how this half second is too terrible.  =]

I mean I'm not against Robert's complain, I don't know how complex his software is, and CT maybe is really a problem for him.

But for example where I work we use C# and so on, and CT is not great either, and if it is a web project, well just to start a local server and so on takes much more than 1 second, in fact I'd trade many things to have this 1 second or anything closer to that.

Of course if you measure things and keep watching you will see the difference, but in daily basis, while changing code, compiling etc. I barely see any difference between 500ms to 1000ms.

Matheus.

PS: I'm ESL please if anything sound rough, forgive me! I'm just sharing a thought.
August 25
On Friday, 25 August 2023 at 01:20:28 UTC, matheus wrote:
> Reading over about other languages CT problems I wonder how this half second is too terrible.  =]
> I barely see any difference between 500ms to 1000ms.

I love the fast compile time of D. It feels so freeing.

Contrast that to a Visual Studio solution I have with over 700 projects and it takes 5 minutes to compile. Just compiling one project takes around 30s. That doesn't include the 75000 tests. It does bother me, but it's not so far outside the norm.

I'm also curious why a 500ms compile time would be generally recognized as way too long. Is it because it has potential to be faster or does it cause some legitimate problem? That's a question at large, not for Matheus.
August 30
On Friday, 25 August 2023 at 02:10:25 UTC, harakim wrote:
> I'm also curious why a 500ms compile time would be generally recognized as way too long. Is it because it has potential to be faster or does it cause some legitimate problem? That's a question at large, not for Matheus.

When you want to react to ~every keypress in a language server impl with updated coloring, it starts to be problematic. Think in terms of "UI feedback" rather than "project compilation".

September 02
On Wednesday, 30 August 2023 at 14:19:03 UTC, FeepingCreature wrote:
> On Friday, 25 August 2023 at 02:10:25 UTC, harakim wrote:
>> I'm also curious why a 500ms compile time would be generally recognized as way too long. Is it because it has potential to be faster or does it cause some legitimate problem? That's a question at large, not for Matheus.
>
> When you want to react to ~every keypress in a language server impl with updated coloring, it starts to be problematic. Think in terms of "UI feedback" rather than "project compilation".

Thanks. That makes a lot of sense.
September 02

On Wednesday, 30 August 2023 at 14:19:03 UTC, FeepingCreature wrote:

>

When you want to react to ~every keypress in a language server impl with updated coloring, it starts to be problematic. Think in terms of "UI feedback" rather than "project compilation".

You only need an AST for this though right? Not a full compilation?

September 03

On Saturday, 2 September 2023 at 20:27:04 UTC, Bonarc wrote:

>

On Wednesday, 30 August 2023 at 14:19:03 UTC, FeepingCreature wrote:

>

When you want to react to ~every keypress in a language server impl with updated coloring, it starts to be problematic. Think in terms of "UI feedback" rather than "project compilation".

You only need an AST for this though right? Not a full compilation?

Depends if you want language server info, ie. types, docs, etc. You can do very clever things here like caching symbols between runs, but the naive way is to run the frontend at least all the way through to sema2.