September 18, 2023
On Friday, 15 September 2023 at 20:22:50 UTC, Atila Neves wrote:
> An argument could be made that it could/should install the dependencies such that only one `-I` flag is needed.

Indeed, this would be god tier.

> ~190k SLOC (not counting the many dub dependencies) killed dmd on a system with 64GB RAM + 64GB swap after over a minute. Even if it worked, it'd be much, much slower.

What you do with the lines of code is *far* more important than how many there are.

The arsd library has about 219,000 lines of text if you delete the Windows-only and obsolete modules (doing so just so I can actually dmd *.d here on my Linux box). This includes comments and such; dscanner --sloc reports about 98,000.

$ wc *.d
<snip>
 218983  870208 7134770 total
$ dscanner --sloc *.d
<snip>
total:  98645

Let's compile it all:

$ /usr/bin/time dmd *.d -L-L/usr/local/pgsql/lib  -unittest -L-lX11
5.35user 0.72system 0:06.08elapsed 99%CPU (0avgtext+0avgdata 1852460maxresident)k
0inputs+70464outputs (0major+536358minor)pagefaults 0swaps

That's a little bit slow, over 5 seconds. About 1.3 of those seconds are spent in the linker, the other 4 remain with dmd -c. Similarly, that's almost 2 GB of RAM it used, more than it probably should, but it worked fine.

My computer btw is a budget model circa 2016. Nothing extraordinary about its hardware.

But notice it isn't actually running out of RAM or melting the CPU over a period of minutes, despite being about six figures lines of code but any measure.


On the other hand, compile:

enum a = () {
   string s;
   foreach(i; 0 .. 20_000_000_000)
     s ~= 'a';
   return s;
}();


Don't actually do it, but you can imagine what will happen. 6 lines that can spin your cpu and explode your memory. Indeed, even just importing this module, even if the build system tried not to compile it again, will cause the same problem.

The arsd libs are written - for the most part, there's some exceptions - with compile speed in mind. If I see my build slow down, I investigate why. Most problems like this can be fixed!

In fact, let's take that snippet and talk about it. I had to remove *several* zeroes to make it even work without freezing up my computer, but with a 100,000 item loop, it just barely worked. Even 200,000 made it OOM.

But ok, a 100,000 item append:

0.53user 1.52system 0:02.17elapsed 95%CPU (0avgtext+0avgdata 4912656maxresident)k

About 5 GB of RAM devoured by these few lines, taking 2 seconds to run. What are some ways we can fix this? The ~= operator is actually *awful* at CTFE, its behavior is quadratic (...or worse, i didn't confirm this today, but it is obviously bad). So you can fix this pretty easily:

enum string a = () {
   // preallocate the buffer instead of append
   char[] s = new char[](100000);
   foreach(ref ch; s)
     ch = 'a';
   return s;
}();

0.17user 0.03system 0:00.21elapsed 98%CPU (0avgtext+0avgdata 45748maxresident)k 16inputs+1408outputs (0major+21995minor)pagefaults 0swaps

Over 10x faster to compile, 1/100th of the RAM, ram result. Real world code is frequently doing more than this example and rewriting it to work like this might take some real effort.... but the results are worth it.

And btw try this: import this module and check your time/memory stats. Even if it isn't compiled, since ctfe is run when the module is even just imported, you gain *nothing* by separate compilation!

...but there are times when you can gain a LOT by separate compilation in situations like this, if you can move the ctfe to be some private thing not exposed in the interface. This requires some work by the lib author too though in most cases. An example where you can gain a lot is when something does a lot of internal code generation but exposes a small interface, for example a scripting language wrapper (though script wrappers can also be made to compile reasonably efficiently if you use things like preallocation of buffers, keep your generated functions short (again, the codegen has quadratic behavior, so many small functions work better than a big one, and if you factor the code well, you can minimize the amount of generated code and call back to generic things, e.g. type erasure), collapse template instances, and keep ctfe things ctfe only with a variety of techniques, so they are not codegened unless they are actually necessary).

My arsd.script and arsd.cgi can wrap large numbers of functions and classes reasonably well, but that's why programs using them tend to be multi-second builds.... just note that's programs using them. Separate compiling the libraries doesn't help. You'd have to structure the code to keep those codegen parts internal to a package with a minimal interface, then separate compiling those internal components might win.

But this is a fairly niche case. Yes, I know there's one major commercial D user who do exactly this. But that's the exception, not the rule.


September 20, 2023

On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

>

https://code.dlang.org/packages/reggae

For those who don't know, reggae is a meta-build system for and in D. It's like CMake, if you replace their terrible language with D*. Like CMake, it can output make and ninja files. Unlike CMake, it can also output tup files and has its own binary generator in case the dependency on one of those programs is undesired. Also unlike CMake, it supports dub projects out-of-the-box by using dub as a library.

Out of curiosity, why do we need one more build tool?

TBH I had the same thought - we should have "good" build tool for D that has D as a language. Since I have some experience with CMake and agree with a lot of people that it have horrible language, I thought it would be beneficial. But then I looked round and realized that it will be like this:

So I dropped that idea and think that investing into existing build tool to support D will be much more efficient.
Have you looked as Meson, for example - it even has a section about D?

September 20, 2023

On Wednesday, 20 September 2023 at 15:24:52 UTC, Andrey Zherikov wrote:

>

On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

>

https://code.dlang.org/packages/reggae

For those who don't know, reggae is a meta-build system for and in D. It's like CMake, if you replace their terrible language with D*. Like CMake, it can output make and ninja files. Unlike CMake, it can also output tup files and has its own binary generator in case the dependency on one of those programs is undesired. Also unlike CMake, it supports dub projects out-of-the-box by using dub as a library.

Out of curiosity, why do we need one more build tool?

Because we don't have one now. Using CMake for D is horrible, and none of the alternatives are that much better. Hand-written Makefiles are a nightmare and will never get dependencies right. dub isn't a build system, although it includes a very limited one that can't be extended. Want to build D and C++ code in the same project? Hah, good luck.

>

TBH I had the same thought - we should have "good" build tool for D that has D as a language. Since I have some experience with CMake and agree with a lot of people that it have horrible language, I thought it would be beneficial. But then I looked round and realized that it will be like this:

In practice, this doesn't seem to happen with build systems. Instead each language ecosystem ends up using its own: CMake, Gradle, Rake, ...

>

Have you looked as Meson, for example - it even has a section about D?

Yes, and even had to use it for D. I don't know what state it's in now, but it was pretty much unusable if one had dub dependencies which... is nearly always the case.

Then there's the language: I'd rather use D or Python. If anything, Meson's use case could be questioned just as well by "why another build language?".

September 21, 2023

On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves wrote:

>

Then there's the language: I'd rather use D or Python.

Someone, somewhere, almost 20 years ago ...

https://forum.dlang.org/post/40BB3D47.9030007@inwind.it

In the D land, everything always changes, to never really change ... it's really a big wheel, or better a pendulum! :-P

/Paolo

September 21, 2023

On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves wrote:

>

Because we don't have one now. Using CMake for D is horrible,

I would say just using CMake is horrible :-D But there are a lot of people using it (even myself) and they all are trapped with the language.

>

Hand-written Makefiles are a nightmare and will never get dependencies right. dub isn't a build system, although it includes a very limited one that can't be extended.

I agree that dub is not a build tool, it's more a package manager.

> >

TBH I had the same thought - we should have "good" build tool for D that has D as a language. Since I have some experience with CMake and agree with a lot of people that it have horrible language, I thought it would be beneficial. But then I looked round and realized that it will be like this:

In practice, this doesn't seem to happen with build systems. Instead each language ecosystem ends up using its own: CMake, Gradle, Rake, ...

IMHO this does not prove that D must have its own build system.

> >

Have you looked as Meson, for example - it even has a section about D?

Yes, and even had to use it for D. I don't know what state it's in now, but it was pretty much unusable if one had dub dependencies which... is nearly always the case.

I believe this is applicable to all build systems atm - they don't know what dub is and how to deal with it.

>

Then there's the language: I'd rather use D or Python. If anything, Meson's use case could be questioned just as well by "why another build language?".

TBH I thought Meson uses python but just found out it's not. My bad.
Just took a quick look at known build tools and noticed that there are (at least) two build tools that have Python as a build language and support D: SCons and Waf. Although I don't how well they support it.

My whole comment was about the reasoning why we need new build system and can't improve existing one. So I would expect more wide analysis and comparison here, not just with CMake. My main concern is that if we create a brand new build tool, years later it might turn out that other existing tool got wide adoption for D so this project become abandoned and all efforts are thrown away.

September 21, 2023

On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves wrote:

>

On Wednesday, 20 September 2023 at 15:24:52 UTC, Andrey Zherikov wrote:

>

On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

>

https://code.dlang.org/packages/reggae

For those who don't know, reggae is a meta-build system for and in D. It's like CMake, if you replace their terrible language with D*. Like CMake, it can output make and ninja files. Unlike CMake, it can also output tup files and has its own binary generator in case the dependency on one of those programs is undesired. Also unlike CMake, it supports dub projects out-of-the-box by using dub as a library.

Out of curiosity, why do we need one more build tool?

Because we don't have one now. Using CMake for D is horrible, and none of the alternatives are that much better. Hand-written Makefiles are a nightmare and will never get dependencies right. dub isn't a build system, although it includes a very limited one that can't be extended. Want to build D and C++ [snip]

You can call CMake with a pre-build command right? Would it make sense to add an example of that?

It also might make sense to include in the tests/projects folder an importC example (not sure it makes sense to include a CMake example there).

September 21, 2023

On Thursday, 21 September 2023 at 14:19:27 UTC, jmh530 wrote:

>

On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves wrote:

>

On Wednesday, 20 September 2023 at 15:24:52 UTC, Andrey Zherikov wrote:

>

On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:

>

[...]

Out of curiosity, why do we need one more build tool?

Because we don't have one now. Using CMake for D is horrible, and none of the alternatives are that much better. Hand-written Makefiles are a nightmare and will never get dependencies right. dub isn't a build system, although it includes a very limited one that can't be extended. Want to build D and C++ [snip]

You can call CMake with a pre-build command right? Would it make sense to add an example of that?

Yes, and I've done that, but it's not great. I'm going to add CMake support to reggae by making CMake export its compilation DB and importing that.

>

It also might make sense to include in the tests/projects folder an importC example (not sure it makes sense to include a CMake example there).

That's a great idea, thanks!

September 21, 2023

On Thursday, 21 September 2023 at 13:38:30 UTC, Andrey Zherikov wrote:

>

On Wednesday, 20 September 2023 at 21:19:22 UTC, Atila Neves wrote:

>

Because we don't have one now. Using CMake for D is horrible,

I would say just using CMake is horrible :-D But there are a lot of people using it (even myself) and they all are trapped with the language.

Not anymore! ;)

> > >

Have you looked as Meson, for example - it even has a section about D?

Yes, and even had to use it for D. I don't know what state it's in now, but it was pretty much unusable if one had dub dependencies which... is nearly always the case.

I believe this is applicable to all build systems atm - they don't know what dub is and how to deal with it.

Well, no, since reggae does know what dub is and uses it as a library.

> >

Then there's the language: I'd rather use D or Python. If anything, Meson's use case could be questioned just as well by "why another build language?".

TBH I thought Meson uses python but just found out it's not. My bad.
Just took a quick look at known build tools and noticed that there are (at least) two build tools that have Python as a build language and support D: SCons and Waf. Although I don't how well they support it.

I'm aware of both and looked at them many, many, years ago.

>

My whole comment was about the reasoning why we need new build system and can't improve existing one.

dub, for starters, and the fact that dub describe doesn't work.

Then there's building by D package, which isn't going to be easy anywhere else.

September 21, 2023

On Thursday, 21 September 2023 at 15:54:26 UTC, Atila Neves wrote:

>

[snip]

Yes, and I've done that, but it's not great. I'm going to add CMake support to reggae by making CMake export its compilation DB and importing that.

Interesting.

September 26, 2023

On Thursday, 21 September 2023 at 15:59:10 UTC, Atila Neves wrote:

>

...

I got your point.

Why does it have multiple languages (front-ends)? Is there anyone willing to write build script for C++ on JavaScript or Ruby, for example?

Also why is it meta build system? Why can't it just build by itself without calling external tools (make, ninja)?

PS I tried to reach you out on Discord and LinkedIn but no response so far.