Alo!

I just watched this talk from Jonathan Blow [0] about his programming language called Jai, and he can now compile an 80,000 line game in about 1.5 seconds on a laptop (of course I have no idea what laptop he's using), under 1 second on a desktop.

And he claims it's he wants to eventually hit compilation of 1,000,000 lines per second and that he thinks that's a realistic goal.

I just tried compiling an optional utility library that has a grand total of 1794 lines in .d files and:

time dmd -c -debug -g -unittest -w -Isource/ -Itests  -I../../.dub/packages/bolts-0.7.1/bolts/source/ source/optional/dispatch.d source/optional/internal.d source/optional/notnull.d source/optional/optional.d source/optional/package.d source/optional/traits.d tests/dispatch.d tests/match.d tests/notnull.d tests/optional.d tests/orelse.d tests/unwrap.d

1.92s user 0.23s system 99% cpu 2.157 total (2.6ghz, 4 cores, plenty ram)

Have compile times gotten worse in D over the years or better or just stayed the same? And is there anyway to get even near the performance of Jai when it comes to compilations (parallelize stuff here and there maybe)? Or is DMD in a state where that is just not feasible?

On a related note: He also mentions some really cool compilation features like having compiler hooks that tell you when compilation is done, when executable and where it will be written so you can create your build recipe inside the program itself. Also allows you do do things like:

whenCompilationFinishes(exeLocation) => loadExecutableIcon(myIcon, exeLocation)

During the build!

Your source knows how to build itself as a concept is awesome! There's actually a D runner [1] that kind of allows for source files to set stuff up.

[0] https://www.youtube.com/watch?v=uZgbKrDEzAs
[1] https://github.com/marler8997/rund

On Thursday, 20 September 2018 at 23:13:38 UTC, aliak wrote:
> Alo!
>
> I just watched this talk from Jonathan Blow [0] about his programming language called Jai, and he can now compile an 80,000 line game in about 1.5 seconds on a laptop (of course I have no idea what laptop he's using), under 1 second on a desktop.

Jai is in the hands of maybe a dozen people and , so it's hard to compare. But with a sufficiently simple language with no metaprogramming, 80k lines of code in 1.5 seconds seems doable.

And in that situation, dmd does just fine -- 0.73 seconds to compile 84k lines of simple generated code on i5 2400, or 0.20 seconds with -c -o-.

It's just that D code tends toward heavy metaprogramming. That's a lot safer (consider C-style varargs writefln versus the template version), and it's slower to compile.

> On a related note: He also mentions some really cool compilation features like having compiler hooks that tell you when compilation is done, when executable and where it will be written so you can create your build recipe inside the program itself. Also allows you do do things like:
>
> whenCompilationFinishes(exeLocation) => loadExecutableIcon(myIcon, exeLocation)
>
> During the build!
>
> Your source knows how to build itself as a concept is awesome! There's actually a D runner [1] that kind of allows for source files to set stuff up.

It's awesome for demos and terrible otherwise. It takes "it builds on my machine" to a new level.

On Thursday, 20 September 2018 at 23:13:38 UTC, aliak wrote:
> he can now compile an 80,000 line game in about 1.5 seconds on a laptop

D can compile similar amounts of code in half the time.

For example, the entire D1 runtime and standard library can be built (compiled and linked!) in 0.6 seconds on my computer, and it is about 82,000 lines of code.

A good chunk of my gui libs in D: terminal.d, simpledisplay.d, minigui.d, nanovega.d, color.d, and dom.d for good measure, can be compiled in 1 second on my computer. That is ~49,000 lines of code. Back when I didn't use phobos in them, it compiled in about 1/3 that time - see, that's the hidden cost of builds: you frequently need to compile parts of the standard library too. In C++, this is caused by #include (sort of, C++ mitigates it in practice though). In D, it is templates. Any templates you use from the stdlib will be compiled and instantiated too - and this is slow.


Of course, D can also take ages to compile one line of code. It all depends on that that line is doing... ctfe and templates are slow. C or Java style code compiling in D is very fast.

> Have compile times gotten worse in D over the years or better or just stayed the same?

Well, if you still compile D code written in and older style - like my example of the D1 phobos - it still builds exceedingly quickly. Just that older style is less common  nowadays - using Phobos function is frequently the slowest part of compiling my code with D2, whereas D1 it was all written C style, and would build in the blink of an eye.

So dmd hasn't gotten much slower, but the typical D style has moved toward using more of the slower parts of the compiler instead of the faster parts.

On Friday, 21 September 2018 at 00:47:27 UTC, Adam D. Ruppe wrote:
> Of course, D can also take ages to compile one line of code. It all depends on that that line is doing... ctfe and templates are slow. C or Java style code compiling in D is very fast.

I was going to say this too, ie how much of that Jai code is run at compile-time, how much is uninstantiated templates that is just skipped over like D does, and how much is templates instantiated many times? Lines of code is not a good enough measure with those programming constructs.

I was just building the stdlib tests with LDC yesterday and they took so much memory on a new Linux/x64 VPS with 2GB of RAM that I had spun up that I couldn't even ssh in anymore. I eventually had to restart the VPS and add a swapfile, which I usually have but simply hadn't bothered with yet for this new Ubuntu 18.04 VPS. The stdlib tests instantiate a ton of templates.

On 09/20/2018 07:13 PM, aliak wrote:
> 
> On a related note: He also mentions some really cool compilation features like having compiler hooks that tell you when compilation is done, when executable and where it will be written so you can create your build recipe inside the program itself. Also allows you do do things like:
> 
> whenCompilationFinishes(exeLocation) => loadExecutableIcon(myIcon, exeLocation)
> 
> During the build!
> 
> Your source knows how to build itself as a concept is awesome! There's actually a D runner [1] that kind of allows for source files to set stuff up.
> 
> [0] https://www.youtube.com/watch?v=uZgbKrDEzAs
> [1] https://github.com/marler8997/rund

Just watched that part of the video. It sounds like it really just boils down to these three things:

1. He did away with C/C++'s awful header system, just like every other language out there.

(It's a good thing, but it's also a bare minimum expectation in ANY language that isn't C or C++. Yawn.)

2. He created a standard (and presumably mandatory) build tool to go along with his language, but is avoiding *calling* it a build tool...even though it's clearly just a build tool that happens to be built into the compiler.

(I've already used a language that did that: Haxe. There are definitely some nice things about it, but the problem is: What happens when the official buildsystem doesn't fit your needs? Your pretty much SOL. Just imagine if DUB was the ONLY way to compile D and you get the picture of why this isn't as great as it sounds. And even if his compiler DOES let you opt-out of the official built-in buildsystem, it's ultimately still just yet another buildsystem, nothing inherently special or new except for...)

3. You can embed your buildscript into one of your project's existing source files, instead of putting it in a dedicated buildscript file.

(We can do that in D too, by utilizing version identifiers, but we don't because its messy and mostly pointless. Just like you wouldn't put your graphics code in an XML-parsing module, or your JSON serializer in an audio-mixing module. They each get their own file or dir. Basic separation of concerns. For one-file scripts, this IS actually really, really nice though, which is why I like both rund and DUB's embedded dub.sdl. But aside from one-file scripts, there's not much point to avoiding basic separation-of-concerns.)

On 9/20/2018 7:44 PM, Nick Sabalausky (Abscissa) wrote:
> 3. You can embed your buildscript into one of your project's existing source files, instead of putting it in a dedicated buildscript file.
> 
> (We can do that in D too, by utilizing version identifiers, but we don't because its messy and mostly pointless. Just like you wouldn't put your graphics code in an XML-parsing module, or your JSON serializer in an audio-mixing module. They each get their own file or dir. Basic separation of concerns. For one-file scripts, this IS actually really, really nice though, which is why I like both rund and DUB's embedded dub.sdl. But aside from one-file scripts, there's not much point to avoiding basic separation-of-concerns.)

In D1, you could embed D code in HTML files. It was a cool, and completely useless, feature.

On Friday, 21 September 2018 at 02:44:57 UTC, Nick Sabalausky (Abscissa) wrote:
> 2. He created a standard (and presumably mandatory) build tool to go along with his language, but is avoiding *calling* it a build tool...even though it's clearly just a build tool that happens to be built into the compiler.

> 3. You can embed your buildscript into one of your project's existing source files, instead of putting it in a dedicated buildscript file.

I understand that the build tool is the compiler itself, presumably providing some functions/hooks for the build script, which is part of program itself written in the language itself. It seems to me that this approach is superior to the usual build systems in many ways: complete control of the build, no external program/lib needed apart from compiler, no other language to learn.

Note that the build can be done at compile time because the metaprogramming capabilities of the language are not limited in terms of system calls.

On Friday, 21 September 2018 at 05:11:32 UTC, mate wrote:
> Note that the build can be done at compile time because the metaprogramming capabilities of the language are not limited in terms of system calls.

Good luck bisecting that code base when any version of it did anything even mildly specific to the author's PC.

On Friday, 21 September 2018 at 05:39:35 UTC, Vladimir Panteleev wrote:
> On Friday, 21 September 2018 at 05:11:32 UTC, mate wrote:
>> Note that the build can be done at compile time because the metaprogramming capabilities of the language are not limited in terms of system calls.
>
> Good luck bisecting that code base when any version of it did anything even mildly specific to the author's PC.

Indeed. I am actually not sure if there really are no limitations to Jai’s CTFE, in its current state. There are probably facilities in the stdlib to avoid the need for doing system specific things; also the build instructions would hopefully be contained in some function/file either by convention or as required by the compiler, limiting the scope of build debugging.

Moreover, I got the feeling that the language is geared towards “good programmers” and is less concerned by mistakes happening because the author did something stupid.

On 9/20/2018 10:11 PM, mate wrote:
> Note that the build can be done at compile time because the metaprogramming capabilities of the language are not limited in terms of system calls.

Back in the naive olden days, Microsoft released ActiveX, where a web page could load executable objects (!) from the internet and run them in the browser.

It quickly became apparent that this was a disaster, as lots of people on the internet aren't to be trusted.

CTFE on D doesn't allow making any system calls. This is on purpose.