Linking is the slowest part of D's compilation process. It is what makes its compilation speed uncompetitive with interrpreted scripting languages like Python and JavaScript.

This project to make a faster linker is in alpha: https://github.com/rui314/mold

> Concretely speaking, I wanted to use the linker to link a Chromium executable with full debug info (~2 GiB in size) just in 1 second. LLVM's lld, the fastest open-source linker which I originally created a few years ago, takes about 12 seconds to link Chromium on my machine. So the goal is 12x performance bump over lld. Compared to GNU gold, it's more than 50x.

> It looks like mold has achieved the goal. It can link Chromium in 2 seconds with 8-cores/16-threads, and if I enable the preloading feature (I'll explain it later), the latency of the linker for an interactive use is less than 900 milliseconds. It is actualy faster than cat.

On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
>
> This project to make a faster linker is in alpha: https://github.com/rui314/mold

We would need to spend substantial engineering effort to
make mold production-ready and ported to macOS and Windows,
but it fits the spirit of D to improve the compilation speed.

On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
> Linking is the slowest part of D's compilation process. It is what makes its compilation speed uncompetitive with interrpreted scripting languages like Python and JavaScript.
>
> This project to make a faster linker is in alpha: https://github.com/rui314/mold
>
>> [...]
>
>> [...]

Have you looked at gold or lld?

On Wednesday, 24 February 2021 at 20:05:17 UTC, Imperatorn wrote:
> On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
>> Linking is the slowest part of D's compilation process. It is what makes its compilation speed uncompetitive with interrpreted scripting languages like Python and JavaScript.
>>
>> This project to make a faster linker is in alpha: https://github.com/rui314/mold
>>
>>> [...]
>>
>>> [...]
>
> Have you looked at gold or lld?

From the readme file:

> LLVM's lld, the fastest open-source linker which I originally created a few years ago, takes about 12 seconds to link Chromium on my machine. So the goal is 12x performance bump over lld. Compared to GNU gold, it's more than 50x.

On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
> Linking is the slowest part of D's compilation process. It is what makes its compilation speed uncompetitive with interrpreted scripting languages like Python and JavaScript.
>
> This project to make a faster linker is in alpha: https://github.com/rui314/mold
>
>> Concretely speaking, I wanted to use the linker to link a Chromium executable with full debug info (~2 GiB in size) just in 1 second. LLVM's lld, the fastest open-source linker which I originally created a few years ago, takes about 12 seconds to link Chromium on my machine. So the goal is 12x performance bump over lld. Compared to GNU gold, it's more than 50x.
>
>> It looks like mold has achieved the goal. It can link Chromium in 2 seconds with 8-cores/16-threads, and if I enable the preloading feature (I'll explain it later), the latency of the linker for an interactive use is less than 900 milliseconds. It is actualy faster than cat.

This project is definitely worth watching, but it's worth saying that people continuously try and make their compiler the fastest, or their linker the fastest etc - progress is always good, but these things tend to slow down quite a bit as they mature.

On Wednesday, 24 February 2021 at 20:48:49 UTC, Petar Kirov [ZombineDev] wrote:
> On Wednesday, 24 February 2021 at 20:05:17 UTC, Imperatorn wrote:
>> On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
>>> Linking is the slowest part of D's compilation process. It is what makes its compilation speed uncompetitive with interrpreted scripting languages like Python and JavaScript.
>>>
>>> This project to make a faster linker is in alpha: https://github.com/rui314/mold
>>>
>>>> [...]
>>>
>>>> [...]
>>
>> Have you looked at gold or lld?
>
> From the readme file:
>
>> LLVM's lld, the fastest open-source linker which I originally created a few years ago, takes about 12 seconds to link Chromium on my machine. So the goal is 12x performance bump over lld. Compared to GNU gold, it's more than 50x.

👍

We should indoctrinate this rui314 to use D! 😁

On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
> Linking is the slowest part of D's compilation process.

Not necessarily:

/tmp % cat foo.d
import std.uni;
void main() {}
/tmp % time dmd -c -unittest foo.d
dmd -c -unittest foo.d  0.27s user 0.06s system 99% cpu 0.331 total
/tmp % time dmd foo.o
dmd foo.o  0.04s user 0.05s system 154% cpu 0.058 total


I'm using lld as the linker.

The linker can be a bottleneck, yes, especially since it doesn't do work in parallel. But in my experience, if the linker takes a while, compiling took a lot longer still. Of course, any improvements in this area are welcome, and I hope mold is production-ready as soon as possible.

I recently bought an NVMe drive just in the hope that it'd help with the "linker tax".

On Wednesday, 24 February 2021 at 22:12:46 UTC, Atila Neves wrote:
> On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
> The linker can be a bottleneck, yes, especially since it doesn't do work in parallel. But in my experience, if the linker takes a while, compiling took a lot longer still. Of course, any improvements in this area are welcome, and I hope mold is production-ready as soon as possible.
>

This is true for a fresh build, but often not the case for incremental builds, which dev often have to go through. This is because the work you have to do for sources grows with the size of the changeset, while the work you have to do link grows with the size of the project as a whole, changed or not. On large projects, it is very common that linking dominates incremental builds.

zld is another interesting project that tries to do enable incremental linking: https://github.com/kubkon/zld

Just like mold, it is fairly new and probably not battle tested enough for production yet.

On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
> Linking is the slowest part of D's compilation process. It is what makes its compilation speed uncompetitive with interrpreted scripting languages like Python and JavaScript.
>
> This project to make a faster linker is in alpha: https://github.com/rui314/mold
>
>> Concretely speaking, I wanted to use the linker to link a Chromium executable with full debug info (~2 GiB in size) just in 1 second. LLVM's lld, the fastest open-source linker which I originally created a few years ago, takes about 12 seconds to link Chromium on my machine. So the goal is 12x performance bump over lld. Compared to GNU gold, it's more than 50x.
>
>> It looks like mold has achieved the goal. It can link Chromium in 2 seconds with 8-cores/16-threads, and if I enable the preloading feature (I'll explain it later), the latency of the linker for an interactive use is less than 900 milliseconds. It is actualy faster than cat.

I've run a quick internal test with a 34MB object.

gold: 0.82s
mold: 0.20s

That's pretty amazing. Of course, it took a lot of poking at linker flags to make it happen, and I had to change hash style cause its gnu hash impl is pretty slow right now (0.6s). But still.

On Wednesday, 24 February 2021 at 22:12:03 UTC, Imperatorn wrote:
> On Wednesday, 24 February 2021 at 20:48:49 UTC, Petar Kirov [ZombineDev] wrote:
>> On Wednesday, 24 February 2021 at 20:05:17 UTC, Imperatorn wrote:
>>> On Tuesday, 23 February 2021 at 14:41:59 UTC, James Lu wrote:
>>>>> [...]
>>>
>>> Have you looked at gold or lld?
>>
>> From the readme file:
>>
>>> LLVM's lld, the fastest open-source linker which I originally created a few years ago, takes about 12 seconds to link Chromium on my machine. So the goal is 12x performance bump over lld. Compared to GNU gold, it's more than 50x.
>
> 👍
>
> We should indoctrinate this rui314 to use D! 😁

😈