Hi,

I am compiling a small vibe.d project with dmd via dub. The linker takes most of the time when invoking "dub build" (total: 11 seconds). I've tried compiling with gdc and it's the same (11 seconds). With ldc, it's faster (6 seconds).

I did not time anything except "dub build" but I get the feeling that dmd compiles much faster but is slowed down by the linker and that ldc takes longer to compile but has a faster linker.

Is there any way to get faster link time? I'm on Linux (Gentoo).

Thanks!

On Fri, 09 Nov 2018 01:04:51 +0000, bennygui wrote:
> Is there any way to get faster link time? I'm on Linux (Gentoo).

Use dynamic libraries.

Dub doesn't currently support them. Fortunately, the main project I've had to do this with is gtkd, which has no dependencies. The hack I used was:

* use git submodules instead of dub for the dependency
* update the dub script for gtkd to build dynamic libraries
* tell dub to use gtkd's library artifacts manually
* tell dub about the gtkd import paths manually

On Fri, Nov 09, 2018 at 01:04:51AM +0000, bennygui via Digitalmars-d wrote:
> Hi,
> 
> I am compiling a small vibe.d project with dmd via dub. The linker
> takes most of the time when invoking "dub build" (total: 11 seconds).
> I've tried compiling with gdc and it's the same (11 seconds). With
> ldc, it's faster (6 seconds).
> 
> I did not time anything except "dub build" but I get the feeling that dmd compiles much faster but is slowed down by the linker and that ldc takes longer to compile but has a faster linker.
> 
> Is there any way to get faster link time? I'm on Linux (Gentoo).
[...]

Are you sure it's the linker, and not the compiler, or dub itself?

If it's the linker, then you could look into using gold instead of the BFD linker.

I suspect most of your build time is spent in dub, though.  You could try various options to turn off network access and dependency resolution when running dub, which should improve total running time. (Of course, you need to run it at least once without those options so that it pulls in required dependencies, but once those are downloaded, there's no need to check dependencies every single time.)  You could also ditch dub and use your own build system, by creating an empty dummy project in a subdirectory and running dub once to pull in your dependencies, then link to the artifacts yourself directly. (I did that in one of my own projects, and it sped up build time by an order of magnitude.)

Another possible cause is that Diet templates are very template- and CTFE-heavy, and will often soak up gobs of RAM and CPU every time they are compiled.  A possible solution is to separate out the Diet templates into their own source files, and use separate compilation to avoid rebuilding those files unless there was an actual change in the Diet code. (I've also run into this in my vibe.d project. I'm considering to replace Diet templates with something lighter-weight, but haven't taken that step yet.)


T

-- 
It is the quality rather than the quantity that matters. -- Lucius Annaeus Seneca

On 11/8/18 9:20 PM, H. S. Teoh wrote:
> On Fri, Nov 09, 2018 at 01:04:51AM +0000, bennygui via Digitalmars-d wrote:
>> Hi,
>>
>> I am compiling a small vibe.d project with dmd via dub. The linker
>> takes most of the time when invoking "dub build" (total: 11 seconds).
>> I've tried compiling with gdc and it's the same (11 seconds). With
>> ldc, it's faster (6 seconds).
>>
>> I did not time anything except "dub build" but I get the feeling that
>> dmd compiles much faster but is slowed down by the linker and that ldc
>> takes longer to compile but has a faster linker.
>>
>> Is there any way to get faster link time? I'm on Linux (Gentoo).
> [...]
> 
> Are you sure it's the linker, and not the compiler, or dub itself?

Yeah, a vibe.d project with diet templates takes most of its time in the linker, I've found. Basically a big hang after dub prints:

Linking...

> If it's the linker, then you could look into using gold instead of the
> BFD linker.
> 
> I suspect most of your build time is spent in dub, though.  You could
> try various options to turn off network access and dependency resolution
> when running dub, which should improve total running time. (Of course,
> you need to run it at least once without those options so that it pulls
> in required dependencies, but once those are downloaded, there's no need
> to check dependencies every single time.)  You could also ditch dub and
> use your own build system, by creating an empty dummy project in a
> subdirectory and running dub once to pull in your dependencies, then
> link to the artifacts yourself directly. (I did that in one of my own
> projects, and it sped up build time by an order of magnitude.)
> 
> Another possible cause is that Diet templates are very template- and
> CTFE-heavy, and will often soak up gobs of RAM and CPU every time they
> are compiled.  A possible solution is to separate out the Diet templates
> into their own source files, and use separate compilation to avoid
> rebuilding those files unless there was an actual change in the Diet
> code. (I've also run into this in my vibe.d project. I'm considering to
> replace Diet templates with something lighter-weight, but haven't taken
> that step yet.)

I suspect the issue is not exactly the linker, but the symbol name sizes again (probably related to the diet templates). I haven't looked into it enough to see where the problem is coming from, but one of either vibe.d using naming tricks to limit the name size, or D doing something clever with string template parameters should help.

-Steve

On Friday, 9 November 2018 at 02:33:42 UTC, Steven Schveighoffer wrote:
> Yeah, a vibe.d project with diet templates takes most of its time in the linker, I've found. Basically a big hang after dub prints:
>
> Linking...

Yep, that's it.

>
> I suspect the issue is not exactly the linker, but the symbol name sizes again (probably related to the diet templates). I haven't looked into it enough to see where the problem is coming from, but one of either vibe.d using naming tricks to limit the name size, or D doing something clever with string template parameters should help.
>

I've tried commenting out all "render" call so I have no diet templates and I save 1 second of total time (10.5 to 9.6) and it's still the linker that takes all the time.

I looked at the gold linker and it's already a possible configuration with Gentoo Linux! I've switched and it now compiles and link in 4 seconds! Yah!

But I don't want to switch my whole system to use gold, is there a way to configure dub or dmd to use usr/bin/ld.gold instead of /usr/bin/ld ?

Thanks!

On Fri, Nov 09, 2018 at 02:47:38AM +0000, bennygui via Digitalmars-d wrote: [...]
> I looked at the gold linker and it's already a possible configuration with Gentoo Linux! I've switched and it now compiles and link in 4 seconds! Yah!
> 
> But I don't want to switch my whole system to use gold, is there a way to configure dub or dmd to use usr/bin/ld.gold instead of /usr/bin/ld ?
[...]

You just need to pass the `-L-fuse-ld=gold` flag to your dmd invocation, or `-fuse-ld=gold` for LDC.  Not 100% sure how to configure this in dub, though.


T

-- 
Computers aren't intelligent; they only think they are.

On Friday, 9 November 2018 at 02:47:38 UTC, bennygui wrote:
> On Friday, 9 November 2018 at 02:33:42 UTC, Steven Schveighoffer wrote:
>> [...]
>
> Yep, that's it.
>
>>[...]
>
> I've tried commenting out all "render" call so I have no diet templates and I save 1 second of total time (10.5 to 9.6) and it's still the linker that takes all the time.
>
> I looked at the gold linker and it's already a possible configuration with Gentoo Linux! I've switched and it now compiles and link in 4 seconds! Yah!
>
> But I don't want to switch my whole system to use gold, is there a way to configure dub or dmd to use usr/bin/ld.gold instead of /usr/bin/ld ?
>
> Thanks!

Yes, LDC has the -linker=gold flag, that you can use to pass a different linker to your C compiler. Try it with lld, it's the fastest of all the linkers.

On Friday, 9 November 2018 at 02:47:38 UTC, bennygui wrote:
> On Friday, 9 November 2018 at 02:33:42 UTC, Steven Schveighoffer wrote:
>> [...]
>
> Yep, that's it.
>
>>[...]
>
> I've tried commenting out all "render" call so I have no diet templates and I save 1 second of total time (10.5 to 9.6) and it's still the linker that takes all the time.
>
> I looked at the gold linker and it's already a possible configuration with Gentoo Linux! I've switched and it now compiles and link in 4 seconds! Yah!
>
> But I don't want to switch my whole system to use gold, is there a way to configure dub or dmd to use usr/bin/ld.gold instead of /usr/bin/ld ?
>
> Thanks!

For dmd there is the dmd.conf file which has the DFLAGS attribute.
I am not an expert in this area but you may could set here the
default linker.

Kind regards
André

On Friday, 9 November 2018 at 03:47:09 UTC, Joakim wrote:
> On Friday, 9 November 2018 at 02:47:38 UTC, bennygui wrote:
>> [...]
>
> Yes, LDC has the -linker=gold flag, that you can use to pass a different linker to your C compiler. Try it with lld, it's the fastest of all the linkers.

lld mostly produces crashing binaries for me on Linux.

On Friday, 9 November 2018 at 03:41:18 UTC, H. S. Teoh wrote:
> You just need to pass the `-L-fuse-ld=gold` flag to your dmd invocation, or `-fuse-ld=gold` for LDC.

As Joakim said, the linker switch for LDC is `-linker=gold`, not `-fuse-ld` (that's the gcc/clang switch). If you guys read the release notes, you'd have seen that LDC v1.13 newly defaults to the gold linker on Linux (due to various issues with older bfd versions), which is apparently the sole reason the OP saw much better performance with LDC.