Thread overview
How to accelerate the linking time?
Jul 10, 2020
zoujiaqing
Jul 10, 2020
Stefan Koch
Jul 10, 2020
Jacob Carlborg
July 10, 2020
zoujiaqing@osx:~/projects/hunt-skeleton (*)
> time dub build                                                                              master [fe3d6bc] modified
Performing "debug" build using /Library/D/dmd/bin/dmd for x86_64.
hunt 1.6.1: target for configuration "library" is up to date.
hunt-extra 1.0.0: target for configuration "library" is up to date.
hunt-net 0.5.1: building configuration "default"...
hunt-time 1.1.1: building configuration "library"...
hunt-proton 1.1.0-beta.3: building configuration "library"...
hunt-amqp 1.1.0-beta.4: building configuration "library"...
hunt-pool 1.1.1: building configuration "library"...
hunt-amqp-client 1.1.0-beta.2: building configuration "library"...
hunt-redis 1.2.0: building configuration "library"...
hunt-cache 0.8.1: building configuration "default"...
hunt-console 0.2.0: target for configuration "hunt-console" is up to date.
hunt-sql 1.4.0: target for configuration "library" is up to date.
hunt-database 2.1.0: building configuration "default"...
hunt-validation 0.3.0: target for configuration "library" is up to date.
hunt-entity 2.6.0: building configuration "library"...
hunt-http 0.6.2: building configuration "default"...
hunt-jwt 0.1.0-beta.1: target for configuration "library" is up to date.
hunt-security 0.4.0: target for configuration "library" is up to date.
hunt-shiro 1.1.3: building configuration "library"...
poodinis 8.1.0-beta.2: target for configuration "library" is up to date.
hunt-framework 3.2.0: building configuration "library"...
hunt-skeleton 3.1.0+commit.4.gfe3d6bc: building configuration "application"...
Linking...
To force a rebuild of up-to-date targets, run again with --force.
dub build  96.60s user 14.75s system 94% cpu 1:57.62 total


My laptop is MacBook Pro 15, but linking time so long.
July 10, 2020
On Friday, 10 July 2020 at 05:45:03 UTC, zoujiaqing wrote:
> [...]

I think the answer is use lld instead of gnu ld
July 10, 2020
On Friday, 10 July 2020 at 07:06:44 UTC, Stefan Koch wrote:
> On Friday, 10 July 2020 at 05:45:03 UTC, zoujiaqing wrote:
>> [...]
>
> I think the answer is use lld instead of gnu ld

He's running macOS, which doesn't use GNU ld. LLD does not work for D programs on macOS. At least not when compiling them using LDC.

--
/Jacob Carlborg
July 10, 2020
On 7/10/20 1:45 AM, zoujiaqing wrote:
> zoujiaqing@osx:~/projects/hunt-skeleton (*)
>> time dub build                             master [fe3d6bc] modified
> Performing "debug" build using /Library/D/dmd/bin/dmd for x86_64.
> hunt 1.6.1: target for configuration "library" is up to date.
> hunt-extra 1.0.0: target for configuration "library" is up to date.
> hunt-net 0.5.1: building configuration "default"...
> hunt-time 1.1.1: building configuration "library"...
> hunt-proton 1.1.0-beta.3: building configuration "library"...
> hunt-amqp 1.1.0-beta.4: building configuration "library"...
> hunt-pool 1.1.1: building configuration "library"...
> hunt-amqp-client 1.1.0-beta.2: building configuration "library"...
> hunt-redis 1.2.0: building configuration "library"...
> hunt-cache 0.8.1: building configuration "default"...
> hunt-console 0.2.0: target for configuration "hunt-console" is up to date.
> hunt-sql 1.4.0: target for configuration "library" is up to date.
> hunt-database 2.1.0: building configuration "default"...
> hunt-validation 0.3.0: target for configuration "library" is up to date.
> hunt-entity 2.6.0: building configuration "library"...
> hunt-http 0.6.2: building configuration "default"...
> hunt-jwt 0.1.0-beta.1: target for configuration "library" is up to date.
> hunt-security 0.4.0: target for configuration "library" is up to date.
> hunt-shiro 1.1.3: building configuration "library"...
> poodinis 8.1.0-beta.2: target for configuration "library" is up to date.
> hunt-framework 3.2.0: building configuration "library"...
> hunt-skeleton 3.1.0+commit.4.gfe3d6bc: building configuration "application"...
> Linking...
> To force a rebuild of up-to-date targets, run again with --force.
> dub build  96.60s user 14.75s system 94% cpu 1:57.62 total
> 
> 
> My laptop is MacBook Pro 15, but linking time so long.

The linking time is high because of the symbol count/sizes generally. Lots of templates means the linker takes a while to sort through the mess you made ;)

I have the same problem on vibe.d projects.

typically for me, the compilation step takes at least 50% of the time (hard to tell, as dub does not output the timestamp at that point).

-Steve
July 11, 2020
On Friday, 10 July 2020 at 12:15:56 UTC, Steven Schveighoffer wrote:
> On 7/10/20 1:45 AM, zoujiaqing wrote:
>> [...]
>
> The linking time is high because of the symbol count/sizes generally. Lots of templates means the linker takes a while to sort through the mess you made ;)
>

Mike Franklin optimized the mangling a while ago. That was a crucial change, IMO (imagine how big the original was!).

Any idea why the unnecessary symbols gets generated by the compiler (foreach, etc) in release mode? I can understand for debug builds though...
July 10, 2020
On 7/10/20 8:17 PM, Arun Chandrasekaran wrote:
> On Friday, 10 July 2020 at 12:15:56 UTC, Steven Schveighoffer wrote:
>> On 7/10/20 1:45 AM, zoujiaqing wrote:
>>> [...]
>>
>> The linking time is high because of the symbol count/sizes generally. Lots of templates means the linker takes a while to sort through the mess you made ;)
>>
> 
> Mike Franklin optimized the mangling a while ago. That was a crucial change, IMO (imagine how big the original was!).

Did you mean Rainer's change to use backreferences? That was for exponential symbols for Voldemort types with a lot of repetitive pieces. It's still quite possible to get large symbols if you wrap a ton of unrelated templates together (vibe.d and probably Hunt have quite a bit of templates that wrap each other).

Apologies if Mike did something else, I don't recall what that is.

There was a recent check of the basic hello world, and the number of symbols generated by format was alarming.

> 
> Any idea why the unnecessary symbols gets generated by the compiler (foreach, etc) in release mode? I can understand for debug builds though...

I don't know if anyone's actually done any research on what is "unnecessary". If there are a lot of unnecessary symbols in the binaries, it would be good to know where they are coming from, and how we can possibly remove them.

This area of D development (binary object symbols) is not very sexy, so I don't imagine many people want to dive into it.

-Steve
July 13, 2020
On Saturday, 11 July 2020 at 00:36:47 UTC, Steven Schveighoffer wrote:
> On 7/10/20 8:17 PM, Arun Chandrasekaran wrote:
>> On Friday, 10 July 2020 at 12:15:56 UTC, Steven Schveighoffer wrote:
>>> On 7/10/20 1:45 AM, zoujiaqing wrote:
>>>> [...]
>>>
>>> The linking time is high because of the symbol count/sizes generally. Lots of templates means the linker takes a while to sort through the mess you made ;)
>>>
>> 
>> Mike Franklin optimized the mangling a while ago. That was a crucial change, IMO (imagine how big the original was!).
>
> Did you mean Rainer's change to use backreferences?

Yeah, sorry about that. It was indeed Rainer's change!