Thread overview | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
April 25, 2020 dmdcache | ||||
---|---|---|---|---|
| ||||
A colleague of mine has written dmdcache which may be very useful for some projects: https://github.com/seeraven/dmdcache It drops our build time from 8 minutes to 45 seconds on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow! This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size. Enjoy! Ali |
April 25, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
> A colleague of mine has written dmdcache which may be very useful for some projects:
>
> https://github.com/seeraven/dmdcache
>
> It drops our build time
>
> from 8 minutes
> to 45 seconds
>
> on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!
>
> This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.
>
> Enjoy!
>
> Ali
The main problem with this is that it does not take string imports into account, (or does it ???, I don't see how it could ....)
Also the compilers output can depend on the timestamp at which the compilation was done.
|
April 25, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
> A colleague of mine has written dmdcache which may be very useful for some projects:
>
> https://github.com/seeraven/dmdcache
>
> It drops our build time
>
> from 8 minutes
> to 45 seconds
Hey Ali,
Did you also try with LDC's built-in object file caching (asm codegen caching) ?
Cheers,
Johan
|
April 25, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
> A colleague of mine has written dmdcache which may be very useful for some projects:
>
> https://github.com/seeraven/dmdcache
>
> It drops our build time
>
> from 8 minutes
> to 45 seconds
>
> on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!
>
> This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.
>
> Enjoy!
>
> Ali
Perhaps I'm being very dumb, but how does this differ from just using make?
|
April 25, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to Stefan Koch | On Saturday, 25 April 2020 at 10:35:49 UTC, Stefan Koch wrote:
> On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
>> A colleague of mine has written dmdcache which may be very useful for some projects:
>>
>> https://github.com/seeraven/dmdcache
>>
>> It drops our build time
>>
>> from 8 minutes
>> to 45 seconds
>>
>> on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!
>>
>> This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.
>>
>> Enjoy!
>>
>> Ali
>
> The main problem with this is that it does not take string imports into account, (or does it ???, I don't see how it could ....)
> Also the compilers output can depend on the timestamp at which the compilation was done.
Yeah, doesn't look like it which means it might not be useful in projects that does a lot of compile-time stuff.
There is no way to determine the import statements either as those themselves can be generated at compile-time.
I can see this being useful for large projects that does not use CTFE but other than that I think it might just create subtle bugs because you won't immediately know that some part of your code didn't update and isn't working as intended because the code for it was imported from another file etc.
|
April 26, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to bauss | On 4/25/20 9:01 AM, bauss wrote: > On Saturday, 25 April 2020 at 10:35:49 UTC, Stefan Koch wrote: >> The main problem with this is that it does not take string imports >> into account [...] > Yeah, doesn't look like it which means it might not be useful in > projects that does a lot of compile-time stuff. > > There is no way to determine the import statements either as those > themselves can be generated at compile-time. Luckily, dmd gives all of that information with both -v and -X command line switches. However, currently dmdcache doesn't seem to look for string imports; opportunity for improvement. :) Ali |
April 26, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to Johan | On 4/25/20 4:39 AM, Johan wrote:
> On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
>> A colleague of mine has written dmdcache which may be very useful for some projects:
>>
>> https://github.com/seeraven/dmdcache
>>
>> It drops our build time
>>
>> from 8 minutes
>> to 45 seconds
>
> Hey Ali,
> Did you also try with LDC's built-in object file caching (asm codegen caching) ?
>
> Cheers,
> Johan
>
That sounds very promising. Time to introduce ldc to the project.
Ali
|
April 26, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to John Colvin | On 4/25/20 5:30 AM, John Colvin wrote: > how does this differ from just using make? make is great and I love it (really) but it works at a coarser level. There is no way for it to know that a particular command will produce the same output. As I understand it, dmdcache is supposed to be similar to ccache, which we already use with make to speed up our C++ compilations: https://ccache.dev/ Ali |
April 27, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On 27/04/2020 1:52 AM, Ali Çehreli wrote:
> However, currently dmdcache doesn't seem to look for string imports; opportunity for improvement. :)
That should not be necessary if you base it on -J instead.
|
December 29, 2020 Re: dmdcache | ||||
---|---|---|---|---|
| ||||
Posted in reply to John Colvin | On Saturday, 25 April 2020 at 12:30:17 UTC, John Colvin wrote: > But how does this differ from just using make? make doesn't support total-dependency-content-digest keying of cached artifacts similar to what, for instance, Bazel does. |
Copyright © 1999-2021 by the D Language Foundation