A colleague of mine has written dmdcache which may be very useful for some projects:

  https://github.com/seeraven/dmdcache

It drops our build time

  from 8 minutes
  to 45 seconds

on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!

This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.

Enjoy!

Ali

On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
> A colleague of mine has written dmdcache which may be very useful for some projects:
>
>   https://github.com/seeraven/dmdcache
>
> It drops our build time
>
>   from 8 minutes
>   to 45 seconds
>
> on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!
>
> This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.
>
> Enjoy!
>
> Ali

The main problem with this is that it does not take string imports into account, (or does it ???, I don't see how it could ....)
Also the compilers output can depend on the timestamp at which the compilation was done.

On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
> A colleague of mine has written dmdcache which may be very useful for some projects:
>
>   https://github.com/seeraven/dmdcache
>
> It drops our build time
>
>   from 8 minutes
>   to 45 seconds

Hey Ali,
  Did you also try with LDC's built-in object file caching (asm codegen caching) ?

Cheers,
  Johan

On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
> A colleague of mine has written dmdcache which may be very useful for some projects:
>
>   https://github.com/seeraven/dmdcache
>
> It drops our build time
>
>   from 8 minutes
>   to 45 seconds
>
> on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!
>
> This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.
>
> Enjoy!
>
> Ali

Perhaps I'm being very dumb, but how does this differ from just using make?

On Saturday, 25 April 2020 at 10:35:49 UTC, Stefan Koch wrote:
> On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
>> A colleague of mine has written dmdcache which may be very useful for some projects:
>>
>>   https://github.com/seeraven/dmdcache
>>
>> It drops our build time
>>
>>   from 8 minutes
>>   to 45 seconds
>>
>> on a particular build environment for about half a dozen D programs, one of which ends up being a 2G executable! WAT! :) And the total cache size is 5.5G. Wow!
>>
>> This build is with dmd 2.084.1 and that one particular application uses tons of template instantiations most of which are in generated source code. If I remember correctly, 2.084.1 does not contain template symbol name improvements and that may be the reason for the large size.
>>
>> Enjoy!
>>
>> Ali
>
> The main problem with this is that it does not take string imports into account, (or does it ???, I don't see how it could ....)
> Also the compilers output can depend on the timestamp at which the compilation was done.

Yeah, doesn't look like it which means it might not be useful in projects that does a lot of compile-time stuff.

There is no way to determine the import statements either as those themselves can be generated at compile-time.

I can see this being useful for large projects that does not use CTFE but other than that I think it might just create subtle bugs because you won't immediately know that some part of your code didn't update and isn't working as intended because the code for it was imported from another file etc.

On 4/25/20 9:01 AM, bauss wrote:

> On Saturday, 25 April 2020 at 10:35:49 UTC, Stefan Koch wrote:

>> The main problem with this is that it does not take string imports
>> into account

[...]

> Yeah, doesn't look like it which means it might not be useful in
> projects that does a lot of compile-time stuff.
>
> There is no way to determine the import statements either as those
> themselves can be generated at compile-time.

Luckily, dmd gives all of that information with both -v and -X command line switches.

However, currently dmdcache doesn't seem to look for string imports; opportunity for improvement. :)

Ali

On 4/25/20 4:39 AM, Johan wrote:
> On Saturday, 25 April 2020 at 10:17:50 UTC, Ali Çehreli wrote:
>> A colleague of mine has written dmdcache which may be very useful for some projects:
>>
>>   https://github.com/seeraven/dmdcache
>>
>> It drops our build time
>>
>>   from 8 minutes
>>   to 45 seconds
> 
> Hey Ali,
>    Did you also try with LDC's built-in object file caching (asm codegen caching) ?
> 
> Cheers,
>    Johan
> 

That sounds very promising. Time to introduce ldc to the project.

Ali

On 4/25/20 5:30 AM, John Colvin wrote:

> how does this differ from just using make?

make is great and I love it (really) but it works at a coarser level. There is no way for it to know that a particular command will produce the same output.

As I understand it, dmdcache is supposed to be similar to ccache, which we already use with make to speed up our C++ compilations:

  https://ccache.dev/

Ali

On 27/04/2020 1:52 AM, Ali Çehreli wrote:
> However, currently dmdcache doesn't seem to look for string imports; opportunity for improvement. :)

That should not be necessary if you base it on -J instead.

On Saturday, 25 April 2020 at 12:30:17 UTC, John Colvin wrote:
> But how does this differ from just using make?

make doesn't support total-dependency-content-digest keying of cached artifacts similar to what, for instance, Bazel does.