Hello,

I have a project that is taking very long to compile (I'm on Windows).

It consists of a bunch of precompiled library (.lib) files and some .d source files inside of my "source" directory. Currently I am using DUB and the .lib files are added to "sourceFiles" in dub.json, which works, but if I edit a single letter it will recompile everything.

One alternative would be to split up my package into "subPackages" to avoid recompiling everything, but that would mean splitting up my current directory structure into a structure that is harder to navigate.

Another alternative I tried is Reggae, but it doesn't work on Windows:
1. When calling .reggae/dcompile, it fails, because the Windows shell (intuitively as always) splits of the ".reggae" from "/dcompile" and treats them as separate arguments (managed to fix this by editing the Reggae source slightly, in payload\reggae\build.d : expandCmd).
2. Either it takes FOREVER to compile and link the thing, or it gets stuck in some kind of infinite loop.

Anyway, it doesn't work.

What kind of build system could I use to speed up compilation when editing very little, or could I configure DUB so it recompiles per file instead of per package?

Thank you.

On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:
> Hello,
>
> I have a project that is taking very long to compile (I'm on Windows).
>
> It consists of a bunch of precompiled library (.lib) files and some .d source files inside of my "source" directory. Currently I am using DUB and the .lib files are added to "sourceFiles" in dub.json, which works, but if I edit a single letter it will recompile everything.
>
> One alternative would be to split up my package into "subPackages" to avoid recompiling everything, but that would mean splitting up my current directory structure into a structure that is harder to navigate.
>
> Another alternative I tried is Reggae, but it doesn't work on Windows:
> 1. When calling .reggae/dcompile, it fails, because the Windows shell (intuitively as always) splits of the ".reggae" from "/dcompile" and treats them as separate arguments (managed to fix this by editing the Reggae source slightly, in payload\reggae\build.d : expandCmd).
> 2. Either it takes FOREVER to compile and link the thing, or it gets stuck in some kind of infinite loop.
>
> Anyway, it doesn't work.
>
> What kind of build system could I use to speed up compilation when editing very little, or could I configure DUB so it recompiles per file instead of per package?
>
> Thank you.

What project are you compiling? It would help to diagnose the problem, as well as what the specs of the computer you are using are.

On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:
> Hello,
>
> I have a project that is taking very long to compile (I'm on Windows).
>
> It consists of a bunch of precompiled library (.lib) files and some .d source files inside of my "source" directory. Currently I am using DUB and the .lib files are added to "sourceFiles" in dub.json, which works, but if I edit a single letter it will recompile everything.
>
> One alternative would be to split up my package into "subPackages" to avoid recompiling everything, but that would mean splitting up my current directory structure into a structure that is harder to navigate.
>
> Another alternative I tried is Reggae, but it doesn't work on Windows:
> 1. When calling .reggae/dcompile, it fails, because the Windows shell (intuitively as always) splits of the ".reggae" from "/dcompile" and treats them as separate arguments (managed to fix this by editing the Reggae source slightly, in payload\reggae\build.d : expandCmd).
> 2. Either it takes FOREVER to compile and link the thing, or it gets stuck in some kind of infinite loop.
>
> Anyway, it doesn't work.
>
> What kind of build system could I use to speed up compilation when editing very little, or could I configure DUB so it recompiles per file instead of per package?
>
> Thank you.

to speed up dub I recommend

1) split up your code base into multiple subpackages (isolated packages of code as seen in vibe.d)
2a) try using --build-mode=allAtOnce which will compile the entire project in one go (note that this will increase memory usage)
2b) try using --build-mode=singleFile which will compile each file separately and link them all together (note that this eliminates any possible caching by the compiler, but reduces memory usage which could also be a bottle neck)

Of these steps 1) will definitely help while developing as it will not recompile the packages which didn't change. 2a) or 2b) can be useful if you are on a very beefy machine or have memory constraints on a low level machine

On Saturday, 15 August 2020 at 15:45:44 UTC, Avrina wrote:
> What project are you compiling? It would help to diagnose the problem, as well as what the specs of the computer you are using are.

It is basically an implementation for a DSL, which has to parse, process and interpret it. The interpretation part has already been implemented and doesn't have to be recompiled (.lib), but it does have to link with it (which takes up quite a bit of time as well, because it has to redo it every time).

On Saturday, 15 August 2020 at 16:15:08 UTC, WebFreak001 wrote:
> [...]
>
> to speed up dub I recommend
>
> 1) split up your code base into multiple subpackages (isolated packages of code as seen in vibe.d)
> 2a) try using --build-mode=allAtOnce which will compile the entire project in one go (note that this will increase memory usage)
> 2b) try using --build-mode=singleFile which will compile each file separately and link them all together (note that this eliminates any possible caching by the compiler, but reduces memory usage which could also be a bottle neck)
>
> Of these steps 1) will definitely help while developing as it will not recompile the packages which didn't change. 2a) or 2b) can be useful if you are on a very beefy machine or have memory constraints on a low level machine

Splitting it up into subpackages helped indeed. --build-mode=singleFile did not speed it up much, but it did show me where the bottleneck was: the compiler was taking FOREVER on parser.d (I mean minutes, for a 3000-line file), so it sure helped to split that one off. Maybe there's a ton of hidden template instantiations going on? Or there's a compiler bug somewhere? I don't know.

Linking is still taking very long; I'm compiling for 64-bit with the MS linker. Could using another linker or some other linker technique speed it up?

Thank you for your response.

On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:
> Hello,
>
> I have a project that is taking very long to compile (I'm on Windows).
>
> [...]

As side information, it is possible to introduce sub packages without changing the folder structure. That said, I do not recommend it. Example https://www.github.com/adamdruppe/arsd/tree/master/

Kind regards
Andre

On Saturday, 15 August 2020 at 18:24:15 UTC, Andre Pany wrote:
> As side information, it is possible to introduce sub packages without changing the folder structure. That said, I do not recommend it. Example https://www.github.com/adamdruppe/arsd/tree/master/

The problem is that since all sub packages still have the same import path, any change to any sub package triggers a rebuild of all sub packages since they may import modules from each other, so you don't gain compile time there.

On Saturday, 15 August 2020 at 14:22:44 UTC, burt wrote:
> Hello,
>
> I have a project that is taking very long to compile (I'm on Windows).

Your project should take 10 secs to compile even with --build=release.
It's possible that, in case you would use DMD, you hit a problem of counter performance of the inliner/optimizer. If this is the case try ldc2. In some **rare occasions**, such the inler problem I mentioned, a release build can be faster with ldc2.

Side question, would your parser would be pegged based ?

On Saturday, 15 August 2020 at 18:33:06 UTC, Dennis wrote:
> The problem is that since all sub packages still have the same import path, any change to any sub package triggers a rebuild of all sub packages since they may import modules from each other, so you don't gain compile time there.

It shouldn't do that; if it does, I'd call it an implementation bug. You can list out the specific files that make up a subpackage.

On Saturday, 15 August 2020 at 19:20:39 UTC, Adam D. Ruppe wrote:
> It shouldn't do that; if it does, I'd call it an implementation bug. You can list out the specific files that make up a subpackage.

But an enum / template in a different module could still affect the subpackage even if it's not linked in, so dub conservatively assumes it invalidated the build. The same applies to  stringImportPaths. Maybe if dub supported also specifying `importFiles` instead of `importPaths` it could assume those other modules are not imported.

On Saturday, 15 August 2020 at 19:48:51 UTC, Dennis wrote:
> On Saturday, 15 August 2020 at 19:20:39 UTC, Adam D. Ruppe wrote:
>> It shouldn't do that; if it does, I'd call it an implementation bug. You can list out the specific files that make up a subpackage.
>
> But an enum / template in a different module could still affect the subpackage even if it's not linked in, so dub conservatively assumes it invalidated the build. The same applies to  stringImportPaths. Maybe if dub supported also specifying `importFiles` instead of `importPaths` it could assume those other modules are not imported.

If files from stringImportPaths are changed, this won't cause a rebuild of the dub package as the change is not detected by dub. For this reason a new attribute extraDependencyFiles were added. This attribute lists files which should cause a dub rebuild if file content changes.

Kind regards
Andre