Is there any way to speed of compilation in Visual D? Seems everything is being recompiled from scratch even with minute changes. I'd think compiling to object files might help but maybe not?

On 23/04/2019 02:50, Alex wrote:
> Is there any way to speed of compilation in Visual D? Seems everything is being recompiled from scratch even with minute changes. I'd think compiling to object files might help but maybe not?

With visualdproj project files, you can select between building all at once (which would cause what you describe unless you separate your code into multiple libraries) and single file compilation which will only recompile what (transitively) imports the file with the changes, but that's often only faster if there are only few files to update.

The vcproj projects can compile per package, i.e. there is some kind of automatic separation into libraries. In addition, the latest version allows you to enable parallel compilation on multiple cores.

On Tuesday, 23 April 2019 at 07:03:36 UTC, Rainer Schuetze wrote:
>
>
> On 23/04/2019 02:50, Alex wrote:
>> Is there any way to speed of compilation in Visual D? Seems everything is being recompiled from scratch even with minute changes. I'd think compiling to object files might help but maybe not?
>
> With visualdproj project files, you can select between building all at once (which would cause what you describe unless you separate your code into multiple libraries) and single file compilation which will only recompile what (transitively) imports the file with the changes, but that's often only faster if there are only few files to update.

How do I enable this? There are probably 10 thousand files and I updating just a few.

I tried some of the different compilation options[combined compile and link and separate compile and link] and they all took about the same time except with separate compile I'd get the warning:

x64\Debug DMD\_.obj : warning LNK4042: object specified more than once; extras ignored

x4

> The vcproj projects can compile per package, i.e. there is some kind of automatic separation into libraries. In addition, the latest version allows you to enable parallel compilation on multiple cores.

Is there a way to move a visualdproj to a vcproj?

What it sounds like is that you are saying vcproj is better?

On 23/04/2019 18:03, Alex wrote:
> On Tuesday, 23 April 2019 at 07:03:36 UTC, Rainer Schuetze wrote:
>>
>>
>> On 23/04/2019 02:50, Alex wrote:
>>> Is there any way to speed of compilation in Visual D? Seems everything is being recompiled from scratch even with minute changes. I'd think compiling to object files might help but maybe not?
>>
>> With visualdproj project files, you can select between building all at once (which would cause what you describe unless you separate your code into multiple libraries) and single file compilation which will only recompile what (transitively) imports the file with the changes, but that's often only faster if there are only few files to update.
> 
> How do I enable this? There are probably 10 thousand files and I updating just a few.
> 
> I tried some of the different compilation options[combined compile and link and separate compile and link] and they all took about the same time except with separate compile I'd get the warning:
> 
> x64\Debug DMD\_.obj : warning LNK4042: object specified more than once; extras ignored
> 
> x4

I guess you have multiple file _.d in different directories? You should enable "keep path from source" on the output page so these are not written to the same object files.

"combined" and "separate" compile and link only have subtle differences that won't have an effect on large projects. "combined" will try to let dmd also do the link in the same invocation, but that is often prohibited by other options.

You might want to try "single file compilation" with "keep path from source" instead.

> 
>> The vcproj projects can compile per package, i.e. there is some kind of automatic separation into libraries. In addition, the latest version allows you to enable parallel compilation on multiple cores.
> 
> Is there a way to move a visualdproj to a vcproj?

Unfortunately, no. You can include all files from a directory recursively, as used by dmd (https://github.com/dlang/dmd/blob/master/src/vcbuild/dmd.vcxproj#L214), but it won't recreate the folder structure in the project. VC projects also don't let you drop directories into the solution, so folders have to be created manually.

> 
> What it sounds like is that you are saying vcproj is better?

It's probably the future, but there are a couple of features that haven't made it to the msbuild based projects, e.g. OMF support, private phobos build.

On Wednesday, 24 April 2019 at 05:34:10 UTC, Rainer Schuetze wrote:
>
>
> On 23/04/2019 18:03, Alex wrote:
>> On Tuesday, 23 April 2019 at 07:03:36 UTC, Rainer Schuetze wrote:
>>>
>>>
>>> On 23/04/2019 02:50, Alex wrote:
>>>> Is there any way to speed of compilation in Visual D? Seems everything is being recompiled from scratch even with minute changes. I'd think compiling to object files might help but maybe not?
>>>
>>> With visualdproj project files, you can select between building all at once (which would cause what you describe unless you separate your code into multiple libraries) and single file compilation which will only recompile what (transitively) imports the file with the changes, but that's often only faster if there are only few files to update.
>> 
>> How do I enable this? There are probably 10 thousand files and I updating just a few.
>> 
>> I tried some of the different compilation options[combined compile and link and separate compile and link] and they all took about the same time except with separate compile I'd get the warning:
>> 
>> x64\Debug DMD\_.obj : warning LNK4042: object specified more than once; extras ignored
>> 
>> x4
>
> I guess you have multiple file _.d in different directories? You should enable "keep path from source" on the output page so these are not written to the same object files.
>
> "combined" and "separate" compile and link only have subtle differences that won't have an effect on large projects. "combined" will try to let dmd also do the link in the same invocation, but that is often prohibited by other options.
>
> You might want to try "single file compilation" with "keep path from source" instead.
>
When I try single file compilation I get errors about models not being found. the problem doesn't exist in combined compilation.

I think, but not sure, it's because I have some files in different directories that access files across "realms".

So I'm having to add include's for all these directories just to get things to compile ;/

Since I use a hierarchy of folders, this is a real pain in the ass ;/ D doesn't seem to let one recursively search included subdirectories for the includes ;/

It wants a flat list of files to work well, it seems, which is asinine given that it is very hard to maintain projects that way.

Why it works on combined compile I don't know. The build log is vastly different and seems to be figuring out the import paths and doing something with rsp files.

I won't be able to use single file compilation the way it is working since I'd have to include about 100 directories ;/

....

Ok, so I went ahead and did it, it wasn't 100 but about 20. The copy full path came in handy.

Now I get a compiler crash!

It seems because it is due to the fact that I have some code that reads modules and imports their text, which requires a -J for every directory, and I have to add it for every new file path I end up using(One thing I hate D about because it forces me to keep track of all this mess rather than just working).

The thing is, I have all the proper -J's under command line. But they are not showing up on the command line but they are when I do combined compile.

So possibly this is a bug in Visual D where single compile mode is ignoring the additional command line options?

As if when generating the command line for single file compilation you forgot to append the additional options string to it.




May I make a suggestion?


Allow visual D to automatically generate paths for the command line by using wild cards.

This might require a special input box like

-J [                         ] [+]

Where one can then add paths like one can do with imports, versions, etc

but one can use wild cards like

C:\Project\*;C:\WaltersSecureInfoToHack\*

and visual D will parse all the sub directories and add them manually.

This will alleviate some major headaches with D's security but it should be valid since it's used inside an IDE. (since no one passes around visual D project files anyways.

The same could be done for import paths too.

It would simply allow me to use one import such as C:\Projects\*. Since all ones I added are hard coded, if I ever move things around I have to modify all the paths(and remember to do so).


Thanks.

On 26/04/2019 06:50, Alex wrote:
>>
>> You might want to try "single file compilation" with "keep path from source" instead.
>>
> When I try single file compilation I get errors about models not being found. the problem doesn't exist in combined compilation.
> 
> I think, but not sure, it's because I have some files in different directories that access files across "realms".
>> So I'm having to add include's for all these directories just to get
> things to compile ;/
> 
> Since I use a hierarchy of folders, this is a real pain in the ass ;/ D doesn't seem to let one recursively search included subdirectories for the includes ;/
> 
> It wants a flat list of files to work well, it seems, which is asinine given that it is very hard to maintain projects that way.
> 
> Why it works on combined compile I don't know. The build log is vastly different and seems to be figuring out the import paths and doing something with rsp files.

If the modules are passed on the command line, the compiler already knows about them by the module statements and doesn't have to search any directories.

> 
> I won't be able to use single file compilation the way it is working since I'd have to include about 100 directories ;/
> 
> ....
> 
> Ok, so I went ahead and did it, it wasn't 100 but about 20. The copy full path came in handy.
> 
> Now I get a compiler crash!
> 
> It seems because it is due to the fact that I have some code that reads modules and imports their text, which requires a -J for every directory, and I have to add it for every new file path I end up using(One thing I hate D about because it forces me to keep track of all this mess rather than just working).
> 
> The thing is, I have all the proper -J's under command line. But they are not showing up on the command line but they are when I do combined compile.

You can also add them to the "String import paths" option.

> So possibly this is a bug in Visual D where single compile mode is ignoring the additional command line options?

> As if when generating the command line for single file compilation you forgot to append the additional options string to it.

Yes, looks like a bug in Visual D: it adds the additional command line options to the link, but not to the single file compiler invocations.

> 
> 
> 
> May I make a suggestion?
> 
> 
> Allow visual D to automatically generate paths for the command line by using wild cards.
> 
> This might require a special input box like
> 
> -J [                         ] [+]
> 
> Where one can then add paths like one can do with imports, versions, etc
> 
> but one can use wild cards like
> 
> C:\Project\*;C:\WaltersSecureInfoToHack\*
> 
> and visual D will parse all the sub directories and add them manually.
> 
> This will alleviate some major headaches with D's security but it should be valid since it's used inside an IDE. (since no one passes around visual D project files anyways.
> 
> The same could be done for import paths too.
> 
> It would simply allow me to use one import such as C:\Projects\*. Since all ones I added are hard coded, if I ever move things around I have to modify all the paths(and remember to do so).

I'm not so sure this is a recommended workflow.

If the modules in the different folders depend on each other, you might start the module hierarchy a bit further up, e.g. make c:\projects the root of the package hierarchy. Then only a single import path is needed.

If you have independent packages in your projects folder, you could create separate projects building static libraries. Enabling "Add import paths of dependent projects" can automatically add the folders to the main project (but it might miss string imports ATM). This would also help the initial problem with only recompiling parts that have changed.

You could also generate the import path options to a response file and add that to the additional command lines as "@import_options.rsp", but it needs the bug above to be fixed.

On 26/04/2019 09:52, Rainer Schuetze wrote:
>> As if when generating the command line for single file compilation you forgot to append the additional options string to it.
> Yes, looks like a bug in Visual D: it adds the additional command line options to the link, but not to the single file compiler invocations.
> 

fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2

On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze wrote:
>
>
> On 26/04/2019 09:52, Rainer Schuetze wrote:
>>> As if when generating the command line for single file compilation you forgot to append the additional options string to it.
>> Yes, looks like a bug in Visual D: it adds the additional command line options to the link, but not to the single file compiler invocations.
>> 
>
> fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2

Man, it's really slow compiling!

It took over 3 minutes to compile the same project that compiles in 10 seconds with combined compile.

Even after compilation of all the object files it takes 10 seconds just to recompile 1 file.

There is absolutely no speed up.

Also, I always get

x64\Debug DMD\S.exe not up to date: d:\repos\s\s\x64\debug dmd\s.exe older than d:\repos\s\s\$(outdir)\g\d\set.obj

and so it has to build this one obj file every time.
I'm not sure what is going on here set.d is not being changed as far as I know.

In any case, unless there is a huge bug in single file compilation I don't see it being at all faster than combined compile.

On 28/04/2019 13:47, Alex wrote:
> On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze wrote:
>>
>>
>> On 26/04/2019 09:52, Rainer Schuetze wrote:
>>>> As if when generating the command line for single file compilation you forgot to append the additional options string to it.
>>> Yes, looks like a bug in Visual D: it adds the additional command line options to the link, but not to the single file compiler invocations.
>>>
>>
>> fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2
> 
> Man, it's really slow compiling!
> 
> It took over 3 minutes to compile the same project that compiles in 10 seconds with combined compile.

Yes, it's usually that bad if you have a good number of dependencies
(imports).

> 
> Even after compilation of all the object files it takes 10 seconds just to recompile 1 file.
> 

If a single file needs 10 sec, there are probably quite a few imports. You can check the dep-files in the intermediate directory to see detected dependencies.

> There is absolutely no speed up.
> 
> Also, I always get
> 
> x64\Debug DMD\S.exe not up to date: d:\repos\s\s\x64\debug dmd\s.exe older than d:\repos\s\s\$(outdir)\g\d\set.obj

That sounds like it is only linking again, though "$(outdir)" in the name looks wrong. Is that really your build directory?

> and so it has to build this one obj file every time.
> I'm not sure what is going on here set.d is not being changed as far as
> I know.
> 
> In any case, unless there is a huge bug in single file compilation I don't see it being at all faster than combined compile.
> 

Combined compilation is preferred in most cases. If you can separate into non-dependent libraries that could help.

On Sunday, 28 April 2019 at 12:22:15 UTC, Rainer Schuetze wrote:
>
>
> On 28/04/2019 13:47, Alex wrote:
>> On Sunday, 28 April 2019 at 08:25:18 UTC, Rainer Schuetze wrote:
>>>
>>>
>>> On 26/04/2019 09:52, Rainer Schuetze wrote:
>>>>> As if when generating the command line for single file compilation you forgot to append the additional options string to it.
>>>> Yes, looks like a bug in Visual D: it adds the additional command line options to the link, but not to the single file compiler invocations.
>>>>
>>>
>>> fixed in https://github.com/dlang/visuald/releases/tag/v0.49.2
>> 
>> Man, it's really slow compiling!
>> 
>> It took over 3 minutes to compile the same project that compiles in 10 seconds with combined compile.
>
> Yes, it's usually that bad if you have a good number of dependencies
> (imports).
>
So I removed most of the gtkd import modules I wasn't using and it went from 810 to 540. Didn't speed anything up but maybe 1/2 a second.

I guess the compiler has no self profiling to figure out where it is spending most of it's time?