Hello Community,

I thought there's a separate forum for VisualD. It did exist when VisualD was on DSource, so why not add it here as well? Or am I to blind to see?

Anyway, this thread is an addition to my previous one in this forum:
http://forum.dlang.org/thread/wkkuvzkzeupyfdpweals@forum.dlang.org

I noticed increasing compile times in my projects. As I use only VisualD I would like to understand how I would shorten compile time with this tool.
Brief description to my setup, more details can be fond in the other post.
I have one Lib, called MyLib consisting of around 15 modules, one class per module, zero to two template methods per class with one type parameter each.
All my projects use MyLib, and call the template methods with different types. Most of the compile time of any of theses projects is spent in rebuilding MyLib.
I am not sure why and where so much time is spent, but is there a way to profile my COMPILE time with VisualD?

There are several VisualD project properties which I do not understand fully, but hope that they might help, namely:

Configuration Properties - General - Files to clean
Configuration Properties - Compiler - Output - Multiple Object Files
Configuration Properties - Compiler - Output - Keep Path From Source File

Could not cleaning some files improve compilation ?
Could Multiple Object Files be used to separately compile non-template and template code blocks ?
What does Keep Path From Source File do at all ?

It is possible to remove the template methods from my classes, create free functions instead and use them in a UFCS way.
Unfortunately I have not figured out UFCS properly, as my approaches do not work ( guess due to UFCS restrictions ).

Anyway, would it help ( and is it possible at all ) to put the template functions in a separate module so that only the corresponding object file requires a recompile ?
I am asking in the context of the Multiple Object Files setting.

I tried to not build MyLib at all, and use the sources directly, but the project setup is kind of confusing. Again, there is a setting which I seem to not understand fully. So one project, lets say MyProject, has only its own module files added to the VisualD project. I want the compiler to also use the MyLib source files, but do not want to add them to MyProject, reasoning bellow. There is one entry in the project properties, which should make this behavior possible, but it doses not. Why ?

Configuration Properties - Compiler - General - Additional Imports

As far as I understand this setting, I am supposed to enter module search paths. But I do get linker errors when I do not add either MyLib.lib to the linker settings or all the source files of MyLib to MyProject.

Reasoning why I do not want to do this: I have one solution file with the MyLib project and ten projects like MyProject using MyLib. I would need to add all the source files to all the projects, they would be reachable and editable at 11 different location within my solution file. This does not not seem to be a clean way to set it up.

Any advice to any or all the thoughts and issues ?

Thanks in advance.

Cheers, ParticlePeter

On 05.07.2014 16:05, ParticlePeter wrote:
> Hello Community,
>
> I thought there's a separate forum for VisualD. It did exist when
> VisualD was on DSource, so why not add it here as well? Or am I to blind
> to see?

The forum digitalmars.D.ide is probably the best fit.

>
> Anyway, this thread is an addition to my previous one in this forum:
> http://forum.dlang.org/thread/wkkuvzkzeupyfdpweals@forum.dlang.org
>
> I noticed increasing compile times in my projects. As I use only VisualD
> I would like to understand how I would shorten compile time with this tool.
> Brief description to my setup, more details can be fond in the other post.
> I have one Lib, called MyLib consisting of around 15 modules, one class
> per module, zero to two template methods per class with one type
> parameter each.
> All my projects use MyLib, and call the template methods with different
> types. Most of the compile time of any of theses projects is spent in
> rebuilding MyLib.
> I am not sure why and where so much time is spent, but is there a way to
> profile my COMPILE time with VisualD?

The longer compile time is very likely caused by the compiler. You could check "Verbose compile" (command line option -v) to see what templates are expanded and what functions are compiled. This might give you a hint where the compiler spends the time.

> There are several VisualD project properties which I do not understand
> fully, but hope that they might help, namely:
>
> Configuration Properties - General - Files to clean
> Configuration Properties - Compiler - Output - Multiple Object Files
> Configuration Properties - Compiler - Output - Keep Path From Source File
>
> Could not cleaning some files improve compilation ?

No.

> Could Multiple Object Files be used to separately compile non-template
> and template code blocks ?

No. This option exposes an undocumented command line option that splits a module into multiple object files as if dmd creates a library. Do not use it.

> What does Keep Path From Source File do at all ?

This instructs dmd to not strip the path components from module source file name when generating the object file name. This avoids troubles if you build to multiple object files, but there are modules with identical names in different packages.

>
> It is possible to remove the template methods from my classes, create
> free functions instead and use them in a UFCS way.
> Unfortunately I have not figured out UFCS properly, as my approaches do
> not work ( guess due to UFCS restrictions ).
>

That would not help, templates are instantiated and compiled when they are used, not when they are declared.

> Anyway, would it help ( and is it possible at all ) to put the template
> functions in a separate module so that only the corresponding object
> file requires a recompile ?
> I am asking in the context of the Multiple Object Files setting.

Incremental building of single object files is not well supported by dmd, there are too many troubles with instantiating templates in different object files.

>
> I tried to not build MyLib at all, and use the sources directly, but the
> project setup is kind of confusing. Again, there is a setting which I
> seem to not understand fully. So one project, lets say MyProject, has
> only its own module files added to the VisualD project. I want the
> compiler to also use the MyLib source files, but do not want to add them
> to MyProject, reasoning bellow. There is one entry in the project
> properties, which should make this behavior possible, but it doses not.
> Why ?
>
> Configuration Properties - Compiler - General - Additional Imports
>
> As far as I understand this setting, I am supposed to enter module
> search paths.

This is correct.

> But I do get linker errors when I do not add either
> MyLib.lib to the linker settings or all the source files of MyLib to
> MyProject.
>

You should add a dependency in the "Project -> Project Dependencies" dialog. This will ensure proper rebuilds and add the library to the command line of dependent projects.


> Reasoning why I do not want to do this: I have one solution file with
> the MyLib project and ten projects like MyProject using MyLib. I would
> need to add all the source files to all the projects, they would be
> reachable and editable at 11 different location within my solution file.
> This does not not seem to be a clean way to set it up.
>
> Any advice to any or all the thoughts and issues ?
>
> Thanks in advance.
>
> Cheers, ParticlePeter

On Sunday, 6 July 2014 at 08:09:07 UTC, Rainer Schuetze wrote:
>
>
> On 05.07.2014 16:05, ParticlePeter wrote:
...
>> It is possible to remove the template methods from my classes, create
>> free functions instead and use them in a UFCS way.
>> Unfortunately I have not figured out UFCS properly, as my approaches do
>> not work ( guess due to UFCS restrictions ).
>>
>
> That would not help, templates are instantiated and compiled when they are used, not when they are declared.

Hence the idea of splitting the class. After the split, the class has only non-template methods and is part of MyLib. Template UFCS functions working with this class are in another module, and not part of the lib. With this approach MyLib does not need to recompile, only the UFCS functions module needs to recompile due to different instantiations. UFCS works now for me btw.


>> I tried to not build MyLib at all, and use the sources directly, but the
>> project setup is kind of confusing. Again, there is a setting which I
>> seem to not understand fully. So one project, lets say MyProject, has
>> only its own module files added to the VisualD project. I want the
>> compiler to also use the MyLib source files, but do not want to add them
>> to MyProject, reasoning bellow. There is one entry in the project
>> properties, which should make this behavior possible, but it doses not.
>> Why ?
>>
>> Configuration Properties - Compiler - General - Additional Imports
>>
>> As far as I understand this setting, I am supposed to enter module
>> search paths.
>
> This is correct.
>
> > But I do get linker errors when I do not add either
>> MyLib.lib to the linker settings or all the source files of MyLib to
>> MyProject.
>>
>
> You should add a dependency in the "Project -> Project Dependencies" dialog. This will ensure proper rebuilds and add the library to the command line of dependent projects.

The idea is to NOT create a lib, but just use the source code from MyLib in MyProject as additional includes. No project dependencies. I was hoping that some .obj files are still available from last builds, and do not need to recompile, as they are up to date ( reason for question about not cleaning files ).
The modules form MyProject do import the MyLib modules properly, I do not get compiler errors. However, the compiler should create Object files from MyLib modules, and the linker should link them. But he does not.
On the other hand, when I add MyLib modules to MyProject ( Rightclick MyProject - add - existing item... MyLib source files ) then linking works. I do not understand why the later step is necessary.

On 06.07.2014 19:51, ParticlePeter wrote:
> On Sunday, 6 July 2014 at 08:09:07 UTC, Rainer Schuetze wrote:
>>
>>
>> On 05.07.2014 16:05, ParticlePeter wrote:
> ...
>>> It is possible to remove the template methods from my classes, create
>>> free functions instead and use them in a UFCS way.
>>> Unfortunately I have not figured out UFCS properly, as my approaches do
>>> not work ( guess due to UFCS restrictions ).
>>>
>>
>> That would not help, templates are instantiated and compiled when they
>> are used, not when they are declared.
>
> Hence the idea of splitting the class. After the split, the class has
> only non-template methods and is part of MyLib. Template UFCS functions
> working with this class are in another module, and not part of the lib.
> With this approach MyLib does not need to recompile, only the UFCS
> functions module needs to recompile due to different instantiations.
> UFCS works now for me btw.

Ok, that allows separate compilation of the class, but templates are still compiled with the rest of the program. I thought the templates were the part that cause the slow compilation.


>
>
>>> I tried to not build MyLib at all, and use the sources directly, but the
>>> project setup is kind of confusing. Again, there is a setting which I
>>> seem to not understand fully. So one project, lets say MyProject, has
>>> only its own module files added to the VisualD project. I want the
>>> compiler to also use the MyLib source files, but do not want to add them
>>> to MyProject, reasoning bellow. There is one entry in the project
>>> properties, which should make this behavior possible, but it doses not.
>>> Why ?
>>>
>>> Configuration Properties - Compiler - General - Additional Imports
>>>
>>> As far as I understand this setting, I am supposed to enter module
>>> search paths.
>>
>> This is correct.
>>
>> > But I do get linker errors when I do not add either
>>> MyLib.lib to the linker settings or all the source files of MyLib to
>>> MyProject.
>>>
>>
>> You should add a dependency in the "Project -> Project Dependencies"
>> dialog. This will ensure proper rebuilds and add the library to the
>> command line of dependent projects.
>
> The idea is to NOT create a lib, but just use the source code from MyLib
> in MyProject as additional includes. No project dependencies. I was
> hoping that some .obj files are still available from last builds, and do
> not need to recompile, as they are up to date ( reason for question
> about not cleaning files ).

These object files are in the library ;-) That means manual selection, though, as incremental builds to multiple object files don't work with dmd, and single file compilation is painfully slow.

> The modules form MyProject do import the MyLib modules properly, I do
> not get compiler errors. However, the compiler should create Object
> files from MyLib modules, and the linker should link them. But he does not.
> On the other hand, when I add MyLib modules to MyProject ( Rightclick
> MyProject - add - existing item... MyLib source files ) then linking
> works. I do not understand why the later step is necessary.

dmd does not compile imported modules, but rdmd does.

On Sunday, 6 July 2014 at 19:27:38 UTC, Rainer Schuetze wrote:

> Ok, that allows separate compilation of the class, but templates are still compiled with the rest of the program. I thought the templates were the part that cause the slow compilation.

I had no chance to profile till now, but I don't think that the templates are slow. They just extract type information from primitive or struct arrays, and pass the type info and void array on. I have the impression that the whole rebuild-relink process of the library itself was taking so long, so I wanted ensure that the non-templte code ( the majority ) does not have to be compiled over and over again.

> These object files are in the library ;-) That means manual selection, though, as incremental builds to multiple object files don't work with dmd, and single file compilation is painfully slow.

Not sure if I am getting this right, so when one object file has to be recompiled all other object files, even if up to date, would be recompiled ?

>> The modules form MyProject do import the MyLib modules properly, I do
>> not get compiler errors. However, the compiler should create Object
>> files from MyLib modules, and the linker should link them. But he does not.
>> On the other hand, when I add MyLib modules to MyProject ( Rightclick
>> MyProject - add - existing item... MyLib source files ) then linking
>> works. I do not understand why the later step is necessary.
>
> dmd does not compile imported modules, but rdmd does.

Ähm ... not seeing the connection here either, why is this significant ?

I feel that I could not explain my problem properly, so one example:
Importing phobos modules. I do not have to define any import path or lib file in the project settings, I just need to import std.somthing. That's because the import path for phobos modules are stored in the dmd sc.ini file.
When I want to import my modules which are somewhere on my hard-drive and not added to my project I need to tell the compiler where these modules can be found, using the additional import path project setting. That's fine, doing this.

But result is, std.somthing works, my modules in a path known by the compiler don't work, giving me linker errors. Why ? ( I do not create a lib, I just want to import the module. )

On 07.07.2014 12:46, ParticlePeter wrote:
> On Sunday, 6 July 2014 at 19:27:38 UTC, Rainer Schuetze wrote:
>
>> These object files are in the library ;-) That means manual selection,
>> though, as incremental builds to multiple object files don't work with
>> dmd, and single file compilation is painfully slow.
>
> Not sure if I am getting this right, so when one object file has to be
> recompiled all other object files, even if up to date, would be
> recompiled ?

That's how it is currently done if you don't use "single file compilation". Compiling only modified and dependent modules in one step could work incrementally, but especially template instantiations make this hard to do correctly.

>
>>> The modules form MyProject do import the MyLib modules properly, I do
>>> not get compiler errors. However, the compiler should create Object
>>> files from MyLib modules, and the linker should link them. But he
>>> does not.
>>> On the other hand, when I add MyLib modules to MyProject ( Rightclick
>>> MyProject - add - existing item... MyLib source files ) then linking
>>> works. I do not understand why the later step is necessary.
>>
>> dmd does not compile imported modules, but rdmd does.
>
> Ähm ... not seeing the connection here either, why is this significant ?

dmd just compiles the files given on the command line. rdmd makes two passes, one to collect imported files, and another to compile all the collected files. So rdmd works the way you want dmd to work (if I understand you correctly).

> I feel that I could not explain my problem properly, so one example:
> Importing phobos modules. I do not have to define any import path or lib
> file in the project settings, I just need to import std.somthing. That's
> because the import path for phobos modules are stored in the dmd sc.ini
> file.
> When I want to import my modules which are somewhere on my hard-drive
> and not added to my project I need to tell the compiler where these
> modules can be found, using the additional import path project setting.
> That's fine, doing this.
>
> But result is, std.somthing works, my modules in a path known by the
> compiler don't work, giving me linker errors. Why ? ( I do not create a
> lib, I just want to import the module. )
>

phobos is precompiled to a library and is automatically included in the link. If you want your custom modules to work the same way, you have to compile them to a library.

On Monday, 7 July 2014 at 22:00:51 UTC, Rainer Schuetze wrote:
>
>
> On 07.07.2014 12:46, ParticlePeter wrote:
>> On Sunday, 6 July 2014 at 19:27:38 UTC, Rainer Schuetze wrote:
>>
>>> These object files are in the library ;-) That means manual selection,
>>> though, as incremental builds to multiple object files don't work with
>>> dmd, and single file compilation is painfully slow.
>>
>> Not sure if I am getting this right, so when one object file has to be
>> recompiled all other object files, even if up to date, would be
>> recompiled ?
>
> That's how it is currently done if you don't use "single file compilation". Compiling only modified and dependent modules in one step could work incrementally, but especially template instantiations make this hard to do correctly.
>
>>
>>>> The modules form MyProject do import the MyLib modules properly, I do
>>>> not get compiler errors. However, the compiler should create Object
>>>> files from MyLib modules, and the linker should link them. But he
>>>> does not.
>>>> On the other hand, when I add MyLib modules to MyProject ( Rightclick
>>>> MyProject - add - existing item... MyLib source files ) then linking
>>>> works. I do not understand why the later step is necessary.
>>>
>>> dmd does not compile imported modules, but rdmd does.
>>
>> Ähm ... not seeing the connection here either, why is this significant ?
>
> dmd just compiles the files given on the command line. rdmd makes two passes, one to collect imported files, and another to compile all the collected files. So rdmd works the way you want dmd to work (if I understand you correctly).
>
>> I feel that I could not explain my problem properly, so one example:
>> Importing phobos modules. I do not have to define any import path or lib
>> file in the project settings, I just need to import std.somthing. That's
>> because the import path for phobos modules are stored in the dmd sc.ini
>> file.
>> When I want to import my modules which are somewhere on my hard-drive
>> and not added to my project I need to tell the compiler where these
>> modules can be found, using the additional import path project setting.
>> That's fine, doing this.
>>
>> But result is, std.somthing works, my modules in a path known by the
>> compiler don't work, giving me linker errors. Why ? ( I do not create a
>> lib, I just want to import the module. )
>>
>
> phobos is precompiled to a library and is automatically included in the link. If you want your custom modules to work the same way, you have to compile them to a library.

Thanks for claryfying all the above. If the std.library is rebuild whenever a template is used, than my assumption that MyLib compiles slow due to usage of ( very simple ) templates must be wrong. I will dive deeper into profiling. Thanks a lot.

Cheers, ParticlePeter