Hello Folks,

I am currently integrating the tracy profiler (https://github.com/wolfpld/tracy)
with dmd.

Such that instead of the profiler in druntime, tracy can be used for instrumented profiling.


The current progress is here: https://github.com/dlang/dmd/compare/master...UplinkCoder:tracy?expand=1
 there are still some teething problems, but I am confident that it will be useful soon.

Cheers,

Stefan

On Monday, 3 August 2020 at 20:36:51 UTC, Stefan Koch wrote:
> Hello Folks,
>
> I am currently integrating the tracy profiler (https://github.com/wolfpld/tracy)
> with dmd.
>
> Such that instead of the profiler in druntime, tracy can be used for instrumented profiling.
>
>
> The current progress is here: https://github.com/dlang/dmd/compare/master...UplinkCoder:tracy?expand=1
>  there are still some teething problems, but I am confident that it will be useful soon.
>
> Cheers,
>
> Stefan


Wouldn't it be more useful for everyone, and for you too, to complete a project instead of starting dozens at the same time? I have completely lost hope of being able to see benefits from the newCTFE

On Tuesday, 4 August 2020 at 08:13:15 UTC, DepresseD wrote:
> On Monday, 3 August 2020 at 20:36:51 UTC, Stefan Koch wrote:
>> Hello Folks,
>>
>> I am currently integrating the tracy profiler (https://github.com/wolfpld/tracy)
>> with dmd.
>>
>> Such that instead of the profiler in druntime, tracy can be used for instrumented profiling.
>>
>>
>> The current progress is here: https://github.com/dlang/dmd/compare/master...UplinkCoder:tracy?expand=1
>>  there are still some teething problems, but I am confident that it will be useful soon.
>>
>> Cheers,
>>
>> Stefan
>
>
> Wouldn't it be more useful for everyone, and for you too, to complete a project instead of starting dozens at the same time? I have completely lost hope of being able to see benefits from the newCTFE

newCTFE in the end, has a lot less benefit than I first assumed.

Currently it is missing execption handling, and associative array support.
And of course there are some bugs in the featureset which is supported.

If I had had a good integrated profiler back then, and some of the code which I have access to now, I would probably never have started newCTFE, and would have tried to fix the template system itself.

On Tuesday, 4 August 2020 at 08:44:08 UTC, Stefan Koch wrote:
> newCTFE in the end, has a lot less benefit than I first assumed.

The speedup will certainly be useful to me since I am doing more and more at CTFE.

> If I had had a good integrated profiler back then, and some of the code which I have access to now, I would probably never have started newCTFE, and would have tried to fix the template system itself.

There is still time :)

On Tuesday, 4 August 2020 at 08:44:08 UTC, Stefan Koch wrote:
> [...]
>
> newCTFE in the end, has a lot less benefit than I first assumed.
>
> Currently it is missing execption handling, and associative array support.
> And of course there are some bugs in the featureset which is supported.
>
> If I had had a good integrated profiler back then, and some of the code which I have access to now, I would probably never have started newCTFE, and would have tried to fix the template system itself.

I think it's great that you are doing this. Tracing is an extremely lacking part in D, so I am grateful for what you have started here.

I'm curious though, why does this need to be a compiler change instead of a library addition? It says in the user manual that "There can be no more than 65534 unique source locations", so wouldn't automatically inserting the boilerplate into every function pretty quickly blow that up?

Also where does the referenced TracyClientNoExit.o file in your changes come from?

On Tuesday, 4 August 2020 at 09:09:41 UTC, WebFreak001 wrote:
> On Tuesday, 4 August 2020 at 08:44:08 UTC, Stefan Koch wrote:
>> [...]
>
> I think it's great that you are doing this. Tracing is an extremely lacking part in D, so I am grateful for what you have started here.
>
> I'm curious though, why does this need to be a compiler change instead of a library addition? It says in the user manual that "There can be no more than 65534 unique source locations", so wouldn't automatically inserting the boilerplate into every function pretty quickly blow that up?
>
Yeah I am running into that limit right now :p

I'll have to hack tracy :)

> Also where does the referenced TracyClientNoExit.o file in your changes come from?

TracyClientNoExit.so is produced by a build of the tracy cpp codebase.
I'll provide a script and instructions later.

On 8/4/20 4:44 AM, Stefan Koch wrote:
> newCTFE in the end, has a lot less benefit than I first assumed.

If CTFE becomes way less expensive (in CPU usage and memory usage), then the template problem becomes easier as well, as we can do more CTFE to replace templates.

> If I had had a good integrated profiler back then, and some of the code which I have access to now, I would probably never have started newCTFE, and would have tried to fix the template system itself.

I still think newCTFE has worthwhile benefit, even if it alone cannot fix all the problems.

I think newCTFE and type functions (along with Manu's ... DIP) would be a good combination to attack the problem.

-Steve

On Tuesday, 4 August 2020 at 08:13:15 UTC, DepresseD wrote:
> On Monday, 3 August 2020 at 20:36:51 UTC, Stefan Koch wrote:
>> Hello Folks,
>>
>> I am currently integrating the tracy profiler (https://github.com/wolfpld/tracy)
>> with dmd.
>>
>> Such that instead of the profiler in druntime, tracy can be used for instrumented profiling.
>>
>>
>> The current progress is here: https://github.com/dlang/dmd/compare/master...UplinkCoder:tracy?expand=1
>>  there are still some teething problems, but I am confident that it will be useful soon.
>>
>> Cheers,
>>
>> Stefan
>
>
> Wouldn't it be more useful for everyone, and for you too, to complete a project instead of starting dozens at the same time? I have completely lost hope of being able to see benefits from the newCTFE
Of course tracy integration also helps to profile newCTFE ;)

There's one more thing, I am almost done with tracy.
It was like 2 days of work.

In the timescale that newCTFE took, and still takes that's nothing!

On Tuesday, 4 August 2020 at 12:47:54 UTC, Steven Schveighoffer wrote:
> On 8/4/20 4:44 AM, Stefan Koch wrote:
>> newCTFE in the end, has a lot less benefit than I first assumed.
>
> If CTFE becomes way less expensive (in CPU usage and memory usage), then the template problem becomes easier as well, as we can do more CTFE to replace templates.

CTFE takes 500 ms for my project. It takes a total of about 10 seconds for the frontend to do everything, without -inline. The more significant problem is definitely templates, their expansion and how everything is processed back into a AST. Such as the case if your run CTFE, even "newCTFE" the result is still going to have to be expanded back into an AST, which is the core problem.

>> If I had had a good integrated profiler back then, and some of the code which I have access to now, I would probably never have started newCTFE, and would have tried to fix the template system itself.
>
> I still think newCTFE has worthwhile benefit, even if it alone cannot fix all the problems.
>
> I think newCTFE and type functions (along with Manu's ... DIP) would be a good combination to attack the problem.
>
> -Steve

It doesn't help, it just introduces a whole lot of more complexity into the compiler as well. Effectively it creates a second compiler within the compiler. It is much more complicated than the current solution, and I don't imagine the speed up is going to be that much as, for my case it will only be able to reduce the build time by a maximum of 500 ms.

On Tuesday, 4 August 2020 at 16:01:54 UTC, Avrina wrote:
> On Tuesday, 4 August 2020 at 12:47:54 UTC, Steven Schveighoffer wrote:
>> On 8/4/20 4:44 AM, Stefan Koch wrote:
>>> newCTFE in the end, has a lot less benefit than I first assumed.
>>
>> If CTFE becomes way less expensive (in CPU usage and memory usage), then the template problem becomes easier as well, as we can do more CTFE to replace templates.
>
> CTFE takes 500 ms for my project. It takes a total of about 10 seconds for the frontend to do everything, without -inline. The more significant problem is definitely templates, their expansion and how everything is processed back into a AST. Such as the case if your run CTFE, even "newCTFE" the result is still going to have to be expanded back into an AST, which is the core problem.
>
It's not quite that AST insertion is slow. It's the fact that you
have to do semantic processing piece by piece, which is expensive.
If you have completely semantically processed nodes, linking them
into the tree is quite painless.
>
> It doesn't help, it just introduces a whole lot of more complexity into the compiler as well. Effectively it creates a second compiler within the compiler. It is much more complicated than the current solution, and I don't imagine the speed up is going to be that much as, for my case it will only be able to reduce the build time by a maximum of 500 ms.

What exactly do you mean.
type functions are a relatively simple extension of ctfe.

"..." is a little more involved but also not massively complicated.
both should stay under 1000 lines when implemented
*fingers crossed*