On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:
> dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created.
>
> There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed).
>
> Would be in your debt for any ideas. Thanks!

You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers.

I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d

When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications.

dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d.

For dmd you can call it like this:
heaptrack dmd -lowmem test.d

It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.

On 6/19/20 12:03 PM, Tim wrote:
> On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:
>> dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created.
>>
>> There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed).
>>
>> Would be in your debt for any ideas. Thanks!
> 
> You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers.
> 
> I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d
> 
> When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications.
> 
> dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d.
> 
> For dmd you can call it like this:
> heaptrack dmd -lowmem test.d
> 
> It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.

Thanks. Does it provide type information about the objects allocated?

On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:
>
> Thanks. Does it provide type information about the objects allocated?

No, it does not save type information.
It generates a stack trace for every allocation and can combine those into a bottom-up or top-down tree.

On 6/19/2020 8:36 AM, Andrei Alexandrescu wrote:
> A quick experiment (built dmd itself with -profile=gc, ran it on a few simple files) seems to show that only array allocations are traced, but not allocations of individual objects.

Please post to bugzilla.

On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:
> Does it provide type information about the objects allocated?

All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/root/rmem.d#L222-L310. There you have the TypeInfos.

On 6/19/20 4:59 PM, Walter Bright wrote:
> On 6/19/2020 8:36 AM, Andrei Alexandrescu wrote:
>> A quick experiment (built dmd itself with -profile=gc, ran it on a few simple files) seems to show that only array allocations are traced, but not allocations of individual objects.
> 
> Please post to bugzilla.

https://issues.dlang.org/show_bug.cgi?id=20960

On 6/19/20 6:05 PM, kinke wrote:
> On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:
>> Does it provide type information about the objects allocated?
> 
> All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/root/rmem.d#L222-L310. There you have the TypeInfos.

Thank you! Just to clarify: this is advice on where to insert my own tracing, not a pointer to an existing instrumentation, right?

On Saturday, 20 June 2020 at 01:20:02 UTC, Andrei Alexandrescu wrote:
> On 6/19/20 6:05 PM, kinke wrote:
>> On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:
>>> Does it provide type information about the objects allocated?
>> 
>> All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/root/rmem.d#L222-L310. There you have the TypeInfos.
>
> Thank you! Just to clarify: this is advice on where to insert my own tracing, not a pointer to an existing instrumentation, right?

Yes, instrumentation to be inserted.

On Friday, 19 June 2020 at 16:03:55 UTC, Tim wrote:
> On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:
>> dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created.
>>
>> There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed).
>>
>> Would be in your debt for any ideas. Thanks!
>
> You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers.
>
> I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d
>
> When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications.
>
> dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d.
>
> For dmd you can call it like this:
> heaptrack dmd -lowmem test.d
>
> It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.

Reply to bump. This is one of the best "secret weapons" in the D ecosystem. Just yesterday we used it to find the reason why a service was ballooning to 11GB.