Thread overview
std.experimental.allocator API and nearly useless StatsCollector
Jun 20, 2021
Luis
Jun 20, 2021
Basile B.
Jun 20, 2021
Luis
Jun 20, 2021
Basile B.
Jun 20, 2021
max haughton
Jun 20, 2021
Luis
Jun 20, 2021
max haughton
June 20, 2021

Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...

What I initially did, was implementing my own allocator API plus my own mallocator, seeing how ECS buble engine does this.
Later, I take a look how std.experimental.allocator API works and I ended writing my own wrapper above make, makeArray, dispose, expandArray & shrinkArray that implements this profiler/stats, and using std.experimental.mallocator .
Finally, I catch that I can only use my wrapper on my code, and that I couldn't use on other third party libraries that allow to use allocators (like EMSI Containers), because (obviously) uses the std.experimental.allocator API.
Then, I discover std.experimental.allocator.building_blocks.stats_collector that looks that does what my littler profiler does, and more. But I see tiny detail that I don't see totally ok.

The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator!

Like this :

/usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40]

This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code!

On my wrapper, What I did was to have a string file = __FILE__, int line = __LINE__ parameters append to the make, makeArray, dispose, etc... functions and uses the file and & line to record where something was allocated/deallocated/reallocated.

So, why std.experimental.allocator API not does something similar to this ? The allocator implementations, would need to pass & ignore these two additional parameters. Except StatsCollector that can use it to show where the allocation/deallocation/reallocation it's happening

June 20, 2021

On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:

>

Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...

[...]

The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator!

Like this :

/usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40]

This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code!

To understand leaks I'd use valgrind instead. As it is based on DWARF info, reported leaking blocks are accompanied with a pretty (after demangling) back trace. So the first entry for each block will be unsurprinsingly always the same (e.g glibc malloc) but the other can be more interesting.

June 20, 2021

On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:

>

Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...

[...]

You aren't using address sanitizer?

June 20, 2021

On Sunday, 20 June 2021 at 17:35:23 UTC, max haughton wrote:

>

On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:

>

Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...

[...]

You aren't using address sanitizer?

I'm using DMD, not LDC or GDC.

June 20, 2021

On Sunday, 20 June 2021 at 17:42:49 UTC, Luis wrote:

>

On Sunday, 20 June 2021 at 17:35:23 UTC, max haughton wrote:

>

On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:

>

Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...

[...]

You aren't using address sanitizer?

I'm using DMD, not LDC or GDC.

You should set up a way to use LDC then, ASan is too good not to use.

June 20, 2021

On Sunday, 20 June 2021 at 16:53:06 UTC, Basile B. wrote:

>

On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:

>

Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...

[...]

The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator!

Like this :

/usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40]

This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code!

To understand leaks I'd use valgrind instead. As it is based on DWARF info, reported leaking blocks are accompanied with a pretty (after demangling) back trace. So the first entry for each block will be unsurprinsingly always the same (e.g glibc malloc) but the other can be more interesting.

Yeah. I know Valgrind. I used it a lot, when I did C or C++ stuff, and it's a wonderful tool. However have his problems :

  1. Makes anything to run painfull slow.
  2. At least with DMD, the mangled information & stack traces are hard to follow.

Compare this two outputs when I comment the code that calls to allocator.dispose():

My simple & dirty profiler :

$ ./bin/ddiv-test-unittest -t 1 -i "Stack"
 ✓ ddiv.container.stack SimpleStack

Summary: 1 passed, 0 failed in 0 ms
WARNING! Possible memory leaks!
----- Memory allocation information -----
Total amount of allocated memory that has not been released: 65536 bytes
        Addr: 0x55C70F4D0E60 - [int[]] 65536 bytes - source/ddiv/container/stack.d:84

Valgrind (fun thing. I need to use --leak-check=full to see the memory leak of my code, at at 0x483DFAF realloc. The rest looks that it's from Silly and/or D runtime):

$ valgrind --leak-check=full ./bin/ddiv-test-unittest -t 1 -i "Stack"
==185906== Memcheck, a memory error detector
==185906== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==185906== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==185906== Command: ./bin/ddiv-test-unittest -t 1 -i Stack
==185906==
--185906-- WARNING: Serious error when reading debug info
--185906-- When reading debug info from /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest:
--185906-- DWARF line info appears to be corrupt - the section is too small
--185906-- WARNING: Serious error when reading debug info
--185906-- When reading debug info from /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest:
--185906-- read_filename_table: .debug_line is missing?
 ✓ ddiv.container.stack SimpleStack

Summary: 1 passed, 0 failed in 89 ms
==185906== Conditional jump or move depends on uninitialised value(s)
==185906==    at 0x44B2C8: _D4core8internal2gc4impl12conservativeQw3Gcx__T4markVbi0Vbi0ZQoMFNbNlSQCqQCoQCiQCiQCgQCrQBw__T9ScanRangeVbi0ZQpZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x445469: _D4core8internal2gc4impl12conservativeQw3Gcx16markConservativeMFNbNlPvQcZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x44BFC7: _D4core8internal2gc4impl12conservativeQw3Gcx__T7markAllS_DQCeQCcQBwQBwQBuQCfQBk16markConservativeMFNbNlPvQcZvZQClMFNbbZ14__foreachbody3MFNbKSQFjQEy11gcinterface5RangeZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x44FFB3: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi7opApplyMFNbMDFNbKQBwZiZ9__lambda2MFNbKxSQEhQCsQCsQCiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x450359: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi13opApplyHelperFNbxPSQDnQDlQDfQCy__TQCvTQCsZQDd4NodeMDFNbKxSQFaQDlQDlQDbZiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x44FF81: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi7opApplyMFNbMDFNbKQBwZiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x44BF60: _D4core8internal2gc4impl12conservativeQw3Gcx__T7markAllS_DQCeQCcQBwQBwQBuQCfQBk16markConservativeMFNbNlPvQcZvZQClMFNbbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x446752: _D4core8internal2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x44B017: _D4core8internal2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCsQCqQCkQCkQCiQCtQBy18fullCollectNoStackMFNbZ2goFNbPSQEuQEsQEmQEmQEkQEv3GcxZmTQBbZQDsMFNbKQBnZm (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x443397: _D4core8internal2gc4impl12conservativeQw14ConservativeGC18fullCollectNoStackMFNbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x44332D: _D4core8internal2gc4impl12conservativeQw14ConservativeGC14collectNoStackMFNbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x4177B6: gc_term (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==
==185906==
==185906== HEAP SUMMARY:
==185906==     in use at exit: 66,568 bytes in 6 blocks
==185906==   total heap usage: 454 allocs, 448 frees, 346,252 bytes allocated
==185906==
==185906== 32 bytes in 1 blocks are possibly lost in loss record 2 of 6
==185906==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==185906==    by 0x441335: _D4core8internal2gc4impl12conservativeQw10initializeFZCQCbQBq11gcinterface2GC (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x43F608: _D4core2gc8registry16createGCInstanceFAyaZCQBpQBn11gcinterface2GC (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x43A2CA: gc_init_nothrow (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x417C93: _D4core8internal2gc4impl5protoQo7ProtoGC6qallocMFNbmkxC8TypeInfoZSQCm6memory8BlkInfo_ (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x3DD216: gc_qalloc (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x3E33FF: _d_newitemT (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x41F66B: _D3std5array__T8AppenderTAAyaZQp6__ctorMFNaNbNcNeQyZSQBzQBy__TQBvTQBpZQCd (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x41F564: _D3std5array__T8appenderTAAyaZQpFNaNbNfZSQBnQBm__T8AppenderTQBjZQo (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x41F4E8: _D3std5array__TQjTSQr9algorithm9iteration__T8splitterVAyaa6_61203d3d2062VEQCu8typecons__T4FlagVQBpa14_6b656570536570617261746f7273ZQBqi0TQDfTQDjZQDxFQDrQDuZ6ResultZQGcFNaNbNfQGaZAQEv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x41F4CD: _D3std5array__T5splitTAyaTQeZQoFNaNbNfQqQsZAQw (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x3EC601: _D3std6getopt11splitAndGetFNaNbNeAyaZSQBkQBj6Option (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==
==185906== 65,536 bytes in 1 blocks are definitely lost in loss record 6 of 6
==185906==    at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==185906==    by 0x4206AA: _D4core6memory__T11pureReallocZQoFNaNbNiPvmZQe (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x3FB799: _D3std12experimental9allocator10mallocator10Mallocator10reallocateMOFNaNbNiKAvmZb (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest)
==185906==    by 0x3A611B: _D3std12experimental9allocator__T11expandArrayTiTOSQBxQBwQBl10mallocator10MallocatorZQCaFNaNbNiKOQBvKAimZb (package.d:2143)
==185906==    by 0x3A6024: _D4ddiv4core6memory9allocator__T11expandArrayTiTOS3std12experimentalQBx10mallocator10MallocatorZQCmFNbNiKOQCfKAimAyaiZb (allocator.d:178)
==185906==    by 0x3A04A7: _D4ddiv9container5stack__T11SimpleStackTiTS3std12experimental9allocator10mallocator10MallocatorZQCs6expandMFNbNiNemZv (stack.d:84)
==185906==    by 0x3A0578: _D4ddiv9container5stack__T11SimpleStackTiTS3std12experimental9allocator10mallocator10MallocatorZQCs4pushMFNbNiNfiZv (stack.d:105)
==185906==    by 0x39FAE9: _D4ddiv9container5stack18__unittest_L142_C1FNfZv (stack.d:166)
==185906==    by 0x3BD004: _D5silly11executeTestFSQv4TestZSQBe10TestResult (silly.d:189)
==185906==    by 0x3BC89B: _D5silly24_sharedStaticCtor_L24_C1FZ9__lambda1FZ15__foreachbody15MFKSQCp4TestZi (silly.d:109)
==185906==    by 0x3C2614: _D3std11parallelism__T14doSizeZeroCaseTAS5silly4TestTDFKQqZiZQBnFKSQCnQCm__T15ParallelForeachTQCdZQwQBvZi (parallelism.d:3748)
==185906==    by 0x3C2013: _D3std11parallelism__T15ParallelForeachTAS5silly4TestZQBg7opApplyMFMDFKQBeZiZi (parallelism.d-mixin-4039:4043)
==185906==
==185906== LEAK SUMMARY:
==185906==    definitely lost: 65,536 bytes in 1 blocks
==185906==    indirectly lost: 0 bytes in 0 blocks
==185906==      possibly lost: 32 bytes in 1 blocks
==185906==    still reachable: 1,000 bytes in 4 blocks
==185906==         suppressed: 0 bytes in 0 blocks
==185906== Reachable blocks (those to which a pointer was found) are not shown.
==185906== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==185906==
==185906== Use --track-origins=yes to see where uninitialised values come from
==185906== For lists of detected and suppressed errors, rerun with: -s
==185906== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)
June 20, 2021

On Sunday, 20 June 2021 at 17:58:41 UTC, Luis wrote:

>

On Sunday, 20 June 2021 at 16:53:06 UTC, Basile B. wrote:

>

On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
Yeah. I know Valgrind. I used it a lot, when I did C or C++ stuff, and it's a wonderful tool. However have his problems :

  1. Makes anything to run painfull slow.

alas nothing can be done against this to my knowledge.

>
  1. At least with DMD, the mangled information & stack traces are hard to follow.

here you can definitively use ddemangle.

June 21, 2021
On 6/20/21 11:49 AM, Luis wrote:
> Well, this last week I was playing using malloc/free to do some @nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc...
> 
> What I initially did, was implementing my own allocator API plus my own mallocator, seeing how ECS buble engine does this.
> Later, I take a look how std.experimental.allocator API works and I ended writing my own wrapper above make, makeArray, dispose, expandArray & shrinkArray that implements this profiler/stats, and using std.experimental.mallocator .
> Finally, I catch that I can only use my wrapper on my code, and that I couldn't use on other third party libraries that allow to use allocators (like EMSI Containers), because (obviously) uses the std.experimental.allocator API.
> Then, I discover std.experimental.allocator.building_blocks.stats_collector that looks that does what my littler profiler does, and more. But I see tiny detail that I don't see totally ok.
> 
> The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator!
> 
> Like this :
> 
> ```
> /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40]
> ```
> 
> This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code!
> 
> On my wrapper, What I did was to have a `string file = __FILE__, int line = __LINE__` parameters append to the make, makeArray, dispose, etc... functions and uses the file and & line to record where something was allocated/deallocated/reallocated.
> 
> So, why std.experimental.allocator API not does something similar to this ? The allocator implementations, would need to pass & ignore these two additional parameters. Except StatsCollector that can use it to show where the allocation/deallocation/reallocation it's happening

If I understand the problem correctly, in order for statsCollector to work, it needs to be at the "top" of the allocator type tree. The design indeed does not support line numbers when statsAllcator is inserted somewhere in the middle.