I made a proposal to quantatively measure and tabulate all GC allocations in Phobos before coming up with solutions to "@nogc Phobos".

After approving node from Andrei I've come up with a piece of automation to extract this data and post it on wiki.

So here is the exhustive list of everything calling into GC in Phobos (-vgc compiler flag):

http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage

Including source links, a wild guess at function's name and the compiler's warning message for potential GC call.

As far as data goes this is about as good as we can get, the next phase is labeling this stuff with potential solution(s). Again doing all by hand is tedious and hardly useful.

Instead we need to observe patterns and label it automatically until the non-trivial subset remains. So everybody, please take time and identify simple patterns and post back your ideas on solution(s).

So far I see the most frequent cases:
- `new SomeException` - switch to RC exceptions
- AA access - ??? (use user-defined AA type as parameter?)
- array concat - ???
- closure - ???



---
Dmitry Olshansky

1.) It may be helpful to reduce the noise in that every match after a new is ignored (and probaly multiple 'operator ~' alarms within the same statement).

2.) There seems to be a problem with repeated alarms:
When viewing the page source, this link shows up numerous times. See
https://github.com/D-Programming-Language//phobos/blob/d4d98124ab6cbef7097025a7cfd1161d1963c87e/std/conv.d#L688

/Gerhard

On Tuesday, 7 October 2014 at 16:23:19 UTC, grm wrote:
> 2.) There seems to be a problem with repeated alarms:
> When viewing the page source, this link shows up numerous times. See
> https://github.com/D-Programming-Language//phobos/blob/d4d98124ab6cbef7097025a7cfd1161d1963c87e/std/conv.d#L688

That's because of multiple template instantiations of the same function. These should probably be filtered for this use case.

On 10/7/2014 8:57 AM, Dmitry Olshansky wrote:
> I made a proposal to quantatively measure and tabulate all GC allocations in
> Phobos before coming up with solutions to "@nogc Phobos".
>
> After approving node from Andrei I've come up with a piece of automation to
> extract this data and post it on wiki.

Thanks, Dmitri, this is great work. I suggest at a minimum that all of those get notes added to their documentation that they gc allocate.

On Tuesday, 7 October 2014 at 19:24:34 UTC, Walter Bright wrote:
> Thanks, Dmitri, this is great work. I suggest at a minimum that all of those get notes added to their documentation that they gc allocate.

Seems like that's something that should just be automated where possible instead of trying to update the documentation of hundreds of functions.

On 2014-10-07 17:57, Dmitry Olshansky wrote:
> I made a proposal to quantatively measure and tabulate all GC
> allocations in Phobos before coming up with solutions to "@nogc Phobos".
>
> After approving node from Andrei I've come up with a piece of automation
> to extract this data and post it on wiki.
>
> So here is the exhustive list of everything calling into GC in Phobos
> (-vgc compiler flag):
>
> http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage
>
> Including source links, a wild guess at function's name and the
> compiler's warning message for potential GC call.
>
> As far as data goes this is about as good as we can get, the next phase
> is labeling this stuff with potential solution(s). Again doing all by
> hand is tedious and hardly useful.
>
> Instead we need to observe patterns and label it automatically until the
> non-trivial subset remains. So everybody, please take time and identify
> simple patterns and post back your ideas on solution(s).
>
> So far I see the most frequent cases:
> - `new SomeException` - switch to RC exceptions
> - AA access - ??? (use user-defined AA type as parameter?)
> - array concat - ???
> - closure - ???

I did some processing of the data and this is the results I got:

772 | 'new' causes GC allocation
515 | operator ~= may cause GC allocation
380 | operator ~ may cause GC allocation
113 | array literal may cause GC allocation
90  | setting 'length' may cause GC allocation
77  | indexing an associative array may cause GC allocation
34  | using closure causes GC allocation
16  | 'delete' requires GC
5   | associative array literal may cause GC allocation

Total 9

I didn't look at any source code to see what "new" is actually allocating, for example.

-- 
/Jacob Carlborg

On Tuesday, 7 October 2014 at 20:13:32 UTC, Jacob Carlborg wrote:
> I didn't look at any source code to see what "new" is actually allocating, for example.

I did some random sampling, and it's 90% exceptions, with the occasional array allocation.

I noticed that a lot of the ~ and ~= complaints are in code that only ever runs at compile time (generating strings for mixin). I wonder if there's any way we can silence these false positives.

On 10/7/14, 12:31 PM, Brad Anderson wrote:
> On Tuesday, 7 October 2014 at 19:24:34 UTC, Walter Bright wrote:
>> Thanks, Dmitri, this is great work. I suggest at a minimum that all of
>> those get notes added to their documentation that they gc allocate.
>
> Seems like that's something that should just be automated where possible
> instead of trying to update the documentation of hundreds of functions.

Could ddoc do that? -- Andrei

On 10/7/14, 8:57 AM, Dmitry Olshansky wrote:
> I made a proposal to quantatively measure and tabulate all GC
> allocations in Phobos before coming up with solutions to "@nogc Phobos".
>
> After approving node from Andrei I've come up with a piece of automation
> to extract this data and post it on wiki.
>
> So here is the exhustive list of everything calling into GC in Phobos
> (-vgc compiler flag):
>
> http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage

Awesome! I've started adding explanations to the first few entries, let's use that crowdsourcing thing to fill this! -- Andrei

On Tuesday, 7 October 2014 at 15:57:59 UTC, Dmitry Olshansky wrote:
> So here is the exhustive list of everything calling into GC in Phobos (-vgc compiler flag):
>
> http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage

A correction on TimSortImpl, it does actually generate garbage by calling uninitializedArray to allocate the buffer. The check for __ctfe is unnecessary (it may have been needed sometime ago or was added naively).

Timsort is an O(n/2) algorithm and requires a buffer, but there's no reason for it to be GC-allocated. It could simply be malloc'd and free'd before the function returns.