On Wednesday, 1 July 2020 at 08:15:12 UTC, NilsLankila wrote:
> There's already a micro optim for AliasDeclaration, see https://github.com/dlang/dmd/pull/11354. It should cut the use by 25% (so the number for AliasDecl *only*), minus the problem of allocations blocks, so maybe more 10% to 20% IRL.

For 64-bit, the size of a Declaration instance is 200 bytes (2.092 frontend), for AliasDeclaration 232 bytes, and with your change, down to 224 bytes, so the correct number is more like -3.5%. Thx for doing it anyway.

Another thing to keep in mind is that all new'd allocations are padded to a multiple of 16-bytes, see https://github.com/dlang/dmd/blob/d97a908d35c8e6c22571688f12138862ef089337/src/dmd/root/rmem.d#L176 (for the GC/-lowmem too, which has a bigger overhead).

On Wednesday, 1 July 2020 at 08:26:05 UTC, NilsLankila wrote:
> On Tuesday, 30 June 2020 at 01:54:54 UTC, rikki cattermole [...]
> idPool is already used but only to guarantee identifiers uniqueness (i.e reject dups). I think that a second idpool could be used, this one to share those who has not be be unique, just like dparse's StringCache/internString()

I've ~~spent~~ wasted some time yesterday evening to work on puting an intern system directly in the two Identifier __ctor that allocate, and this is not worth. only 300Kb won. but the overhead of the new table waste more. Actually most of the identifiers are already interned.

Then, there's no more easy possible gain in the AST. Basically for Expression and Statement derived nodes there's nothing anymore.

Walter has made a good optimisation for Type (-40Mo used for dparse).

I've made 3 minors ones (-3Mo for dparse).

Andrei made one for template instance but unfortunately it hasn't saved a full block of memory. What you have to understand, if you wish to propose something is that to reduce usage a class must have it's size div 16 taking one less, eg 32 to 17 has not impact but 32 to 15 does.

Honestly I think this will be hard to do better **without** serious work and deep changes.

Anyway for now compiling dparse using ~master and on x86_64 takes 904Mo VS 950Mo previously.
That'd be nice if 3 or 4 micro optims would be found, so that it goes below 900 Megs, symbolically.

On Wednesday, 1 July 2020 at 08:38:13 UTC, kinke wrote:
> On Wednesday, 1 July 2020 at 08:15:12 UTC, NilsLankila wrote:
>> There's already a micro optim for AliasDeclaration, see https://github.com/dlang/dmd/pull/11354. It should cut the use by 25% (so the number for AliasDecl *only*), minus the problem of allocations blocks, so maybe more 10% to 20% IRL.
>
> For 64-bit, the size of a Declaration instance is 200 bytes (2.092 frontend), for AliasDeclaration 232 bytes, and with your change, down to 224 bytes, so the correct number is more like -3.5%. Thx for doing it anyway.
>
> Another thing to keep in mind is that all new'd allocations are padded to a multiple of 16-bytes, see https://github.com/dlang/dmd/blob/d97a908d35c8e6c22571688f12138862ef089337/src/dmd/root/rmem.d#L176 (for the GC/-lowmem too, which has a bigger overhead).

Yeah you're right, I've realized later that I forgot the size of the inherited members.

On Tuesday, 30 June 2020 at 18:18:32 UTC, Andrei Alexandrescu wrote:
> On 6/29/20 9:56 PM, Bruce Carneal wrote:
>> On Tuesday, 30 June 2020 at 01:18:48 UTC, Andrei Alexandrescu wrote:
>>> I got a few numbers on what types dmd allocates the most while compiling a large project.
>> 
>> [snip of interesting description of how the numbers were obtained]
>> 
>>>
>>> Looking at the top offenders:
>>>
>>> 42634177 dmd.mtype.TypeIdentifier
>>> 20452075 dmd.dtemplate.TemplateInstance
>>> 20202329 dmd.dsymbol.DsymbolTable
>>> 18783004 dmd.declaration.AliasDeclaration
>>> 18224199 dmd.dsymbol.ScopeDsymbol
>>> 18172133 dmd.mtype.Parameter
>>> 14124126 dmd.expression.IntegerExp
>>>
>>> The drop-off is fast, as expected, more than 2x from #1 to #2, and 3x from #1 to #7. That means any work that improves on TypeIdentifier is likely to greatly relieve the number of allocation calls. Here are a few thoughts on possible improvements:
>>>
>> [snip of improvement descriptions]
>> 
>> Do you know, or can you easily find out how many of these type identifiers might be eliminated by the improvements Stefan Koch and others have talked about recently?
>
> I don't know, maybe Stefan would. Anyway, I attach one more file in the format:
>
> total_bytes total_objects object_size type_name
>
> The top offenders are about the same, but now the top 2 are much closer to each other.

Can you post the project? If not, is there a similar one? I don't know of many big open source projects, especially ones that eat over 16 GB of memory.

On Thursday, 9 July 2020 at 12:29:57 UTC, Avrina wrote:
> On Tuesday, 30 June 2020 at 18:18:32 UTC, Andrei Alexandrescu wrote:
>> On 6/29/20 9:56 PM, Bruce Carneal wrote:
>>> [...]
>>
>> I don't know, maybe Stefan would. Anyway, I attach one more file in the format:
>>
>> total_bytes total_objects object_size type_name
>>
>> The top offenders are about the same, but now the top 2 are much closer to each other.
>
> Can you post the project? If not, is there a similar one? I don't know of many big open source projects, especially ones that eat over 16 GB of memory.

Check out libdparse.
Anything which uses std.regex.
or the std.range unittests.