I'm fighting some out of memory problems using DMD and some super-template heavy code.

I have ideas on how to improve the situation, but it involves redesigning a large portion of the design. I want to do it incrementally, but I need to see things improving.

Is there a straightforward way to figure out how much memory the compiler uses during compilation? I though maybe /usr/bin/time, but I feel like I don't trust the output to be the true max resident size to be what I'm looking for (or that it's 100% accurate). Is there a sure-fire way to have DMD print it's footprint?

-Steve

On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:
> Is there a straightforward way to figure out how much memory the compiler uses during compilation?

So this isn't a great solution but what I have done in the past is just have a little script running

ps aux | grep dmd
sleep 1 # or whatever time you want

in a loop running in the background to gather some data as it goes. If combined with dmd -v you can interpolate the -v and ps output to get an idea of what the memory use is at a certain point of what the compiler is working on.

again i know it isn't great but it is easy to play with and better than nothing

On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:
> I'm fighting some out of memory problems using DMD and some super-template heavy code.
>
> I have ideas on how to improve the situation, but it involves redesigning a large portion of the design. I want to do it incrementally, but I need to see things improving.
>
> Is there a straightforward way to figure out how much memory the compiler uses during compilation? I though maybe /usr/bin/time, but I feel like I don't trust the output to be the true max resident size to be what I'm looking for (or that it's 100% accurate). Is there a sure-fire way to have DMD print it's footprint?
>
> -Steve

Hello,
You can look at
http://man7.org/linux/man-pages/man2/getrusage.2.html
But, probably /usr/bin/time already use it.

On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:
> I'm fighting some out of memory problems using DMD and some super-template heavy code.

Try to enable the GC using the `-lowmem` flag [1].

[1] https://dlang.org/changelog/2.086.0.html#lowmem

—
/Jacob Carlborg

On 11/17/19 7:33 PM, Adam D. Ruppe wrote:
> On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:
>> Is there a straightforward way to figure out how much memory the compiler uses during compilation?
> 
> So this isn't a great solution but what I have done in the past is just have a little script running
> 
> ps aux | grep dmd
> sleep 1 # or whatever time you want
> 
> in a loop running in the background to gather some data as it goes. If combined with dmd -v you can interpolate the -v and ps output to get an idea of what the memory use is at a certain point of what the compiler is working on.
> 
> again i know it isn't great but it is easy to play with and better than nothing

Yeah, I started trying out valgrind, which seems to give a decent report. It slows down the compiler a bit, but that's ok.

In any case, I realized that I need to write some more targeted tests, because the template I want to replace is used all over the place, so I would need to change them all in order to reduce memory usage.

-Steve

On 11/18/19 3:16 AM, Jacob Carlborg wrote:
> On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:
>> I'm fighting some out of memory problems using DMD and some super-template heavy code.
> 
> Try to enable the GC using the `-lowmem` flag [1].
> 
> [1] https://dlang.org/changelog/2.086.0.html#lowmem
> 

I don't think this will work for me, as this isn't CTFE, and I'm assuming the instantiated templates won't be collected because they stay referenced for the whole program.

-Steve

On 11/17/19 7:48 PM, ikod wrote:
> On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:
>> I'm fighting some out of memory problems using DMD and some super-template heavy code.
>>
>> I have ideas on how to improve the situation, but it involves redesigning a large portion of the design. I want to do it incrementally, but I need to see things improving.
>>
>> Is there a straightforward way to figure out how much memory the compiler uses during compilation? I though maybe /usr/bin/time, but I feel like I don't trust the output to be the true max resident size to be what I'm looking for (or that it's 100% accurate). Is there a sure-fire way to have DMD print it's footprint?
> 
> Hello,
> You can look at
> http://man7.org/linux/man-pages/man2/getrusage.2.html
> But, probably /usr/bin/time already use it.
> 

Yeah, that looks like what /usr/bin/time is doing.

-Steve

On Monday, 18 November 2019 at 15:31:02 UTC, Steven Schveighoffer wrote:
> I don't think this will work for me, as this isn't CTFE, and I'm assuming the instantiated templates won't be collected because they stay referenced for the whole program.

You could also combine that flag with the GC profiling flags.

—
/Jacob Carlborg

On 11/18/19 10:36 AM, Jacob Carlborg wrote:
> On Monday, 18 November 2019 at 15:31:02 UTC, Steven Schveighoffer wrote:
>> I don't think this will work for me, as this isn't CTFE, and I'm assuming the instantiated templates won't be collected because they stay referenced for the whole program.
> 
> You could also combine that flag with the GC profiling flags.
> 

That's a good point. I tried the profiling flags, but realized really quickly that it doesn't use the GC. But if you told it to use the GC, then that info becomes relevant.

I also noticed that dub suppresses the output of the profiler somehow (I had to run it with -v, and then run the compile line by hand to get it to do the profiling).

-Steve

On 11/18/19 11:15 AM, Steven Schveighoffer wrote:
> On 11/18/19 10:36 AM, Jacob Carlborg wrote:
>> On Monday, 18 November 2019 at 15:31:02 UTC, Steven Schveighoffer wrote:
>>> I don't think this will work for me, as this isn't CTFE, and I'm assuming the instantiated templates won't be collected because they stay referenced for the whole program.
>>
>> You could also combine that flag with the GC profiling flags.
>>
> 
> That's a good point. I tried the profiling flags, but realized really quickly that it doesn't use the GC. But if you told it to use the GC, then that info becomes relevant.
> 
> I also noticed that dub suppresses the output of the profiler somehow (I had to run it with -v, and then run the compile line by hand to get it to do the profiling).
> 

Yeah, as I expected it still gets killed by the OOM killer.

But still, good to know that switch exists.

-Steve