On Wednesday, 17 April 2019 at 16:27:02 UTC, Adam D. Ruppe wrote:
> D programs are a vital part of my home computer infrastructure. I run some 60 D processes at almost any time.... and have recently been running out of memory.

I usually run program under valgrind in this case. Though it will not help you to debug GC problems, but will cut off memory leaked malloc-s.

On Wednesday, 17 April 2019 at 16:27:02 UTC, Adam D. Ruppe wrote:
> D programs are a vital part of my home computer infrastructure. I run some 60 D processes at almost any time.... and have recently been running out of memory.
>
> Each individual process eats ~30-100 MB, but that times 60 = trouble. They start off small, like 5 MB, and grow over weeks or months, so it isn't something I can easily isolate in a debugger after recompiling.
>
> I'm pretty sure this is the result of wasteful code somewhere in my underlying libraries, but nothing is obviously jumping out at me in the code. So I want to look at some of my existing processes and see just what is responsible for this.
>
> I tried attaching to one and `call gc_stats()` in gdb... and it segfaulted. Whoops.
>
>
>
>
> I am willing to recompile and run again, though I need to actually use the programs, so if instrumenting makes them unusable it won't really help. Is there a magic --DRT- argument perhaps? Or some trick with gdb attaching to a running process I don't know?
>
> What I'm hoping to do is get an idea of which line of code allocates the most that isn't subsequently freed.


Curious, what are these programs?

You might have hook in to the GC and just write out stats, I believe there is a stats collector somewhere though.

I did this by replacing new and monitored and calculated the allocations. This didn't help for any GC issues but at least made sure all my allocations were correct.

What you could do is something similar to this and just output stuff to a text file(that is written every so often).

If they programs are not too large you could used named allocations that could then be graphed individually(or use the file locations, __FILE__, etc).

Search and replace all new's and allocs with your custom ones and override the GC's.

Should give you a good idea of what's going on.

On Friday, 19 April 2019 at 02:58:34 UTC, Alex wrote:
> Curious, what are these programs?

A terminal emulator gui (like xterm), a detachable terminal emulator (like gnu screen), a slack client, an irc client, and a bunch of http servers including d doc search, a work program, and a personal utility.

All of them would show growing memory time, some worse than others. You can see a lot of difference in them - gui programs, terminal programs, network server programs. But, I did write it all myself, so it could be a mistake I keep on making.

So far, I basically tracked down the terminal emulator things to being inefficient scrollback storage. I made the structs smaller and limited the amount saved more than before and cut this by half. The ddoc search was keeping the index in memory, that's fixed, but it still shows growing usage over time. Of course, restarting that is trivial if need be, but still, I wanna make sure I am doing it right too - especially if it is one of my underlying libraries to blame.

> You might have hook in to the GC and just write out stats, I believe there is a stats collector somewhere though.

Yes, indeed. I am starting to make serious progress now - mostly the fault is me storing excessive histories inefficiently. Should have been obvious in retrospect, but I didn't realize just how much overhead there was in my designs!

On Friday, 19 April 2019 at 03:27:04 UTC, Adam D. Ruppe wrote:
> On Friday, 19 April 2019 at 02:58:34 UTC, Alex wrote:
>> Curious, what are these programs?
>
> A terminal emulator gui (like xterm), a detachable terminal emulator (like gnu screen), a slack client, an irc client, and a bunch of http servers including d doc search, a work program, and a personal utility.
>

Ok, nothing useful ;) I was thinking you might be doing stuff like running a security system that did computer vision, or some type of advanced house monitoring and control(voice activated doors or something) ;)

Could you not, as a quick fix, just reboot and automate restarting those?

Maybe design an auto-restart which saves the state, shuts itself down and then relaunches itself and loads the data?

This could be done as a fail safe when memory consumption get's too high.

> All of them would show growing memory time, some worse than others. You can see a lot of difference in them - gui programs, terminal programs, network server programs. But, I did write it all myself, so it could be a mistake I keep on making.
>
> So far, I basically tracked down the terminal emulator things to being inefficient scrollback storage. I made the structs smaller and limited the amount saved more than before and cut this by half. The ddoc search was keeping the index in memory, that's fixed, but it still shows growing usage over time. Of course, restarting that is trivial if need be, but still, I wanna make sure I am doing it right too - especially if it is one of my underlying libraries to blame.

Gonna have to be one of those things you track down because it could be something as simple as the GC or something more serious.


>> You might have hook in to the GC and just write out stats, I believe there is a stats collector somewhere though.
>
> Yes, indeed. I am starting to make serious progress now - mostly the fault is me storing excessive histories inefficiently. Should have been obvious in retrospect, but I didn't realize just how much overhead there was in my designs!


D should have a very good memory statistics library built(I guess it has something with the switches)... since it should have no issues tracking memory usage.

Every memory allocation must have a corresponding deallocation for stable programs. All allocations and deallocations have specific file locations or occur in the GC.

I don't see why it would be difficult to monitor this stuff. As I mentioned, I generally never use new precisely so I can track this stuff myself and have a nice printout of memory usage when I need it and even verify the net memory allocation 0 zero on program exit. I haven't messed with the GC but I imagine it too shouldn't be hard to add the info too.

On Friday, 19 April 2019 at 17:30:50 UTC, Alex wrote:
> I was thinking you might be doing stuff like running a security system that did computer vision, or some type of advanced house monitoring and control(voice activated doors or something) ;)

LOL, now *that* would be totally useless!

Of course, I can restart stuff, it is just a hassle, and besides, I also wanna make sure my libs aren't too badly written.

> D should have a very good memory statistics library built(I guess it has something with the switches)... since it should have no issues tracking memory usage.

Indeed, I am sorta starting to make a hacky add-on module that will provide some of that info. But I need to hook deallocations too and haven't gotten that yet.

It'll be cool to get a report out of the program at any time that tells me which lines have how much outstanding allocations.

On Thursday, 18 April 2019 at 12:00:10 UTC, ikod wrote:
> On Wednesday, 17 April 2019 at 16:27:02 UTC, Adam D. Ruppe wrote:
>> D programs are a vital part of my home computer infrastructure. I run some 60 D processes at almost any time.... and have recently been running out of memory.
>
> I usually run program under valgrind in this case. Though it will not help you to debug GC problems, but will cut off memory leaked malloc-s.

Even valgrind tool=massif ?

On Sunday, 21 April 2019 at 21:04:52 UTC, Patrick Schluter wrote:
> On Thursday, 18 April 2019 at 12:00:10 UTC, ikod wrote:
>> On Wednesday, 17 April 2019 at 16:27:02 UTC, Adam D. Ruppe wrote:
>>> D programs are a vital part of my home computer infrastructure. I run some 60 D processes at almost any time.... and have recently been running out of memory.
>>
>> I usually run program under valgrind in this case. Though it will not help you to debug GC problems, but will cut off memory leaked malloc-s.
>
> Even valgrind tool=massif ?

I rarely use massif. It is heap profiler, and my primary target usually are memory leaks.

Does your question mean that massif can help to debug GC?