Have you try use VibeManualMemoryManagement

https://github.com/TechEmpower/FrameworkBenchmarks/blob/3b24d0a21463edc536b30e2cea647fd425915401/frameworks/D/vibed/dub.json#L22


On Fri, Jun 29, 2018 at 3:20 PM Anton Fediushin via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:

> On Friday, 29 June 2018 at 11:42:18 UTC, bauss wrote:
> > On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote:
> >> On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote:
> >>> On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:
> >>>> On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:
> >>>>> Almost forgot, there are two timers which call this function for two different streams.
> >>>>>
> >>>>> Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long.
> >>>>>
> >>>>> This gives us... 16KiB per one nowPlaying() call. Why
> >>>>> doesn't it free the memory?
> >>>>
> >>>> Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html
> instead of theAllocator as it defaults to GC.
> >>>
> >>> Thanks, I'll try that.
> >>> ...
> >>> I will deploy that and see if it changes anything.
> >>
> >> It did! Memory usage went down to 7MiB yet it still grows slightly. I'll monitor if it changes in a couple of hours but it is much better.
> >>
> >> Thank you a lot, Radu. It turns out that theAllocator is so tricky.
> >
> > Again you could do @nogc and see what memory is possibly allocated by the GC and perhaps that way you can see what memory the GC is holding on to.
>
> @nogc tells nothing new, just an error on every single line because neither `res.bodyReader.read` nor Mallocator's functions are marked as @nogc.
>
> Compiling with dmd's `-vgc` flag shows nothing but the last line.
>
> >
> > non-GC memory should be freed right away and those there shouldn't be a leak from that.
>
> Using Mallocator instead of theAllocator improved the situation, but it still leaks for some reason. After 2 hours it went from 7MiB to 18MiB.
>
> I will compile it with profile-gc again and look for the possible cause of that, maybe I'll try valgrind too.
>
>
>

On Friday, 29 June 2018 at 14:10:26 UTC, Daniel Kozak wrote:
> Have you try use VibeManualMemoryManagement
>
> https://github.com/TechEmpower/FrameworkBenchmarks/blob/3b24d0a21463edc536b30e2cea647fd425915401/frameworks/D/vibed/dub.json#L22


I'll try, not quite sure it'll help much.

On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:
> On 29/06/2018 11:09 PM, Anton Fediushin wrote:
>> It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory?
>
> Probably doesn't know that it should deallocate so eagerly.
> A GC.collect(); call may help.

That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage.

Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back.

On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:
> On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:
>> On 29/06/2018 11:09 PM, Anton Fediushin wrote:
>>> It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory?
>>
>> Probably doesn't know that it should deallocate so eagerly.
>> A GC.collect(); call may help.
>
> That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage.
>
> Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back.

Which language that you had write apps in that utilize GC? Java? C#? You shouldn't treat D GC the same as other languages GC.

Alexander

On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:
> On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:
>> On 29/06/2018 11:09 PM, Anton Fediushin wrote:
>>> It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory?
>>
>> Probably doesn't know that it should deallocate so eagerly.
>> A GC.collect(); call may help.
>
> That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage.
>
> Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back.

I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory.

On Friday, 29 June 2018 at 16:19:39 UTC, 12345swordy wrote:
> On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:
>> Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back.
>
> Which language that you had write apps in that utilize GC? Java? C#? You shouldn't treat D GC the same as other languages GC.
>
> Alexander

Talking about D here.

GC can be the best option for some languages and environments, but it doesn't fit D that well. Writing programs in D I always know where stack-allocated structs get deleted and such, but I have no idea on what's going on with the GC. Does it collect anything at all? Why doesn't it collect this? How do I force it to collect this?

On Friday, 29 June 2018 at 16:49:41 UTC, Bauss wrote:
> On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:
>> On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:
>>> On 29/06/2018 11:09 PM, Anton Fediushin wrote:
>>>> It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory?
>>>
>>> Probably doesn't know that it should deallocate so eagerly.
>>> A GC.collect(); call may help.
>>
>> That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage.
>>
>> Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back.
>
> I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory.

I am not quite sure what should I blame now, because even if I use malloc for memory allocation, memory goes... somewhere?

So, long story short:
- Usage of Mallocator instead of theAllocator made it a little bit better
- VibeManualMemoryManagement had no (or little) effect
- Manually calling GC.collect had no (or little) effect

It makes me think that error is somewhere else. I made a code snippet of my testing program: https://gitlab.com/snippets/1729304

There are some changes to it:
- It uses different stream with metaint of 32KB
- It calls nowPlaying() every second

Now I will take a break from this, dealing with this kind of nonsense annoys me.

On 30/06/2018 4:49 AM, Bauss wrote:
> I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory.

Let's be honest, I don't think it was meant to live in a container with 64mb of ram. I just don't think it is kicking in to collect.

On Saturday, 30 June 2018 at 05:00:35 UTC, rikki cattermole wrote:
> On 30/06/2018 4:49 AM, Bauss wrote:
>> I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory.
>
> Let's be honest, I don't think it was meant to live in a container with 64mb of ram. I just don't think it is kicking in to collect.

It doesn't, I'm experimenting with different GC configurations [1]. By default [2] `maxPoolSize` is set to 64MB, so maybe program gets killed by docker right before GC decides to collect.

[1] https://dlang.org/spec/garbage.html#gc_config
[2] https://github.com/dlang/druntime/blob/master/src/gc/config.d#L23

On 30/06/2018 7:42 PM, Anton Fediushin wrote:
> On Saturday, 30 June 2018 at 05:00:35 UTC, rikki cattermole wrote:
>> On 30/06/2018 4:49 AM, Bauss wrote:
>>> I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory.
>>
>> Let's be honest, I don't think it was meant to live in a container with 64mb of ram. I just don't think it is kicking in to collect.
> 
> It doesn't, I'm experimenting with different GC configurations [1]. By default [2] `maxPoolSize` is set to 64MB, so maybe program gets killed by docker right before GC decides to collect.
> 
> [1] https://dlang.org/spec/garbage.html#gc_config
> [2] https://github.com/dlang/druntime/blob/master/src/gc/config.d#L23

The OS ext. takes memory too.

34mb might be safer.