On Friday, 29 June 2018 at 17:40:07 UTC, Anton Fediushin wrote:
> So, long story short:
> - Usage of Mallocator instead of theAllocator made it a little bit better
> - VibeManualMemoryManagement had no (or little) effect
> - Manually calling GC.collect had no (or little) effect

You could try to call GC.minimize in pair with GC.collect:

```
GC.collect();
GC.minimize();
```

to return all freed memory back to the OS.

Not sure that the leakage of this type is possible because if you're running your program on 64bit Linux the probability of it is very low. AFAIK the GC is launched every (almost) time you allocate the memory, and if it finds "dead" pointers, it definitely must clean them out.

Vibe.d may also leak. Have you tried to run the same code without Vibe.d, say, using https://github.com/ikod/dlang-requests as an HTTP client?

Also, have you tried to change vibe.d's event loop engine, like libevent or libasync?

On Saturday, 30 June 2018 at 22:06:50 UTC, Jacob Shtokolov wrote:
> On Friday, 29 June 2018 at 17:40:07 UTC, Anton Fediushin wrote:
>> So, long story short:
>> - Usage of Mallocator instead of theAllocator made it a little bit better
>> - VibeManualMemoryManagement had no (or little) effect
>> - Manually calling GC.collect had no (or little) effect
>
> You could try to call GC.minimize in pair with GC.collect:
>
> ```
> GC.collect();
> GC.minimize();
> ```
>
> to return all freed memory back to the OS.


With vibe.d this had no effect too.

>
> Not sure that the leakage of this type is possible because if you're running your program on 64bit Linux the probability of it is very low. AFAIK the GC is launched every (almost) time you allocate the memory, and if it finds "dead" pointers, it definitely must clean them out.
>
> Vibe.d may also leak. Have you tried to run the same code without Vibe.d, say, using https://github.com/ikod/dlang-requests as an HTTP client?

Now I tried it and indeed, it's vibe.d's fault. I'm not quite sure what causes it and if this problem is known, I'll look into that later and open an issue if it doesn't exist already.

On Sunday, 1 July 2018 at 05:20:17 UTC, Anton Fediushin wrote:
> Now I tried it and indeed, it's vibe.d's fault. I'm not quite sure what causes it and if this problem is known, I'll look into that later and open an issue if it doesn't exist already.

Yes, please do this when you have time. That would be really helpful for further vibe.d improvement.

I remember a pretty old (and closed) bug of HTTP client here:
https://github.com/vibe-d/vibe.d/issues/1321

So it might be somehow related to this one. Probably something wrong with HTTP client or TLS/SSL related logic. You example code is very good and I was able to reproduce the same issue with the latest stable compiler, so I hope the guys will find the problem.

Thanks!

On Sunday, 1 July 2018 at 12:32:25 UTC, Jacob Shtokolov wrote:
> On Sunday, 1 July 2018 at 05:20:17 UTC, Anton Fediushin wrote:
>> Now I tried it and indeed, it's vibe.d's fault. I'm not quite sure what causes it and if this problem is known, I'll look into that later and open an issue if it doesn't exist already.
>
> Yes, please do this when you have time. That would be really helpful for further vibe.d improvement.
>
> I remember a pretty old (and closed) bug of HTTP client here:
> https://github.com/vibe-d/vibe.d/issues/1321
>
> So it might be somehow related to this one. Probably something wrong with HTTP client or TLS/SSL related logic. You example code is very good and I was able to reproduce the same issue with the latest stable compiler, so I hope the guys will find the problem.
>
> Thanks!

I reduced the test case to _one_ line:
```
1.seconds.setTimer(() => "http://google.com".requestHTTP((scope req) {}, (scope res) {res.disconnect;}), true);
```

What happens is `res.disconnect` doesn't free all of the internal buffers, causing leakage. One way to avoid that is to call `res.dropBody`, but it isn't always wanted (just like in my example).

I submitted an issue: https://github.com/vibe-d/vibe.d/issues/2179

On Sunday, 1 July 2018 at 13:44:23 UTC, Anton Fediushin wrote:

> I reduced the test case to _one_ line:
> ```
> 1.seconds.setTimer(() => "http://google.com".requestHTTP((scope req) {}, (scope res) {res.disconnect;}), true);
> ```
>
> What happens is `res.disconnect` doesn't free all of the internal buffers, causing leakage. One way to avoid that is to call `res.dropBody`, but it isn't always wanted (just like in my example).
 The problem is known and mentioned in the documentation:
http://vibed.org/api/vibe.http.client/requestHTTP

>Note that it is highly recommended to use one of the overloads that take a responder callback, as they can avoid some memory allocations and are safe against accidentally leaving stale response objects (objects whose response body wasn't fully read). For the returning overloads of the function it is recommended to put a scope(exit) right after the call in which HTTPClientResponse.dropBody is called to avoid this.

As I understand the situation, request object will reside in memory until you fully read or do something with response body.

On Sunday, 1 July 2018 at 20:15:02 UTC, crimaniak wrote:
> On Sunday, 1 July 2018 at 13:44:23 UTC, Anton Fediushin wrote:
>
>> I reduced the test case to _one_ line:
>> ```
>> 1.seconds.setTimer(() => "http://google.com".requestHTTP((scope req) {}, (scope res) {res.disconnect;}), true);
>> ```
>>
>> What happens is `res.disconnect` doesn't free all of the internal buffers, causing leakage. One way to avoid that is to call `res.dropBody`, but it isn't always wanted (just like in my example).
>  The problem is known and mentioned in the documentation:
> http://vibed.org/api/vibe.http.client/requestHTTP
>
>>Note that it is highly recommended to use one of the overloads that take a responder callback, as they can avoid some memory allocations and are safe against accidentally leaving stale response objects (objects whose response body wasn't fully read). For the returning overloads of the function it is recommended to put a scope(exit) right after the call in which HTTPClientResponse.dropBody is called to avoid this.
>
> As I understand the situation, request object will reside in memory until you fully read or do something with response body.

It says so "for the returning overloads". Callback-based ones should be "safe against accidentally leaving stale response objects". Actually, in this example I don't 'accidentally' leave objects, I do that on purpose and call `res.disconnect` to forcefully close everything. Yet it still doesn't free memory.

There's nothing much to do with the response body - it can be either read and destroyed or just destroyed, and `res.disconect` should do this.