Thread overview
Memory leak in rare cases using foreach and parallel
Oct 17
Seb
Dec 07
Kagamin
October 14
Hello,

In a number-crunching app that tries many different configuration parameters, I observed in some rare cases a memory leak when doing:


    class Cfg { /* ...configuration parameters...*/ }

    auto cfg_range; // range that spits out the many configurations (instances of Cfg)

    foreach (cfg; parallel( cfg_range ))
    {
       // Do one computation for the set of parameters `cfg`
    }

using LDC 1.10.0 (based on dmd 2.080.1) on a linux machine.

This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.

The workaround I found looked like this:

    foreach (i; parallel( cfg_range.length.iota ))
    {
      auto cfg = cfg_range[ i ];
      // Do one computation for the set of parameters `cfg`
    }

Hopefully this helps anyone encountering a similar issue and/or LDC developers.

Best regards,
Guillaume
October 15
Hi Guillaume,

On 14 Oct 2019, at 8:52, Guillaume Lathoud via digitalmars-d-ldc wrote:
> This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.

Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?

And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?

Best,
David

October 16
On Tuesday, 15 October 2019 at 20:36:05 UTC, David Nadlinger wrote:

> Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
>
> And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?

In those particular cases, I observed in htop a growing memory usage for the whole executable. I recall that all threads were concerned. As soon as I switched to the iota implementation, the problem disappeared. I cannot say much more than this.

Best regards,
Guillaume
October 17
On Wednesday, 16 October 2019 at 05:39:22 UTC, Guillaume Lathoud wrote:
> On Tuesday, 15 October 2019 at 20:36:05 UTC, David Nadlinger wrote:
>
>> Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
>>
>> And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?
>
> In those particular cases, I observed in htop a growing memory usage for the whole executable. I recall that all threads were concerned. As soon as I switched to the iota implementation, the problem disappeared. I cannot say much more than this.
>
> Best regards,
> Guillaume

Do you have by any chance a small reproducible example?
October 17
On Thursday, 17 October 2019 at 06:43:38 UTC, Seb wrote:

> Do you have by any chance a small reproducible example?

No. The use case was difficult to reduce, to reproduce the bug in a usable manner
If I manage to get one I'll definitely post it!

Best,
Guillaume

December 04
On Thursday, 17 October 2019 at 14:44:51 UTC, Guillaume Lathoud wrote:
> On Thursday, 17 October 2019 at 06:43:38 UTC, Seb wrote:
>
>> Do you have by any chance a small reproducible example?
>
> No. The use case was difficult to reduce, to reproduce the bug in a usable manner
> If I manage to get one I'll definitely post it!

I still could not reproduce this in a small example.

But here is an additional hint that helped me as well, and could help someone having similar memory usage issues, in the case of running long tasks using `parallel()`: run

core.memory.GC.collect();
core.memory.GC.minimize();

at the end of each task. See the example below.

Best regards,
Guillaume
.

#!/usr/bin/env rdmd

import core.memory;
import core.thread;
import std.parallelism;
import std.range;
import std.stdio;

double[] do_one()
{
  auto arr = new double[ 50_000_000 ];

  Thread.sleep( 500.msecs );

  return arr;
}

void main()
{
  foreach (i; parallel( 100.iota ))
    {
      do_one; // Some long task (several seconds or minutes)

      // The next two lines divided the peak memory usage by a
      // factor of about 2 or 3
      core.memory.GC.collect();
      core.memory.GC.minimize();
    }

  writeln( "Done. Sleeping so that you can get the peak memory usage using `grep VmHWM /proc/<id>/status or similar..." );
  Thread.sleep( 10.seconds );
}





December 07
On Monday, 14 October 2019 at 06:52:32 UTC, Guillaume Lathoud wrote:
> This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.

By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?
December 09
On Saturday, 7 December 2019 at 07:19:44 UTC, Kagamin wrote:

> By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?

No, from what I remember the problem remained after recompiling everything. In those cases I ended up doing the change mentioned in the OP.
December 20
On Monday, 9 December 2019 at 13:18:29 UTC, Guillaume Lathoud wrote:
> On Saturday, 7 December 2019 at 07:19:44 UTC, Kagamin wrote:
>
>> By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?
>
> No, from what I remember the problem remained after recompiling everything. In those cases I ended up doing the change mentioned in the OP.