Hello,

In a number-crunching app that tries many different configuration parameters, I observed in some rare cases a memory leak when doing:


    class Cfg { /* ...configuration parameters...*/ }

    auto cfg_range; // range that spits out the many configurations (instances of Cfg)

    foreach (cfg; parallel( cfg_range ))
    {
       // Do one computation for the set of parameters `cfg`
    }

using LDC 1.10.0 (based on dmd 2.080.1) on a linux machine.

This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.

The workaround I found looked like this:

    foreach (i; parallel( cfg_range.length.iota ))
    {
      auto cfg = cfg_range[ i ];
      // Do one computation for the set of parameters `cfg`
    }

Hopefully this helps anyone encountering a similar issue and/or LDC developers.

Best regards,
Guillaume

Hi Guillaume,

On 14 Oct 2019, at 8:52, Guillaume Lathoud via digitalmars-d-ldc wrote:
> This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.

Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?

And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?

Best,
David

On Tuesday, 15 October 2019 at 20:36:05 UTC, David Nadlinger wrote:

> Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
>
> And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?

In those particular cases, I observed in htop a growing memory usage for the whole executable. I recall that all threads were concerned. As soon as I switched to the iota implementation, the problem disappeared. I cannot say much more than this.

Best regards,
Guillaume

On Wednesday, 16 October 2019 at 05:39:22 UTC, Guillaume Lathoud wrote:
> On Tuesday, 15 October 2019 at 20:36:05 UTC, David Nadlinger wrote:
>
>> Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
>>
>> And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?
>
> In those particular cases, I observed in htop a growing memory usage for the whole executable. I recall that all threads were concerned. As soon as I switched to the iota implementation, the problem disappeared. I cannot say much more than this.
>
> Best regards,
> Guillaume

Do you have by any chance a small reproducible example?

On Thursday, 17 October 2019 at 06:43:38 UTC, Seb wrote:

> Do you have by any chance a small reproducible example?

No. The use case was difficult to reduce, to reproduce the bug in a usable manner
If I manage to get one I'll definitely post it!

Best,
Guillaume

On Thursday, 17 October 2019 at 14:44:51 UTC, Guillaume Lathoud wrote:
> On Thursday, 17 October 2019 at 06:43:38 UTC, Seb wrote:
>
>> Do you have by any chance a small reproducible example?
>
> No. The use case was difficult to reduce, to reproduce the bug in a usable manner
> If I manage to get one I'll definitely post it!

I still could not reproduce this in a small example.

But here is an additional hint that helped me as well, and could help someone having similar memory usage issues, in the case of running long tasks using `parallel()`: run

core.memory.GC.collect();
core.memory.GC.minimize();

at the end of each task. See the example below.

Best regards,
Guillaume
.

#!/usr/bin/env rdmd

import core.memory;
import core.thread;
import std.parallelism;
import std.range;
import std.stdio;

double[] do_one()
{
  auto arr = new double[ 50_000_000 ];

  Thread.sleep( 500.msecs );

  return arr;
}

void main()
{
  foreach (i; parallel( 100.iota ))
    {
      do_one; // Some long task (several seconds or minutes)

      // The next two lines divided the peak memory usage by a
      // factor of about 2 or 3
      core.memory.GC.collect();
      core.memory.GC.minimize();
    }

  writeln( "Done. Sleeping so that you can get the peak memory usage using `grep VmHWM /proc/<id>/status or similar..." );
  Thread.sleep( 10.seconds );
}

On Monday, 14 October 2019 at 06:52:32 UTC, Guillaume Lathoud wrote:
> This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.

By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?

On Saturday, 7 December 2019 at 07:19:44 UTC, Kagamin wrote:

> By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?

No, from what I remember the problem remained after recompiling everything. In those cases I ended up doing the change mentioned in the OP.

On Monday, 9 December 2019 at 13:18:29 UTC, Guillaume Lathoud wrote:
> On Saturday, 7 December 2019 at 07:19:44 UTC, Kagamin wrote:
>
>> By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?
>
> No, from what I remember the problem remained after recompiling everything. In those cases I ended up doing the change mentioned in the OP.