Thread overview | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
|
October 14, 2019 Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Hello, In a number-crunching app that tries many different configuration parameters, I observed in some rare cases a memory leak when doing: class Cfg { /* ...configuration parameters...*/ } auto cfg_range; // range that spits out the many configurations (instances of Cfg) foreach (cfg; parallel( cfg_range )) { // Do one computation for the set of parameters `cfg` } using LDC 1.10.0 (based on dmd 2.080.1) on a linux machine. This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened. The workaround I found looked like this: foreach (i; parallel( cfg_range.length.iota )) { auto cfg = cfg_range[ i ]; // Do one computation for the set of parameters `cfg` } Hopefully this helps anyone encountering a similar issue and/or LDC developers. Best regards, Guillaume |
October 15, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Guillaume Lathoud | Hi Guillaume,
On 14 Oct 2019, at 8:52, Guillaume Lathoud via digitalmars-d-ldc wrote:
> This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.
Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?
Best,
David
|
October 16, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to David Nadlinger | On Tuesday, 15 October 2019 at 20:36:05 UTC, David Nadlinger wrote:
> Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
>
> And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?
In those particular cases, I observed in htop a growing memory usage for the whole executable. I recall that all threads were concerned. As soon as I switched to the iota implementation, the problem disappeared. I cannot say much more than this.
Best regards,
Guillaume
|
October 17, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Guillaume Lathoud | On Wednesday, 16 October 2019 at 05:39:22 UTC, Guillaume Lathoud wrote:
> On Tuesday, 15 October 2019 at 20:36:05 UTC, David Nadlinger wrote:
>
>> Just for future reference, do you know when the leak happened – once per execution? Once per thread? On every iteration?
>>
>> And to clarify, by leak do you mean actual leaked memory, or excessive consumption of GC memory?
>
> In those particular cases, I observed in htop a growing memory usage for the whole executable. I recall that all threads were concerned. As soon as I switched to the iota implementation, the problem disappeared. I cannot say much more than this.
>
> Best regards,
> Guillaume
Do you have by any chance a small reproducible example?
|
October 17, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Seb | On Thursday, 17 October 2019 at 06:43:38 UTC, Seb wrote:
> Do you have by any chance a small reproducible example?
No. The use case was difficult to reduce, to reproduce the bug in a usable manner
If I manage to get one I'll definitely post it!
Best,
Guillaume
|
December 04, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Guillaume Lathoud | On Thursday, 17 October 2019 at 14:44:51 UTC, Guillaume Lathoud wrote:
> On Thursday, 17 October 2019 at 06:43:38 UTC, Seb wrote:
>
>> Do you have by any chance a small reproducible example?
>
> No. The use case was difficult to reduce, to reproduce the bug in a usable manner
> If I manage to get one I'll definitely post it!
I still could not reproduce this in a small example.
But here is an additional hint that helped me as well, and could help someone having similar memory usage issues, in the case of running long tasks using `parallel()`: run
core.memory.GC.collect();
core.memory.GC.minimize();
at the end of each task. See the example below.
Best regards,
Guillaume
.
#!/usr/bin/env rdmd
import core.memory;
import core.thread;
import std.parallelism;
import std.range;
import std.stdio;
double[] do_one()
{
auto arr = new double[ 50_000_000 ];
Thread.sleep( 500.msecs );
return arr;
}
void main()
{
foreach (i; parallel( 100.iota ))
{
do_one; // Some long task (several seconds or minutes)
// The next two lines divided the peak memory usage by a
// factor of about 2 or 3
core.memory.GC.collect();
core.memory.GC.minimize();
}
writeln( "Done. Sleeping so that you can get the peak memory usage using `grep VmHWM /proc/<id>/status or similar..." );
Thread.sleep( 10.seconds );
}
|
December 07, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Guillaume Lathoud | On Monday, 14 October 2019 at 06:52:32 UTC, Guillaume Lathoud wrote:
> This was difficult to reduce to a simple use case. The memory leak happened in rare cases, but in those cases, it always happened.
By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?
|
December 09, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Kagamin | On Saturday, 7 December 2019 at 07:19:44 UTC, Kagamin wrote:
> By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?
No, from what I remember the problem remained after recompiling everything. In those cases I ended up doing the change mentioned in the OP.
|
December 20, 2019 Re: Memory leak in rare cases using foreach and parallel | ||||
---|---|---|---|---|
| ||||
Posted in reply to Guillaume Lathoud | On Monday, 9 December 2019 at 13:18:29 UTC, Guillaume Lathoud wrote:
> On Saturday, 7 December 2019 at 07:19:44 UTC, Kagamin wrote:
>
>> By rare cases you mean that if a particular compiled executable always leaked? If it was recompiled (or relinked) from exactly the same source, the problem could disappear?
>
> No, from what I remember the problem remained after recompiling everything. In those cases I ended up doing the change mentioned in the OP.
|
Copyright © 1999-2021 by the D Language Foundation