Dear Community.

Thank you for helping me so much in the last thread concerning our AI library (https://forum.dlang.org/thread/rrpmgzqqtkqgeicjdgps@forum.dlang.org). Thanks to you, we can now debug our AI library with GDB. My suspicion was confirmed: At least one of our parallel foreach loops will only be processed item by item instead of parallel.

It is the loop that contains the largest workload: An external process is started for each agent. I log the PID, so I see that the processes are started one after the other.

Our foreach loops are constructed as:

auto taskPool = new TaskPool();
foreach (n, individual; taskPool.parallel(individuals, params.workUnitSize4ParallelExecution)) {
    [...]
    pipes = pipeProcess(args, Redirect.stdout | Redirect.stdin);
    [...]
}

taskPool.finish(true);

The looped individuals are sourced by such a statement:
return this.individuals.filter!( [...] ).array;

Currently, for debugging, params.workUnitSize4ParallelExecution is set to 100 and individuals contain approx. 10 agents. Therefore, we would expect 10 external processes to be started simultaneously. Each process runs approx. 4 seconds. Accordingly, the processing should take about 4 seconds instead of 40 seconds.

Do you have any ideas what might be the reason for this? If you don't know a solution, then maybe approaches to further investigation?


Best regards
Thorsten

Okay: Sometimes it is enough to write down the problem so that someone outside the team understands it. Obviously, workUnitSize4ParallelExecution must be set to 1 for all to run parallel. I have read the documentation on this subject just today and I have overlooked it anyway.