On 6/24/21 1:41 PM, seany wrote:

> Is there any way to control the number of CPU cores used in
> parallelization ?

Yes. You have to create a task pool explicitly:

import std.parallelism;

void main() {
  enum threadCount = 2;
  auto myTaskPool = new TaskPool(threadCount);
  scope (exit) {
    myTaskPool.finish();
  }

  enum workUnitSize = 1; // Or 42 or something else. :)
  foreach (e; myTaskPool.parallel([ 1, 2, 3 ], workUnitSize)) {
    // ...
  }
}

I've touched on a few parallelism concepts at this point in a presentation:

  https://www.youtube.com/watch?v=dRORNQIB2wA&t=1332s

Ali

On 6/25/21 6:53 AM, seany wrote:

> I tried this .
>
>                  int[][] pnts ;
>          pnts.length = fld.length;
>
>          enum threadCount = 2;
>          auto prTaskPool = new TaskPool(threadCount);
>
>          scope (exit) {
>              prTaskPool.finish();
>          }
>
>          enum workUnitSize = 1;
>
>          foreach(i, fLine; prTaskPool.parallel(fld, workUnitSize)) {
>                    //....
>                  }
>
>
> This is throwing random segfaults.
> CPU has 2 cores, but usage is not going above 37%

Performance is not guaranteed depending on many factors. For example, inserting a writeln() call in the loop would make all threads compete with each other for stdout. There can be many contention points some of which depending on your program logic. (And "Amdahl's Law" applies.)

Another reason: 1 can be a horrible value for workUnitSize. Try 100, 1000, etc. and see whether it helps with performance.

> Even much deeper down in program, much further down the line...
> And the location of segfault is random.

Do you still have two parallel loops? Are both with explicit TaskPool objects? If not, I wonder whether multiple threads are using the convenient 'parallel' function, stepping over each others' toes. (I am not sure about this because perhaps it's safe to do this; never tested.)

It is possible that the segfaults are caused by your code. The code you showed in your original post (myFunction0() and others), they all work on independent data structures, right?

Ali

On Friday, 25 June 2021 at 14:10:52 UTC, Ali Çehreli wrote:
> On 6/25/21 6:53 AM, seany wrote:
>
> >          [...]
> workUnitSize)) {
> > [...]
>
> Performance is not guaranteed depending on many factors. For example, inserting a writeln() call in the loop would make all threads compete with each other for stdout. There can be many contention points some of which depending on your program logic. (And "Amdahl's Law" applies.)
>
> Another reason: 1 can be a horrible value for workUnitSize. Try 100, 1000, etc. and see whether it helps with performance.
>
> > [...]
> line...
> > [...]
>
> Do you still have two parallel loops? Are both with explicit TaskPool objects? If not, I wonder whether multiple threads are using the convenient 'parallel' function, stepping over each others' toes. (I am not sure about this because perhaps it's safe to do this; never tested.)
>
> It is possible that the segfaults are caused by your code. The code you showed in your original post (myFunction0() and others), they all work on independent data structures, right?
>
> Ali

The code without the parallel foreach works fine. No segfault.

In several instances, I do have multiple nested loops, but in every case. only the outer one in parallel foreach.

All of them are with explicit taskpool definition.

On 6/25/21 7:21 AM, seany wrote:

> The code without the parallel foreach works fine. No segfault.

That's very common.

What I meant is, is the code written in a way to work safely in a parallel foreach loop? (i.e. Is the code "independent"?) (But I assume it is because it's been the common theme in this thread; so there must be something stranger going on.)

Ali

On Friday, 25 June 2021 at 15:08:38 UTC, Ali Çehreli wrote:
> On 6/25/21 7:21 AM, seany wrote:
>
> > The code without the parallel foreach works fine. No segfault.
>
> That's very common.
>
> What I meant is, is the code written in a way to work safely in a parallel foreach loop? (i.e. Is the code "independent"?) (But I assume it is because it's been the common theme in this thread; so there must be something stranger going on.)
>
> Ali

I have added MWP. did you have a chance to look at it?