I have seen this.

I can't call break form parallel foreach.

Okey, Is there a way to easily call .stop() from such a case?

Here is a case to consider:

outer: foreach(i, a; parallel(array_of_a)) {
   foreach(j, b; parallel(array_of_b)) {
     auto c = myFunction0(i,j);
     auto d = myFunction1(a,b);
     auto f = myFunction2(i,b);
     auto g = myFunction3(a,j);

     if(someConditionCheck(c,d,f,g)) {
       // stop the outer foreach loop here
     }

   }
}

Thank you

On Thursday, 24 June 2021 at 18:23:01 UTC, seany wrote:

outer: foreach(i, a; parallel(array_of_a)) {
   foreach(j, b; parallel(array_of_b)) {
     auto c = myFunction0(i,j);
     auto d = myFunction1(a,b);
     auto f = myFunction2(i,b);
     auto g = myFunction3(a,j);

     if(someConditionCheck(c,d,f,g)) {
       // stop the outer foreach loop here
     }

   }
}

Maybe I'm wrong here, but I don't think there is any way to do that with parallel.
What I would do is negate someConditionCheck and instead only do work when there is work to be done.
Obviously that may or may not be suitable.
But with parallel I don't see any way to make it happen.

On Thursday, 24 June 2021 at 19:46:52 UTC, Jerry wrote:

The parallel() function is running a taskpool. I should be able to stop that in any case...

On Thursday, 24 June 2021 at 20:08:06 UTC, seany wrote:

PS :

I have done this :

parallelContainer: while(true) {
outer: foreach(i, a; parallel(array_of_a)) {
   foreach(j, b; parallel(array_of_b)) {
     auto c = myFunction0(i,j);
     auto d = myFunction1(a,b);
     auto f = myFunction2(i,b);
     auto g = myFunction3(a,j);

     if(someConditionCheck(c,d,f,g)) {
       break parallelContainer;
     }

   }
break;
}

Is this safe? Will this cause any problem that I can't foresee now? Thank you

On Thursday, 24 June 2021 at 18:23:01 UTC, seany wrote:

Yes there is, but it won’t break the foreach:

auto tp = taskPool;
foreach (i, ref e; tp.parallel(a))
{
    // …
    tp.stop;
}

The reason this does not work is because stop terminates the worker threads as soon as they are finished with their current Task, but no sooner. parallel creates the Tasks before it presents a range to foreach, so no new Tasks are created during iteration. Therefore all elements are iterated.

outer: foreach(i, a; parallel(array_of_a)) {
   foreach(j, b; parallel(array_of_b)) {

By the way, nesting parallel foreach does not make much sense, as one level already distributes the load across all cores (but one). Additional parallelisation will likely just add overhead, and have a net negative effect.

— Bastiaan.

On Thursday, 24 June 2021 at 20:33:00 UTC, Bastiaan Veelo wrote:

Okey. So consider :

foreach(array_elem; parallel(an_array)) {
  dothing(array_elem);
}

and then in dothing() :

foreach(subelem; array_elem) {

  dootherthing(subelem);
}

• Will this ALSO cause the same overhead?

Is there any way to control the number of CPU cores used in parallelization ?

E.g : take 3 cores for the first parallel foreach - and then for the second one, take 3 cores each -> so 3 + 3 * 3 = 12 cores out of a 16 core system? Thank you.

On Thursday, 24 June 2021 at 20:41:40 UTC, seany wrote:

foreach(array_elem; parallel(an_array)) {
  dothing(array_elem);
}

foreach(subelem; array_elem) {

  dootherthing(subelem);
}

You can nest multiple foreach, but only parallelise one like so:

foreach(array_elem; parallel(an_array))
      foreach(subelem; array_elem)
          dootherthing(subelem);

So there is no need to hide one of them in a function.

There might be, by using various TaskPools with a smaller number of work threads: https://dlang.org/phobos/std_parallelism.html#.TaskPool.this.2. But I cannot see the benefit of doing this. It will just distribute the same amount of work in a different way.

— Bastiaan.

On Thursday, 24 June 2021 at 21:05:28 UTC, Bastiaan Veelo wrote:

Actually, I think this would be suboptimal as well, as the three outer threads seem to do no real work.

— Bastiaan.