View mode: basic / threaded / horizontal-split · Log in · Help
February 11, 2012
Bug? taskPool.map() with bufSize and writeln() gets stuck
Sorry for the double-post; I have asked the same question on D.learn 
earlier but I think this is more of a question to this forum.

Tested on Ubuntu 11.10 64-bit dmd.

The following program gets stuck during the writeln() call.

- Note that the foo() call alone works fine.

- Also note that the program works fine when there is no writeln() call 
nor foo() call. All elements get processed in that case and the results 
are ignored.

Am I using taskPool.map incorrectly or is this a bug? Can you help 
identify where the problem may be? How is writeln() using the range 
differently than foo() to cause this behavior?

import std.stdio;
import std.parallelism;
import core.thread;

int func(int i)
{
    writeln("processing ", i);
    return i;
}

void main()
{
    auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

    writeln(results);  // <-- Gets stuck HERE

    foo(results); // this works fine
}

void foo(R)(R range)
{
    for ( ; !range.empty; range.popFront()) {
        writeln(range.front);
    }
}

Thank you,
Ali
February 11, 2012
Re: Bug? taskPool.map() with bufSize and writeln() gets stuck
On Saturday, 11 February 2012 at 01:31:29 UTC, Ali Çehreli wrote:
> Sorry for the double-post; I have asked the same question on 
> D.learn earlier but I think this is more of a question to this 
> forum.
>
> Tested on Ubuntu 11.10 64-bit dmd.
>
> The following program gets stuck during the writeln() call.
>
> - Note that the foo() call alone works fine.
>
> - Also note that the program works fine when there is no 
> writeln() call nor foo() call. All elements get processed in 
> that case and the results are ignored.
>
> Am I using taskPool.map incorrectly or is this a bug? Can you 
> help identify where the problem may be? How is writeln() using 
> the range differently than foo() to cause this behavior?
>
> import std.stdio;
> import std.parallelism;
> import core.thread;
>
> int func(int i)
> {
>    writeln("processing ", i);
>    return i;
> }
>
> void main()
> {
>    auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);
>
>    writeln(results);  // <-- Gets stuck HERE
>
>    foo(results); // this works fine
> }
>
> void foo(R)(R range)
> {
>    for ( ; !range.empty; range.popFront()) {
>        writeln(range.front);
>    }
> }
>
> Thank you,
> Ali

For what I see, maybe that's because it's a "non-random access 
ranges". So to access "results", you need access from it's 
aggregate...

try this inside your main:


  foreach(int r; results)
      writeln(r);

//   foo(results); // this works fin
February 11, 2012
Re: Bug? taskPool.map() with bufSize and writeln() gets stuck
On Sat, 11 Feb 2012 02:31:29 +0100, Ali Çehreli <acehreli@yahoo.com> wrote:

> Sorry for the double-post; I have asked the same question on D.learn  
> earlier but I think this is more of a question to this forum.
>
> Tested on Ubuntu 11.10 64-bit dmd.
>
> The following program gets stuck during the writeln() call.
>
> - Note that the foo() call alone works fine.
>
> - Also note that the program works fine when there is no writeln() call  
> nor foo() call. All elements get processed in that case and the results  
> are ignored.
>
> Am I using taskPool.map incorrectly or is this a bug? Can you help  
> identify where the problem may be? How is writeln() using the range  
> differently than foo() to cause this behavior?
>
> import std.stdio;
> import std.parallelism;
> import core.thread;
>
> int func(int i)
> {
>      writeln("processing ", i);
>      return i;
> }
>
> void main()
> {
>      auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);
>
>      writeln(results);  // <-- Gets stuck HERE
>
>      foo(results); // this works fine
> }
>
> void foo(R)(R range)
> {
>      for ( ; !range.empty; range.popFront()) {
>          writeln(range.front);
>      }
> }
>
> Thank you,
> Ali

Yeah, you have a deadlock in there, it's somewhat hidden though.
The issue is that writeln will take lock on stdout once. This will deadlock
with the lazy processing of the map range.
February 11, 2012
Re: Bug? taskPool.map() with bufSize and writeln() gets stuck
On 02/11/2012 12:56 AM, Martin Nowak wrote:
> On Sat, 11 Feb 2012 02:31:29 +0100, Ali Çehreli <acehreli@yahoo.com> 
wrote:
>
>> Sorry for the double-post; I have asked the same question on D.learn
>> earlier but I think this is more of a question to this forum.
>>
>> Tested on Ubuntu 11.10 64-bit dmd.
>>
>> The following program gets stuck during the writeln() call.
>>
>> - Note that the foo() call alone works fine.
>>
>> - Also note that the program works fine when there is no writeln()
>> call nor foo() call. All elements get processed in that case and the
>> results are ignored.
>>
>> Am I using taskPool.map incorrectly or is this a bug? Can you help
>> identify where the problem may be? How is writeln() using the range
>> differently than foo() to cause this behavior?
>>
>> import std.stdio;
>> import std.parallelism;
>> import core.thread;
>>
>> int func(int i)
>> {
>> writeln("processing ", i);
>> return i;
>> }
>>
>> void main()
>> {
>> auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);
>>
>> writeln(results); // <-- Gets stuck HERE
>>
>> foo(results); // this works fine
>> }
>>
>> void foo(R)(R range)
>> {
>> for ( ; !range.empty; range.popFront()) {
>> writeln(range.front);
>> }
>> }
>>
>> Thank you,
>> Ali
>
> Yeah, you have a deadlock in there, it's somewhat hidden though.
> The issue is that writeln will take lock on stdout once. This will 
deadlock
> with the lazy processing of the map range.

Thank you.

I was trying to visualize the semi-lazy nature of taskPool.map. Now I 
get what I want when the writeln() call in main() is changed to be on 
stderr:

import std.stdio;
import std.parallelism;
import core.thread;

int func(int i)
{
    writeln("processing ", i);
    Thread.sleep(dur!"seconds"(1));
    return i;
}

void main()
{
    auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

    stderr.writeln(results);    // <-- now on stderr
}

Now the output hints at how taskPool.map is semi-lazy:

processing 1
processing 2
[processing 3
1, 2processing 4
, 3processing 5
, 4processing 6
, 5, 6processing 7
processing 8
, 7, 8]

Good. :)

Ali
February 11, 2012
Re: Bug? taskPool.map() with bufSize and writeln() gets stuck
On Sat, 11 Feb 2012 17:18:21 +0100, Ali Çehreli <acehreli@yahoo.com> wrote:

> On 02/11/2012 12:56 AM, Martin Nowak wrote:
>  > On Sat, 11 Feb 2012 02:31:29 +0100, Ali Çehreli <acehreli@yahoo.com>  
> wrote:
>  >
>  >> Sorry for the double-post; I have asked the same question on D.learn
>  >> earlier but I think this is more of a question to this forum.
>  >>
>  >> Tested on Ubuntu 11.10 64-bit dmd.
>  >>
>  >> The following program gets stuck during the writeln() call.
>  >>
>  >> - Note that the foo() call alone works fine.
>  >>
>  >> - Also note that the program works fine when there is no writeln()
>  >> call nor foo() call. All elements get processed in that case and the
>  >> results are ignored.
>  >>
>  >> Am I using taskPool.map incorrectly or is this a bug? Can you help
>  >> identify where the problem may be? How is writeln() using the range
>  >> differently than foo() to cause this behavior?
>  >>
>  >> import std.stdio;
>  >> import std.parallelism;
>  >> import core.thread;
>  >>
>  >> int func(int i)
>  >> {
>  >> writeln("processing ", i);
>  >> return i;
>  >> }
>  >>
>  >> void main()
>  >> {
>  >> auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);
>  >>
>  >> writeln(results); // <-- Gets stuck HERE
>  >>
>  >> foo(results); // this works fine
>  >> }
>  >>
>  >> void foo(R)(R range)
>  >> {
>  >> for ( ; !range.empty; range.popFront()) {
>  >> writeln(range.front);
>  >> }
>  >> }
>  >>
>  >> Thank you,
>  >> Ali
>  >
>  > Yeah, you have a deadlock in there, it's somewhat hidden though.
>  > The issue is that writeln will take lock on stdout once. This will  
> deadlock
>  > with the lazy processing of the map range.
>
> Thank you.
>
> I was trying to visualize the semi-lazy nature of taskPool.map. Now I  
> get what I want when the writeln() call in main() is changed to be on  
> stderr:
>
> import std.stdio;
> import std.parallelism;
> import core.thread;
>
> int func(int i)
> {
>      writeln("processing ", i);
>      Thread.sleep(dur!"seconds"(1));
>      return i;
> }
>
> void main()
> {
>      auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);
>
>      stderr.writeln(results);    // <-- now on stderr
> }
>
> Now the output hints at how taskPool.map is semi-lazy:
>
> processing 1
> processing 2
> [processing 3
> 1, 2processing 4
> , 3processing 5
> , 4processing 6
> , 5, 6processing 7
> processing 8
> , 7, 8]
>
> Good. :)
>
> Ali
>
Going with foreach(e; results) will work too and you can write to the  
right output.
Top | Discussion index | About this forum | D home