I've talked about this before, there's a problem with spawning multiple processes and letting them write to stderr asynchronously It seems like this might be a Windows-only problem, I couldn't recreate on Ubuntu but maybe that's because it was virtualized.

Take buggy.d:

import std.concurrency;
import std.process;

void test(int i)
{
    system("dmd buggy.d buggy.d");
}

void main()
{
    foreach (i; 0 .. 100)
    {
        spawn(&test, i);
    }
}

Run it via rdmd: rdmd buggy.d

The output is messy (ran via cmd.exe): http://paste.pocoo.org/show/513763/

So it just gets worse the more processes are spawned. One workaround is to redirect each process' stderr to a unique file. Example:

import core.thread;
import std.concurrency;
import std.process;
import std.string;
import std.stdio;
import std.file : readText;

void test(int i)
{
    // redirect each process stderr to its own file
    system(format("dmd workaround.d workaround.d 2> error_%s.txt", i));
}

void main()
{
    foreach (i; 0 .. 100)
    {
        spawn(&test, i);
    }

    thread_joinAll();

    foreach (i; 0 .. 100)
    {
        writeln(format("Output %s = %s", i,
readText(format("error_%s.txt", i))));
    }
}

So now the output is perfect: http://paste.pocoo.org/show/513771/

But writing to files is rather slow. Is there anything in Phobos I can use to spawn a process and redirect stderr to an internal buffer?

Tango had this sort of thing for D1, but I'm not seeing anything in Phobos. Since I can only recreate this on Windows I'm ok with using a WinAPI function if it provides this feature.

On Sun, 27 Nov 2011 17:37:24 -0500, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> I've talked about this before, there's a problem with spawning
> multiple processes and letting them write to stderr asynchronously It
> seems like this might be a Windows-only problem, I couldn't recreate
> on Ubuntu but maybe that's because it was virtualized.
>
> Take buggy.d:
>
> import std.concurrency;
> import std.process;
>
> void test(int i)
> {
>     system("dmd buggy.d buggy.d");
> }
>
> void main()
> {
>     foreach (i; 0 .. 100)
>     {
>         spawn(&test, i);
>     }
> }
>
> Run it via rdmd: rdmd buggy.d
>
> The output is messy (ran via cmd.exe): http://paste.pocoo.org/show/513763/
>
> So it just gets worse the more processes are spawned. One workaround
> is to redirect each process' stderr to a unique file. Example:
>
> import core.thread;
> import std.concurrency;
> import std.process;
> import std.string;
> import std.stdio;
> import std.file : readText;
>
> void test(int i)
> {
>     // redirect each process stderr to its own file
>     system(format("dmd workaround.d workaround.d 2> error_%s.txt", i));
> }
>
> void main()
> {
>     foreach (i; 0 .. 100)
>     {
>         spawn(&test, i);
>     }
>
>     thread_joinAll();
>
>     foreach (i; 0 .. 100)
>     {
>         writeln(format("Output %s = %s", i,
> readText(format("error_%s.txt", i))));
>     }
> }
>
> So now the output is perfect: http://paste.pocoo.org/show/513771/
>
> But writing to files is rather slow. Is there anything in Phobos I can
> use to spawn a process and redirect stderr to an internal buffer?
>
> Tango had this sort of thing for D1, but I'm not seeing anything in
> Phobos. Since I can only recreate this on Windows I'm ok with using a
> WinAPI function if it provides this feature.

the yet-to-be-reviewed std.process upgrade should fix all your problems.  However, I need to get Walter to include a fix in dmc so reading past the end of a pipe produces EOF instead of an invalid handle error.

I'll work on getting that pull request to Walter, this keeps coming up...

-Steve

Cool stuff Steve, I'll be eager to review it when the time comes.

Btw, is the new std.process hosted anywhere? My solution is windows-specific and probably doesn't handle all the edge-cases. Even if the new std.process isn't ready yet I'd love to look at your implementation.

On Mon, 28 Nov 2011 17:42:54 -0500, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> Btw, is the new std.process hosted anywhere? My solution is
> windows-specific and probably doesn't handle all the edge-cases. Even
> if the new std.process isn't ready yet I'd love to look at your
> implementation.

Yes,but in order for it to work you need a new dmc runtime.

it's on my github account.  I'd have to send you the dmc runtime privately, send me a private email.

-Steve