Thread overview
How do I redirect stderr of a spawned process into an internal buffer?
Nov 27, 2011
Andrej Mitrovic
Nov 28, 2011
Andrej Mitrovic
Nov 28, 2011
Andrej Mitrovic
November 27, 2011
I've talked about this before, there's a problem with spawning multiple processes and letting them write to stderr asynchronously It seems like this might be a Windows-only problem, I couldn't recreate on Ubuntu but maybe that's because it was virtualized.

Take buggy.d:

import std.concurrency;
import std.process;

void test(int i)
{
    system("dmd buggy.d buggy.d");
}

void main()
{
    foreach (i; 0 .. 100)
    {
        spawn(&test, i);
    }
}

Run it via rdmd: rdmd buggy.d

The output is messy (ran via cmd.exe): http://paste.pocoo.org/show/513763/

So it just gets worse the more processes are spawned. One workaround is to redirect each process' stderr to a unique file. Example:

import core.thread;
import std.concurrency;
import std.process;
import std.string;
import std.stdio;
import std.file : readText;

void test(int i)
{
    // redirect each process stderr to its own file
    system(format("dmd workaround.d workaround.d 2> error_%s.txt", i));
}

void main()
{
    foreach (i; 0 .. 100)
    {
        spawn(&test, i);
    }

    thread_joinAll();

    foreach (i; 0 .. 100)
    {
        writeln(format("Output %s = %s", i,
readText(format("error_%s.txt", i))));
    }
}

So now the output is perfect: http://paste.pocoo.org/show/513771/

But writing to files is rather slow. Is there anything in Phobos I can use to spawn a process and redirect stderr to an internal buffer?

Tango had this sort of thing for D1, but I'm not seeing anything in Phobos. Since I can only recreate this on Windows I'm ok with using a WinAPI function if it provides this feature.
November 28, 2011
On Sun, 27 Nov 2011 17:37:24 -0500, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> I've talked about this before, there's a problem with spawning
> multiple processes and letting them write to stderr asynchronously It
> seems like this might be a Windows-only problem, I couldn't recreate
> on Ubuntu but maybe that's because it was virtualized.
>
> Take buggy.d:
>
> import std.concurrency;
> import std.process;
>
> void test(int i)
> {
>     system("dmd buggy.d buggy.d");
> }
>
> void main()
> {
>     foreach (i; 0 .. 100)
>     {
>         spawn(&test, i);
>     }
> }
>
> Run it via rdmd: rdmd buggy.d
>
> The output is messy (ran via cmd.exe): http://paste.pocoo.org/show/513763/
>
> So it just gets worse the more processes are spawned. One workaround
> is to redirect each process' stderr to a unique file. Example:
>
> import core.thread;
> import std.concurrency;
> import std.process;
> import std.string;
> import std.stdio;
> import std.file : readText;
>
> void test(int i)
> {
>     // redirect each process stderr to its own file
>     system(format("dmd workaround.d workaround.d 2> error_%s.txt", i));
> }
>
> void main()
> {
>     foreach (i; 0 .. 100)
>     {
>         spawn(&test, i);
>     }
>
>     thread_joinAll();
>
>     foreach (i; 0 .. 100)
>     {
>         writeln(format("Output %s = %s", i,
> readText(format("error_%s.txt", i))));
>     }
> }
>
> So now the output is perfect: http://paste.pocoo.org/show/513771/
>
> But writing to files is rather slow. Is there anything in Phobos I can
> use to spawn a process and redirect stderr to an internal buffer?
>
> Tango had this sort of thing for D1, but I'm not seeing anything in
> Phobos. Since I can only recreate this on Windows I'm ok with using a
> WinAPI function if it provides this feature.

the yet-to-be-reviewed std.process upgrade should fix all your problems.  However, I need to get Walter to include a fix in dmc so reading past the end of a pipe produces EOF instead of an invalid handle error.

I'll work on getting that pull request to Walter, this keeps coming up...

-Steve
November 28, 2011
Cool stuff Steve, I'll be eager to review it when the time comes.
November 28, 2011
Btw, is the new std.process hosted anywhere? My solution is windows-specific and probably doesn't handle all the edge-cases. Even if the new std.process isn't ready yet I'd love to look at your implementation.
November 29, 2011
On Mon, 28 Nov 2011 17:42:54 -0500, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> Btw, is the new std.process hosted anywhere? My solution is
> windows-specific and probably doesn't handle all the edge-cases. Even
> if the new std.process isn't ready yet I'd love to look at your
> implementation.

Yes,but in order for it to work you need a new dmc runtime.

it's on my github account.  I'd have to send you the dmc runtime privately, send me a private email.

-Steve