I'm rather new to D, so apologies if this is a silly question:

I'd like to be able to fork a number of instances of a process, write to their stdins and read from their stdouts in parallel. That is, I want to write some data to the stdin of all these processes and collect lines written to their stdouts as soon as they're emitted by the process.

I tried for quite a while last night using permutations of pipeProcess and i/o redirection but nothing did exactly what I was looking for.

Is such a thing possible? If so, where might I go to educate myself?

On Monday, 23 November 2015 at 20:02:16 UTC, Cameron Reid wrote:
> I'm rather new to D, so apologies if this is a silly question:
>
> I'd like to be able to fork a number of instances of a process, write to their stdins and read from their stdouts in parallel. That is, I want to write some data to the stdin of all these processes and collect lines written to their stdouts as soon as they're emitted by the process.
>
> I tried for quite a while last night using permutations of pipeProcess and i/o redirection but nothing did exactly what I was looking for.
>
> Is such a thing possible? If so, where might I go to educate myself?

Take a look at message passing concurrency (http://ddili.org/ders/d.en/concurrency.html). You can make a program that automatically send what its stdin as a message to the owner thread and that can receive messages to print to stdout

On Monday, 23 November 2015 at 20:02:16 UTC, Cameron Reid wrote:
> Is such a thing possible? If so, where might I go to educate myself?

Yes, though the D stdlib doesn't help a whole lot, unless you want to use threads and that's blargh, I hate using threads and recommend you avoid them when you can. If you do want to try them you can just spawn a new thread for each child then block until it talks to you.

But without threads, well, Phobos helps a little, but you'll want to get platform-specific to go the rest of the way (and at that point, you might as well just use the OS functions all the way down but whatever).

Creating the process is simple enough: you can DIY with pipe/fork/exec/dup2 (on Posix), CreatePipe/CreateProcess (on Windows), or go ahead and let Phobos help you with pipeProcess.

Whatever you're more comfortable with at that point. Now the program is running and you have a handle to the pipes to talk to it.

Next, you want to be alerted when one of them is ready. This is where Phobos won't help much (though a lib like vibe.d might, I don't know though) and you want to use an operating system function.

On Posix, one of the select/poll family of functions will do what you want. If you have just a handful of processes, select is easy enough to use and will do the job. On Windows, you'll want WaitForMultipleObjects which works similarly (or you could use overlapped I/O on the pipe which is actually pretty elegant but pretty different to use too).

These functions take an array of file numbers/file handles and tells you when one of them is ready to read. If you pass all your read pipes from the children to them, you'll be alerted when any of them writes back to you. You will handle them serially when a message comes in (so you prolly want to handle it quickly if you can), but all the children run in parallel.

I find this generally easier to code than the threaded solution and it is also generally more efficient while keeping a lot of the same benefits.



Anyway, if you used Phobos' pipeProcess to create the pipes, you get the platform-specific handles through these functions: http://dlang.org/phobos/std_stdio.html#.File.fileno file.fileno and File.windowsHandle for Windows (use `version(Windows) { win version } else version(Posix) { posix version } else static assert(0, "unsupported OS");`).


Once you have the handles, you call the OS functions just like you would in C. If you need an example, I can write one up, but I suggest just looking up a C example and remembering:

import core.sys.posix.sys.select; // same as #include<sys/select.h> in a C example

import core.sys.windows.windows; // has WaitForMultipleObjects from C in there


will get you started. So import them, add the files to your list, then call the function so the OS waits and tells you when any of them are ready, then loop over it and handle the input again.

On Monday, 23 November 2015 at 20:02:16 UTC, Cameron Reid wrote:
> I'm rather new to D, so apologies if this is a silly question:
>
> I'd like to be able to fork a number of instances of a process, write to their stdins and read from their stdouts in parallel. That is, I want to write some data to the stdin of all these processes and collect lines written to their stdouts as soon as they're emitted by the process.
>
> I tried for quite a while last night using permutations of pipeProcess and i/o redirection but nothing did exactly what I was looking for.
>
> Is such a thing possible? If so, where might I go to educate myself?

pipeProcess will give you the input/output pair (and believe the output is accessible at the available time the program writes a line). To publish the same input to multiple pipes will be something have to create along with a receiver of multiple outputs.

That is to say, you can do it, but managing read/write is on you.