https://dlang.org/library/std/process.html

How do I pipe (|) through three programs using std.process?

I'm getting confused again about what functions should I use to efficiently mimick piping like it is done in some shells (Linux bash, Windows cmd):

Here I will provide something to test against on Windows Operating System.

Shell example on Windows:

echo This is a sample text | find "sample" | find "text"

Output if piping works:

This is a sample text

Output if it does not:

Explanation:
echo is used to print output to stdout.
The output is redirected as input to a find command.
The find "textToFind" command finds matching text in the stdin input.

Windows find command:

• If text matches it returns the whole given input.
• If the text does not match, it returns nothing.

On Saturday, 11 November 2023 at 17:29:14 UTC, BoQsc wrote:

echo This is a sample text | find "sample" | find "text"

import std.stdio;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }

int main (string [] args)
{
   auto p1 = pipe;
   auto p2 = pipe;

   auto pid1 = spawnProcess (args [1..$], stdin, p1.writeEnd);
   auto pid2 = spawnProcess ([Find, "sample"], p1.readEnd, p2.writeEnd);
   auto pid3 = spawnProcess ([Find, "text"], p2.readEnd, stdout);
   wait (pid1);
   wait (pid2);
   wait (pid3);
   return 0;
}

$ ./pip echo This is a sample text
This is a sample text
$ ./pip echo This is an ample text

On Windows:

While trying to use spawnshell I discovered that I can not use any alphabetical letters inside the spawnProcess([Find, "Hello"]) it all works when they are numerical [Find, "6515"].

As of recent testing [Find, "df123"] also is acceptable,
but not when letter is on the right [Find, "df123d"]

Unable to figure why this is the case and how to resolve it.

Working example without problems:

import std.stdio;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }

int main (string [] args)
{
   auto p1 = pipe;
   auto p2 = pipe;

   auto pid1 = spawnShell("echo 123", stdin, p1.writeEnd);
   auto pid2 = spawnProcess([Find, "123"], p1.readEnd, p2.writeEnd);
   auto pid3 = spawnProcess([Find, "123"], p2.readEnd, stdout);
   wait (pid1);
   wait (pid2);
   wait (pid3);
   return 0;
}

Output:

123

Immediate issue if alphabetical letters are used:

import std.stdio;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }

int main (string [] args)
{
   auto p1 = pipe;
   auto p2 = pipe;

   auto pid1 = spawnShell("echo HelloWorld", stdin, p1.writeEnd);
   auto pid2 = spawnProcess([Find, "HelloWorld"], p1.readEnd, p2.writeEnd);
   auto pid3 = spawnProcess([Find, "HelloWorld"], p2.readEnd, stdout);
   wait (pid1);
   wait (pid2);
   wait (pid3);
   return 0;
}

Output:

FIND: Parameter format not correct
FIND: Parameter format not correct
The process tried to write to a nonexistent pipe.

Using spawnShell it all seem to work.

However the question of why spawnProcess(["find", "string to find"] is not working and produces error is still unresolved.

Works with spawnShell:

import std.stdio;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }

int main (string [] args)
{
   auto p1 = pipe;
   auto p2 = pipe;

   auto pid1 = spawnShell("echo HelloWorld", stdin, p1.writeEnd);
   auto pid2 = spawnShell("find  \"HelloWorld\"", p1.readEnd, p2.writeEnd);
   auto pid3 = spawnShell("find  \"HelloWorld\"", p2.readEnd, stdout);

   wait (pid1);
   wait (pid2);
   wait (pid3);
   return 0;
}

On Sunday, 12 November 2023 at 13:39:25 UTC, BoQsc wrote:

spawnProcess always encodes its arguments in a very specific way and the receiving programs are not always compatible with that thing.

A Windows process does not take an array of args, but rather a single string command line. spawnProcess tries to turn the unix-style array into a single string that can then be turned back into arguments by the receiving program. But it makes a lot of assumptions in how that happens that just don't always match reality.

To make this thread more complete, here is the final version.

import std.stdio;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }


int main (string [] args)
{
   auto p1 = pipe;
   auto p2 = pipe;

   auto pid1 = spawnShell("echo HelloWorld", stdin, p1.writeEnd);
   auto pid2 = spawnShell(Find ~ " \"HelloWorld\"", p1.readEnd, p2.writeEnd);
   auto pid3 = spawnShell(Find ~ " \"HelloWorld\"", p2.readEnd, stdout);

   wait (pid1);
   wait (pid2);
   wait (pid3);
   return 0;
}

It is equal to:

• Windows: echo HelloWorld | find "HelloWorld | find "HelloWorld"
• Linux: echo HelloWorld | grep "HelloWorld | grep "HelloWorld"

On Saturday, 11 November 2023 at 17:29:14 UTC, BoQsc wrote:

https://dev.to/jessekphillips/piping-process-output-1cai

Your issue with [Find, "Hello"] might be

[Find, ""Hello""]

But I'm not testing the theory...

Latest iteration on this thread.

Limitations:

• pipes through two programs.
• very verbose, hard to use.

import std;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }

void pipeTo(Pipe p, string nextprogram){
	spawnShell(nextprogram, p.readEnd, stdout);
 }

auto program(string name){
    Pipe p = std.process.pipe;
	spawnShell(name, stdin, p.writeEnd);
    return p;
}

void main()
{
    program("echo HelloWorld").pipeTo(nextprogram: Find ~ ` "HelloWorld"`);
}

On Saturday, 18 November 2023 at 18:09:53 UTC, BoQsc wrote:

What exactly are you trying to achieve?

import std;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }

void pipeTo(Pipe p, string nextprogram){
	spawnShell(nextprogram, p.readEnd, stdout);

If you allow invoking the shell from within your program why don't you use one of its facilities, i.e. the shell's pipe operator |, in the first place? spawnShell does not execute nextprogram but it executes it under the shell.

 }

auto program(string name){
    Pipe p = std.process.pipe;
	spawnShell(name, stdin, p.writeEnd);
    return p;
}

void main()
{
    program("echo HelloWorld").pipeTo(nextprogram: Find ~ ` "HelloWorld"`);

Have I missed the advent of named function arguments in D?

}

Your whole program shrinks considerably:

import std;
import std.process;

version (Windows) { enum Find = "find"; }
version (Posix) { enum Find = "grep"; }
void main()
{
    spawnShell("echo HelloWorld | " ~ Find ~ ` "HelloWorld"`).wait;
}