I an using pipeShell, I have redirected stdout, stderr, and stdin.

I am trying to read from the output and display it in my app. I have followed this code almost exactly except I use try wait and flush because the app is continuously updating the output. (it outputs a progress text on the same line and I'm trying to poll it to report to the user)


auto pipes = pipeProcess("my_application", Redirect.stdout | Redirect.stderr);
scope(exit) wait(pipes.pid);

// Store lines of output.
string[] output;
foreach (line; pipes.stdout.byLine) output ~= line.idup;

// Store lines of errors.
string[] errors;
foreach (line; pipes.stderr.byLine) errors ~= line.idup;


My code

auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | Redirect.stdin | Redirect.stderr);

		
			while(!tryWait(p.pid).terminated)
			{
				string[] output;
				foreach (line; p.stdout.byLine)
				{
					output ~= line.idup;
					writeln(line);
				}

				string[] errors;
				foreach (line; p.stderr.byLine)
				{
					errors ~= line.idup;
					writeln("Err:"~line);
				}
			}

wait(p.pid);

None of this works though. What is strange is that when I close out the debugger the app starts working(no console output but I able to see that it is doing something) but is very slow.

auto p = executeShell(`app.exe "`~f.name~`"`);

Does work, except I have no output or input. I have another app that I do the exact same code and I can get the output and parse it, but this is after the app terminates. I imagine the issue here is that I'm trying to get the output while the app is running.


I want to be able to get the output so I can reduce much of the clutter and give a progress report. I am ok with simply hooking up the in and out of the console of the app to mine just as if I ran app.exe directly.

On 2/17/21 1:58 AM, Jedi wrote:
> I an using pipeShell, I have redirected stdout, stderr, and stdin.
> 
> I am trying to read from the output and display it in my app. I have followed this code almost exactly except I use try wait and flush because the app is continuously updating the output. (it outputs a progress text on the same line and I'm trying to poll it to report to the user)
> 
> 
> auto pipes = pipeProcess("my_application", Redirect.stdout | Redirect.stderr);
> scope(exit) wait(pipes.pid);
> 
> // Store lines of output.
> string[] output;
> foreach (line; pipes.stdout.byLine) output ~= line.idup;
> 
> // Store lines of errors.
> string[] errors;
> foreach (line; pipes.stderr.byLine) errors ~= line.idup;
> 
> 
> My code
> 
> auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | Redirect.stdin | Redirect.stderr);
> 
> 
>              while(!tryWait(p.pid).terminated)
>              {
>                  string[] output;
>                  foreach (line; p.stdout.byLine)

You need to be careful here -- this will wait until stdout is *closed*.

>                  {
>                      output ~= line.idup;
>                      writeln(line);
>                  }
> 
>                  string[] errors;
>                  foreach (line; p.stderr.byLine)
>                  {
>                      errors ~= line.idup;
>                      writeln("Err:"~line);
>                  }

Same thing here.

>              }
> 
> wait(p.pid);
> 
> None of this works though. What is strange is that when I close out the debugger the app starts working(no console output but I able to see that it is doing something) but is very slow.
> 
> auto p = executeShell(`app.exe "`~f.name~`"`);
> 
> Does work, except I have no output or input. I have another app that I do the exact same code and I can get the output and parse it, but this is after the app terminates. I imagine the issue here is that I'm trying to get the output while the app is running.

Without knowing the pattern of what your app is outputting, it's hard to tell what will happen.

The most common problem with people dealing with piped output is not reading data off the pipe, which then makes the child process hang trying to write to the pipe, because the buffer is full.

For instance, if your process outputs tons of stuff to stderr, you will hang, because you are waiting for stdout to be closed first before you read anything from stderr, the child process fills up stderr pipe, and is put to sleep waiting for it to be writable, never closing stdout.

Unfortunately, std.process wraps all the pipes in File structs, so you have almost no good mechanisms to properly read the data.

-Steve

On Wednesday, 17 February 2021 at 14:36:58 UTC, Steven Schveighoffer wrote:
> On 2/17/21 1:58 AM, Jedi wrote:
>> I an using pipeShell, I have redirected stdout, stderr, and stdin.
>> 
>> I am trying to read from the output and display it in my app. I have followed this code almost exactly except I use try wait and flush because the app is continuously updating the output. (it outputs a progress text on the same line and I'm trying to poll it to report to the user)
>> 
>> 
>> auto pipes = pipeProcess("my_application", Redirect.stdout | Redirect.stderr);
>> scope(exit) wait(pipes.pid);
>> 
>> // Store lines of output.
>> string[] output;
>> foreach (line; pipes.stdout.byLine) output ~= line.idup;
>> 
>> // Store lines of errors.
>> string[] errors;
>> foreach (line; pipes.stderr.byLine) errors ~= line.idup;
>> 
>> 
>> My code
>> 
>> auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | Redirect.stdin | Redirect.stderr);
>> 
>> 
>>              while(!tryWait(p.pid).terminated)
>>              {
>>                  string[] output;
>>                  foreach (line; p.stdout.byLine)
>
> You need to be careful here -- this will wait until stdout is *closed*.
>
>>                  {
>>                      output ~= line.idup;
>>                      writeln(line);
>>                  }
>> 
>>                  string[] errors;
>>                  foreach (line; p.stderr.byLine)
>>                  {
>>                      errors ~= line.idup;
>>                      writeln("Err:"~line);
>>                  }
>
> Same thing here.
>
>>              }
>> 
>> wait(p.pid);
>> 
>> None of this works though. What is strange is that when I close out the debugger the app starts working(no console output but I able to see that it is doing something) but is very slow.
>> 
>> auto p = executeShell(`app.exe "`~f.name~`"`);
>> 
>> Does work, except I have no output or input. I have another app that I do the exact same code and I can get the output and parse it, but this is after the app terminates. I imagine the issue here is that I'm trying to get the output while the app is running.
>
> Without knowing the pattern of what your app is outputting, it's hard to tell what will happen.
>
> The most common problem with people dealing with piped output is not reading data off the pipe, which then makes the child process hang trying to write to the pipe, because the buffer is full.
>
> For instance, if your process outputs tons of stuff to stderr, you will hang, because you are waiting for stdout to be closed first before you read anything from stderr, the child process fills up stderr pipe, and is put to sleep waiting for it to be writable, never closing stdout.
>
> Unfortunately, std.process wraps all the pipes in File structs, so you have almost no good mechanisms to properly read the data.
>
WTF?

> -Steve

Seriously, I can't simply get the output in real time? Come on, that is lame, Surely D can do better than that? How hard is it to get a buffer? Is there any hacks? How can one communicate with an app using std io if one can't actually communicate until the app is closed? It makes no sense.

But note that even executeShell doesn't display the output of the app.exe so it is more than just pipeShell.

The app just outputs text, just like almost every other text. One shouldn't have to know any pattern, that defeats the purpose. I should just be able to get the output of the app.exe, and also if the app is requesting input. This isn't rocket science but it seems someone wants to turn it in to it?

When the app.exe is running it just prints stuff out, every once in a while it might ask for input(e.g., to overwrite the file if it exists, but I can get around that by checking in D)... but ultimately I just want to consolidate the output it gives so I need access to it BEFORE the app closes. The app.exe processes files, takes some time to do so so if I have to wait to display anything nothing will be displayed for a long time.

On Thursday, 18 February 2021 at 06:04:13 UTC, Jedi wrote:

>> Unfortunately, std.process wraps all the pipes in File structs, so you have almost no good mechanisms to properly read the data.
>>
> WTF?
>
>> -Steve

I'm wonder about this message. You can always use readln() and eof() on such kind of streams. byLine() is just not the best option here.

On 2/18/21 4:40 AM, frame wrote:
> On Thursday, 18 February 2021 at 06:04:13 UTC, Jedi wrote:
> 
>>> Unfortunately, std.process wraps all the pipes in File structs, so you have almost no good mechanisms to properly read the data.
>>>
>> WTF?

It's just the way it is. Everything in Phobos is a C FILE * (wrapped in a File). You need to use alternative i/o systems to get the information.

you can I believe get the file descriptor out of the File, which should help with better mechanisms. Try `fileno` and `windowsHandle`.

> 
> I'm wonder about this message. You can always use readln() and eof() on such kind of streams. byLine() is just not the best option here.

readln will block. eof doesn't tell you that there is no data in the pipe, it just says whether the pipe has been closed.

What you need is non-blocking ways to check "does this pipe have more output for me?" If you read the implementation of execute, it gets around this by redirecting stderr to stdout (so you can wait on just one pipe).

If you need them separated, then you need to do something more asynchronous. And Phobos does not wrap that, you have to use OS primitives.

-Steve

On Thursday, 18 February 2021 at 17:27:48 UTC, Steven Schveighoffer wrote:

> readln will block. eof doesn't tell you that there is no data in the pipe, it just says whether the pipe has been closed.

Of course, I must have been thinking of another language - I should take a coffee before posting.

Is there a d-library that can handle process/named pipes on Windows and Linux? I did not found any.

On Thursday, 18 February 2021 at 17:27:48 UTC, Steven Schveighoffer wrote:
> On 2/18/21 4:40 AM, frame wrote:
>>> [...]
>
> It's just the way it is. Everything in Phobos is a C FILE * (wrapped in a File). You need to use alternative i/o systems to get the information.
>
> [...]

Does your iopipe handle... Pipes? 😀

Like someone wrote, can it? 🤔

On Friday, 19 February 2021 at 08:37:50 UTC, Imperatorn wrote:
> Does your iopipe handle... Pipes? 😀

BTW: What about SIGPIPE?

In an experimental code I have this

   :
   fout.rawWrite (buf);
   fout.rawWrite ("\n");
   writeln ("flushing");
   fout.flush ();                                    // (a)
   enforce (! fout.eof, "eof on write to child");    // (b)
   writeln ("reading from pipe");
   :

fout actually is the child's stdin. Sometimes between (a) and (b)
a SIGPIPE occurs and terminates the process (exit code 141). That
prevents the proper handling of eof.

Why isn't SIGPIPE blocked or handled by default?

On 2/19/21 3:37 AM, Imperatorn wrote:
> On Thursday, 18 February 2021 at 17:27:48 UTC, Steven Schveighoffer wrote:
>> On 2/18/21 4:40 AM, frame wrote:
>>>> [...]
>>
>> It's just the way it is. Everything in Phobos is a C FILE * (wrapped in a File). You need to use alternative i/o systems to get the information.
>>
>> [...]
> 
> Does your iopipe handle... Pipes? 😀
> 
> Like someone wrote, can it? 🤔

Not yet. Important to remember that iopipe does not do actual i/o, it's a library to add *buffering* to i/o.

The std.io (which is the intended source for iopipe) library does not support pipes at the moment. But I plan to add them.

https://github.com/MartinNowak/io/issues/24

-Steve

On 2/19/21 5:41 AM, kdevel wrote:
> On Friday, 19 February 2021 at 08:37:50 UTC, Imperatorn wrote:
>> Does your iopipe handle... Pipes? 😀
> 
> BTW: What about SIGPIPE?
> 
> In an experimental code I have this
> 
>     :
>     fout.rawWrite (buf);
>     fout.rawWrite ("\n");
>     writeln ("flushing");
>     fout.flush ();                                    // (a)
>     enforce (! fout.eof, "eof on write to child");    // (b)
>     writeln ("reading from pipe");
>     :
> 
> fout actually is the child's stdin. Sometimes between (a) and (b)
> a SIGPIPE occurs and terminates the process (exit code 141). That
> prevents the proper handling of eof.
> 
> Why isn't SIGPIPE blocked or handled by default?

ignoring SIGPIPE is a process-wide thing, and so it's not appropriate for Phobos to make that decision for you. But it's trivial to ignore it.

I've never been a fan of SIGPIPE. If you look around on the Internet, you'll find that most people agree that the reasoning for SIGPIPE is to fix poor programming (i.e. ignoring of error codes). But it doesn't give you any good way to handle it. A SIGPIPE can be due to any pipe being written, it doesn't tell you which one. In order to know which one caused it, well, you have to look at the error code of the call!

The end result is -- it makes poor programming the standard. If you ignore SIGPIPE for a child process, then if that process' operation depends on SIGPIPE killing it, then you have screwed over that child process, or rather, exposed the lack of error checking in the child process.

-Steve