On Friday, 19 February 2021 at 13:42:46 UTC, Steven Schveighoffer wrote:

[...]

> ignoring SIGPIPE is a process-wide thing, and so it's not appropriate for Phobos to make that decision for you. But it's trivial to ignore it.

Sure.

> I've never been a fan of SIGPIPE. If you look around on the Internet, you'll find that most people agree that the reasoning for SIGPIPE is to fix poor programming (i.e. ignoring of error codes).

As application programmer I don't want to check any error codes.
Thankfully I don't have to in D. There is a nice off-topic example
from the linux kernel [1] what happens when people do not check
return values [1].

> But it doesn't give you any good way to handle it. A SIGPIPE can be due to any pipe being written, it doesn't tell you which one. In order to know which one caused it, well, you have to look at the error code of the call!

Fortunately the D runtime /does/ take care and it throws---if the signal
is ignored beforehand. I filed issue 21649.

[1] <https://lore.kernel.org/patchwork/patch/260181/>
    move RLIMIT_NPROC check from set_user() to do_execve_common()

On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:
> On Friday, 19 February 2021 at 13:42:46 UTC, Steven Schveighoffer wrote:
>
> [...]
>
>> [...]
>
> Sure.
>
>> [...]
>
> As application programmer I don't want to check any error codes.
> Thankfully I don't have to in D. There is a nice off-topic example
> from the linux kernel [1] what happens when people do not check
> return values [1].
>
>> [...]
>
> Fortunately the D runtime /does/ take care and it throws---if the signal
> is ignored beforehand. I filed issue 21649.
>
> [1] <https://lore.kernel.org/patchwork/patch/260181/>
>     move RLIMIT_NPROC check from set_user() to do_execve_common()

Perhaps a bit late, but this is how I deal with pipes and spawnShell.
Read one byte at a time from stdout and stderr:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:

> https://github.com/DannyArends/DaNode/blob/master/danode/process.d
>
> Danny

This example shows how easy it is to implement a non-blocking stream. Phobos knows this for sockets but not for pipes?

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:
> On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:
>> [...]
>
> Perhaps a bit late, but this is how I deal with pipes and spawnShell.
> Read one byte at a time from stdout and stderr:
>
> https://github.com/DannyArends/DaNode/blob/master/danode/process.d
>
> Danny

Interesting, do you have any benchmarks for DaNode?

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:
> On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:

[...]

>> Fortunately the D runtime /does/ take care and it throws---if the signal
>> is ignored beforehand. I filed issue 21649.

[...]

> Perhaps a bit late,

It's never too late.™ :-)

> but this is how I deal with pipes and spawnShell.
> Read one byte at a time from stdout and stderr:
>
> https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Is this immune to SIGPIPE and is this design able to serve infinite
streams? BTW: Why does run use spawnShell and not spawnProcess (would
save one File object).

If the design is not intended to serve infinite streams I would
suggest to open two temporary files "out" and "err", delete them,
and let the child process write stdout/stderr into those files.
IFAICS this avoid threads, sleep, pipe and reading with fgetc.

On Thursday, 25 February 2021 at 15:28:25 UTC, kdevel wrote:
> On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:
>> On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:
>
> [...]
>
>>> Fortunately the D runtime /does/ take care and it throws---if the signal
>>> is ignored beforehand. I filed issue 21649.
>
> [...]
>
>> Perhaps a bit late,
>
> It's never too late.™ :-)
>
>> but this is how I deal with pipes and spawnShell.
>> Read one byte at a time from stdout and stderr:
>>
>> https://github.com/DannyArends/DaNode/blob/master/danode/process.d
>
> Is this immune to SIGPIPE and is this design able to serve infinite
> streams?

No I have linked up a signal handler to just ignore sigpipe, the web server closes connections after not seeing a valid output from the script for 5min (e.g. no header)

> BTW: Why does run use spawnShell and not spawnProcess (would
> save one File object).

I tried different approaches, this one worked for me™ and I just went with it. I need the stdin (for the get/post/cookies) stdout (php/d/brainf*ck script output) and stderr for the error stream from he external script

>
> If the design is not intended to serve infinite streams I would
> suggest to open two temporary files "out" and "err", delete them,
> and let the child process write stdout/stderr into those files.

I used to do that, but it generated a lot of temporary files, and I would need to parse in the files after the process is done to serve the output to the client. Using pipes is cleaner code wise, since I can just stream back the output to the client (e.g. in keepalive connections)

> IFAICS this avoid threads, sleep, pipe and reading with fgetc.

On Tuesday, 23 February 2021 at 10:07:03 UTC, Imperatorn wrote:
> On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:
>> On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:
>>> [...]
>>
>> Perhaps a bit late, but this is how I deal with pipes and spawnShell.
>> Read one byte at a time from stdout and stderr:
>>
>> https://github.com/DannyArends/DaNode/blob/master/danode/process.d
>>
>> Danny
>
> Interesting, do you have any benchmarks for DaNode?

I used to run Apache bench on the code to make sure it was as fast as possible, for static files it is pretty performant since it buffers the files, and serves them directly from memory.

The main overhead comes from the external process booting up, rdmd is nice since it only does the compile once then reuses the compiled binary for external scripts. PHP and such are always hit with the additional overhead.

Feel free to run your own tests, the development branch has the latest version with some additional bug fixes not yet available in the master branch, and feedback is welcome and can be posted as an issue in Github

On Monday, 22 February 2021 at 14:52:22 UTC, frame wrote:
> On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:
>
>> https://github.com/DannyArends/DaNode/blob/master/danode/process.d
>>
>> Danny
>
> This example shows how easy it is to implement a non-blocking stream. Phobos knows this for sockets but not for pipes?

Sockets seem to be more OS independent and are way more mature in Phobos.

Pipes seem to have been added as an afterthought in std.process and std.stdio

I had to add code for windows to deal with non-blocking/buffering pipes, Linux uses fcntl/fileno to enable non-blocking

No idea why non-blocking pipes aren't in Phobos, but pipes should not be an afterthought but a first class citizen imho

On Wednesday, 17 February 2021 at 06:58:55 UTC, Jedi wrote:
> I an using pipeShell, I have redirected stdout, stderr, and stdin.
>
> I am trying to read from the output and display it in my app. I have followed this code almost exactly except I use try wait and flush because the app is continuously updating the output. (it outputs a progress text on the same line and I'm trying to poll it to report to the user)
>

I think this post is going to answer your need.

https://dev.to/jessekphillips/piping-process-output-1cai

I haven't read all the replies, so maybe you have it working and this will benefit someone else.

On Wednesday, 3 March 2021 at 20:43:54 UTC, Danny Arends wrote:
> On Monday, 22 February 2021 at 14:52:22 UTC, frame wrote:
>> On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:
>>
>>> https://github.com/DannyArends/DaNode/blob/master/danode/process.d
>>>
>>> Danny
>>
>> This example shows how easy it is to implement a non-blocking stream. Phobos knows this for sockets but not for pipes?
>
> Sockets seem to be more OS independent and are way more mature in Phobos.
>
> Pipes seem to have been added as an afterthought in std.process and std.stdio
>
> I had to add code for windows to deal with non-blocking/buffering pipes, Linux uses fcntl/fileno to enable non-blocking
>
> No idea why non-blocking pipes aren't in Phobos, but pipes should not be an afterthought but a first class citizen imho

True, we should add this ☀️