Hi all,

I'm on macOS 10.11.6 with dmd 2.081.2 and I've a problem with std.process.

---
gork ():foo gorker$ gcc -c -Iinclude -o foo.cpp.o src/foo.cpp
In file included from src/foo.cpp:2:
include/foo/foo.hpp:22:10: warning: scoped enumerations are a C++11 extension [-Wc++11-extensions]
    enum class foo_event_type_t
         ^
    <snip, about 14k in STDERR>

    56 warnings and 9 errors generated.
---
No output, (zero bytes) in stdout.

If I use standard process to collect both stream with:
---
auto processPipes = pipeProcess(args, Redirect.all, null, Config.none, workDir);
foreach (c; pipes.stdout.byChunk(100)) writeln(cast(string) c); // <<<---- it halts here: stdout file is empty, but not EOF
foreach (c; pipes.stderr.byChunk(100)) writeln(cast(string) c);
---
Everything is fine if I don't redirect the stderr to a pipe.

Suggestions?

stderr buffer is full (it's about 8kb or so) and gcc waits when you read from it.

On Wednesday, 10 October 2018 at 08:02:29 UTC, Kagamin wrote:
> stderr buffer is full (it's about 8kb or so) and gcc waits when you read from it.

Thank you for your kind reply,

How to just try to read from stdout (not blocking), and then try to read from stderr (not blocking)?
I mean, how to check both pipes for data?

As an alternative, how to raise the 8kb limit?

Maybe read them with parallelism? http://dpldocs.info/experimental-docs/std.parallelism.parallel.2.html

On Wednesday, 10 October 2018 at 08:31:36 UTC, Kagamin wrote:
> Maybe read them with parallelism? http://dpldocs.info/experimental-docs/std.parallelism.parallel.2.html

thanks, but I'd rather avoid having to use threads just for this reason.
some other suggestion?

On Wednesday, 10 October 2018 at 09:16:43 UTC, Gorker wrote:
> On Wednesday, 10 October 2018 at 08:31:36 UTC, Kagamin wrote:
>> Maybe read them with parallelism? http://dpldocs.info/experimental-docs/std.parallelism.parallel.2.html
>
> thanks, but I'd rather avoid having to use threads just for this reason.
> some other suggestion?

This might be a way to do it under Linux / Posix systems (it reads byte by byte), adapted from my web server which uses it to read the output from php/perl/cgi scripts:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d

It uses fcntl to make sure the pipe is non-blocking
you can read 1 byte from the stdout pipe, then 1 byte from stderr pipe

bool nonblocking(ref File file) {
  version(Posix) {
    return(fcntl(fileno(file.getFP()), F_SETFL, O_NONBLOCK) != -1);
  }else{
    return(false);
  }
}

int readpipe(ref Pipe pipe, int verbose = NORMAL){
  File fp = pipe.readEnd;
  try{
    if(fp.isOpen()){
      if(!nonblocking(fp) && verbose >= DEBUG) writeln("[WARN]   unable to create nonblocking pipe for command");
      return(fgetc(fp.getFP()));
    }
  }catch(Exception e){
    writefln("[WARN]   Exception during readpipe command: %s", e.msg);
    fp.close();
  }
  return(EOF);
}


pStdIn = File(inputfile, "r");
pStdOut = pipe();
pStdErr = pipe();
auto cpid = spawnShell(command, pStdIn, pStdOut.writeEnd, pStdErr.writeEnd, null);
while(true){
  ch = readpipe(pStdOut);
  outbuffer.put(cast(char)ch);
  ch = readpipe(pStdErr);
  errbuffer.put(cast(char)ch);
}

Hope this works for your usecase

On Tuesday, 9 October 2018 at 09:15:09 UTC, Gorker wrote:
> Hi all,
>
> I'm on macOS 10.11.6 with dmd 2.081.2 and I've a problem with std.process.
>
> ---
> gork ():foo gorker$ gcc -c -Iinclude -o foo.cpp.o src/foo.cpp
> In file included from src/foo.cpp:2:
> include/foo/foo.hpp:22:10: warning: scoped enumerations are a C++11 extension [-Wc++11-extensions]
>     enum class foo_event_type_t
>          ^
>     <snip, about 14k in STDERR>
>
>     56 warnings and 9 errors generated.
> ---
> No output, (zero bytes) in stdout.
>
> If I use standard process to collect both stream with:
> ---
> auto processPipes = pipeProcess(args, Redirect.all, null, Config.none, workDir);
> foreach (c; pipes.stdout.byChunk(100)) writeln(cast(string) c); // <<<---- it halts here: stdout file is empty, but not EOF
> foreach (c; pipes.stderr.byChunk(100)) writeln(cast(string) c);
> ---
> Everything is fine if I don't redirect the stderr to a pipe.
>
> Suggestions?

If you want to use asynch io - you'll have to fork and use NONBLOCK output on the file descriptor.

I wrote this years ago (not sure if it still compiles tbh, but it should)

https://github.com/grogancolin/dexpect

Specifically this block should help you: https://github.com/grogancolin/dexpect/blob/master/source/dexpect.d#L343-L352

On 10/10/18 4:07 AM, Gorker wrote:
> On Wednesday, 10 October 2018 at 08:02:29 UTC, Kagamin wrote:
>> stderr buffer is full (it's about 8kb or so) and gcc waits when you read from it.
> 
> Thank you for your kind reply,
> 
> How to just try to read from stdout (not blocking), and then try to read from stderr (not blocking)?
> I mean, how to check both pipes for data?

Probably you could use OS mechanisms like poll or select. Given your usage of gcc, I'm assuming you are on some non-Windows system? Those tools should be available.

Of course, it's hard to write a cross platform way to do this, so I don't think std.process has ways to check if the pipe has data asynchronously. Things are complicated by the fact that it wraps the pipes in FILE * constructs, which means the file descriptor may not reflect if data is available if it's already in the buffer.

I think the most reasonable way is to use multiple threads to process the output. It may seem heavy-handed, but I don't know of a better way given how std.process works.

> As an alternative, how to raise the 8kb limit?

You would have to use the OS mechanisms to do this (e.g. https://stackoverflow.com/questions/5218741/set-pipe-buffer-size). I strongly suggest not taking this route, I'm not sure how portable it is to raise the buffer size after the pipe is in use. Plus, how much do you raise the limit? What if you compile an even worse C program and get 100k of errors? ;)

-Steve