I'd like to do the following:

auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

while(true){
version(A1)
  string line=pipes.stdout.readln;
version(A2)
  auto line=pipes.stdout.readChunk(10);
version(A3)
  auto line=pipes.stdout.readChar();

  // do something with line

  if(tryWait(pipes.pid).terminated)
    break;
}


The problem is that 'string line=pipes.stdout.readln;' seems to block until
the process is terminated, ie if the command is a long running command that
prints a line every 1 second for 10 seconds, this program will wait 10
seconds before starting the processing.
I also tried with rawRead, readf, fgetc but couldn't make it work.
I'm on OSX, if that matters.

Is there any way to achieve this?

On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <thelastmammoth@gmail.com> wrote:

> I'd like to do the following:
>
> auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);
>
> while(true){
> version(A1)
>   string line=pipes.stdout.readln;
> version(A2)
>   auto line=pipes.stdout.readChunk(10);
> version(A3)
>   auto line=pipes.stdout.readChar();
>
>   // do something with line
>
>   if(tryWait(pipes.pid).terminated)
>     break;
> }
>
>
> The problem is that 'string line=pipes.stdout.readln;' seems to block until
> the process is terminated, ie if the command is a long running command that
> prints a line every 1 second for 10 seconds, this program will wait 10
> seconds before starting the processing.
> I also tried with rawRead, readf, fgetc but couldn't make it work.
> I'm on OSX, if that matters.
>
> Is there any way to achieve this?

I think the issue is on the child process side.  If you are using buffered I/O you have to flush the buffer.

For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes.  You can flush early by calling flush on stdout, or fflush in C.  Note that C will auto-detect if it is an interactive console, and flush via newlines instead.  So running the same program from the console will flush every line!

Alternatively, you can set the flush policy to flush after every line.  See here:

https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html

And here:

http://dlang.org/phobos/std_stdio.html#.File.setvbuf

-Steve

On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer <schveiguy@yahoo.com>wrote:

> On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour < thelastmammoth@gmail.com> wrote:
>
>  I'd like to do the following:
>>
>> auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);
>>
>> while(true){
>> version(A1)
>>   string line=pipes.stdout.readln;
>> version(A2)
>>   auto line=pipes.stdout.readChunk(**10);
>> version(A3)
>>   auto line=pipes.stdout.readChar();
>>
>>   // do something with line
>>
>>   if(tryWait(pipes.pid).**terminated)
>>     break;
>> }
>>
>>
>> The problem is that 'string line=pipes.stdout.readln;' seems to block
>> until
>> the process is terminated, ie if the command is a long running command
>> that
>> prints a line every 1 second for 10 seconds, this program will wait 10
>> seconds before starting the processing.
>> I also tried with rawRead, readf, fgetc but couldn't make it work.
>> I'm on OSX, if that matters.
>>
>> Is there any way to achieve this?
>>
>
> I think the issue is on the child process side.  If you are using buffered I/O you have to flush the buffer.
>

yes


> For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes.  You can flush early by calling flush on stdout, or fflush in C.  Note that C will auto-detect if it is an interactive console, and flush via newlines instead.  So running the same program from the console will flush every line!
>
> Alternatively, you can set the flush policy to flush after every line.
>  See here:
>
> https://developer.apple.com/**library/ios/#documentation/** System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>
>
> And here:
>
> http://dlang.org/phobos/std_**stdio.html#.File.setvbuf<http://dlang.org/phobos/std_stdio.html#.File.setvbuf>
>
> -Steve
>

Thanks, that does indeed work if I have source code for the child program and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main. However I want to make it work without modifying source of child program.

I tried
http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipewith
script:
auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
Redirect.stderr);
that works but has issues : only buffers stdout, not stderr; and I may not
want to redirect stderr to stdout; also it won't work in more complex
cases, eg if program contains '|' etc, and it requires replacing \r\n by \n.

I tried replacing fork with forkpty inside std.process. That doesn't seem
to work: calling isatty(1),isatty(2)   on child process still returns 0.
Not sure why.

I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
creation in std.process (after the fork with case 0); doesn't work either.

Timothee Cour wrote:

> On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer <schveiguy@yahoo.com>wrote:
> 
>> On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour < thelastmammoth@gmail.com> wrote:
>>
>>  I'd like to do the following:
>>>
>>> auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);
>>>
>>> while(true){
>>> version(A1)
>>>   string line=pipes.stdout.readln;
>>> version(A2)
>>>   auto line=pipes.stdout.readChunk(**10);
>>> version(A3)
>>>   auto line=pipes.stdout.readChar();
>>>
>>>   // do something with line
>>>
>>>   if(tryWait(pipes.pid).**terminated)
>>>     break;
>>> }
>>>
>>>
>>> The problem is that 'string line=pipes.stdout.readln;' seems to block
>>> until
>>> the process is terminated, ie if the command is a long running command
>>> that
>>> prints a line every 1 second for 10 seconds, this program will wait 10
>>> seconds before starting the processing.
>>> I also tried with rawRead, readf, fgetc but couldn't make it work.
>>> I'm on OSX, if that matters.
>>>
>>> Is there any way to achieve this?
>>>
>>
>> I think the issue is on the child process side.  If you are using buffered I/O you have to flush the buffer.
>>
> 
> yes
> 
> 
>> For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes.  You can flush early by calling flush on stdout, or fflush in C.  Note that C will auto-detect if it is an interactive console, and flush via newlines instead.  So running the same program from the console will flush every line!
>>
>> Alternatively, you can set the flush policy to flush after every line.
>>  See here:
>>
>> https://developer.apple.com/**library/ios/#documentation/**
>> 
System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>
>>
>> And here:
>>
>> 
http://dlang.org/phobos/std_**stdio.html#.File.setvbuf<http://dlang.org/phobos/std_stdio.html#.File.setvbuf>
>>
>> -Steve
>>
> 
> Thanks, that does indeed work if I have source code for the child program and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main. However I want to make it work without modifying source of child program.
> 
> I tried http://stackoverflow.com/questions/1401002/trick-an-application-into-
thinking-its-stdin-is-interactive-not-a-pipewith
> script:
> auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
> Redirect.stderr);
> that works but has issues : only buffers stdout, not stderr; and I may not
> want to redirect stderr to stdout; also it won't work in more complex
> cases, eg if program contains '|' etc, and it requires replacing \r\n by
> \n.
> 
> I tried replacing fork with forkpty inside std.process. That doesn't seem
> to work: calling isatty(1),isatty(2)   on child process still returns 0.
> Not sure why.
> 
> I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
> creation in std.process (after the fork with case 0); doesn't work either.

I ran into this also.  My solution is the makeShell() function at https://github.com/klamonte/d-tui/blob/master/tterminal.d .  I'd like to see forkpty in phobos myself, so I just opened bug 10464 for that request.