I have just redone the interface of my std.process proposal, and I am very pleased with the result.

Calling "ls -l" is now as simple as writing

    spawnProcess("ls -l");

while "ls -l | grep foobar > filesNamedFoobar.txt" isn't that big of a deal either:

     auto pipe = Pipe.create();
     auto file = File("filesNamedFoobar.txt", "w");

     auto lsPid = spawnProcess("ls -l", stdin, pipe.writeEnd);
     scope(exit) lsPid.wait();

     auto grPid = spawnProcess("grep foobar", pipe.readEnd, file);
     scope(exit) grPid.wait();

The documentation is also a lot better now (though the spawnProcess() signature looks horrible, but that's a DDoc issue).  As always, please check out

     http://github.com/kyllingstad/ltk/blob/master/ltk/process.d
     http://kyllingen.net/code/ltk/doc/process.html

and give me your opinions.

-Lars


-- 
Lars Tandle Kyllingstad
@: lars at kyllingen.net
#: 40233221
w: http://www.kyllingen.net

This looks great!

Where would the Windows gui flag fit in?

Also, note that one can redirect both stdout and stderr to the same File, which means if you are closing both, you should only call close once on those files (I'm not sure how bad it is, but it probably will result in an ignored error from the OS).

Another thing, I liked how the pid struct used to contain the pipes when I wanted to control the std handles, and how the spawnProcess function handled the pipe creation for me.  I wonder if it's possible to pass in a default argument for the handles, or have a separate function that does the pipe creation, and store the right ends of the pipe in the returned struct.

For example, if I want control over all three handles, I have to do:

auto lsstdin = Pipe.create();
auto lsstdout = Pipe.create();
auto lsstderr = Pipe.create();

auto lspid = spawnProcess("ls -l", lsstdin.readEnd, lsstdout.writeEnd, lsstderr.writeEnd);

If I do that a lot, I have a lot of pipes littering my local variables, and the code is very verbose.

I think we can build this on top of your function, into a struct/class that handles this stuff for you.  Or we could make the spawnProcess arguments variadic and put the created pipe handles back in the pid struct.

-Steve


----- Original Message ----
> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
> To: Phobos mailing list <phobos at puremagic.com>
> Sent: Thu, March 11, 2010 6:33:29 AM
> Subject: [phobos] Major improvements to std.process
> 
> I have just redone the interface of my std.process proposal, and I am very pleased with the result.
> 
> Calling "ls -l" is now as simple as writing
> 
>    spawnProcess("ls -l");
> 
> while "ls -l | grep foobar > filesNamedFoobar.txt" isn't that big of a deal either:
> 
>     auto pipe = Pipe.create();
>     auto file = File("filesNamedFoobar.txt", "w");
> 
>     auto lsPid = spawnProcess("ls -l", stdin, pipe.writeEnd);
>     scope(exit) lsPid.wait();
> 
>     auto grPid = spawnProcess("grep foobar", pipe.readEnd, file);
>     scope(exit) grPid.wait();
> 
> The documentation is also a lot better now (though the spawnProcess() signature looks horrible, but that's a DDoc issue).  As always, please check out
> 
>     http://github.com/kyllingstad/ltk/blob/master/ltk/process.d
>     http://kyllingen.net/code/ltk/doc/process.html
> 
> and give me your opinions.
> 
> -Lars
> 
> 
> -- Lars Tandle Kyllingstad
> @: lars at kyllingen.net
> #: 40233221
> w: http://www.kyllingen.net
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

The signal termination aspect of wait is not portable to Windows.  Should we put a note that the function only occurs on POSIX systems, or is that enough to remove the feature?

-Steve

Steve Schveighoffer wrote:
> This looks great!

Thanks!


> Where would the Windows gui flag fit in?

Um...  I kinda forgot about that one. :)  What would you say is the most common setting, and to which degree?

If we can assume that the user almost always wants it to be off, say, we can just add a bool argument to the end of spawnProcess() that defaults to false.

Since using a GUI is in most cases a design choice, another option is to make it a template parameter.  But that will require either of bugs 1820 or 2972 to be fixed, otherwise every spawnProcess() call where you want to use its default value will looke like

    spawnProcess!()(...);


> Also, note that one can redirect both stdout and stderr to the same File, which means if you are closing both, you should only call close once on those files (I'm not sure how bad it is, but it probably will result in an ignored error from the OS).

Which close() operations are you referring to?

If it's the core.sys.posix.unistd.close() in the child process, I'm pretty sure nothing bad happens, except close() setting errno to EBADF and returning -1.  It probably good form to add the check, though.

If it's the std.stdio.File.close() in the parent process, then Andrei has designed it to be a no-op when the file is already closed.


> Another thing, I liked how the pid struct used to contain the pipes when I wanted to control the std handles, and how the spawnProcess function handled the pipe creation for me.  I wonder if it's possible to pass in a default argument for the handles, or have a separate function that does the pipe creation, and store the right ends of the pipe in the returned struct.
> 
> For example, if I want control over all three handles, I have to do:
> 
> auto lsstdin = Pipe.create();
> auto lsstdout = Pipe.create();
> auto lsstderr = Pipe.create();
> 
> auto lspid = spawnProcess("ls -l", lsstdin.readEnd, lsstdout.writeEnd, lsstderr.writeEnd);
> 
> If I do that a lot, I have a lot of pipes littering my local variables, and the code is very verbose.

But how often do you do that?  I mean, how often do you start several programs in a row, capturing all their input output streams, all inside one scope?

I'm asking out of curiosity.  I'm mostly a scientific programmer, not an application programmer, so maybe I'm not thinking big enough.  The reason I started working on this stuff was that I've been writing a Linux session manager -- just a lightweight one that takes care of starting other applications on login and restarting them if they die.  I can hardly imagine a program that needs to spawn more processes than that, but even that one contained just a single spawn command.

I think the current approach fits with the "systems programming language" aspect of D, and it will feel familiar to C programmers.  Only it will be a lot less bug-prone than the corresponding C code, because you no longer have to worry about manually closing the pipes, duplicating the right file descriptors, etc.


> I think we can build this on top of your function, into a struct/class that handles this stuff for you.  Or we could make the spawnProcess arguments variadic and put the created pipe handles back in the pid struct.

That's a possibility.  Something like this might do the trick:

   struct SlaveProcess
   {
       Pid pid;
       File stdin, stdout, stderr;
       this(string command, Redirect redirectFlags = Redirect.all);
       void spawn();
   }

I'm still not entirely convinced it is necessary, though.  :)

-Lars



> ----- Original Message ----
>> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
>> To: Phobos mailing list <phobos at puremagic.com>
>> Sent: Thu, March 11, 2010 6:33:29 AM
>> Subject: [phobos] Major improvements to std.process
>>
>> I have just redone the interface of my std.process proposal, and I am very pleased with the result.
>>
>> Calling "ls -l" is now as simple as writing
>>
>>    spawnProcess("ls -l");
>>
>> while "ls -l | grep foobar > filesNamedFoobar.txt" isn't that big of a deal either:
>>
>>     auto pipe = Pipe.create();
>>     auto file = File("filesNamedFoobar.txt", "w");
>>
>>     auto lsPid = spawnProcess("ls -l", stdin, pipe.writeEnd);
>>     scope(exit) lsPid.wait();
>>
>>     auto grPid = spawnProcess("grep foobar", pipe.readEnd, file);
>>     scope(exit) grPid.wait();
>>
>> The documentation is also a lot better now (though the spawnProcess() signature looks horrible, but that's a DDoc issue).  As always, please check out
>>
>>     http://github.com/kyllingstad/ltk/blob/master/ltk/process.d
>>     http://kyllingen.net/code/ltk/doc/process.html
>>
>> and give me your opinions.
>>
>> -Lars
>>
>>
>> -- Lars Tandle Kyllingstad
>> @: lars at kyllingen.net
>> #: 40233221
>> w: http://www.kyllingen.net
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos


-- 
Lars Tandle Kyllingstad
@: lars at kyllingen.net
#: 40233221
w: http://www.kyllingen.net

Steve Schveighoffer wrote:
> The signal termination aspect of wait is not portable to Windows.  Should we put a note that the function only occurs on POSIX systems, or is that enough to remove the feature?

Is there no way of detecting abnormal termination, such as a process being killed by the process manager?

If not, then we can do what is currently done in std.process, namely

    if (terminatedNormally)       return exitCode;
    else if (terminatedBySignal)  return -signal;

This works because POSIX exit codes are restricted to the range 0-255.

-Lars

This is a perfect solution, as I think it's similar to what Windows does.  If Windows is terminated by an unhandled exception, then it returns the exception number.  I don't think this always happens correctly.  For instance, I think if you terminate a process via program manager, it just exits with a 0 (I never trust exit codes from programs on Windows anyways, there seems to be a lack of respect for it among windows developers).  But at least we get a best effort for all OSes.

Thanks!

-Steve



----- Original Message ----
> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
> 
> Steve Schveighoffer wrote:
> > The signal termination aspect of wait is not portable to Windows.  Should we
> put a note that the function only occurs on POSIX systems, or is that enough to remove the feature?
> 
> Is there no way of detecting abnormal termination, such as a process being killed by the process manager?
> 
> If not, then we can do what is currently done in std.process, namely
> 
>    if (terminatedNormally)       return exitCode;
>    else if (terminatedBySignal)  return -signal;
> 
> This works because POSIX exit codes are restricted to the range 0-255.
> 
> -Lars
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

----- Original Message ----
> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
> 
> Steve Schveighoffer wrote:
> > Where would the Windows gui flag fit in?
> 
> Um...  I kinda forgot about that one. :)  What would you say is the most common setting, and to which degree?

It depends on whether you want a console or not.  I think the default should be unset, because that mimics the default for Linux.  Plus I think it's the default (no flags) in windows.

> If we can assume that the user almost always wants it to be off, say, we can just add a bool argument to the end of spawnProcess() that defaults to false.

That sounds fine to me.  We could also possibly remove that argument unless the version is windows.  But we can leave it in as a noop for portability.

> Since using a GUI is in most cases a design choice, another option is to make it
> a template parameter.  But that will require either of bugs 1820 or 2972 to be
> fixed, otherwise every spawnProcess() call where you want to use its default
> value will looke like
>    spawnProcess!()(...);

No :)

> > Also, note that one can redirect both stdout and stderr to the same File,
> which means if you are closing both, you should only call close once on those files (I'm not sure how bad it is, but it probably will result in an ignored error from the OS).
> 
> Which close() operations are you referring to?
> 
> If it's the core.sys.posix.unistd.close() in the child process, I'm pretty sure nothing bad happens, except close() setting errno to EBADF and returning -1.  It probably good form to add the check, though.
> 
> If it's the std.stdio.File.close() in the parent process, then Andrei has designed it to be a no-op when the file is already closed.

I was thinking of the parent.  It's probably fine in that case.

> > Another thing, I liked how the pid struct used to contain the pipes when I
> wanted to control the std handles, and how the spawnProcess function handled the pipe creation for me.  I wonder if it's possible to pass in a default argument for the handles, or have a separate function that does the pipe creation, and store the right ends of the pipe in the returned struct.
> > 
> > For example, if I want control over all three handles, I have to do:
> > 
> > auto lsstdin = Pipe.create();
> > auto lsstdout = Pipe.create();
> > auto lsstderr = Pipe.create();
> > 
> > auto lspid = spawnProcess("ls -l", lsstdin.readEnd, lsstdout.writeEnd,
> lsstderr.writeEnd);
> > 
> > If I do that a lot, I have a lot of pipes littering my local variables, and
> the code is very verbose.
> 
> But how often do you do that?  I mean, how often do you start several programs in a row, capturing all their input output streams, all inside one scope?

In the case where I used Tango's Process object, I redirected all three for all processes, because I was sending the results via a custom network protocol to a server.  The default option in Tango was to redirect all three to pipes contained in the instance.  If you are not writing a shell, I think this is a very common situation (wanting control over all three standard handles).

> I'm asking out of curiosity.  I'm mostly a scientific programmer, not an application programmer, so maybe I'm not thinking big enough.  The reason I started working on this stuff was that I've been writing a Linux session manager -- just a lightweight one that takes care of starting other applications on login and restarting them if they die.  I can hardly imagine a program that needs to spawn more processes than that, but even that one contained just a single spawn command.
> 
> I think the current approach fits with the "systems programming language" aspect of D, and it will feel familiar to C programmers.  Only it will be a lot less bug-prone than the corresponding C code, because you no longer have to worry about manually closing the pipes, duplicating the right file descriptors, etc.

It's just that the creation of the pipes and creation of the process always go hand in hand, you very seldom want to separate those functions.  It's somewhat of a nuisance to write many times "create this pipe, now this one, then create the process that uses them".  It would be much nicer to just say "create the process, use pipes for these handles, and give me everything back when you're done."  I liken it to how constructors unify the process of creation and initialization in one line.  Yes, you can split them, but why?

I'm not saying there isn't use for your signatures, I agree having fine-grained control over which file descriptors are used is a good thing.  I just think we should cater to common cases, even if it's via wrapper functions/structs.

-Steve

I've committed a new version with the following changes:
  - Removed ChildTerminatedException, made wait() return -signal.
  - Added GUI flag which does nothing on POSIX.


Steve Schveighoffer wrote:
> [...]
> 
> It's just that the creation of the pipes and creation of the process always go hand in hand, you very seldom want to separate those functions.  It's somewhat of a nuisance to write many times "create this pipe, now this one, then create the process that uses them".  It would be much nicer to just say "create the process, use pipes for these handles, and give me everything back when you're done."  I liken it to how constructors unify the process of creation and initialization in one line.  Yes, you can split them, but why?
> 
> I'm not saying there isn't use for your signatures, I agree having fine-grained control over which file descriptors are used is a good thing.  I just think we should cater to common cases, even if it's via wrapper functions/structs.

Ok.  I agree there should be a set of convenience functions to take care of the most common tasks.  Here's a suggestion:

   struct ProcessPipes
   {
       Pid pid;
       File stdin, stdout, stderr;
   }

   // Spawn process with redirected input/output, return immediately.
   // Our version of popen().
   ProcessPipes pipeProcess(string command, RedirectFlags flags);
   ProcessPipes pipeShell(string command, RedirectFlags flags);

   // Spawn process, optionally capture output, wait for it to finish,
   // return exit code.
   int execute(string command)
   int execute(string command, out string output);

   // Run command in shell, optionally capture output, wait for
   // it to finish, return exit code.  (Already implemented.)
   int shell(string command);
   int shell(string command, out string output);

-Lars

This all looks good.  Do you want me to access your repository to put in the necessary Windows stuff, or should we start putting this in phobos directly?

-Steve



----- Original Message ----
> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Fri, March 12, 2010 10:20:09 AM
> Subject: Re: [phobos] Major improvements to std.process
> 
> I've committed a new version with the following changes:
> - Removed ChildTerminatedException, made wait() return -signal.
> - Added GUI flag which does nothing on POSIX.
> 
> 
> Steve Schveighoffer wrote:
> > [...]
> > 
> > It's just that the creation of the pipes and creation of the process always go
> hand in hand, you very seldom want to separate those functions.  It's somewhat of a nuisance to write many times "create this pipe, now this one, then create the process that uses them".  It would be much nicer to just say "create the process, use pipes for these handles, and give me everything back when you're done."  I liken it to how constructors unify the process of creation and initialization in one line.  Yes, you can split them, but why?
> > 
> > I'm not saying there isn't use for your signatures, I agree having
> fine-grained control over which file descriptors are used is a good thing.  I just think we should cater to common cases, even if it's via wrapper functions/structs.
> 
> Ok.  I agree there should be a set of convenience functions to take care of the most common tasks.  Here's a suggestion:
> 
>   struct ProcessPipes
>   {
>       Pid pid;
>       File stdin, stdout, stderr;
>   }
> 
>   // Spawn process with redirected input/output, return immediately.
>   // Our version of popen().
>   ProcessPipes pipeProcess(string command, RedirectFlags flags);
>   ProcessPipes pipeShell(string command, RedirectFlags flags);
> 
>   // Spawn process, optionally capture output, wait for it to finish,
>   // return exit code.
>   int execute(string command)
>   int execute(string command, out string output);
> 
>   // Run command in shell, optionally capture output, wait for
>   // it to finish, return exit code.  (Already implemented.)
>   int shell(string command);
>   int shell(string command, out string output);
> 
> -Lars
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

I guess we should have the Windows functionality more or less in place before committing it to Phobos.  Besides, I don't have write access there yet.

If it's OK with you, I suggest we work out of my repo for the time being.  I don't mind giving you access to it.  Do you have a GitHub user account?

In the meantime I'll contact Andrei and arrange access to the Phobos SVN repository.

Btw, I've added the aforementioned convenience functions as well now, and I've version()ed out the things I believe to be POSIX specific. Also, check out the test_process.d file in the parent directory of ltk/, it contains a few tests that should also work on Windows.  (I had to put them in a separate file because of a failing Phobos unittest.)

-Lars


Steve Schveighoffer wrote:
> This all looks good.  Do you want me to access your repository to put in the necessary Windows stuff, or should we start putting this in phobos directly?
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
>> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Fri, March 12, 2010 10:20:09 AM
>> Subject: Re: [phobos] Major improvements to std.process
>>
>> I've committed a new version with the following changes:
>> - Removed ChildTerminatedException, made wait() return -signal.
>> - Added GUI flag which does nothing on POSIX.
>>
>>
>> Steve Schveighoffer wrote:
>>> [...]
>>>
>>> It's just that the creation of the pipes and creation of the process always go
>> hand in hand, you very seldom want to separate those functions.  It's somewhat of a nuisance to write many times "create this pipe, now this one, then create the process that uses them".  It would be much nicer to just say "create the process, use pipes for these handles, and give me everything back when you're done."  I liken it to how constructors unify the process of creation and initialization in one line.  Yes, you can split them, but why?
>>> I'm not saying there isn't use for your signatures, I agree having
>> fine-grained control over which file descriptors are used is a good thing.  I just think we should cater to common cases, even if it's via wrapper functions/structs.
>>
>> Ok.  I agree there should be a set of convenience functions to take care of the most common tasks.  Here's a suggestion:
>>
>>   struct ProcessPipes
>>   {
>>       Pid pid;
>>       File stdin, stdout, stderr;
>>   }
>>
>>   // Spawn process with redirected input/output, return immediately.
>>   // Our version of popen().
>>   ProcessPipes pipeProcess(string command, RedirectFlags flags);
>>   ProcessPipes pipeShell(string command, RedirectFlags flags);
>>
>>   // Spawn process, optionally capture output, wait for it to finish,
>>   // return exit code.
>>   int execute(string command)
>>   int execute(string command, out string output);
>>
>>   // Run command in shell, optionally capture output, wait for
>>   // it to finish, return exit code.  (Already implemented.)
>>   int shell(string command);
>>   int shell(string command, out string output);
>>
>> -Lars