Hi,

Recently, I found myself in need of the functionality that is (or should have been) in std.process, but unfortunately I found it lacking in many respects.  Assuming that improving it currently isn't at the top of the to-do list for Phobos, I decided not to wait, and rather to write my own version.  If you want you can check it out here:

Code:   http://github.com/kyllingstad/ltk/blob/master/ltk/process.d Docs:   http://kyllingen.net/code/ltk/doc/process.html

I don't know if any of it is usable for Phobos, but if it is, I'd be happy to contribute.

I've tried to write it in the style of std.concurrency, with a function spawnProcess() that returns a Pid struct.  Currently it is for POSIX only, since I have no experience at all with the Windows API.

-Lars

Looks good to me. If everybody else agrees, I'd be glad to extend you an offer for write access to Phobos so you can commit the changes yourself.

Andrei

Lars Tandle Kyllingstad wrote:
> Hi,
> 
> Recently, I found myself in need of the functionality that is (or should have been) in std.process, but unfortunately I found it lacking in many respects.  Assuming that improving it currently isn't at the top of the to-do list for Phobos, I decided not to wait, and rather to write my own version.  If you want you can check it out here:
> 
> Code:   http://github.com/kyllingstad/ltk/blob/master/ltk/process.d Docs:   http://kyllingen.net/code/ltk/doc/process.html
> 
> I don't know if any of it is usable for Phobos, but if it is, I'd be happy to contribute.
> 
> I've tried to write it in the style of std.concurrency, with a function spawnProcess() that returns a Pid struct.  Currently it is for POSIX only, since I have no experience at all with the Windows API.
> 
> -Lars
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

I can help you with some aspects of it, I revamped tangos' Process object a while back.

Some things about my experience with Tango's Process:

1. I included a similar "redirect flags" option for the process, with the added ability to redirect stdout to stderr and vice versa.
2. About the searchPath flag -- Windows' function to create a process already takes into account the PATH variable, so I'm guessing that flag will be a noop on Windows?  Also, execlp and execvp already take into account the PATH.  I can't imagine you would need to build your own PATH parsing/searching method, or to run a command without taking the PATH into account (you can do this anyways by prepending a './').  My advice is to simply use one of those versions of exec, and get rid of the searchPath flag.
3. Tango's Process object included a means to specify the environment for the child process (a useful feature).  A simple argument with a string[string] for the environment variables would be useful.
4. For Windows, you do not pass in the command as an array of parameters, you pass it in as one string.  The parsing required for doing so is extremely convoluted and I only could figure it out by trial and error (documentation was incomplete).  I can help you with that.
5. Some version that parses a command line would be useful.  For example, it would be nice to be able to simply do spawn("ls -l");
6. I think your method of getting the stdout/stderr/stdin handles is a bad idea.  One can see someone not simply storing the stdout/stderr/stdin properties in an external variable and using them later, but rather always accessing the property to work with it.  This means you create a new FILE* and a new File object for each access.  I think you should store the File object directly in the Pid struct, and simply refer to it later.
7. Windows has an additional flag to specify that no console should be created.  This is very essential for things like plugins (where you don't want a black console box to pop up while running a child process).  The way I handled this in Tango is to add a gui flag which was a noop in Linux.

That's all I can think of right now.  I agree that std.process is very lacking, it is good someone is working on this.

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Thu, March 4, 2010 8:18:50 AM
> Subject: Re: [phobos] Suggestions for std.process
> 
> Looks good to me. If everybody else agrees, I'd be glad to extend you an offer for write access to Phobos so you can commit the changes yourself.
> 
> Andrei
> 
> Lars Tandle Kyllingstad wrote:
> > Hi,
> > 
> > Recently, I found myself in need of the functionality that is (or should have
> been) in std.process, but unfortunately I found it lacking in many respects. Assuming that improving it currently isn't at the top of the to-do list for Phobos, I decided not to wait, and rather to write my own version.  If you want you can check it out here:
> > 
> > Code:   http://github.com/kyllingstad/ltk/blob/master/ltk/process.d Docs:   http://kyllingen.net/code/ltk/doc/process.html
> > 
> > I don't know if any of it is usable for Phobos, but if it is, I'd be happy to
> contribute.
> > 
> > I've tried to write it in the style of std.concurrency, with a function
> spawnProcess() that returns a Pid struct.  Currently it is for POSIX only, since I have no experience at all with the Windows API.
> > 
> > -Lars
> > _______________________________________________
> > phobos mailing list
> > phobos at puremagic.com
> > http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Sounds great. You guys please have at it!

Andrei

Steve Schveighoffer wrote:
> I can help you with some aspects of it, I revamped tangos' Process object a while back.
> 
> Some things about my experience with Tango's Process:
> 
> 1. I included a similar "redirect flags" option for the process, with the added ability to redirect stdout to stderr and vice versa.
> 2. About the searchPath flag -- Windows' function to create a process already takes into account the PATH variable, so I'm guessing that flag will be a noop on Windows?  Also, execlp and execvp already take into account the PATH.  I can't imagine you would need to build your own PATH parsing/searching method, or to run a command without taking the PATH into account (you can do this anyways by prepending a './').  My advice is to simply use one of those versions of exec, and get rid of the searchPath flag.
> 3. Tango's Process object included a means to specify the environment for the child process (a useful feature).  A simple argument with a string[string] for the environment variables would be useful.
> 4. For Windows, you do not pass in the command as an array of parameters, you pass it in as one string.  The parsing required for doing so is extremely convoluted and I only could figure it out by trial and error (documentation was incomplete).  I can help you with that.
> 5. Some version that parses a command line would be useful.  For example, it would be nice to be able to simply do spawn("ls -l");
> 6. I think your method of getting the stdout/stderr/stdin handles is a bad idea.  One can see someone not simply storing the stdout/stderr/stdin properties in an external variable and using them later, but rather always accessing the property to work with it.  This means you create a new FILE* and a new File object for each access.  I think you should store the File object directly in the Pid struct, and simply refer to it later.
> 7. Windows has an additional flag to specify that no console should be created.  This is very essential for things like plugins (where you don't want a black console box to pop up while running a child process).  The way I handled this in Tango is to add a gui flag which was a noop in Linux.
> 
> That's all I can think of right now.  I agree that std.process is very lacking, it is good someone is working on this.
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
>> From: Andrei Alexandrescu <andrei at erdani.com>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Thu, March 4, 2010 8:18:50 AM
>> Subject: Re: [phobos] Suggestions for std.process
>>
>> Looks good to me. If everybody else agrees, I'd be glad to extend you an offer for write access to Phobos so you can commit the changes yourself.
>>
>> Andrei
>>
>> Lars Tandle Kyllingstad wrote:
>>> Hi,
>>>
>>> Recently, I found myself in need of the functionality that is (or should have
>> been) in std.process, but unfortunately I found it lacking in many respects. Assuming that improving it currently isn't at the top of the to-do list for Phobos, I decided not to wait, and rather to write my own version.  If you want you can check it out here:
>>> Code:   http://github.com/kyllingstad/ltk/blob/master/ltk/process.d Docs:   http://kyllingen.net/code/ltk/doc/process.html
>>>
>>> I don't know if any of it is usable for Phobos, but if it is, I'd be happy to
>> contribute.
>>> I've tried to write it in the style of std.concurrency, with a function
>> spawnProcess() that returns a Pid struct.  Currently it is for POSIX only, since I have no experience at all with the Windows API.
>>> -Lars
>>> _______________________________________________
>>> phobos mailing list
>>> phobos at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/phobos
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Steve Schveighoffer wrote:
> I can help you with some aspects of it, I revamped tangos' Process object a while back.

That would be great!


> Some things about my experience with Tango's Process:
> 
> 1. I included a similar "redirect flags" option for the process, with the added ability to redirect stdout to stderr and vice versa.

That's a good idea.

I've actually considered making it even more general, by allowing the user to provide File objects to replace the standard streams.  This would make it easy to use the output from one process as input to another, or to pass the contents of a file to a process' input stream. A simplified example that emulates "foo | bar":

   Pid spawnProcess(string executable, File useThisAsInput);

   // foo | bar
   auto fooPid = spawnProcess("foo");
   auto barPid = spawnProcess("bar", foo.stdout);

   // baz < myfile.txt
   auto bazPid = spawnProcess("baz", File("myfile.txt");

What do you think?


> 2. About the searchPath flag -- Windows' function to create a process already takes into account the PATH variable, so I'm guessing that flag will be a noop on Windows?  Also, execlp and execvp already take into account the PATH.  I can't imagine you would need to build your own PATH parsing/searching method, or to run a command without taking the PATH into account (you can do this anyways by prepending a './').  My advice is to simply use one of those versions of exec, and get rid of the searchPath flag.

I really can't decide what I think is best.  On one hand, having the function always search the path is convenient, on the other hand, if your program is going to execute another, there should be as little chance as possible of it executing the wrong program.  (The latter may be of small concern, though, since one can just include a directory in the executable name.)

Importantly, though, I think code in Phobos should work in the same way on both Windows and POSIX, so that more or less settles it for me.

The main reason I wrote my own path-searching mechanism is that the execvp function not only searches the path, but if that fails it also tries to run the command through the shell.  For some reason I find that annoying.


> 3. Tango's Process object included a means to specify the environment for the child process (a useful feature).  A simple argument with a string[string] for the environment variables would be useful.

That's coming soon, just haven't gotten around to it yet. :)  In fact I have a few ideas regarding environment variables.  The getenv() function has no business being in std.process.  Rather, the following functions should be in std.system:

   string getEnv(string varName);
   string[string] getEnv();


> 4. For Windows, you do not pass in the command as an array of parameters, you pass it in as one string.  The parsing required for doing so is extremely convoluted and I only could figure it out by trial and error (documentation was incomplete).  I can help you with that.

Awesome!


> 5. Some version that parses a command line would be useful.  For example, it would be nice to be able to simply do spawn("ls -l");

(That being the default on Windows, right?)  I have considered a spawnShell() function, would that be anything like what you're suggesting?


> 6. I think your method of getting the stdout/stderr/stdin handles is a bad idea.  One can see someone not simply storing the stdout/stderr/stdin properties in an external variable and using them later, but rather always accessing the property to work with it.  This means you create a new FILE* and a new File object for each access.  I think you should store the File object directly in the Pid struct, and simply refer to it later.

That's a very good point, and now that you mention it, a rather obvious mistake.  Luckily it's also trivial to implement.  Consider it done!


> 7. Windows has an additional flag to specify that no console should be created.  This is very essential for things like plugins (where you don't want a black console box to pop up while running a child process).  The way I handled this in Tango is to add a gui flag which was a noop in Linux.

My biggest problem here, which I mentioned in my first e-mail, is that I have no experience whatsoever with the Windows API.  I don't even have a computer with Windows on it.  (Though I may be able to get a Windows licence through work, I'll have to check that out.  Then I'll set up a virtual machine or something.)

In the event that I'm able to get Windows, we are left with three options, either of which is fine by me:

1. I spend some time on it, and write it myself, from scratch.  This will take a while.

2. You, as the author of Tango's Process class, if possible, give me
permission to take the Windows part of that code and adapt it to Phobos.
  So far I haven't looked at it, due to those dreaded licensing issues.

3. Someone else writes the Windows version.  This is the fastest option by far, but I guess manpower is in short demand.

-Lars


> That's all I can think of right now.  I agree that std.process is very lacking, it is good someone is working on this.
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
>> From: Andrei Alexandrescu <andrei at erdani.com>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Thu, March 4, 2010 8:18:50 AM
>> Subject: Re: [phobos] Suggestions for std.process
>>
>> Looks good to me. If everybody else agrees, I'd be glad to extend you an offer for write access to Phobos so you can commit the changes yourself.
>>
>> Andrei
>>
>> Lars Tandle Kyllingstad wrote:
>>> Hi,
>>>
>>> Recently, I found myself in need of the functionality that is (or should have
>> been) in std.process, but unfortunately I found it lacking in many respects. Assuming that improving it currently isn't at the top of the to-do list for Phobos, I decided not to wait, and rather to write my own version.  If you want you can check it out here:
>>> Code:   http://github.com/kyllingstad/ltk/blob/master/ltk/process.d Docs:   http://kyllingen.net/code/ltk/doc/process.html
>>>
>>> I don't know if any of it is usable for Phobos, but if it is, I'd be happy to
>> contribute.
>>> I've tried to write it in the style of std.concurrency, with a function
>> spawnProcess() that returns a Pid struct.  Currently it is for POSIX only, since I have no experience at all with the Windows API.
>>> -Lars
>>> _______________________________________________
>>> phobos mailing list
>>> phobos at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/phobos
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos


-- 
Lars Tandle Kyllingstad
@: lars at kyllingen.net
#: 40233221
w: http://www.kyllingen.net

----- Original Message ----
> From: Lars Tandle Kyllingstad <lars at kyllingen.net>
> 
> Steve Schveighoffer wrote:
> > Some things about my experience with Tango's Process:
> > 
> > 1. I included a similar "redirect flags" option for the process, with the
> added ability to redirect stdout to stderr and vice versa.
> 
> That's a good idea.
> 
> I've actually considered making it even more general, by allowing the user to provide File objects to replace the standard streams.  This would make it easy to use the output from one process as input to another, or to pass the contents of a file to a process' input stream. A simplified example that emulates "foo | bar":
> 
>   Pid spawnProcess(string executable, File useThisAsInput);
> 
>   // foo | bar
>   auto fooPid = spawnProcess("foo");
>   auto barPid = spawnProcess("bar", foo.stdout);
> 
>   // baz < myfile.txt
>   auto bazPid = spawnProcess("baz", File("myfile.txt");
> 
> What do you think?

That sounds like a good idea.  Hm... I read the std.stream code, and I wasn't aware that Phobos used its own buffering layer in place of C's for everything but the standard handles.  That is good news!  However, you are still using FILE * in your code, you should get rid of that (I don't even know where the File.wrapFile function is).

> > 2. About the searchPath flag -- Windows' function to create a process already
> takes into account the PATH variable, so I'm guessing that flag will be a noop on Windows?  Also, execlp and execvp already take into account the PATH.  I can't imagine you would need to build your own PATH parsing/searching method, or to run a command without taking the PATH into account (you can do this anyways by prepending a './').  My advice is to simply use one of those versions of exec, and get rid of the searchPath flag.
> 
> I really can't decide what I think is best.  On one hand, having the function always search the path is convenient, on the other hand, if your program is going to execute another, there should be as little chance as possible of it executing the wrong program.  (The latter may be of small concern, though, since one can just include a directory in the executable name.)
> 
> Importantly, though, I think code in Phobos should work in the same way on both Windows and POSIX, so that more or less settles it for me.
> 
> The main reason I wrote my own path-searching mechanism is that the execvp function not only searches the path, but if that fails it also tries to run the command through the shell.  For some reason I find that annoying.

Hm... that is annoying.  Now that I look at Tango, it actually uses execve, and does the path parsing (only on Linux/Mac/BSD however).  As you say, it should work the same on both Windows and POSIX, so I think that path search should be implicit.

> > 3. Tango's Process object included a means to specify the environment for the
> child process (a useful feature).  A simple argument with a string[string] for the environment variables would be useful.
> 
> That's coming soon, just haven't gotten around to it yet. :)  In fact I have a few ideas regarding environment variables.  The getenv() function has no business being in std.process.  Rather, the following functions should be in std.system:
> 
>   string getEnv(string varName);
>   string[string] getEnv();

Sounds good.

> > 5. Some version that parses a command line would be useful.  For example, it
> would be nice to be able to simply do spawn("ls -l");
> 
> (That being the default on Windows, right?)  I have considered a spawnShell() function, would that be anything like what you're suggesting?

What I mean is, having a method that parses the command line according to simple sane rules and calls the array-style version.  I don't suggest simply passing the command to Windows, as the parsing and quote escaping is so ridiculous, nobody should have to deal with it.  Wait until you see it :)

> > 7. Windows has an additional flag to specify that no console should be
> created.  This is very essential for things like plugins (where you don't want a black console box to pop up while running a child process).  The way I handled this in Tango is to add a gui flag which was a noop in Linux.
> 
> My biggest problem here, which I mentioned in my first e-mail, is that I have no experience whatsoever with the Windows API.  I don't even have a computer with Windows on it.  (Though I may be able to get a Windows licence through work, I'll have to check that out.  Then I'll set up a virtual machine or something.)
> 
> In the event that I'm able to get Windows, we are left with three options, either of which is fine by me:
> 
> 1. I spend some time on it, and write it myself, from scratch.  This will take a while.
> 
> 2. You, as the author of Tango's Process class, if possible, give me permission to take the Windows part of that code and adapt it to Phobos.  So far I haven't looked at it, due to those dreaded licensing issues.
> 
> 3. Someone else writes the Windows version.  This is the fastest option by far, but I guess manpower is in short demand.
> 

As long as we agree on the interface, I can write the Windows version.  I am not the original author of Tango's Process class, that was Reagan Heath, and then Juan Comellas.  I took over because I needed some functionality that wasn't there, and I found some bugs, and Juan had stopped maintaining it.  Therefore, I can't really give you full permission to copy Tango's Process class.

But I did write the Windows quote parsing part of it, so I have no problems copying that over.  The rest is pretty straightforward code that can easily be rewritten from scratch.

-Steve

----- Original Message ----
> From: Steve Schveighoffer <schveiguy at yahoo.com>
> That sounds like a good idea.  Hm... I read the std.stream code, and I wasn't
> aware that Phobos used its own buffering layer in place of C's for everything
> but the standard handles.  That is good news!  However, you are still using FILE
> * in your code, you should get rid of that (I don't even know where the
> File.wrapFile function is).

Oh, wow, I just realized there's a std.stdio.File struct.  That needs to be changed.  Having File as a class in std.stream and as a struct in std.stdio is horrible :(

In any case, I think you want to use the std.stream File class, not the File struct.  It provides no buffering, and has the appropriate "getHandle" function that would be needed for passing to a child process.

-Steve

I plan to throw away all of std.stream.

Andrei

Steve Schveighoffer wrote:
> 
> 
> 
> ----- Original Message ----
>> From: Steve Schveighoffer <schveiguy at yahoo.com>
>> That sounds like a good idea.  Hm... I read the std.stream code, and I wasn't
>> aware that Phobos used its own buffering layer in place of C's for everything
>> but the standard handles.  That is good news!  However, you are still using FILE
>> * in your code, you should get rid of that (I don't even know where the
>> File.wrapFile function is).
> 
> Oh, wow, I just realized there's a std.stdio.File struct.  That needs to be changed.  Having File as a class in std.stream and as a struct in std.stdio is horrible :(
> 
> In any case, I think you want to use the std.stream File class, not the File struct.  It provides no buffering, and has the appropriate "getHandle" function that would be needed for passing to a child process.
> 
> -Steve
> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Will there be an unbuffered interface to file handles?  std.stdio.File wraps a FILE* which I find substandard.  I don't think D should be relying on C buffering for D-only constructs.

I actually find the whole notion of relying on libc a little suspect, even for the standard handles.

Is there a document somewhere which lists which parts of phobos will be trimmed?  I am not very familiar with the library, and I'd like to know what parts to avoid when contributing.

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> 
> I plan to throw away all of std.stream.
> 
> Andrei

I think mostly std.stream and std.bind are for deprecation and not yet documented as deprecated, there are a few others (std.boxer, std.bit???) that are already marked as such.

There's a long discussion about relying on FILE*. We could eliminate it if needed, it's a bit (though not a lot) of an annoyance.


Andrei

Steve Schveighoffer wrote:
> Will there be an unbuffered interface to file handles?  std.stdio.File wraps a FILE* which I find substandard.  I don't think D should be relying on C buffering for D-only constructs.
> 
> I actually find the whole notion of relying on libc a little suspect, even for the standard handles.
> 
> Is there a document somewhere which lists which parts of phobos will be trimmed?  I am not very familiar with the library, and I'd like to know what parts to avoid when contributing.
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
>> From: Andrei Alexandrescu <andrei at erdani.com>
>>
>> I plan to throw away all of std.stream.
>>
>> Andrei
> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos