I am using the new std.process and am disappointed with it.

There is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says:

============
These functions are scheduled for deprecation. Please use spawnShell instead (or, alternatively, the homonymous C functions declared in std.c.process.)
============

The problems are:

1. spawnShell does NOT do the same thing as exec*. The former waits for the spawned process and then continues the calling process. The latter _replaces_ the running process with the new process.

2. The homonym (not homonymous btw) functions in std.c.process use in char*, not string, which makes them unpleasant to use.

So... what to do here?


Andrei

On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:
> I am using the new std.process and am disappointed with it.
>
> There is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says:
>
> ============
> These functions are scheduled for deprecation. Please use spawnShell instead (or, alternatively, the homonymous C functions declared in std.c.process.)
> ============
>
> The problems are:
>
> 1. spawnShell does NOT do the same thing as exec*. The former waits for the spawned process and then continues the calling process. The latter _replaces_ the running process with the new process.
>
> 2. The homonym (not homonymous btw) functions in std.c.process use in char*, not string, which makes them unpleasant to use.
>
> So... what to do here?

Since I was the one who added the deprecation notices, I guess I'll have to answer for it. :)

The main rationale for removing them was that I considered them to be low-level, rarely used functionality.  Most applications *by far* will use the "spawn" style of process creation.

While I have no statistics to back this statement with, the very fact that std.process went through several reviews on this forum and the Phobos mailing list, and even went into active use by some people (and one notable project, vibe.d) long before its inclusion in Phobos, without these functions being mentioned even *once*, is testament to its accuracy.

To me, these functions simply fall on the wrong side of the usefulness/bloat line.  And to quote http://dlang.org/phobos/,

"No pointless wrappers around C runtime library functions or OS API functions. [...] Pointless D wrappers around those functions just adds blather, bloat, baggage and bugs."

That said, they are not actually deprecated yet (as in marked with the "deprecated" attribute), so if the community wants them back, it is a simple matter of removing the deprecation notices.

-Lars

On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:
> There is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says:

In addition to what Lars said, I'd like to add that replacing the current process is a rather platform-dependent trick. While process creation in POSIX is done by forking then replacing the forked process with a new one, on Windows it's the other way around - created processes are always isolated from the current one, and Windows implementations of this function simply emulate the behavior by creating a new process, then terminating the current one.

I believe one of the goals of designing the new std.process is to create an interface that would be as platform-independent as possible. This meant that platform-specific functionality (which, incidentally, seems to be rarely used in practice) was abstracted away or scheduled for removal, delegating the task to such platform-specific (= low-level) tasks to C bindings in std.c.*, core.sys.* and core.stdc.*.

On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:
> 2. The homonym (not homonymous btw) functions [...]

Going OT here, but being somewhat of a pedant when it comes to grammar myself, I can't just let this go. :)  What is wrong with this use of "homonymous"?  What would be an example of appropriate use?

Lars

On 10/29/13 11:16 AM, Lars T. Kyllingstad wrote:
> Since I was the one who added the deprecation notices, I guess I'll have
> to answer for it. :)

Great, thanks.

> The main rationale for removing them was that I considered them to be
> low-level, rarely used functionality.  Most applications *by far* will
> use the "spawn" style of process creation.

That would be reason to not add, not to remove. They're there, and I must use them, otherwise my D code is 10% slower than the equivalent C++ code, in a project at work in which speed is of the essence. Why do I need to resort to the C functions?

> While I have no statistics to back this statement with, the very fact
> that std.process went through several reviews on this forum and the
> Phobos mailing list, and even went into active use by some people (and
> one notable project, vibe.d) long before its inclusion in Phobos,
> without these functions being mentioned even *once*, is testament to its
> accuracy.

It's been indeed reviewed, and I followed the reviews and was glad to see people liked it. My understanding was it was a backward compatible (within reason) revamp of std.process. It honestly didn't occur to me to go and see whether essential existing functionality has been deemed deprecated.

> To me, these functions simply fall on the wrong side of the
> usefulness/bloat line.

On usefulness: "Objection, your honor" :o). Forwarding from one process to another is an essential part of process control. Loading another one and waiting until it's done is a very wasteful way to replace it. And I do recall the std.process revamping took pride in being more efficient than the old one.

On bloating: the module std.process defines a couple dozen symbols. It's very far from bloating, and unlikely to grow because OSs are fairly stable when it comes to process control. Bloating doesn't strike me as a valid argument for removing existing useful and relevant functionality.

> And to quote http://dlang.org/phobos/,
>
> "No pointless wrappers around C runtime library functions or OS API
> functions. [...] Pointless D wrappers around those functions just adds
> blather, bloat, baggage and bugs."

This quote is thoroughly misapplied. A pointless wrapper is a one-liner. Writing a passable wrapper for exec* is quite a bit more involved.

> That said, they are not actually deprecated yet (as in marked with the
> "deprecated" attribute), so if the community wants them back, it is a
> simple matter of removing the deprecation notices.

I hope I provided compelling arguments.


Andrei

P.S. Also found this: "abstract final class environment". What happened to the naming conventions? Shouldn't that be capitalized?

On 10/29/13 11:34 AM, Vladimir Panteleev wrote:
> On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:
>> There is no convenient function to replace the running process with a
>> new one. There used to be the exec() family of functions, which
>> conveniently use string[] for arguments etc., but now the doc says:
>
> In addition to what Lars said, I'd like to add that replacing the
> current process is a rather platform-dependent trick. While process
> creation in POSIX is done by forking then replacing the forked process
> with a new one, on Windows it's the other way around - created processes
> are always isolated from the current one, and Windows implementations of
> this function simply emulate the behavior by creating a new process,
> then terminating the current one.
>
> I believe one of the goals of designing the new std.process is to create
> an interface that would be as platform-independent as possible. This
> meant that platform-specific functionality (which, incidentally, seems
> to be rarely used in practice) was abstracted away or scheduled for
> removal, delegating the task to such platform-specific (= low-level)
> tasks to C bindings in std.c.*, core.sys.* and core.stdc.*.

That argument doesn't seem to hold water. So if Windows does it less efficiently than Linux, we should... remove the option altogether so we level the field?

Andrei

On 10/29/13 12:06 PM, Lars T. Kyllingstad wrote:
> On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:
>> 2. The homonym (not homonymous btw) functions [...]
>
> Going OT here, but being somewhat of a pedant when it comes to grammar
> myself, I can't just let this go. :)  What is wrong with this use of
> "homonymous"?  What would be an example of appropriate use?
>
> Lars

The meaning of "homonym" is more narrow:

http://www.merriam-webster.com/dictionary/homonym

http://www.merriam-webster.com/dictionary/homonymous

But I think both work, so my mistake.


Andrei

On Tuesday, 29 October 2013 at 19:13:28 UTC, Andrei Alexandrescu wrote:
> On 10/29/13 11:16 AM, Lars T. Kyllingstad wrote:
> [...]
>> The main rationale for removing them was that I considered them to be
>> low-level, rarely used functionality.  Most applications *by far* will
>> use the "spawn" style of process creation.
>
> That would be reason to not add, not to remove. They're there, and I must use them, otherwise my D code is 10% slower than the equivalent C++ code, in a project at work in which speed is of the essence. Why do I need to resort to the C functions?

Even if you don't buy my arguments, I think Vladimir's point about exec*() basically being POSIX specific, and nothing more than a hack on Windows, is an even stronger argument.

You are doing platform-specific stuff, so you should resort to platform-specific libraries.  Were it up to me, I'd remove them from std.c too, forcing users to import core.sys.posix.unistd instead.

> [...]
>
> On usefulness: "Objection, your honor" :o). Forwarding from one process to another is an essential part of process control.

It is apparently not essential enough for it to be natively supported on Windows.

> Loading another one and waiting until it's done is a very wasteful way to replace it. [...]

I'm not suggesting this as a solution for you, but let me just point out that you wouldn't have to wait until it's done.  You'd spawn it and exit.  On Windows this is, as Vladimir points out, exactly what happens when you call exec*().  On POSIX it comes at the cost of a fork() (which I completely agree is unacceptable in some situations).

> [...]
>
>> And to quote http://dlang.org/phobos/,
>>
>> "No pointless wrappers around C runtime library functions or OS API
>> functions. [...] Pointless D wrappers around those functions just adds
>> blather, bloat, baggage and bugs."
>
> This quote is thoroughly misapplied. A pointless wrapper is a one-liner. Writing a passable wrapper for exec* is quite a bit more involved.

Ok, I'll give you that.

>> That said, they are not actually deprecated yet (as in marked with the
>> "deprecated" attribute), so if the community wants them back, it is a
>> simple matter of removing the deprecation notices.
>
> I hope I provided compelling arguments.

Nope, I don't think so. :)

That said, I know that there are completely legitimate uses for exec*(), and I have no doubt yours is one.  I also agree that the fact that the functionality is already there is an argument against removing it.

Therefore, I would like to suggest a compromise:  I propose we move the functions into an std.posix.process module.  (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)


> P.S. Also found this: "abstract final class environment". What happened to the naming conventions? Shouldn't that be capitalized?

Seems I'm the culprit again. :)  The rationale is: It walks like a variable, it talks like a variable, so let's name it like a variable.  The semantics of "environment" are those of a global variable (of a noncopyable associative-array-like type), and the fact that it's actually a class is an implementation detail.

Btw, environment has been in std.process for ages; it was not part of the revamp.  (Or rather, it was a part of the revamp that got integrated years before the rest.)

Lars

On 10/29/13 1:25 PM, Lars T. Kyllingstad wrote:
> Even if you don't buy my arguments, I think Vladimir's point about
> exec*() basically being POSIX specific, and nothing more than a hack on
> Windows, is an even stronger argument.

I think that's the weakest argument of the lot, see the rebuttal in my answer to it.

> You are doing platform-specific stuff, so you should resort to
> platform-specific libraries.  Were it up to me, I'd remove them from
> std.c too, forcing users to import core.sys.posix.unistd instead.

It's Posix, and Windows also implements the family. Not an argument for removal that Windows implements it in a suboptimal manner.

>> On usefulness: "Objection, your honor" :o). Forwarding from one
>> process to another is an essential part of process control.
>
> It is apparently not essential enough for it to be natively supported on
> Windows.

That's a good point. Nevertheless there are deeper reasons for that. Far as I can tell functions like fork() and exec() are tenuous on Windows due to the way it's architected, so they favor other ways to go about things. Then they also took the time to implement exec(). No fault there. Just don't make it difficult to get to it.

>> Loading another one and waiting until it's done is a very wasteful way
>> to replace it. [...]
>
> I'm not suggesting this as a solution for you, but let me just point out
> that you wouldn't have to wait until it's done.  You'd spawn it and
> exit.  On Windows this is, as Vladimir points out, exactly what happens
> when you call exec*().  On POSIX it comes at the cost of a fork() (which
> I completely agree is unacceptable in some situations).

Interesting, I'll do that. But my point here remains: there is no lower bound in the speed I need.

>>> That said, they are not actually deprecated yet (as in marked with the
>>> "deprecated" attribute), so if the community wants them back, it is a
>>> simple matter of removing the deprecation notices.
>>
>> I hope I provided compelling arguments.
>
> Nope, I don't think so. :)

Then we have a problem. Because I am convinced I am copiously right, and you failed to make any comparable argument to the contrary. To me the only matter to deal with is my being annoying when I know I'm right. (That may, in fact, be the bigger problem because I can be mightily annoying.)

> That said, I know that there are completely legitimate uses for exec*(),
> and I have no doubt yours is one.  I also agree that the fact that the
> functionality is already there is an argument against removing it.
>
> Therefore, I would like to suggest a compromise:  I propose we move the
> functions into an std.posix.process module.  (There is currently no
> std.posix package, but we do have std.windows, so I don't see why we
> can't add it.)

I can only assume Windows made a bona fide effort to make their _exec() implementation (http://msdn.microsoft.com/en-us/library/vstudio/431x4c1w.aspx) as good as possible within the design of that environment. I find it very forced to insist on letting that go to waste and moving exec() to Posix just because... I honestly don't know why. There was no good argument in favor of going any other way but fixing the mistake of deprecation.

>> P.S. Also found this: "abstract final class environment". What
>> happened to the naming conventions? Shouldn't that be capitalized?
>
> Seems I'm the culprit again. :)  The rationale is: It walks like a
> variable, it talks like a variable, so let's name it like a variable.
> The semantics of "environment" are those of a global variable (of a
> noncopyable associative-array-like type), and the fact that it's
> actually a class is an implementation detail.
>
> Btw, environment has been in std.process for ages; it was not part of
> the revamp.  (Or rather, it was a part of the revamp that got integrated
> years before the rest.)

Well that's a bummer but not the focus of my ire.


Andrei

On 10/29/13 1:25 PM, Lars T. Kyllingstad wrote:
> I'm not suggesting this as a solution for you, but let me just point out
> that you wouldn't have to wait until it's done.  You'd spawn it and
> exit.  On Windows this is, as Vladimir points out, exactly what happens
> when you call exec*().  On POSIX it comes at the cost of a fork() (which
> I completely agree is unacceptable in some situations).

Replacing

  return wait(spawnShell(cmd)) != 0;

with

  spawnShell(cmd);
  return 0;

as the last line of main() produces weird errors, including intermittent segfaults. Thoughts?


Andrei