On Tuesday, 29 October 2013 at 19:29:12 UTC, Andrei Alexandrescu wrote:
> On 10/29/13 11:34 AM, Vladimir Panteleev wrote:
>> On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei In addition to what Lars said, I'd like to add that replacing the
>> current process is a rather platform-dependent trick. While process
>> creation in POSIX is done by forking then replacing the forked process
>> with a new one, on Windows it's the other way around - created processes
>> are always isolated from the current one, and Windows implementations of
>> this function simply emulate the behavior by creating a new process,
>> then terminating the current one.
>>
>> [...]
>
> That argument doesn't seem to hold water. So if Windows does it less efficiently than Linux, we should... remove the option altogether so we level the field?

It's not just that it does it less efficiently.  What's worse is that it does it *differently*.  While I don't have a Windows machine available to test this, I've done some research, and it seems the effect of calling _exec*() on Windows is *exactly* as if a new process was created and the parent process exited.  In other words, the child process starts in the "background" (with a new PID), and the parent process returns control to the program which called it (such as cmd.exe).

Furthermore, Windows' exec*() functions do not handle program names with spaces well.

Apparently, Python has included support for the exec*() functionality on all platforms, which has led to several bug reports:

  http://bugs.python.org/issue9148
  http://bugs.python.org/issue19066
  http://bugs.python.org/issue19124

These were only the ones that came up in my first Google search.  The answer from the Python team to all of them seems to be "don't use exec on Windows".  StackOverflow users agree:

  http://stackoverflow.com/q/7004687
  http://stackoverflow.com/q/7264571

I suspect that these functions were only included in Windows to satisfy POSIX requirements (which they fail to do).

We should discourage their use on Windows, and one way to do this is to *not* have a nice cross-platform interface to them in Phobos.

Lars

On Tuesday, 29 October 2013 at 20:44:50 UTC, Andrei Alexandrescu wrote:
> On 10/29/13 1:25 PM, Lars T. Kyllingstad wrote:
>> Even if you don't buy my arguments, I think Vladimir's point about
>> exec*() basically being POSIX specific, and nothing more than a hack on
>> Windows, is an even stronger argument.
>
> I think that's the weakest argument of the lot, see the rebuttal in my answer to it.

I've rebutted your rebuttal in another post.

>> You are doing platform-specific stuff, so you should resort to
>> platform-specific libraries.  Were it up to me, I'd remove them from
>> std.c too, forcing users to import core.sys.posix.unistd instead.
>
> It's Posix, and Windows also implements the family. Not an argument for removal that Windows implements it in a suboptimal manner.

Like I said in the other post, suboptimal is one thing, *wrong* is another.

>>> On usefulness: "Objection, your honor" :o). Forwarding from one
>>> process to another is an essential part of process control.
>>
>> It is apparently not essential enough for it to be natively supported on
>> Windows.
>
> That's a good point. Nevertheless there are deeper reasons for that. Far as I can tell functions like fork() and exec() are tenuous on Windows due to the way it's architected, so they favor other ways to go about things. Then they also took the time to implement exec(). No fault there. Just don't make it difficult to get to it.

I've suggested a way to make it easy to get to, but which discourages its use on Windows:  Put it in std.posix.

> [...]
>>> I hope I provided compelling arguments.
>>
>> Nope, I don't think so. :)
>
> Then we have a problem. Because I am convinced I am copiously right, and you failed to make any comparable argument to the contrary. To me the only matter to deal with is my being annoying when I know I'm right. (That may, in fact, be the bigger problem because I can be mightily annoying.)

Good thing I'm halfway around the globe, then. :)

-Lars

On Tuesday, 29 October 2013 at 20:55:22 UTC, Andrei Alexandrescu wrote:
> Replacing
>
>   return wait(spawnShell(cmd)) != 0;
>
> with
>
>   spawnShell(cmd);
>   return 0;
>
> as the last line of main() produces weird errors, including intermittent segfaults. Thoughts?

That's really weird.  I've tried and failed to reproduce it with this minimal test program on my Linux box:

    int main(string[] args)
    {
        import std.process;
        spawnShell(args[1]);
        return 0;
    }

This worked fine with a variety of app names passed in args[1].

Are you able to provide a bit more context?  Such as:

 - Which platform are you on?
 - What happens before spawnShell()?
 - Which program is spawned?

Lars

On 10/29/13 3:07 PM, Lars T. Kyllingstad wrote:
...
> I suspect that these functions were only included in Windows to satisfy
> POSIX requirements (which they fail to do).
>
> We should discourage their use on Windows, and one way to do this is to
> *not* have a nice cross-platform interface to them in Phobos.

Well argued. I'm okay with moving this to a posix-specific module and mentioning it in std.process.

Andrei

On Tuesday, 29 October 2013 at 20:25:16 UTC, Lars T. Kyllingstad wrote:
> Therefore, I would like to suggest a compromise:  I propose we move the functions into an std.posix.process module.  (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)

I think, a more appropriate name is std.native (std.native.process etc). This package would strive to be platform-specific for the sake of efficiency rather than cross-platform. If someone doesn't need to be cross-platform, he needs an efficient interface to the native efficient capabilities of the current platform. That said, the content of std.native modules will be platform-specific and will provide comprehensive D-ified API for the platform features.

On Wednesday, October 30, 2013 17:42:36 Kagamin wrote:
> On Tuesday, 29 October 2013 at 20:25:16 UTC, Lars T. Kyllingstad
> 
> wrote:
> > Therefore, I would like to suggest a compromise:  I propose we move the functions into an std.posix.process module.  (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)
> 
> I think, a more appropriate name is std.native (std.native.process etc). This package would strive to be platform-specific for the sake of efficiency rather than cross-platform. If someone doesn't need to be cross-platform, he needs an efficient interface to the native efficient capabilities of the current platform. That said, the content of std.native modules will be platform-specific and will provide comprehensive D-ified API for the platform features.

Then you have to worry about which functions in a module work on which OS. It's generally a far cleaner separation to separate the OSes via the module system, which is what druntime does for posix and windows.

- Jonathan M Davis

On Tuesday, 29 October 2013 at 20:44:50 UTC, Andrei Alexandrescu wrote:
> I can only assume Windows made a bona fide effort to make their _exec() implementation (http://msdn.microsoft.com/en-us/library/vstudio/431x4c1w.aspx)

Windows does not have exec! This page is part of Microsoft's implementation of the standard C library used by their C compilers, which DMD32 doesn't even use! There is an important distinction between capabilities of an operating system, and the runtime of one implementation of C for that operating system.

On Thursday, 31 October 2013 at 02:25:11 UTC, Jonathan M Davis wrote:
> Then you have to worry about which functions in a module work on which OS.
> It's generally a far cleaner separation to separate the OSes via the module
> system, which is what druntime does for posix and windows.

posix and windows each end up as a soup of OS-specific feature sets too, and you still have to worry about which functions work on which OS. Is it a clear enough separation? And posix doesn't mean native, it means portable: windows implements it to some extent, so one might want to add windows support to posix modules or implement missing posix functions from scratch.

Am Tue, 29 Oct 2013 16:42:08 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 10/29/13 3:07 PM, Lars T. Kyllingstad wrote:
> ...
> > I suspect that these functions were only included in Windows to satisfy POSIX requirements (which they fail to do).
> >
> > We should discourage their use on Windows, and one way to do this is to *not* have a nice cross-platform interface to them in Phobos.
> 
> Well argued. I'm okay with moving this to a posix-specific module and mentioning it in std.process.
> 
> Andrei

That was an interesting discussion. With every new post I was switching sides.

The fact that on Windows it is just emulated with
CreateProcess reminded me how Phobos still uses "emulation"
layers for I/O, namely C files in one or two places.
If at some point Phobos used OS functionality exclusively
instead of D wrappers around C wrappers, we could fix the
O_CLOEXEC issue.

By default Posix keeps file descriptors from the original process open after an exec(), whereas Windows asks you for 'inheritance' at file creation time and when you spawn a subprocess. We had a discussion about it when std.process was ready for review, so I'll cut it short.

C doesn't offer a standard mode flag for file inheritance, so every C library does it differently. If Phobos is to offer a portable file handling, it needs to have platform specific code and default to not inheriting file descriptors.

I think it was Steven, who argued that with everyone using
std.process, no open file handles would ever slip through an
exec() call (since std.process closes them all manually), but
Andrei's almost use of bare bones C exec() shows once again
that the issue is not mitigated.
I had a patch for Phobos in the pipe, that ensures, O_CLOEXEC
is set everywhere a file is opened (e.g. file streams), but I
had to stop at std.stdio, where the C API is exposed in the
form of mode flag strings - which already offer the
non-portable way of setting O_CLOEXEC through different flags
for each C library. So next you would see me argue for a
replacement of zero-terminated mode strings with something
more D-ish and at that point we are in a full blown debate
about a std.stdio rewrite...

So for now just remember to open all files with O_CLOEXEC (available as the "e" mode flag for glibc) on Linux unless you-know-what-you-are-doing™.

-- 
Marco