On Saturday, 9 March 2013 at 18:44:45 UTC, Lars T. Kyllingstad wrote:
> On Saturday, 9 March 2013 at 18:35:25 UTC, Steven Schveighoffer wrote:
>> How do you loop over all open ones?  Just curious :)
>
> You don't.  That is why I said solution (3) sucks too. :)  You have to loop over all possible non-std file descriptors, i.e. from 3 to the maximum number of open files.  (On my Ubuntu installation, this is by default 1024, but may be as much as 4096.  I don't know about other *NIXes)
>
> Here is how to do it:
>
> import core.sys.posix.unistd, core.sys.posix.sys.resource;
> rlimit r;
> getrlimit(RLIMIT_NOFILE, &r);
> for (int i = 0; i < r.rlim_cur; ++i)
>     close(i);

BTW, core.sys.posix.sys.resource currently doesn't exist, we have to create it.

Lars

On Sat, 09 Mar 2013 13:44:44 -0500, Lars T. Kyllingstad <public@kyllingen.net> wrote:

> On Saturday, 9 March 2013 at 18:35:25 UTC, Steven Schveighoffer wrote:

>> How do you loop over all open ones?  Just curious :)
>
> You don't.  That is why I said solution (3) sucks too. :)  You have to loop over all possible non-std file descriptors, i.e. from 3 to the maximum number of open files.  (On my Ubuntu installation, this is by default 1024, but may be as much as 4096.  I don't know about other *NIXes)
>
> Here is how to do it:
>
> import core.sys.posix.unistd, core.sys.posix.sys.resource;
> rlimit r;
> getrlimit(RLIMIT_NOFILE, &r);
> for (int i = 0; i < r.rlim_cur; ++i)
>      close(i);

Hm... don't close 0, 1, 2 :)

On Linux at least, you could use /proc/self/fd  I suppose it's faster just to loop though.

How long does it take when you close non-open descriptors?  We don't want to hamper performance too much.

I wonder if select on all possible file descriptors in the fd_err parameter would give you a clue as to which were invalid.

-Steve

On Sat, 09 Mar 2013 13:54:32 -0500, Steven Schveighoffer <schveiguy@yahoo.com> wrote:


> I wonder if select on all possible file descriptors in the fd_err parameter would give you a clue as to which were invalid.

Hm... seems like select returns -1 if an invalid descriptor is included.  That probably means it won't flag all of them..

-Steve

On Saturday, 9 March 2013 at 18:54:31 UTC, Steven Schveighoffer wrote:
> On Sat, 09 Mar 2013 13:44:44 -0500, Lars T. Kyllingstad <public@kyllingen.net> wrote:
>
>> On Saturday, 9 March 2013 at 18:35:25 UTC, Steven Schveighoffer wrote:
>
>>> How do you loop over all open ones?  Just curious :)
>>
>> You don't.  That is why I said solution (3) sucks too. :)  You have to loop over all possible non-std file descriptors, i.e. from 3 to the maximum number of open files.  (On my Ubuntu installation, this is by default 1024, but may be as much as 4096.  I don't know about other *NIXes)
>>
>> Here is how to do it:
>>
>> import core.sys.posix.unistd, core.sys.posix.sys.resource;
>> rlimit r;
>> getrlimit(RLIMIT_NOFILE, &r);
>> for (int i = 0; i < r.rlim_cur; ++i)
>>     close(i);
>
> Hm... don't close 0, 1, 2 :)
>
> On Linux at least, you could use /proc/self/fd  I suppose it's faster just to loop though.
>
> How long does it take when you close non-open descriptors?  We don't want to hamper performance too much.

On my computer, with 1024 (minus 3 ;))possible file descriptors, it roughly doubles the time spent inside spawnProcess() up to, but not including, the excecve() call.  (About 0.1 microsecond per file descriptor.)  Considering that execve() probebly dwarfs that number, I think we're in good shape.

Of course, we have a problem if some other platform allows ulong.max open files...

> I wonder if select on all possible file descriptors in the fd_err parameter would give you a clue as to which were invalid.

My guess is that select() uses more or less the same mechanism as close() for checking FD validity, and thus would gain us nothing.

Lars

On Sat, 09 Mar 2013 14:24:49 -0500, Lars T. Kyllingstad <public@kyllingen.net> wrote:

> My guess is that select() uses more or less the same mechanism as close() for checking FD validity, and thus would gain us nothing.

Yeah, I was hoping select would flag each "non-valid" file descriptor in fd_errors, and do it all in an internal loop in the kernel instead of going back and forth between user space and kernel space.

But I don't think it does that.  It just errors the whole function on the first invalid descriptor it sees.

Go with your workaround, sounds reasonable.  I agree that 100us pales in comparison to launching a program.  Perhaps an option to disable this behavior should be available in Config.  It certainly is possible with the F_CLOEXEC flag to manually do what needs to be done.

-Steve

On Saturday, 9 March 2013 at 16:05:15 UTC, Lars T. Kyllingstad wrote:
> 1. Make a "special" spawnProcess() function for pipe redirection.
> 2. Use the "process object" approach, like Tango and Qt.
> 3. After fork(), in the child process, loop over the full range of possible file descriptors and close the ones we don't want open.
>
> The last one would let us keep the current API (and would have the added benefit of cleaning up unused FDs) but I have no idea how it would impact performance.

How about this: Set FD_CLOEXEC on all pipes just after creation, but clear the flag for the relevant pipes before exec?

On Sat, 09 Mar 2013 19:51:49 -0500, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:

> On Saturday, 9 March 2013 at 16:05:15 UTC, Lars T. Kyllingstad wrote:
>> 1. Make a "special" spawnProcess() function for pipe redirection.
>> 2. Use the "process object" approach, like Tango and Qt.
>> 3. After fork(), in the child process, loop over the full range of possible file descriptors and close the ones we don't want open.
>>
>> The last one would let us keep the current API (and would have the added benefit of cleaning up unused FDs) but I have no idea how it would impact performance.
>
> How about this: Set FD_CLOEXEC on all pipes just after creation, but clear the flag for the relevant pipes before exec?

This doesn't help if other threads are randomly opening file descriptors.  That is a problem I don't think we considered.

Unix's design here is very outdated, seems to assume a single threaded app.

This does make me thing of another good point, we should unset the FD_CLOEXEC flag on stdout, stdin, and stderr!

-Steve

On Sat, 09 Mar 2013 11:05:14 -0500, Lars T. Kyllingstad <public@kyllingen.net> wrote:

> On Wednesday, 6 March 2013 at 16:45:51 UTC, Steven Schveighoffer wrote:
>> On Tue, 05 Mar 2013 17:38:09 -0500, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
>>
>>> By the way, I should mention that I ran into several issues while trying to come up with the above example. The test program does not work on Windows, for some reason I get the exception:
>>>
>>> std.process2.ProcessException@std\process2.d(494): Failed to spawn new process (The parameter is incorrect.)
>>
>> I think Lars is on that.
>
> I'm going to need som help with this one.  I only have Linux on my computer, and I can't reproduce the bug in Wine.

Tried as well.  I have only a 32-bit license for Windows XP, so I don't have a 64-bit VM to test with (this is not wine, but vmware, should be exactly the same as running on a real windows box).  I gave away my Windows 7 64-bit box :(

Anyway, on 32-bit XP I get a successful run:

100 lines of stdout, 100 lines of stderr

I can possibly try it on a laptop from work.  But not until Monday.

So it probably is a 64-bit-only issue.  I know you just added this part Lars, and it uses microsoft's runtime.  Very different from the DMC runtime.  But both should use the same OS call.  Will take a closer look at the code around that line.

Vladimir, can you try compiling 32-bit windows and see if it works for you, just to confirm?

-Steve

> So it probably is a 64-bit-only issue.  I know you just added this part Lars, and it uses microsoft's runtime.  Very different from the DMC runtime.  But both should use the same OS call.  Will take a closer look at the code around that line.
>

Looks fine to me.

Doing some searching online, the overwhelming results for that specific error are for failed ANT build or java execution due to a huge class path (which I think is either passed on the command line or via environment?)

Try shortening your executable path/environment?

Just a guess...

-Steve

On Sunday, 10 March 2013 at 02:54:44 UTC, Steven Schveighoffer wrote:
> Vladimir, can you try compiling 32-bit windows and see if it works for you, just to confirm?

I'm seeing the same exception with both 32 and 64 bit, Steven.

I guess it must be something specific to my system, like the HANDLE_FLAG_INHERIT stuff.