Hey,

This code is from std.process:

        if (!(config & Config.inheritFDs))
        {
            import core.sys.posix.sys.resource;
            rlimit r;
            getrlimit(RLIMIT_NOFILE, &r);
            foreach (i; 3 .. cast(int) r.rlim_cur) close(i);
        }

This is a close-loop to make sure no fds are leaked before execve'ing, and can be disabled using Config.inheritFDs.

Here's a program I used to profile:

void main() {
    StopWatch sw;

    import core.sys.posix.sys.resource;
    rlimit r;
    errnoEnforce(getrlimit(RLIMIT_NOFILE, &r) == 0);
    writefln("The rlimit is %d (hard: %s)", r.rlim_cur, r.rlim_max);
    //r.rlim_cur = 1024 * 1024;
    //errnoEnforce(setrlimit(RLIMIT_NOFILE, &r) == 0);

    sw.start();
    foreach (_; 0 .. 100) {
        executeShell("echo hello", null, std.process.Config.inheritFDs);
    }
    sw.stop();

    writefln("Total time: %s", sw.peek().to!Duration.to!string);
}

These are the results I got:

// On a typical linux machine, without inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 3 secs, 875 ms, 759 μs, and 7 hnsecs

// with inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 80 ms, 549 μs, and 2 hnsecs

// osx, without inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 476 ms, 483 μs, and 5 hnsecs

// with inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 352 ms, 637 μs, and 7 hnsecs

So obviously this becomes a problem if the rlimit is high.

A few suggestions:
1. Make inheritFDs the default, since most people aren't aware of the performance cost. Also most programs will run just fine if a few fds are leaked.
1.1. Or, by default, close up to min(rlim_cur, 1024) - this should be enough for most processes. (If I'm not mistaken, that's what they do in python). Also this change won't break existing code.
2. All phobos fds should be opened with O_CLOEXEC, as intended. This eliminates the need to close all the fds when execve'ing.
3. Optimization for linux - use /proc to check which fds are actually opened, and close them, instead of making thousands of syscalls!

On Monday, 3 July 2017 at 14:48:38 UTC, Jonathan Shamir wrote:
>         if (!(config & Config.inheritFDs))
>         {
>             import core.sys.posix.sys.resource;
>             rlimit r;
>             getrlimit(RLIMIT_NOFILE, &r);
>             foreach (i; 3 .. cast(int) r.rlim_cur) close(i);
>         }

That code is pretty old. std.process was changed to use poll when available over a year ago:

https://github.com/dlang/phobos/pull/4114

Update your D installation?

On 07/03/2017 07:53 AM, Vladimir Panteleev wrote:

> That code is pretty old. std.process was changed to use poll when
> available over a year ago:
>
> https://github.com/dlang/phobos/pull/4114
>
> Update your D installation?

Jonathan is hailing from Weka. :) As evidenced from Johan's recent messages, they're in the process of updating their ldc.

Ali

On Monday, 3 July 2017 at 20:34:00 UTC, Ali Çehreli wrote:
> On 07/03/2017 07:53 AM, Vladimir Panteleev wrote:
>
> > That code is pretty old. std.process was changed to use poll
> when
> > available over a year ago:
> >
> > https://github.com/dlang/phobos/pull/4114
> >
> > Update your D installation?
>
> Jonathan is hailing from Weka. :) As evidenced from Johan's recent messages, they're in the process of updating their ldc.
>
> Ali

But then what's the point of this thread?