With a small number of threads, things work as intended in the code below. But with 1000, on my machine it either crashes or throws an exception:


import std.stdio;
import std.parallelism;
import std.range;


void main() {
    stdout = File("/dev/null", "w");
    foreach(t; 1000.iota.parallel) {
        writeln("Oops");
    }
}



I get, depending on the run, "Bad file descriptor", "Attempting to write to a closed file", or segfaults. What am I doing wrong?

Atila

On Saturday, 5 March 2016 at 14:18:31 UTC, Atila Neves wrote:
> With a small number of threads, things work as intended in the code below. But with 1000, on my machine it either crashes or throws an exception:
>
>
> import std.stdio;
> import std.parallelism;
> import std.range;
>
>
> void main() {
>     stdout = File("/dev/null", "w");
>     foreach(t; 1000.iota.parallel) {
>         writeln("Oops");
>     }
> }
>
>
>
> I get, depending on the run, "Bad file descriptor", "Attempting to write to a closed file", or segfaults. What am I doing wrong?
>
> Atila

Could this be a bug in phobos/compiler?

Am Sat, 05 Mar 2016 14:18:31 +0000
schrieb Atila Neves <atila.neves@gmail.com>:

> void main() {
>      stdout = File("/dev/null", "w");
>      foreach(t; 1000.iota.parallel) {
>          writeln("Oops");
>      }
> }

First thing I tried:

void main() {
     stdout = File("/dev/null", "w");
     foreach(t; 1000.iota.parallel) {
         stdout.writeln("Oops");
     }
}

That does NOT segfault ... hmm.

-- 
Marco

On Saturday, 5 March 2016 at 14:18:31 UTC, Atila Neves wrote:
> With a small number of threads, things work as intended in the code below. But with 1000, on my machine it either crashes or throws an exception:
>
>
> import std.stdio;
> import std.parallelism;
> import std.range;
>
>
> void main() {
>     stdout = File("/dev/null", "w");
>     foreach(t; 1000.iota.parallel) {
>         writeln("Oops");
>     }
> }

Note that `1000.iota.parallel` does *not* run 1000 threads. `parallel` just splits the work of the range up between the worker threads (likely 2, 4, or 8, depending on your CPU). I see the effect you describe with any parallel workload. Smaller numbers in place of 1000 aren't necessarily splitting things off to additional threads, which is why smaller numbers avoid the multi-threaded problems you are encountering.

> I get, depending on the run, "Bad file descriptor", "Attempting to write to a closed file", or segfaults. What am I doing wrong?
>
> Atila

`File` uses ref-counting internally to allow it to auto-close. `stdout` and friends are initialized in a special way such that they have a high initial ref-count. When you assign a new file to stdout, the ref count becomes one. As soon as one of your threads exits, this will cause stdout to close, producing the odd errors you are encountering on all the other threads.

I would avoid reassigning `stdout` and friends in favor of using a logger or manually specifying the file to write to if I were you.

Got it now: https://issues.dlang.org/show_bug.cgi?id=15768

writeln() creates a copy of the stdout struct in a non thread-safe way. If stdout has been assigned a File struct created from a file name this copy includes a "racy" increment/decrement of a reference count to the underlying C-library FILE*. In the case that the reference count is erroneously reaching 0, the file is closed prematurely and when Glibc tries to access internal data it results in the observable SIGSEGV.

-- 
Marco

Am Sun, 06 Mar 2016 01:10:58 +0000
schrieb Anon <anon@anon.anon>:

> I would avoid reassigning `stdout` and friends in favor of using a logger or manually specifying the file to write to if I were you.

Meh. Too little drama. :p

-- 
Marco

On Sunday, 6 March 2016 at 01:10:58 UTC, Anon wrote:
> On Saturday, 5 March 2016 at 14:18:31 UTC, Atila Neves wrote:
>> [...]
>
> Note that `1000.iota.parallel` does *not* run 1000 threads. `parallel` just splits the work of the range up between the worker threads (likely 2, 4, or 8, depending on your CPU). I see the effect you describe with any parallel workload. Smaller numbers in place of 1000 aren't necessarily splitting things off to additional threads, which is why smaller numbers avoid the multi-threaded problems you are encountering.

Err, right.

>
>> [...]
>
> `File` uses ref-counting internally to allow it to auto-close. `stdout` and friends are initialized in a special way such that they have a high initial ref-count. When you assign a new file to stdout, the ref count becomes one. As soon as one of your threads exits, this will cause stdout to close, producing the odd errors you are encountering on all the other threads.
>
> I would avoid reassigning `stdout` and friends in favor of using a logger or manually specifying the file to write to if I were you.

I see. Here's my problem: I want to make it so code not under my control doesn't get to write to stdout and stderr. I don't see any other way but to reassign stdout. Maybe I can manually bump up the ref count?

Atila

On Sunday, 6 March 2016 at 01:28:52 UTC, Marco Leise wrote:
> Got it now: https://issues.dlang.org/show_bug.cgi?id=15768
>
> writeln() creates a copy of the stdout struct in a non thread-safe way. If stdout has been assigned a File struct created from a file name this copy includes a "racy" increment/decrement of a reference count to the underlying C-library FILE*. In the case that the reference count is erroneously reaching 0, the file is closed prematurely and when Glibc tries to access internal data it results in the observable SIGSEGV.

Nice, good work!

Atila