Thread overview
[DConf] Ali's talk: parallelism bug is caught by ThreadSanitizer
3 days ago
Johan
2 days ago
Walter Bright
2 days ago
Ali Çehreli
3 days ago
Hi all, Ali,
  Thanks Ali for your talk on DConf Online 2020, it was nice to watch.

The std.parallelism.parallel example [1] immediately triggered me. I had to stop the video, and see whether ThreadSanitizer (TSan) shipped with LDC 1.24 can catch it. It does!

Testcase:
```
import std.parallelism;

void main() {
    // defaultPoolThreads(4); // [2]

    int[] elements = [0, 1, 2, 3];
    int[] results;
    foreach (e; elements.parallel) {  // test.d:8
        results ~= e;                 // test.d:9
    }
}
```

Let's compile it with ThreadSanitizer enabled and run it!
```
❯ bin/ldc2 -g -fsanitize=thread -run test.d
==================
WARNING: ThreadSanitizer: data race (pid=19606)
  Read of size 1 at 0x7f311b105010 by thread T7:
    #0 memcpy <null> (libtsan.so.0+0x32505)
    #1 _d_arrayappendcTX <null> (test-b40741e-06cb93+0x5fe76)
    #2 _D3std11parallelism__T15ParallelForeachTAiZQv7opApplyMFMDFKiZiZ4doItMFZv phobos/std/parallelism.d-mixin-4048:4094 (test-b40741e-06cb93+0x2886e)
    #3 _D3std11parallelism8TaskPool15executeWorkLoopMFZv <null> (test-b40741e-06cb93+0x293af)

  Previous write of size 4 at 0x7f311b105010 by thread T6:
    #0 _D4test4mainFZ14__foreachbody1MFKiZi test.d:9 (test-b40741e-06cb93+0x28284)
    #1 _D3std11parallelism__T15ParallelForeachTAiZQv7opApplyMFMDFKiZiZ4doItMFZv phobos/std/parallelism.d-mixin-4048:4094 (test-b40741e-06cb93+0x2886e)
    #2 _D3std11parallelism8TaskPool15executeWorkLoopMFZv <null> (test-b40741e-06cb93+0x293af)

  Thread T7 (tid=19614, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 _D4core6thread8osthread6Thread5startMFNbZCQBoQBmQBiQBc <null> (test-b40741e-06cb93+0x5508b)
    #2 _Dmain test.d:8 (test-b40741e-06cb93+0x27f97)
    #3 _D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv <null> (test-b40741e-06cb93+0x5aeeb)
    #4 __libc_start_main <null> (libc.so.6+0x21b96)

  Thread T6 (tid=19613, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 _D4core6thread8osthread6Thread5startMFNbZCQBoQBmQBiQBc <null> (test-b40741e-06cb93+0x5508b)
    #2 _Dmain test.d:8 (test-b40741e-06cb93+0x27f97)
    #3 _D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv <null> (test-b40741e-06cb93+0x5aeeb)
    #4 __libc_start_main <null> (libc.so.6+0x21b96)

SUMMARY: ThreadSanitizer: data race (/usr/lib/x86_64-linux-gnu/libtsan.so.0+0x32505) in memcpy
==================
==================
WARNING: ThreadSanitizer: data race (pid=19606)
  Read of size 1 at 0x7f311b105010 by thread T4:
    #0 memcpy <null> (libtsan.so.0+0x32505)
    #1 _d_arrayappendcTX <null> (test-b40741e-06cb93+0x5fe76)
    #2 _D3std11parallelism__T15ParallelForeachTAiZQv7opApplyMFMDFKiZiZ4doItMFZv phobos/std/parallelism.d-mixin-4048:4094 (test-b40741e-06cb93+0x2886e)
    #3 _D3std11parallelism8TaskPool15executeWorkLoopMFZv <null> (test-b40741e-06cb93+0x293af)

  Previous write of size 4 at 0x7f311b105010 by thread T6:
    #0 _D4test4mainFZ14__foreachbody1MFKiZi test.d:9 (test-b40741e-06cb93+0x28284)
    #1 _D3std11parallelism__T15ParallelForeachTAiZQv7opApplyMFMDFKiZiZ4doItMFZv phobos/std/parallelism.d-mixin-4048:4094 (test-b40741e-06cb93+0x2886e)
    #2 _D3std11parallelism8TaskPool15executeWorkLoopMFZv <null> (test-b40741e-06cb93+0x293af)

  Thread T4 (tid=19611, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 _D4core6thread8osthread6Thread5startMFNbZCQBoQBmQBiQBc <null> (test-b40741e-06cb93+0x5508b)
    #2 _Dmain test.d:8 (test-b40741e-06cb93+0x27f97)
    #3 _D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv <null> (test-b40741e-06cb93+0x5aeeb)
    #4 __libc_start_main <null> (libc.so.6+0x21b96)

  Thread T6 (tid=19613, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 _D4core6thread8osthread6Thread5startMFNbZCQBoQBmQBiQBc <null> (test-b40741e-06cb93+0x5508b)
    #2 _Dmain test.d:8 (test-b40741e-06cb93+0x27f97)
    #3 _D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv <null> (test-b40741e-06cb93+0x5aeeb)
    #4 __libc_start_main <null> (libc.so.6+0x21b96)

SUMMARY: ThreadSanitizer: data race (/usr/lib/x86_64-linux-gnu/libtsan.so.0+0x32505) in memcpy
==================
ThreadSanitizer: reported 2 warnings
Error: /tmp/johan/test-b40741e-06cb93 failed with status: 66
```

I tested this using WSL2, the Linux environment available on Windows 10. I think TSan should be supported on Windows and macOS too, but I did not test this example.

Cheers,
  Johan


[1] https://www.youtube.com/watch?v=dRORNQIB2wA&feature=youtu.be&list=PLIldXzSkPUXWsvFA4AuawPoMnq9Bh1lIZ&t=1471
[2] Use this line of code if by default only a single thread is available in the pool (e.g. in a container).
2 days ago
On 11/22/2020 4:49 AM, Johan wrote:
> The std.parallelism.parallel example [1] immediately triggered me. I had to stop the video, and see whether ThreadSanitizer (TSan) shipped with LDC 1.24 can catch it. It does!

Excellent!
2 days ago
On 11/22/20 4:49 AM, Johan wrote:

> The std.parallelism.parallel example [1] immediately triggered me. I had
> to stop the video, and see whether ThreadSanitizer (TSan) shipped with
> LDC 1.24 can catch it. It does!

I love it! I am happy that for once I knew that my code on the slides had a bug. Phew... :)

Ali