So I tried using unit-threaded to run Phobos unit tests again and had problems (which I'll look into later) with its compile-time reflection. Then I realised I was an idiot since I don't need to reflect on anything: all Phobos tests are in unittest blocks so all I need to do is include them in the build and unit-threaded will run them for me.

I tried a basic sanity check by running them in one thread only with the -s option and got a segfault, and a failing test before that. None of this should happen, and I'll be taking a look at that as well.

But I carried on by removing the troublesome modules from the build. These turned out to be:

std.datetime (fails)
std.process (fails and causes the segfault)
std.stdio (fails)

All the others pass in single threaded mode. After this I tried using threads and std.parallelism failed, so I took that away from the build as well.

Another thing to mention is that although the tests are running in threads, since when I wrote the library the getUnitTests __traits wasn't available (and since then I wasn't interested in using it), each module's unit tests run as one test. So they only interleave with other modules, not with each other.

Running in one thread took 39 +/- 1 seconds.
Running in 8 threads took... ~41 seconds.

Oops. I noticed some tests take a lot longer so I tried removing those. They were:

std.file
std.conv
std.regex
std.random
std.container
std.xml
std.utf
std.numeric
std.uuid
std.exception

I also removed any modules that were likely to be problematic like std.concurrency and std.socket. With the reduced sample size the results were:

1 thread: ~1.9s
8 threads: 4.1s +/- 0.2

So the whole threading thing isn't looking so great. Or at least not how I implemented it. This got me thinking about my own projects. The tests run so fast I never really paid attention to how fast they were running. I compared running the unit tests in Cerealed in one or more threads and got the same result: running in one thread was faster.

I have to look to be sure but maybe the bottleneck is output. As in actually printing the results to the screen. I had to jump through a few hoops to make sure the output wasn't interleaved, and in the end decided to have one thread be responsible for that, with the tests sending it output messages.

For reference, I copied all of the std/*.d modules into a local std directory, compiled all of them with dmd -unittest -c, then used this as the build command:

dmd -unittest -I~/coding/d/unit-threaded/source ut.d std/algorithm.o std/array.o std/ascii.o std/base64.o std/bigint.o std/bitmanip.o std/compiler.o std/complex.o std/container.o std/cstream.o std/csv.o std/demangle.o std/encoding.o std/format.o std/functional.o std/getopt.o std/json.o std/math.o std/mathspecial.o std/metastrings.o std/mmfile.o std/numeric.o std/outbuffer.o std/range.o  std/signals.o  std/stdint.o std/stdiobase.o std/stream.o std/string.o std/syserror.o std/system.o std/traits.o std/typecons.o std/typelist.o std/typetuple.o std/uri.o std/variant.o std/zip.o std/zlib.o  libunit-threaded.a -ofphobos_ut

I got libunit-threaded.a by running "dub build" in the root directory of unit-threaded.

I might just implement a random order option now. Hmm.

Atila

I turned off all output to check. It was still slower with multiple threads. That was the only "weird" thing I was doing I could think of as the cause. Otherwise it's just a foreach(test; tests.parallel) { test(); }.

Atila

On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
> So I tried using unit-threaded to run Phobos unit tests again and had problems (which I'll look into later) with its compile-time reflection. Then I realised I was an idiot since I don't need to reflect on anything: all Phobos tests are in unittest blocks so all I need to do is include them in the build and unit-threaded will run them for me.
>
> I tried a basic sanity check by running them in one thread only with the -s option and got a segfault, and a failing test before that. None of this should happen, and I'll be taking a look at that as well.
>
> But I carried on by removing the troublesome modules from the build. These turned out to be:
>
> std.datetime (fails)
> std.process (fails and causes the segfault)
> std.stdio (fails)
>
> All the others pass in single threaded mode. After this I tried using threads and std.parallelism failed, so I took that away from the build as well.
>
> Another thing to mention is that although the tests are running in threads, since when I wrote the library the getUnitTests __traits wasn't available (and since then I wasn't interested in using it), each module's unit tests run as one test. So they only interleave with other modules, not with each other.
>
> Running in one thread took 39 +/- 1 seconds.
> Running in 8 threads took... ~41 seconds.
>
> Oops. I noticed some tests take a lot longer so I tried removing those. They were:
>
> std.file
> std.conv
> std.regex
> std.random
> std.container
> std.xml
> std.utf
> std.numeric
> std.uuid
> std.exception
>
> I also removed any modules that were likely to be problematic like std.concurrency and std.socket. With the reduced sample size the results were:
>
> 1 thread: ~1.9s
> 8 threads: 4.1s +/- 0.2
>
> So the whole threading thing isn't looking so great. Or at least not how I implemented it. This got me thinking about my own projects. The tests run so fast I never really paid attention to how fast they were running. I compared running the unit tests in Cerealed in one or more threads and got the same result: running in one thread was faster.
>
> I have to look to be sure but maybe the bottleneck is output. As in actually printing the results to the screen. I had to jump through a few hoops to make sure the output wasn't interleaved, and in the end decided to have one thread be responsible for that, with the tests sending it output messages.
>
> For reference, I copied all of the std/*.d modules into a local std directory, compiled all of them with dmd -unittest -c, then used this as the build command:
>
> dmd -unittest -I~/coding/d/unit-threaded/source ut.d std/algorithm.o std/array.o std/ascii.o std/base64.o std/bigint.o std/bitmanip.o std/compiler.o std/complex.o std/container.o std/cstream.o std/csv.o std/demangle.o std/encoding.o std/format.o std/functional.o std/getopt.o std/json.o std/math.o std/mathspecial.o std/metastrings.o std/mmfile.o std/numeric.o std/outbuffer.o std/range.o  std/signals.o  std/stdint.o std/stdiobase.o std/stream.o std/string.o std/syserror.o std/system.o std/traits.o std/typecons.o std/typelist.o std/typetuple.o std/uri.o std/variant.o std/zip.o std/zlib.o  libunit-threaded.a -ofphobos_ut
>
> I got libunit-threaded.a by running "dub build" in the root directory of unit-threaded.
>
> I might just implement a random order option now. Hmm.
>
> Atila

On Saturday, 3 May 2014 at 12:08:56 UTC, Atila Neves wrote:
> I turned off all output to check. It was still slower with multiple threads. That was the only "weird" thing I was doing I could think of as the cause. Otherwise it's just a foreach(test; tests.parallel) { test(); }.
>
> Atila
>
> On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
>> So I tried using unit-threaded to run Phobos unit tests again and had problems (which I'll look into later) with its compile-time reflection. Then I realised I was an idiot since I don't need to reflect on anything: all Phobos tests are in unittest blocks so all I need to do is include them in the build and unit-threaded will run them for me.
>>
>> I tried a basic sanity check by running them in one thread only with the -s option and got a segfault, and a failing test before that. None of this should happen, and I'll be taking a look at that as well.
>>
>> But I carried on by removing the troublesome modules from the build. These turned out to be:
>>
>> std.datetime (fails)
>> std.process (fails and causes the segfault)
>> std.stdio (fails)
>>
>> All the others pass in single threaded mode. After this I tried using threads and std.parallelism failed, so I took that away from the build as well.
>>
>> Another thing to mention is that although the tests are running in threads, since when I wrote the library the getUnitTests __traits wasn't available (and since then I wasn't interested in using it), each module's unit tests run as one test. So they only interleave with other modules, not with each other.
>>
>> Running in one thread took 39 +/- 1 seconds.
>> Running in 8 threads took... ~41 seconds.
>>
>> Oops. I noticed some tests take a lot longer so I tried removing those. They were:
>>
>> std.file
>> std.conv
>> std.regex
>> std.random
>> std.container
>> std.xml
>> std.utf
>> std.numeric
>> std.uuid
>> std.exception
>>
>> I also removed any modules that were likely to be problematic like std.concurrency and std.socket. With the reduced sample size the results were:
>>
>> 1 thread: ~1.9s
>> 8 threads: 4.1s +/- 0.2
>>
>> So the whole threading thing isn't looking so great. Or at least not how I implemented it. This got me thinking about my own projects. The tests run so fast I never really paid attention to how fast they were running. I compared running the unit tests in Cerealed in one or more threads and got the same result: running in one thread was faster.
>>
>> I have to look to be sure but maybe the bottleneck is output. As in actually printing the results to the screen. I had to jump through a few hoops to make sure the output wasn't interleaved, and in the end decided to have one thread be responsible for that, with the tests sending it output messages.
>>
>> For reference, I copied all of the std/*.d modules into a local std directory, compiled all of them with dmd -unittest -c, then used this as the build command:
>>
>> dmd -unittest -I~/coding/d/unit-threaded/source ut.d std/algorithm.o std/array.o std/ascii.o std/base64.o std/bigint.o std/bitmanip.o std/compiler.o std/complex.o std/container.o std/cstream.o std/csv.o std/demangle.o std/encoding.o std/format.o std/functional.o std/getopt.o std/json.o std/math.o std/mathspecial.o std/metastrings.o std/mmfile.o std/numeric.o std/outbuffer.o std/range.o  std/signals.o  std/stdint.o std/stdiobase.o std/stream.o std/string.o std/syserror.o std/system.o std/traits.o std/typecons.o std/typelist.o std/typetuple.o std/uri.o std/variant.o std/zip.o std/zlib.o  libunit-threaded.a -ofphobos_ut
>>
>> I got libunit-threaded.a by running "dub build" in the root directory of unit-threaded.
>>
>> I might just implement a random order option now. Hmm.
>>
>> Atila

Out of curiosity are you on Windows?

> Out of curiosity are you on Windows?

No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day.

Atila

On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote:
>> Out of curiosity are you on Windows?
>
> No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day.
>
> Atila

I'm surprised. Threads should be cheap on Linux. Something funky is definitely going on I bet.

Ok, so I went and added __traits(getUnitTests) to unit-threaded. That way each unittest block is its own test case. I registered these modules in std to run:

array, ascii, base64, bigint, bitmanip, concurrency, container, cstream.

On the good news front, they all passed even though they were running concurrently.

On the bad news front, single-threaded operation was still faster (0.22s vs 0.28s). I still don't know why.

I fixed my concurrency bug, now I'm using taskPool.amap.


Atila

I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing.

The Phobos modules are all called "ustd" because I couldn't/didn't know how to get this to work otherwise. So I copied the std/*.d files to a directory called ustd and changed their module declarations. Silly but it works. I'd love to know how to do this properly.

With this file, I consistenly get faster times with "-s" (for single-threaded) than without (multi-threaded):


import std.parallelism;
import std.getopt;


import ustd.array;
import ustd.ascii;
import ustd.base64;
import ustd.bigint;
import ustd.bitmanip;
import ustd.concurrency;
import ustd.container;
import ustd.cstream;


alias TestFunction = void function();

auto getTests(Modules...)() {
    TestFunction[] tests;
    foreach(mod; Modules) {
        foreach(test; __traits(getUnitTests, mod)) {
            tests ~= &test;
        }
    }
    return tests;
}



void main(string[] args) {
    bool single;
    getopt(args,
           "single|s", &single
        );

    enum tests = getTests!(
        ustd.array,
        ustd.ascii,
        ustd.base64,
        ustd.bigint,
        ustd.bitmanip,
        ustd.concurrency,
        ustd.container,
        ustd.cstream,
        );

    if(single) {
        foreach(test; tests) {
            test();
        }
    } else {
        foreach(test; tests.parallel) {
            test();
        }
    }
}

On 5/3/14, 4:54 AM, Atila Neves wrote:
> So I tried using unit-threaded to run Phobos unit tests
[snip]

Thanks. Are you using thread pooling (a limited number of threads e.g. 1.5 * cores running all unittests)? -- Andrei

On 5/3/2014 5:26 AM, Rikki Cattermole wrote:
> Something funky is definitely going on I bet.

No doubt: http://www.youtube.com/watch?v=aZcbDESaxhY

On 5/3/2014 10:22 AM, Atila Neves wrote:
> I can reproduce the slower-with-threads issue without using my library. I've
> included the source file below and would like to know if other people see the
> same thing.

I haven't investigated this, but my suspicions are:

1. thread creation/destruction is dominating the times.

2. since very few of the unittests block, there is no speed advantage from having more threads than cores. If you limit the number of threads to the number of cores on your machine, you might see a speedup.