Hi guys,

I wrote a toy benchmark in C++ [1] and tried to literally translate it to D [2].

The results are quite disappointing. What seems particularly strange to me is that -boundscheck=off leads to a performance decrease.

Am I missing anything?

// $ clang++ -std=c++1z -O3 cmp-bench.cpp
// $ time ./a.out
// 501
//
// real 0m2.777s
// user 0m2.774s
// sys  0m0.004s
//
//
// $ clang++ --version
// clang version 4.0.0 (tags/RELEASE_400/final)
// Target: x86_64-unknown-linux-gnu
// Thread model: posix
// InstalledDir: /usr/bin

// $ ldc2 -O3 -release cmp-bench.d
// $ time ./cmp-bench
// 501
//
// real	0m5.257s
// user	0m5.224s
// sys	0m0.000s
//
//
// $ ldc2 -O3 -release -boundscheck=off cmp-bench.d
// $ time ./cmp-bench
// 501
//
// real	0m11.083s
// user	0m11.083s
// sys	0m0.000s
//
//
// $ ldc2 --version
// LDC - the LLVM D compiler (1.2.0):
//   based on DMD v2.072.2 and LLVM 4.0.0
//   built with DMD64 D Compiler v2.074.0
//   Default target: x86_64-unknown-linux-gnu
//   Host CPU: haswell

[1] C++ Insertion Sort - https://dpaste.dzfl.pl/74fdb92a0579
[2] D Insertion Sort   - https://dpaste.dzfl.pl/b97a1fca1546

On 6/9/17 12:21 PM, Honey wrote:
> Hi guys,
>
> I wrote a toy benchmark in C++ [1] and tried to literally translate it
> to D [2].

Wow, so that's how D code would look like if it were C++ :)

>
> The results are quite disappointing. What seems particularly strange to
> me is that -boundscheck=off leads to a performance decrease.

That doesn't make much sense, but I'm not an ldc2 user. However, it does note in the help that -release disables bounds checks already.

I did replicate that issue on my box, and mucking around with the implementation didn't help.

In answer to the subject, no D is not slow. However, it's quite possible that std.algorithm.bringToFront is slower than std::rotate, or SortedRange.upperBound is slower than std::upper_bound, or both. I don't think it's a design issue per se, probably more of an implementation issue.

-Steve

On 06/09/2017 11:32 AM, Steven Schveighoffer wrote:

> In answer to the subject, no D is not slow. However, it's quite possible
> that std.algorithm.bringToFront is slower than std::rotate, or
> SortedRange.upperBound is slower than std::upper_bound, or both.

That was exactly my conclusion. I found this C++ article which may point at a similar implementation issue in Phobos:

  https://stackoverflow.com/questions/21160875/why-is-stdrotate-so-fast

Ali

On 6/9/17 2:32 PM, Steven Schveighoffer wrote:

> In answer to the subject, no D is not slow. However, it's quite possible
> that std.algorithm.bringToFront is slower than std::rotate, or
> SortedRange.upperBound is slower than std::upper_bound, or both. I don't
> think it's a design issue per se, probably more of an implementation issue.

https://issues.dlang.org/show_bug.cgi?id=17485

-Steve

On Friday, 9 June 2017 at 18:32:06 UTC, Steven Schveighoffer wrote:
> Wow, so that's how D code would look like if it were C++ :)

Well, I cannot (and did not try to) hide where I am coming from. ;-)


>> The results are quite disappointing. What seems particularly strange to
>> me is that -boundscheck=off leads to a performance decrease.
>
> That doesn't make much sense, but I'm not an ldc2 user. However, it does note in the help that -release disables bounds checks already.

Sounds like a bug, then.


> I did replicate that issue on my box, and mucking around with the implementation didn't help.
>
> In answer to the subject, no D is not slow. However, it's quite possible that std.algorithm.bringToFront is slower than std::rotate, or SortedRange.upperBound is slower than std::upper_bound, or both. I don't think it's a design issue per se, probably more of an implementation issue.

Thank you for confirming the results and your factual explanation notwithstanding my pointed question. ;-)

Maybe I was expecting too much given Andrei's performance oriented talks. I realize that the conceptual groundwork is more important than a concrete implementation that can be easily improved. However, I think that real world out-of-the-box performance - particularly with respect to toy examples (since those are small enough to be literally translated) - is important for prospects to gain confidence in buying into D.

At the current state, at least for such benchmarks, I think, I should not rely on standard library facilities. Unfortunately, that does not increase my confidence.

On Friday, 9 June 2017 at 19:29:35 UTC, Honey wrote:
> Unfortunately, that does not increase my confidence.

I should add that, nevertheless, I very much appreciate and respect D and its community. Great work! :-)

On Friday, 9 June 2017 at 18:32:06 UTC, Steven Schveighoffer wrote:
> Wow, so that's how D code would look like if it were C++ :)

 When dipping my toes into C++ to do a quicksort algorithm, I quickly got annoyed I'd have to create all the individual comparison functions rather than just one like in D... Which is one thing I'm seeing from the converted 'toy'. Actually I ended up making an opCmp and then overloading all the individual types to use the opCmp.

On Friday, 9 June 2017 at 19:29:35 UTC, Honey wrote:
> On Friday, 9 June 2017 at 18:32:06 UTC, Steven Schveighoffer wrote:
>> Wow, so that's how D code would look like if it were C++ :)
>
> Well, I cannot (and did not try to) hide where I am coming from. ;-)
>
>
>>> The results are quite disappointing. What seems particularly strange to
>>> me is that -boundscheck=off leads to a performance decrease.
>>
>> That doesn't make much sense, but I'm not an ldc2 user. However, it does note in the help that -release disables bounds checks already.
>
> Sounds like a bug, then.
>
>
>> I did replicate that issue on my box, and mucking around with the implementation didn't help.
>>
>> In answer to the subject, no D is not slow. However, it's quite possible that std.algorithm.bringToFront is slower than std::rotate, or SortedRange.upperBound is slower than std::upper_bound, or both. I don't think it's a design issue per se, probably more of an implementation issue.
>
> Thank you for confirming the results and your factual explanation notwithstanding my pointed question. ;-)
>
> Maybe I was expecting too much given Andrei's performance oriented talks. I realize that the conceptual groundwork is more important than a concrete implementation that can be easily improved. However, I think that real world out-of-the-box performance - particularly with respect to toy examples (since those are small enough to be literally translated) - is important for prospects to gain confidence in buying into D.
>
> At the current state, at least for such benchmarks, I think, I should not rely on standard library facilities. Unfortunately, that does not increase my confidence.

Real world and toy are mutually exclusive categories, and I am not sure the empirical evidence is consistent with your perspective that it is what prospects need to see before exploring D, though that is an interesting perspective.   I highly recommend Weka.io talks if you would like to see how one larger D user has found performance in practice.

If you are expecting a perfectly finished glossy product then I don't think that - at least in the current year - D will be necessarily for you.  Polish improves every year, but it's not the principal focus of the community currently.   It's more the opposite - pay the price up front in different ways and reap the returns again and again over time.

On 06/09/2017 12:29 PM, Honey wrote:

> I think, I should not rely on standard library facilities.

I think you hit a Phobos function with particularly bad performance (which can be improved). Phobos is not a slow library in general.

Ali

On 6/9/17 3:29 PM, Honey wrote:
> On Friday, 9 June 2017 at 18:32:06 UTC, Steven Schveighoffer wrote:
>> Wow, so that's how D code would look like if it were C++ :)
>
> Well, I cannot (and did not try to) hide where I am coming from. ;-)

hehe this was meant more as a dig at C++ syntax and not you, I totally understand the reason you did it that way.

Just to show you what I meant, I changed your code to eliminate the functors completely, the main function now looks like this:


    foreach (i;  0 .. N)
    {
        insertionSort!((a, b) => lt(a, b))(v);
        insertionSort!((a, b) => lt(b, a))(v);
    }

I'm sure there's also a way to reduce the initialization of the array to a few lines (or maybe even one?), but didn't have time to think about it.

>> I did replicate that issue on my box, and mucking around with the
>> implementation didn't help.
>>
>> In answer to the subject, no D is not slow. However, it's quite
>> possible that std.algorithm.bringToFront is slower than std::rotate,
>> or SortedRange.upperBound is slower than std::upper_bound, or both. I
>> don't think it's a design issue per se, probably more of an
>> implementation issue.
>
> Thank you for confirming the results and your factual explanation
> notwithstanding my pointed question. ;-)
>
> Maybe I was expecting too much given Andrei's performance oriented
> talks. I realize that the conceptual groundwork is more important than a
> concrete implementation that can be easily improved. However, I think
> that real world out-of-the-box performance - particularly with respect
> to toy examples (since those are small enough to be literally
> translated) - is important for prospects to gain confidence in buying
> into D.

Well, D is pretty fast, as fast as C++ or C. What I mean is that there is no inherent overhead -- both can produce exactly the same code.

However, there are some parts of C/C++ that have been optimized to death. It's one of those things where D's version of rotate probably hasn't had as much scrutiny as C++'s version. We are always looking to improve such things, and more investigation and suggestions are most welcome! It's why I filed the bug report.

> At the current state, at least for such benchmarks, I think, I should
> not rely on standard library facilities. Unfortunately, that does not
> increase my confidence.

Try to find something besides insertion sort to test with I think ;) D is pretty fast at a lot of things, and pleasant to write. You just found one test case that isn't (and we will fix it, I'm sure).

-Steve