Hi all,

I am currently finalizing my material for the LDC DConf talk, and I thought it would be nice to include a quick runtime performance comparison between the different compilers, just to give the audience a general sense of what to expect.

Thus, I am looking for benchmarks to use in the talk. Specifically, they should:

 - be open source, or at least source-available, so other people can reproduce the results
 - be *reasonably* self-contained, so that I don't have to spend three hours setting up build dependencies
 - be written mostly in D, I don't want to benchmark GCC
 - work with DMD 2.061 or DMD 2.062
 - run on Linux or OS X

I already have a few results (Dmitry's std.regex and std.uni benchmarks, WebDrake's Dregs, some of my own projects, …), but it would be great if some of you could point me to your own set of tests so I can hopefully paint a more complete picture.

There is a host of results if you search for »benchmark« here on the forums, but many of the discussed test cases are trivial micro-benchmarks, and I was hoping to add a few more elaborate performance tests to my collection.

In the future – i.e. as soon as possible, but somebody has to actually spend some time on setting things up –, we might also want to set up a nightly tester with such benchmarks to track performance of the different compilers over time. It's not as crucial for GDC and LDC as it is for the upstream backend projects, but there are still quite a few things to watch out for in druntime/Phobos and the LDC LLVM optimizations specific to D.

David



P.S.: Juan Manuel Cabo's "avgtime" is a really, _really_ useful tool for benchmarking whole programs and actually getting solid statistics out of it. Let's add something similar as a library for more finely-grained use!

>  - be open source, or at least source-available, so other people can reproduce the results
>  - be *reasonably* self-contained, so that I don't have to spend three hours setting up build dependencies
>  - be written mostly in D, I don't want to benchmark GCC
>  - work with DMD 2.061 or DMD 2.062
>  - run on Linux or OS X

You could also use pfft, then. This branch is the most up to date:

https://github.com/jerro/pfft/tree/use-gcc-udas

If you run:

./build
./build --tests

It will generate files test/test_float, test/test_double and test/test_real. You can choose compiler with --dc DMD|GDC|LDC. You can use -s flag on test/test_* executables to do a benchmark. Test executables  and build.d have --help option.

To make the comparison fair, it would probably be good to use --simd sse flag when building tests, because otherwise GDC and LDC versions will use AVX if run on a machine that supports it and DMD won't.

I also have test/benchmarks.d script that runs benchmarks and outputs charts, but I made no attempt to make it user friendly and you would need to read and modify its (a bit messy) code to use it - the only option it takes is the location of output files. It depends on Plot2kill.

> --simd sse flag when building tests

Should be

> --simd sse flag when building the library.

You can try with nBodySim https://github.com/Zardoz89/nBodySim
I wrote it and tested with DMD 2.060, but should be working with 2.061 or 2.062.
I used it to benchmark "parallel for" vs "serial for" in some computers with 2, 4 and 16 cores, getting a speedup like x13 in a 16 core machine.
It have a small bash script to benchmark, running N times the program, and doing a average mean of the total time of execution.

On Sunday, 10 March 2013 at 23:36:26 UTC, David Nadlinger wrote:
> Hi all,
>
> I am currently finalizing my material for the LDC DConf talk, and I thought it would be nice to include a quick runtime performance comparison between the different compilers, just to give the audience a general sense of what to expect.
>
> Thus, I am looking for benchmarks to use in the talk. Specifically, they should:
>
>  - be open source, or at least source-available, so other people can reproduce the results
>  - be *reasonably* self-contained, so that I don't have to spend three hours setting up build dependencies
>  - be written mostly in D, I don't want to benchmark GCC
>  - work with DMD 2.061 or DMD 2.062
>  - run on Linux or OS X
>
> I already have a few results (Dmitry's std.regex and std.uni benchmarks, WebDrake's Dregs, some of my own projects, …), but it would be great if some of you could point me to your own set of tests so I can hopefully paint a more complete picture.
>
> There is a host of results if you search for »benchmark« here on the forums, but many of the discussed test cases are trivial micro-benchmarks, and I was hoping to add a few more elaborate performance tests to my collection.
>
> In the future – i.e. as soon as possible, but somebody has to actually spend some time on setting things up –, we might also want to set up a nightly tester with such benchmarks to track performance of the different compilers over time. It's not as crucial for GDC and LDC as it is for the upstream backend projects, but there are still quite a few things to watch out for in druntime/Phobos and the LDC LLVM optimizations specific to D.
>
> David
>
>
>
> P.S.: Juan Manuel Cabo's "avgtime" is a really, _really_ useful tool for benchmarking whole programs and actually getting solid statistics out of it. Let's add something similar as a library for more finely-grained use!

On Mon, 2013-03-11 at 00:36 +0100, David Nadlinger wrote:
> Hi all,
> 
> I am currently finalizing my material for the LDC DConf talk, and I thought it would be nice to include a quick runtime performance comparison between the different compilers, just to give the audience a general sense of what to expect.
> 
> Thus, I am looking for benchmarks to use in the talk. Specifically, they should:
> 
>   - be open source, or at least source-available, so other people
> can reproduce the results
>   - be *reasonably* self-contained, so that I don't have to spend
> three hours setting up build dependencies
>   - be written mostly in D, I don't want to benchmark GCC
>   - work with DMD 2.061 or DMD 2.062
>   - run on Linux or OS X

I have a variety of D implementations of "Calculating π by
quadrature" (including the one David Simcha contributed for testing
std.parallelism). This is a trivial, data parallel, embarrassingly
parallel problem that I use for comparing scaling in various languages
in comparison to C. There is no experiment set up, just single run. It
uses SCons for the D build which will require the fork of SCons I have
that includes (almost reasonable) support for D build. On the other hand
all the files are fundamentally self-contained except for reliance on
one module in the same directory so rdmd should work just fine. The code
is released under GPLv3 and is available on GitHub:

        git@github.com:russel/Pi_Quadrature.git

and my own website, for cloning:

        http://www.russel.org.uk/Git/Pi_Quadrature.git

or for browsing:

        http://www.russel.org.uk/gitweb/?p=Pi_Quadrature.git;a=summary

If there are any errors or infelicities of D coding I would be very pleased to hear of them, especially if the come with a pull request!

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

You can use my raytracing in D project.

https://github.com/minas1/D_Raytracing

It's very incomplete at the current state by the way (no soft shadows, no texturing, no reflection, no antialiasing).

11-Mar-2013 03:36, David Nadlinger пишет:
> Hi all,
>
> I am currently finalizing my material for the LDC DConf talk, and I
> thought it would be nice to include a quick runtime performance
> comparison between the different compilers, just to give the audience a
> general sense of what to expect.
>
> Thus, I am looking for benchmarks to use in the talk. Specifically, they
> should:
>
>   - be open source, or at least source-available, so other people can
> reproduce the results
>   - be *reasonably* self-contained, so that I don't have to spend three
> hours setting up build dependencies
>   - be written mostly in D, I don't want to benchmark GCC
>   - work with DMD 2.061 or DMD 2.062
>   - run on Linux or OS X
>
[snip]

> In the future – i.e. as soon as possible, but somebody has to actually
> spend some time on setting things up –, we might also want to set up a
> nightly tester with such benchmarks to track performance of the
> different compilers over time. It's not as crucial for GDC and LDC as it
> is for the upstream backend projects, but there are still quite a few
> things to watch out for in druntime/Phobos and the LDC LLVM
> optimizations specific to D.

Hopefully sometime soon benchmarks would become part of the auto-tester framework.


> P.S.: Juan Manuel Cabo's "avgtime" is a really, _really_ useful tool for
> benchmarking whole programs and actually getting solid statistics out of
> it. Let's add something similar as a library for more finely-grained use!

+111

-- 
Dmitry Olshansky