At this years DConf I gave a very short overview of the benchmarking library I'm working on [1]. The source can be found here [2]

The general idea goes as following.
Unit tests in D help us to find bugs and avoid regressions, but they do not cover performance regressions. That is bad, we are basically, currently just hoping that performance diminishes.

So my idea is to eventually get this library into phobos, write benchmarks for all functions in phobos, execute the benchmarks for every merge into master, use gnuplot to display the results on the dlang webpage, profit.

Right now I need people to give feedback on the library. You can find it on code.dlang.org [3]



[1] https://www.youtube.com/watch?v=QELK73JSpFk&list=PL3jwVPmk_PRxo23yyoc0Ip_cP3-rCm7eB&index=30
[2] https://github.com/burner/benchmark
[3] http://code.dlang.org/packages/std_benchmark

On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek wrote:
> they do not cover performance regressions. That is bad, we are basically, currently just hoping that performance diminishes.

diminishes -> does not diminishes

On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek wrote:
> So my idea is to eventually get this library into phobos, write benchmarks for all functions in phobos, execute the benchmarks for every merge into master, use gnuplot to display the results on the dlang webpage

And reject out of hand any PR which reduces performance in order to "ratchet" performance over time.

I would also do this to a lesser degree to DMD as well.

On 06/02/2017 02:49 PM, Jack Stouffer wrote:
> On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek wrote:
>> So my idea is to eventually get this library into phobos, write benchmarks for all functions in phobos, execute the benchmarks for every merge into master, use gnuplot to display the results on the dlang webpage
> 
> And reject out of hand any PR which reduces performance in order to "ratchet" performance over time.
> 
> I would also do this to a lesser degree to DMD as well.

That's indeed worth it. Robert, we discussed first deploying the library and a driver for dmd/druntime/phobos on dub, is this your plan? Thanks! -- Andrei

On Friday, 2 June 2017 at 18:49:49 UTC, Jack Stouffer wrote:
> And reject out of hand any PR which reduces performance in order to "ratchet" performance over time.
>
> I would also do this to a lesser degree to DMD as well.

I think being binary about it is the wrong approach.
Lets say we fix a bug, and the function is slower afterwards. Do we merge the bugfix? I say yes.
Lets say we add a feature to a function that makes more useful. Do we merge it? I say yes.

This is just another hammer, it does not mean everything is a nail now.

On Friday, 2 June 2017 at 18:51:24 UTC, Andrei Alexandrescu wrote:
> That's indeed worth it. Robert, we discussed first deploying the library and a driver for dmd/druntime/phobos on dub, is this your plan? Thanks! -- Andrei

I think a practical approach is to create another dub project that solely consists of benchmarks for functions of phobos, and prints pretty pictures of their performance.
This will properly help to find shortcomings of the benchmark library, and help to develop idioms for benchmarking and random testing.
If that works out, the next step is properly to integrate that in the phobos/druntime/dmd CI. Which properly means putting the benchmark library through the experimental integration process, and finding a place for the benchmarks to life.

On 2017-06-01 23:08, Robert burner Schadek wrote:

> So my idea is to eventually get this library into phobos, write
> benchmarks for all functions in phobos, execute the benchmarks for every
> merge into master, use gnuplot to display the results on the dlang
> webpage, profit.

In Xcode you can actually have benchmarks that behave as tests. That is, you write your benchmark. Define a boundary for a set of platforms, if the benchmark takes longer to run then the boundary it will fail the test. Then you don't need anyone to constantly look at the result. Of course, this only work if you have a limited set of platforms with a consistent performance.

-- 
/Jacob Carlborg

On Friday, 2 June 2017 at 21:56:47 UTC, Robert burner Schadek wrote:
> If that works out, the next step is properly to integrate that in the phobos/druntime/dmd CI. Which properly means putting the benchmark library through the experimental integration process, and finding a place for the benchmarks to life.

As I've pointed out before, having the benchmark support code be part of the public standard library is not required for the Phobos project to make use of it internally.

 — David