Everybody is talking about benchmarks and making code faster, yet phobos is still lacking any long term benchmark gathering and monitoring solution.

PR https://github.com/D-Programming-Language/phobos/pull/2995 provides all that, and is done some time now. It will give us pretty pictures we can put on the webpage.
These will show how awesome phobos already is, and how much we improve performance wise with time progressing.

Please have a look, and let us make phobos the most benchmarked library out there.

On Wednesday, 2 September 2015 at 08:30:40 UTC, Robert burner Schadek wrote:
> Everybody is talking about benchmarks and making code faster, yet phobos is still lacking any long term benchmark gathering and monitoring solution.
>
> PR https://github.com/D-Programming-Language/phobos/pull/2995 provides all that, and is done some time now. It will give us pretty pictures we can put on the webpage.
> These will show how awesome phobos already is, and how much we improve performance wise with time progressing.
>

This does sound like a really good idea. Is the plan to turn every unittest block into a benchmark (automatically), or did you add specific benchmarks blocks?

Might be nice to have a mock up of the webpage with all the results/benchmarks.

On Tuesday, 8 September 2015 at 08:42:59 UTC, Edwin van Leeuwen wrote:
> This does sound like a really good idea. Is the plan to turn every unittest block into a benchmark (automatically), or did you add specific benchmarks blocks?

you would add specific benchmark blocks, because the benchmarks might run to long for a rapid development cycle.

>
> Might be nice to have a mock up of the webpage with all the results/benchmarks.

that is the idea

On Tuesday, 8 September 2015 at 09:01:22 UTC, Robert burner Schadek wrote:
> On Tuesday, 8 September 2015 at 08:42:59 UTC, Edwin van Leeuwen wrote:
>> This does sound like a really good idea. Is the plan to turn every unittest block into a benchmark (automatically), or did you add specific benchmarks blocks?
>
> you would add specific benchmark blocks, because the benchmarks might run to long for a rapid development cycle.

Just to be clear. The PR is to add the needed ground work for adding benchmark, or does it also add some benchmarks itself?

It does seem a bit of a shame not to use the unittests at all for baseline benchmarks. The unittests already exist, with reasonable good coverage. Getting good benchmark coverage by writing specific/new benchmarks would mean a huge amount of extra effort.

On Tuesday, 8 September 2015 at 09:07:10 UTC, Edwin van Leeuwen wrote:
> Just to be clear. The PR is to add the needed ground work for adding benchmark, or does it also add some benchmarks itself?

The PR adds benchmarks for some functions of std.string.

>
> It does seem a bit of a shame not to use the unittests at all for baseline benchmarks. The unittests already exist, with reasonable good coverage. Getting good benchmark coverage by writing specific/new benchmarks would mean a huge amount of extra effort.

I disagree, the unittests in phobos are often very short, only test corner cases, test regression fix, serve add doc unittests or do some basic sanity checking.

Benchmarks have other requirements.

On Tuesday, 8 September 2015 at 09:20:08 UTC, Robert burner Schadek wrote:
> On Tuesday, 8 September 2015 at 09:07:10 UTC, Edwin van Leeuwen wrote:
>> Just to be clear. The PR is to add the needed ground work for adding benchmark, or does it also add some benchmarks itself?
>
> The PR adds benchmarks for some functions of std.string.
>

I sort of like what Rust is using: https://github.com/rust-lang/rust/tree/master/src/test/bench

If we can have similar benchmarks suite which can be tested with every release, it can help to spot speed regressions or promote new optimisations.

On Tuesday, 8 September 2015 at 12:24:19 UTC, tchaloupka wrote:
> On Tuesday, 8 September 2015 at 09:20:08 UTC, Robert burner Schadek wrote:
>> On Tuesday, 8 September 2015 at 09:07:10 UTC, Edwin van Leeuwen wrote:
>>> Just to be clear. The PR is to add the needed ground work for adding benchmark, or does it also add some benchmarks itself?
>>
>> The PR adds benchmarks for some functions of std.string.
>>
>
> I sort of like what Rust is using: https://github.com/rust-lang/rust/tree/master/src/test/bench
>
> If we can have similar benchmarks suite which can be tested with every release, it can help to spot speed regressions or promote new optimisations.

Have you taken a look at the source of the PR. It does exactly that.
Of course not ever function of phobos will get its benchmark in this PR, but this makes it easy to create benchmarks.