Hello,


I finally found the time to complete std.benchmark. I got to a very simple API design, starting where I like it: one line of code.

Code is in the form of a pull request at https://github.com/D-Programming-Language/phobos/pull/529. (There's some noise in there caused by my git n00biness). Documentation is at http://erdani.com/d/web/phobos-prerelease/std_benchmark.html.

If reasonable and at all possible, I'd like to bump the priority of this proposal. Clearly D's user base is highly interested in efficiency, and many of the upcoming libraries have efficiency a virtual part of their design. So we should get std.benchmark in soon and require that new addition come with benchmarks for their essential functionality.

My vision is that in the future Phobos will have a significant benchmarks battery, which will help improving Phobos and porting to new platforms.


Andrei

On Sunday, 8 April 2012 at 03:25:16 UTC, Andrei Alexandrescu wrote:
> Hello,
>
>
> I finally found the time to complete std.benchmark. I got to a very simple API design, starting where I like it: one line of code.

Nice, some comments:

1) Is it possible to do something about the "kilonanoseconds", and print the two-letter SI time abbreviation instead (μs/ns/ms/s/...)?

2) The meaning of "epoch" in this context is unfamiliar to me:

> Measurement is done in epochs. For each function benchmarked, the smallest time is taken over all epochs.

3) "benchmark_relative_file read" should be replaced with a language construct. E.g. a function call like relativeBenchmark("file read"), or an enum value like getopt's.

4) There is a TODO in benchmarkSuspend's example.

On 4/7/12 11:45 PM, Vladimir Panteleev wrote:
> On Sunday, 8 April 2012 at 03:25:16 UTC, Andrei Alexandrescu wrote:
>> Hello,
>>
>>
>> I finally found the time to complete std.benchmark. I got to a very
>> simple API design, starting where I like it: one line of code.
>
> Nice, some comments:
>
> 1) Is it possible to do something about the "kilonanoseconds", and print
> the two-letter SI time abbreviation instead (μs/ns/ms/s/...)?

Cool.

> 2) The meaning of "epoch" in this context is unfamiliar to me:
>
>> Measurement is done in epochs. For each function benchmarked, the
>> smallest time is taken over all epochs.

To me too :o).

> 3) "benchmark_relative_file read" should be replaced with a language
> construct. E.g. a function call like relativeBenchmark("file read"), or
> an enum value like getopt's.

No can do. Need a function name-based convention so we can automate scheduleForBenchmarking.

> 4) There is a TODO in benchmarkSuspend's example.

Will destroy.


Andrei

On Sunday, 8 April 2012 at 05:41:11 UTC, Andrei Alexandrescu wrote:
>> 3) "benchmark_relative_file read" should be replaced with a language
>> construct. E.g. a function call like relativeBenchmark("file read"), or
>> an enum value like getopt's.
>
> No can do. Need a function name-based convention so we can automate scheduleForBenchmarking.

Hmm, maybe there's a misunderstanding, but what I meant was:

The "benchmark_relative_" prefix makes sense for function names (for scheduleForBenchmarking), but not so much for string literals for benchmark names. The string literal "benchmark_relative_file read" looks like the words "benchmark relative file" are grouped together, with "read" added on. So, my suggestion would be to wrap the "benchmark_relative_" prefix - when used with benchmark name strings - into a semantical function / enum / etc. In my example above, relativeBenchmark would be:

string relativeBenchmark(string s) { return "benchmark_relative_" ~ s; }

I suppose it can be summed up as a tradeoff between complexity (you need to explain both the function name usage and the relativeBenchmark wrapper usage) vs. code prettiness.

Le 08/04/2012 05:25, Andrei Alexandrescu a écrit :
> Hello,
> 
> 
> I finally found the time to complete std.benchmark. I got to a very simple API design, starting where I like it: one line of code.
> 
> Code is in the form of a pull request at https://github.com/D-Programming-Language/phobos/pull/529. (There's some noise in there caused by my git n00biness). Documentation is at http://erdani.com/d/web/phobos-prerelease/std_benchmark.html.
> 
> If reasonable and at all possible, I'd like to bump the priority of this proposal. Clearly D's user base is highly interested in efficiency, and many of the upcoming libraries have efficiency a virtual part of their design. So we should get std.benchmark in soon and require that new addition come with benchmarks for their essential functionality.
> 
> My vision is that in the future Phobos will have a significant benchmarks battery, which will help improving Phobos and porting to new platforms.
> 
> 
> Andrei

Like it. Would it be a good idea to add a column with an average memory used ?

Andrei Alexandrescu wrote:
> I finally found the time to complete std.benchmark. I got to a very
> simple API design, starting where I like it: one line of code.
>
> Code is in the form of a pull request at
> https://github.com/D-Programming-Language/phobos/pull/529. (There's some
> noise in there caused by my git n00biness). Documentation is at
> http://erdani.com/d/web/phobos-prerelease/std_benchmark.html.

For algorithms that process sequences, it would be nice to have results represented in cycles per item, This should give more consistent results across different CPU familes and different clock speeds.

Specifically, I think about cycles per byte, see http://en.wikipedia.org/wiki/Cycles_per_byte
Example: http://www.cryptopp.com/benchmarks.html

On 4/8/12 2:02 AM, Vladimir Panteleev wrote:
> The "benchmark_relative_" prefix makes sense for function names (for
> scheduleForBenchmarking), but not so much for string literals for
> benchmark names. The string literal "benchmark_relative_file read" looks
> like the words "benchmark relative file" are grouped together, with
> "read" added on. So, my suggestion would be to wrap the
> "benchmark_relative_" prefix - when used with benchmark name strings -
> into a semantical function / enum / etc. In my example above,
> relativeBenchmark would be:
>
> string relativeBenchmark(string s) { return "benchmark_relative_" ~ s; }
>
> I suppose it can be summed up as a tradeoff between complexity (you need
> to explain both the function name usage and the relativeBenchmark
> wrapper usage) vs. code prettiness.

I understand, thanks.

Andrei

On 4/8/12 3:16 AM, Somedude wrote:
> Like it. Would it be a good idea to add a column with an average memory
> used ?

Interesting idea. I saw http://stackoverflow.com/questions/1674652/c-c-memory-usage-api-in-linux-windows and it looks like it's not an easy problem. Should we make this part of the initial release?

Andrei

On 4/8/12 5:46 AM, Piotr Szturmaj wrote:
> Andrei Alexandrescu wrote:
>> I finally found the time to complete std.benchmark. I got to a very
>> simple API design, starting where I like it: one line of code.
>>
>> Code is in the form of a pull request at
>> https://github.com/D-Programming-Language/phobos/pull/529. (There's some
>> noise in there caused by my git n00biness). Documentation is at
>> http://erdani.com/d/web/phobos-prerelease/std_benchmark.html.
>
> For algorithms that process sequences, it would be nice to have results
> represented in cycles per item, This should give more consistent results
> across different CPU familes and different clock speeds.
>
> Specifically, I think about cycles per byte, see
> http://en.wikipedia.org/wiki/Cycles_per_byte
> Example: http://www.cryptopp.com/benchmarks.html

The framework aims at generality and simplicity. I'd rather have it simple and universal rather than subtle.

Andrei

On 08.04.2012 12:16, Somedude wrote:
[snip]
>
> Like it. Would it be a good idea to add a column with an average memory
> used ?

In general it's next to impossible and/or entirely OS-specific.
What can be done I think is adding a query function to GC interface that returns amount of RAM currently allocated on it's heap.

So adding GC heap usage can be done albeit it's not really a stable metric. This way one gets a nice hint on why something is slow ;)

-- 
Dmitry Olshansky