Am Sun, 08 Apr 2012 09:35:14 -0500
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 4/8/12 3:16 AM, Somedude wrote:
> > Like it. Would it be a good idea to add a column with an average memory used ?
> 
> Interesting idea. I saw http://stackoverflow.com/questions/1674652/c-c-memory-usage-api-in-linux-windows and it looks like it's not an easy problem. Should we make this part of the initial release?
> 
> Andrei

Oh please, I use these functions already and with the similarities amongst operating systems they should be somewhere in Phobos! My thought was more in the line of std.procmgmt/std.process though. Since they are process management functions useful for server monitoring as well, not just benchmarking.

-- 
Marco

Very good but minimum isn't a best guess. Personally I (and there will be a lot of such maniacs I suppose) will think that this (minimum) time can be significantly smaller than average time.

So a parameter (probably with a default value) should be added. Something like enum of flags telling what we want to know. At least these looks usable: minTime, <some mean time>, maxTime, standardDeviation, graph (yes, good old ASCII art).

Yes, graph is needed.

-- 
Денис В. Шеломовский
Denis V. Shelomovskij

On 4/8/12 11:59 AM, Denis Shelomovskij wrote:
> Very good but minimum isn't a best guess. Personally I (and there will
> be a lot of such maniacs I suppose) will think that this (minimum) time
> can be significantly smaller than average time.

I've analyzed this quite a bit at work and the average and median are not very informative. You need the mode, but in most benchmarks the mode is very close to the minimum, so using the minimum is even better.

In speed measurements, all noise is additive (there's no noise that may make a benchmark appear to run faster). There are also quite a few outliers. Recording the average will include a fair amount of noise.

Clearly there is noise during normal use as well, but incorporating it in benchmarks as a matter of course reduces the usefulness of benchmarks as a mean to improve performance of the benchmarked code.

> So a parameter (probably with a default value) should be added.
> Something like enum of flags telling what we want to know. At least
> these looks usable: minTime, <some mean time>, maxTime,
> standardDeviation, graph (yes, good old ASCII art).

Standard deviation is also not very useful because it includes all outliers (some of which are very far away from the mode).

> Yes, graph is needed.

I am not sure about that. We may provide the raw measurement data for programs that want to plot things, but plotting is beyond the charter of std.benchmark.


Andrei

On 08.04.2012 20:59, Denis Shelomovskij wrote:
> Very good but minimum isn't a best guess. Personally I (and there will
> be a lot of such maniacs I suppose) will think that this (minimum) time
> can be significantly smaller than average time.
>

Prime example is networking.

> So a parameter (probably with a default value) should be added.
> Something like enum of flags telling what we want to know. At least
> these looks usable: minTime, <some mean time>, maxTime,
> standardDeviation,

+1

 graph (yes, good old ASCII art).
>
> Yes, graph is needed.
>

No not ASCII art. It's too coarse grained and the only good thing it can do is look nicely in terminal (or give an overall picture but it's limited to say ~25 blocks in a histogram).

What would be nice to have an option to output bare-bones info so that one can easily adapt it and feed the result to e.g. gnuplot.
I think csv or it's version with tabs is good enough in this case.

Another cool addition IMHO would be parametric benchmarks, so there is a function and a set of parameters (one parameter is fine too) to benchmark on. It makes that much more sense with graphs as algorithm profile plotted for various inputs (sizes) can be very revealing.

-- 
Dmitry Olshansky

On 4/8/12 12:35 PM, Dmitry Olshansky wrote:
> Another cool addition IMHO would be parametric benchmarks, so there is a
> function and a set of parameters (one parameter is fine too) to
> benchmark on. It makes that much more sense with graphs as algorithm
> profile plotted for various inputs (sizes) can be very revealing.

This would be interesting to explore, could you give some more detail please?

Andrei

On Sat, Apr 7, 2012 at 10:25 PM, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> Hello,
>
> I finally found the time to complete std.benchmark. I got to a very simple API design, starting where I like it: one line of code.
>
> Andrei


I probably missed this somewhere, but what happens to `std.datetime : benchmark` ?  Is it going to be deprecated ?

On 08.04.2012 21:40, Andrei Alexandrescu wrote:
> On 4/8/12 12:35 PM, Dmitry Olshansky wrote:
>> Another cool addition IMHO would be parametric benchmarks, so there is a
>> function and a set of parameters (one parameter is fine too) to
>> benchmark on. It makes that much more sense with graphs as algorithm
>> profile plotted for various inputs (sizes) can be very revealing.
>
> This would be interesting to explore, could you give some more detail
> please?
>

I thought to just throw it in. But if were to do it in under an hour:

benchmark_func_blah(int problem_size);
auto benchmark_params_func_blah = [ 1, 10, 100, 1_000, 10_000 ...];

or slightly more general - use the range protocol:
auto benchmark_params_func_blah = getRangeOfValues();

I'm not sure the prefix is place to put it but here the rough idea.

-- 
Dmitry Olshansky

On 4/8/12 1:03 PM, Caligo wrote:
> On Sat, Apr 7, 2012 at 10:25 PM, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org>  wrote:
>> Hello,
>>
>> I finally found the time to complete std.benchmark. I got to a very simple
>> API design, starting where I like it: one line of code.
>>
>> Andrei
>
>
> I probably missed this somewhere, but what happens to `std.datetime :
> benchmark` ?  Is it going to be deprecated ?

Yes, I forgot to mention that. Part of the proposal is that std.datetime.StopWatch and std.datetime.benchmark are be deprecated.

Andrei

Andrei Alexandrescu wrote:

> Clearly there is noise during normal use as well, but incorporating it in benchmarks as a matter of course reduces the usefulness of benchmarks

On the contrary:
1) The "noise during normal use" has to be measured in order to detect
the sensibility of the benchmarked program to that noise.
2) The noise the benchmarked program produces has to be measured too,
because the running benchmarked program probably increases the noise
for all other running programs.

In addition: the noise produced by a machine under heavy load might bring the performance of the benchmarked program down to zero.

-manfred

Le 08/04/2012 18:21, Marco Leise a écrit :
> Am Sun, 08 Apr 2012 09:35:14 -0500
> schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
> 
>> On 4/8/12 3:16 AM, Somedude wrote:
>>> Like it. Would it be a good idea to add a column with an average memory used ?
>>
>> Interesting idea. I saw http://stackoverflow.com/questions/1674652/c-c-memory-usage-api-in-linux-windows and it looks like it's not an easy problem. Should we make this part of the initial release?
>>
>> Andrei
> 
> Oh please, I use these functions already and with the similarities amongst operating systems they should be somewhere in Phobos! My thought was more in the line of std.procmgmt/std.process though. Since they are process management functions useful for server monitoring as well, not just benchmarking.
> 

I thought about monitoring as well.
But then again, following your remark, monitoring may in fact be a
completely different matter.
This would need a little bit more thought I guess. Adding monitoring
capabilities to processes would be interesting. But I guess we are
already quite far from benchmarking now.
Unless std.benchmark makes use of these monitoring capabilities.