View mode: basic / threaded / horizontal-split · Log in · Help
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
Am Sun, 08 Apr 2012 09:35:14 -0500
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 4/8/12 3:16 AM, Somedude wrote:
> > Like it. Would it be a good idea to add a column with an average memory
> > used ?
> 
> Interesting idea. I saw 
> http://stackoverflow.com/questions/1674652/c-c-memory-usage-api-in-linux-windows 
> and it looks like it's not an easy problem. Should we make this part of 
> the initial release?
> 
> Andrei

Oh please, I use these functions already and with the similarities amongst operating systems they should be somewhere in Phobos! My thought was more in the line of std.procmgmt/std.process though. Since they are process management functions useful for server monitoring as well, not just benchmarking.

-- 
Marco
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
Very good but minimum isn't a best guess. Personally I (and there will 
be a lot of such maniacs I suppose) will think that this (minimum) time 
can be significantly smaller than average time.

So a parameter (probably with a default value) should be added. 
Something like enum of flags telling what we want to know. At least 
these looks usable: minTime, <some mean time>, maxTime, 
standardDeviation, graph (yes, good old ASCII art).

Yes, graph is needed.

-- 
Денис В. Шеломовский
Denis V. Shelomovskij
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
On 4/8/12 11:59 AM, Denis Shelomovskij wrote:
> Very good but minimum isn't a best guess. Personally I (and there will
> be a lot of such maniacs I suppose) will think that this (minimum) time
> can be significantly smaller than average time.

I've analyzed this quite a bit at work and the average and median are 
not very informative. You need the mode, but in most benchmarks the mode 
is very close to the minimum, so using the minimum is even better.

In speed measurements, all noise is additive (there's no noise that may 
make a benchmark appear to run faster). There are also quite a few 
outliers. Recording the average will include a fair amount of noise.

Clearly there is noise during normal use as well, but incorporating it 
in benchmarks as a matter of course reduces the usefulness of benchmarks 
as a mean to improve performance of the benchmarked code.

> So a parameter (probably with a default value) should be added.
> Something like enum of flags telling what we want to know. At least
> these looks usable: minTime, <some mean time>, maxTime,
> standardDeviation, graph (yes, good old ASCII art).

Standard deviation is also not very useful because it includes all 
outliers (some of which are very far away from the mode).

> Yes, graph is needed.

I am not sure about that. We may provide the raw measurement data for 
programs that want to plot things, but plotting is beyond the charter of 
std.benchmark.


Andrei
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
On 08.04.2012 20:59, Denis Shelomovskij wrote:
> Very good but minimum isn't a best guess. Personally I (and there will
> be a lot of such maniacs I suppose) will think that this (minimum) time
> can be significantly smaller than average time.
>

Prime example is networking.

> So a parameter (probably with a default value) should be added.
> Something like enum of flags telling what we want to know. At least
> these looks usable: minTime, <some mean time>, maxTime,
> standardDeviation,

+1

 graph (yes, good old ASCII art).
>
> Yes, graph is needed.
>

No not ASCII art. It's too coarse grained and the only good thing it can 
do is look nicely in terminal (or give an overall picture but it's 
limited to say ~25 blocks in a histogram).

What would be nice to have an option to output bare-bones info so that 
one can easily adapt it and feed the result to e.g. gnuplot.
I think csv or it's version with tabs is good enough in this case.

Another cool addition IMHO would be parametric benchmarks, so there is a 
function and a set of parameters (one parameter is fine too) to 
benchmark on. It makes that much more sense with graphs as algorithm 
profile plotted for various inputs (sizes) can be very revealing.

-- 
Dmitry Olshansky
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
On 4/8/12 12:35 PM, Dmitry Olshansky wrote:
> Another cool addition IMHO would be parametric benchmarks, so there is a
> function and a set of parameters (one parameter is fine too) to
> benchmark on. It makes that much more sense with graphs as algorithm
> profile plotted for various inputs (sizes) can be very revealing.

This would be interesting to explore, could you give some more detail 
please?

Andrei
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
On Sat, Apr 7, 2012 at 10:25 PM, Andrei Alexandrescu
<SeeWebsiteForEmail@erdani.org> wrote:
> Hello,
>
> I finally found the time to complete std.benchmark. I got to a very simple
> API design, starting where I like it: one line of code.
>
> Andrei


I probably missed this somewhere, but what happens to `std.datetime :
benchmark` ?  Is it going to be deprecated ?
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
On 08.04.2012 21:40, Andrei Alexandrescu wrote:
> On 4/8/12 12:35 PM, Dmitry Olshansky wrote:
>> Another cool addition IMHO would be parametric benchmarks, so there is a
>> function and a set of parameters (one parameter is fine too) to
>> benchmark on. It makes that much more sense with graphs as algorithm
>> profile plotted for various inputs (sizes) can be very revealing.
>
> This would be interesting to explore, could you give some more detail
> please?
>

I thought to just throw it in. But if were to do it in under an hour:

benchmark_func_blah(int problem_size);
auto benchmark_params_func_blah = [ 1, 10, 100, 1_000, 10_000 ...];

or slightly more general - use the range protocol:
auto benchmark_params_func_blah = getRangeOfValues();

I'm not sure the prefix is place to put it but here the rough idea.

-- 
Dmitry Olshansky
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
On 4/8/12 1:03 PM, Caligo wrote:
> On Sat, Apr 7, 2012 at 10:25 PM, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org>  wrote:
>> Hello,
>>
>> I finally found the time to complete std.benchmark. I got to a very simple
>> API design, starting where I like it: one line of code.
>>
>> Andrei
>
>
> I probably missed this somewhere, but what happens to `std.datetime :
> benchmark` ?  Is it going to be deprecated ?

Yes, I forgot to mention that. Part of the proposal is that 
std.datetime.StopWatch and std.datetime.benchmark are be deprecated.

Andrei
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
Andrei Alexandrescu wrote:

> Clearly there is noise during normal use as well, but
> incorporating it in benchmarks as a matter of course reduces the
> usefulness of benchmarks 

On the contrary:
1) The "noise during normal use" has to be measured in order to detect 
the sensibility of the benchmarked program to that noise.
2) The noise the benchmarked program produces has to be measured too, 
because the running benchmarked program probably increases the noise 
for all other running programs.   

In addition: the noise produced by a machine under heavy load might 
bring the performance of the benchmarked program down to zero.

-manfred
April 08, 2012
Re: std.benchmark ready for review. Manager sought after
Le 08/04/2012 18:21, Marco Leise a écrit :
> Am Sun, 08 Apr 2012 09:35:14 -0500
> schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
> 
>> On 4/8/12 3:16 AM, Somedude wrote:
>>> Like it. Would it be a good idea to add a column with an average memory
>>> used ?
>>
>> Interesting idea. I saw 
>> http://stackoverflow.com/questions/1674652/c-c-memory-usage-api-in-linux-windows 
>> and it looks like it's not an easy problem. Should we make this part of 
>> the initial release?
>>
>> Andrei
> 
> Oh please, I use these functions already and with the similarities amongst operating systems they should be somewhere in Phobos! My thought was more in the line of std.procmgmt/std.process though. Since they are process management functions useful for server monitoring as well, not just benchmarking.
> 

I thought about monitoring as well.
But then again, following your remark, monitoring may in fact be a
completely different matter.
This would need a little bit more thought I guess. Adding monitoring
capabilities to processes would be interesting. But I guess we are
already quite far from benchmarking now.
Unless std.benchmark makes use of these monitoring capabilities.
1 2 3 4 5 6
Top | Discussion index | About this forum | D home