Jens Mueller wrote:
> Hi,
> 
> it's my pleasure to announce the begin of the formal review of Andrei's std.benchmark. The review will start today and end in two weeks, on 1st of October. The review is followed by a week of voting which ends on 8th of October.
> 
> Quoting Andrei from his request for formal review:
> "I reworked the benchmarking framework for backward compatibility,
> flexibility, and convenience.
> 
> There are a few enhancement possibilities (such as tracking system/user time separately etc), but there is value in keeping things simple and convenient. Right now it really takes only one line of code and observing a simple naming convention to hook a module into the benchmarking framework."
> 
> Code: https://github.com/D-Programming-Language/phobos/pull/794 Docs: http://dlang.org/phobos-prerelease/std_benchmark.html
> 
> If std.benchmark is accepted it will likely lead to a deprecation of std.datetime's benchmark facilities.
> 
> The code is provided as a pull requested and being (as usual) integrated
> by the auto tester for Mac OS X, FreeBSD, Linux and Windows (see
> (http://d.puremagic.com/test-results/pull-history.ghtml?repoid=3&pullid=794).
> 
> In your comments you can/should address the
> * design
> * implementation
> * documentation
> * usefulness
> of the library.
> 
> Provide information regarding the depth (ranging from very brief to in-depth) of your review and conclude explicitly whether std.benchmark should or shouldn't be included in Phobos.
> 
> Post all feedback to this thread. Constructive feedback is very much appreciated.
> 
> To conclude in more Andrei like words: Happy destruction!
> 
> Jens

The review of std.benchmark is over.
I'd like to give some conclusions regarding the review of the proposed
benchmarking module hoping to reach some consensus on how to proceed.

First of all I'd like to thank all the reviewers for taking part in this review.

Let me first summarize the main point that was intensely discussed in the review.

* benchmarking vs. profiling
  std.benchmark, as the review clarified, is primarily intended for
  micro benchmarking.
  The review questioned this arguably limited scope. In particular, it
  asked for different reductions (max, avg (with std deviation)).
  This whole initial confusion should be addressed by improving the
  documentation and making sure a broader set of use cases can be
  supported without breaking code.

  It is not entirely settled whether this broader scope of usage will be
  addressed by Andrei's proposal. Maybe the community can address this
  issue by providing additional functionality at a later point in time.
  But it needs to made sure that those changes won't break code written
  against the first version of std.benchmark.

  The issues related to this are
  -non-deterministic/randomized functions (currently not supported)
  -benchmarking real time software
  -benchmarking with warm vs. cold caches
  -measuring by using hardware counters

  These seem to be of particular interest to D community.

During the review other issues were identified which the author will
look into.
* make number of function runs configurable
* individual registration of benchmark functions in addition to naming convention
* auto detect when min is stable with some degree of statistical certainty
* specify a problem size
* make scheduleForBenchmarking a template mixin
* benchmark should return the BenchmarkResult array

A major issue is that std.benchmark won't work if two cyclically dependent modules are benchmarked because it uses module/static constructors. Currently there is no idea on how to solve this issue.

I hope this summary captures the most important issues that where
brought up. How to proceed seems difficult at this point. I believe the
issue with cyclic dependent modules has to be solved and we should reach
consensus regarding the intended use cases that should be addressed by
std.benchmark even in the long run.
For the time being I suppose there is no use in proceeding with voting.
Any comments?

Jens

On Tuesday, 2 October 2012 at 01:09:48 UTC, Jens Mueller wrote:
> Any comments?
>
> Jens

Looks like a good synopsis of the discussion I was reading. I like the goal of benchmark and am unsure how I feel on the profiling. I've used the -profile command from DMD, but I'm not sure how this translates to internal code. To profile my program I'm trying to find the place I'm spending most of my time so I can speed it up. Then when I'm improving that area I'd need a good benchmark to reduce that time. If std.benchmark could be used to support profiling I'm for it, but am not sure for my limited desires what that would be.

The other issues listed look like a good idea to address. Hopeful module constructors can be resolved in the language so no change needed by std.benchmark.

On 10/1/12 9:09 PM, Jens Mueller wrote:
> The review of std.benchmark is over.
> I'd like to give some conclusions regarding the review of the proposed
> benchmarking module hoping to reach some consensus on how to proceed.
[snip]

Thanks Jens for shepherding the review and to all for the comments.

I agree there's no need to vote at this point and am preemptively retracting the proposal in order to integrate the feedback. There's a fine line to navigate between offering comprehensive functionality and throwing in the kitchen sink, but it's clear that right now the framework falls short in offering functionality. I'll work more on this and re-submit later with the feedback minded.


Thanks all,

Andrei

Andrei Alexandrescu wrote:
> On 10/1/12 9:09 PM, Jens Mueller wrote:
> >The review of std.benchmark is over.
> >I'd like to give some conclusions regarding the review of the proposed
> >benchmarking module hoping to reach some consensus on how to proceed.
> [snip]
> 
> Thanks Jens for shepherding the review and to all for the comments.

Thanks for your reply.
I'm looking forward to your changes.

Jens