Is there some way to measure the performance of a function so that the results will be same in different computers (all x86, but otherwise different processors)? I'm thinking of making a test suite that could find performance regressions automatically.

I figured out Bochs[1] could be used for that, but it seems an overkill for me to install a whole virtual operating system just to benchmark functions. Does anybody know more lightweight way?

[1] http://bochs.sourceforge.net/

On Thursday, 7 May 2020 at 10:21:07 UTC, Dukc wrote:
> Is there some way to measure the performance of a function so that the results will be same in different computers (all x86, but otherwise different processors)? I'm thinking of making a test suite that could find performance regressions automatically.
>
> I figured out Bochs[1] could be used for that, but it seems an overkill for me to install a whole virtual operating system just to benchmark functions. Does anybody know more lightweight way?
>
> [1] http://bochs.sourceforge.net/

If I understand correctly, you want to measure how many cycles pass, rather than clock time?

If so, it seems perf can do that: https://perf.wiki.kernel.org/index.php/Tutorial

--
  Simen

On Thursday, 7 May 2020 at 10:21:07 UTC, Dukc wrote:
> Is there some way to measure the performance of a function so that the results will be same in different computers (all x86, but otherwise different processors)? I'm thinking of making a test suite that could find performance regressions automatically.

You can make a reference program that you use to get a measure for how fast the computer is that you run the benchmark on. Then you can use that to scale your actual benchmark results.

When testing regressions there's a fairly obvious choice for this reference program: the old version. You can compare those results with the new version and report the relative difference.

On Thursday, 7 May 2020 at 10:51:27 UTC, Simen Kjærås wrote:
>
> If I understand correctly, you want to measure how many cycles pass, rather than clock time?

Something like that. Well, I would also like to eliminate differences based on different memory caches between machines.

In addition, if the profiler could eliminate the randomness of the benchamrks that results from memory aligment, context switches and dunno, that would obviously be a big plus for automatic testing.

>
> If so, it seems perf can do that: https://perf.wiki.kernel.org/index.php/Tutorial


Sounds worth a look. Thanks!

On Thursday, 7 May 2020 at 11:06:17 UTC, Dennis wrote:
>
> You can make a reference program that you use to get a measure for how fast the computer is that you run the benchmark on. Then you can use that to scale your actual benchmark results.
>
> When testing regressions there's a fairly obvious choice for this reference program: the old version. You can compare those results with the new version and report the relative difference.

I have to keep that in mind, but I'd prefer something that does not require keeping old sources working, or floating their binaries around.