Compiletime Vs Runtime bencmarks

Aug 17, 2015

D_Learner

Aug 17, 2015

Edwin van Leeuwen

Aug 17, 2015

Aug 17, 2015

Aug 17, 2015

Aug 18, 2015

Aug 18, 2015

Aug 18, 2015

Aug 18, 2015

August 17, 2015

Compiletime Vs Runtime bencmarks

Posted by D_Learner

Permalink

D_Learner

Permalink

Hello everyone . I need advice on my first D-project . I have uploaded it at :-

https://bitbucket.org/mrjohns/matcher/downloads

IDEA : Benchmarking of 3 runtime algorithms and comparing them to their compile-time variants. The only difference between these is that for the compile time-ones, the lookup tables (i.e. Arrays bmBc, bmGs, and suffixes ) must be computed at compile time( I currently rely on CTFE ) . While for the runtime-ones the lookup tables are computed on runtime.

NB : The pattern matching algorithms themselves need not be executed  at compile-time, only the lookup tables.Having stated this the algorithms which run on known( compile-time computed) tables must be faster than the ones which have to compute them at runtime.

My results seem to show something different, only the first   pair(
BM_Runtime and BM_Compile-time)   yields  admissible results, the other two pair give higher execution time for the compile-time variants. I think am missing something here. Please help.

Current Results  for the pattern="GCAGAGAG"  are as below :-

BM_Runtime          = 366 hnsecs     position=   513
BM_Compile-time     = 294 hnsecs     position   =513

BMH_Runtime         = 174 hnsecs     position=   513
BMH_Compile-time    = 261 hnsecs     position=   513

AG_Run-time         = 258 hnsecs    position=   513
AG_Compile-time     = 268 hnsecs    position=   513


Running the code : dmd -J.  matcher.d inputs.d  rtime_pre.d ctime_pre.d && numactl --physcpubind=0 ./matcher


I would appreciate your suggestions.

Regards,

On Monday, 17 August 2015 at 14:43:35 UTC, D_Learner wrote: > Hello everyone . I need advice on my first D-project . I have uploaded it at :- > > Current Results for the pattern="GCAGAGAG" are as below :- > > BM_Runtime = 366 hnsecs position= 513 > BM_Compile-time = 294 hnsecs position =513 > > BMH_Runtime = 174 hnsecs position= 513 > BMH_Compile-time = 261 hnsecs position= 513 > > AG_Run-time = 258 hnsecs position= 513 > AG_Compile-time = 268 hnsecs position= 513 > > > Running the code : dmd -J. matcher.d inputs.d rtime_pre.d ctime_pre.d && numactl --physcpubind=0 ./matcher > Hi, What happens if you run each algorithm many (say 100000) times. The current times seem to short to be reliable (variation in runtimes would be too great). Regards, Edwin

On Monday, 17 August 2015 at 14:52:18 UTC, Edwin van Leeuwen wrote: > On Monday, 17 August 2015 at 14:43:35 UTC, D_Learner wrote: >> Hello everyone . I need advice on my first D-project . I have uploaded it at :- >> >> Current Results for the pattern="GCAGAGAG" are as below :- >> >> BM_Runtime = 366 hnsecs position= 513 >> BM_Compile-time = 294 hnsecs position =513 >> >> BMH_Runtime = 174 hnsecs position= 513 >> BMH_Compile-time = 261 hnsecs position= 513 >> >> AG_Run-time = 258 hnsecs position= 513 >> AG_Compile-time = 268 hnsecs position= 513 >> >> >> Running the code : dmd -J. matcher.d inputs.d rtime_pre.d ctime_pre.d && numactl --physcpubind=0 ./matcher >> > > Hi, > > What happens if you run each algorithm many (say 100000) times. The current times seem to short to be reliable (variation in runtimes would be too great). > > Regards, Edwin Thyanks, I havent tried that. So it seems you suggest I must fucus on average times. Will try it , the above results are from a single run (no looping), but after executing the program several times I realised that the 1st two seem ok(relatively stable results for same pattern), but the other pair give higher performance for the run-time versions.

On Monday, 17 August 2015 at 14:52:18 UTC, Edwin van Leeuwen wrote: > On Monday, 17 August 2015 at 14:43:35 UTC, D_Learner wrote: >> Hello everyone . I need advice on my first D-project . I have uploaded it at :- >> >> Current Results for the pattern="GCAGAGAG" are as below :- >> >> BM_Runtime = 366 hnsecs position= 513 >> BM_Compile-time = 294 hnsecs position =513 >> >> BMH_Runtime = 174 hnsecs position= 513 >> BMH_Compile-time = 261 hnsecs position= 513 >> >> AG_Run-time = 258 hnsecs position= 513 >> AG_Compile-time = 268 hnsecs position= 513 >> >> >> Running the code : dmd -J. matcher.d inputs.d rtime_pre.d ctime_pre.d && numactl --physcpubind=0 ./matcher >> > > Hi, > > What happens if you run each algorithm many (say 100000) times. The current times seem to short to be reliable (variation in runtimes would be too great). > > Regards, Edwin The surprisingly, the D-profiler gives plausible results:- Algorithm1 2921 int rtime_pre.bm_rmatch (runtime ) 2122 int ctime_pre.bm_cmatch (compiletime ) 1317 int rtime_pre.bmh_rmatch (runtime ) 1099 int ctime_pre.bmh_cmatch (compiletime ) 3959 int rtime_pre.ag_rmatch (runtime ) 2688 pure int ctime_pre.ag_cmatch (compiletime ) This suggests that my timer design has some flaw ;)

On Monday, 17 August 2015 at 17:48:22 UTC, D_Learner wrote: > On Monday, 17 August 2015 at 14:52:18 UTC, Edwin van Leeuwen wrote: >> [...] > > The surprisingly, the D-profiler gives plausible results:- > Algorithm1 > 2921 int rtime_pre.bm_rmatch (runtime ) > 2122 int ctime_pre.bm_cmatch (compiletime ) > > 1317 int rtime_pre.bmh_rmatch (runtime ) > 1099 int ctime_pre.bmh_cmatch (compiletime ) > > > 3959 int rtime_pre.ag_rmatch (runtime ) > 2688 pure int ctime_pre.ag_cmatch (compiletime ) > > This suggests that my timer design has some flaw ;) std.datetime.StopWatch is the easiest way to do timing manually, or just use std.datetime.benchmark

On Monday, 17 August 2015 at 22:01:32 UTC, John Colvin wrote: > On Monday, 17 August 2015 at 17:48:22 UTC, D_Learner wrote: >> On Monday, 17 August 2015 at 14:52:18 UTC, Edwin van Leeuwen wrote: >>> [...] >> >> The surprisingly, the D-profiler gives plausible results:- >> Algorithm1 >> 2921 int rtime_pre.bm_rmatch (runtime ) >> 2122 int ctime_pre.bm_cmatch (compiletime ) >> >> 1317 int rtime_pre.bmh_rmatch (runtime ) >> 1099 int ctime_pre.bmh_cmatch (compiletime ) >> >> >> 3959 int rtime_pre.ag_rmatch (runtime ) >> 2688 pure int ctime_pre.ag_cmatch (compiletime ) >> >> This suggests that my timer design has some flaw ;) > > std.datetime.StopWatch is the easiest way to do timing manually, or just use std.datetime.benchmark My code already uses std.datetime.StopWatch , but I get results which are no match to the D compilers.

On Tuesday, 18 August 2015 at 07:16:32 UTC, D_Learner wrote: > On Monday, 17 August 2015 at 22:01:32 UTC, John Colvin wrote: >> On Monday, 17 August 2015 at 17:48:22 UTC, D_Learner wrote: >>> On Monday, 17 August 2015 at 14:52:18 UTC, Edwin van Leeuwen wrote: >>>> [...] >>> >>> The surprisingly, the D-profiler gives plausible results:- >>> Algorithm1 >>> 2921 int rtime_pre.bm_rmatch (runtime ) >>> 2122 int ctime_pre.bm_cmatch (compiletime ) >>> >>> 1317 int rtime_pre.bmh_rmatch (runtime ) >>> 1099 int ctime_pre.bmh_cmatch (compiletime ) >>> >>> >>> 3959 int rtime_pre.ag_rmatch (runtime ) >>> 2688 pure int ctime_pre.ag_cmatch (compiletime ) >>> >>> This suggests that my timer design has some flaw ;) >> >> std.datetime.StopWatch is the easiest way to do timing manually, or just use std.datetime.benchmark > > My code already uses std.datetime.StopWatch , but I get results which are no match to the D profiler's. Though am not searching for exact, but theyy must be comparable atleast. When you say D profiler, you mean dmd's built-in profiler, yes? That is an instrumenting profiler which adds a not insignificant cost to function calls, which means you have to be careful interpreting the results

On Monday, 17 August 2015 at 14:43:35 UTC, D_Learner wrote: > Hello everyone . I need advice on my first D-project . I have uploaded it at :- > > [...] I wouldn't compare benchmarks without optimisations turned on. Atila

Forums