Make nofib suitable for runtime measurements.
Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless.
This comes with a bunch of issues:
- It takes up build/runtime while not actually improving the accuracy of runtime measurements.
- It skews results as the jump from 1ms to 2ms registers as a 200% change.
- It can hide regression which fall outside of the measured granularity.
- Higher number of runs per benchmark spend far too much time on the few slower benchmarks.
At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms.
For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data.
Below are the runtimes of a recent default nofib run.
benchmark | Time (ms) |
---|---|
ansi | 0 |
awards | 0 |
banner | 0 |
calendar | 0 |
expert | 0 |
gen_regexps | 0 |
grep | 0 |
pretty | 0 |
scc | 0 |
cse | 1 |
eliza | 1 |
lift | 1 |
mkhprog | 1 |
prolog | 1 |
sorting | 1 |
veritas | 1 |
knights | 2 |
minimax | 2 |
x2n1 | 2 |
mandel2 | 3 |
parstof | 3 |
fluid | 4 |
pic | 4 |
boyer2 | 5 |
bspt | 5 |
cryptarithm2 | 5 |
fish | 6 |
gg | 6 |
reptile | 6 |
symalg | 6 |
tak | 6 |
VSD | 7 |
rfib | 7 |
queens | 9 |
rewrite | 9 |
sched | 11 |
fem | 12 |
fibheaps | 13 |
parser | 14 |
rsa | 14 |
genfft | 15 |
clausify | 17 |
gamteb | 17 |
fft | 18 |
boyer | 20 |
gcd | 20 |
sphere | 21 |
fft2 | 23 |
infer | 25 |
maillist | 25 |
mandel | 27 |
nucleic2 | 30 |
primes | 34 |
cichelli | 35 |
ida | 39 |
listcompr | 41 |
listcopy | 43 |
multiplier | 43 |
wang | 43 |
hpg | 45 |
anna | 46 |
reverse-complem | 46 |
integrate | 48 |
primetest | 50 |
paraffins | 54 |
solid | 54 |
puzzle | 55 |
treejoin | 61 |
compress2 | 64 |
compress | 65 |
event | 65 |
bernouilli | 74 |
comp_lab_zift | 78 |
simple | 86 |
wheel-sieve2 | 91 |
life | 100 |
exp3_8 | 102 |
wave4main | 103 |
typecheck | 106 |
atom | 110 |
fulsom | 112 |
CS | 116 |
para | 120 |
kahan | 138 |
transform | 145 |
power | 150 |
cacheprof | 151 |
hidden | 160 |
scs | 168 |
lcss | 175 |
wheel-sieve1 | 199 |
VS | 204 |
pidigits | 216 |
FS | 222 |
last-piece | 230 |
fasta | 263 |
cryptarithm1 | 264 |
CSD | 272 |
digits-of-e1 | 273 |
binary-trees | 342 |
digits-of-e2 | 381 |
circsim | 418 |
S | 451 |
VSM | 497 |
lambda | 617 |
integer | 905 |
n-body | 1043 |
linear | 1070 |
spectral-norm | 1132 |
constraints | 1171 |
exact-reals | 1289 |
mate | 1955 |
fannkuch-redux | 2185 |
k-nucleotide | 2989 |