Make nofib suitable for runtime measurements.
Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless.
This comes with a bunch of issues:
- It takes up build/runtime while not actually improving the accuracy of runtime measurements.
- It skews results as the jump from 1ms to 2ms registers as a 200% change.
- It can hide regression which fall outside of the measured granularity.
- Higher number of runs per benchmark spend far too much time on the few slower benchmarks.
At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms.
For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data.
Below are the runtimes of a recent default nofib run.
| benchmark | Time (ms) |
|---|---|
| ansi | 0 |
| awards | 0 |
| banner | 0 |
| calendar | 0 |
| expert | 0 |
| gen_regexps | 0 |
| grep | 0 |
| pretty | 0 |
| scc | 0 |
| cse | 1 |
| eliza | 1 |
| lift | 1 |
| mkhprog | 1 |
| prolog | 1 |
| sorting | 1 |
| veritas | 1 |
| knights | 2 |
| minimax | 2 |
| x2n1 | 2 |
| mandel2 | 3 |
| parstof | 3 |
| fluid | 4 |
| pic | 4 |
| boyer2 | 5 |
| bspt | 5 |
| cryptarithm2 | 5 |
| fish | 6 |
| gg | 6 |
| reptile | 6 |
| symalg | 6 |
| tak | 6 |
| VSD | 7 |
| rfib | 7 |
| queens | 9 |
| rewrite | 9 |
| sched | 11 |
| fem | 12 |
| fibheaps | 13 |
| parser | 14 |
| rsa | 14 |
| genfft | 15 |
| clausify | 17 |
| gamteb | 17 |
| fft | 18 |
| boyer | 20 |
| gcd | 20 |
| sphere | 21 |
| fft2 | 23 |
| infer | 25 |
| maillist | 25 |
| mandel | 27 |
| nucleic2 | 30 |
| primes | 34 |
| cichelli | 35 |
| ida | 39 |
| listcompr | 41 |
| listcopy | 43 |
| multiplier | 43 |
| wang | 43 |
| hpg | 45 |
| anna | 46 |
| reverse-complem | 46 |
| integrate | 48 |
| primetest | 50 |
| paraffins | 54 |
| solid | 54 |
| puzzle | 55 |
| treejoin | 61 |
| compress2 | 64 |
| compress | 65 |
| event | 65 |
| bernouilli | 74 |
| comp_lab_zift | 78 |
| simple | 86 |
| wheel-sieve2 | 91 |
| life | 100 |
| exp3_8 | 102 |
| wave4main | 103 |
| typecheck | 106 |
| atom | 110 |
| fulsom | 112 |
| CS | 116 |
| para | 120 |
| kahan | 138 |
| transform | 145 |
| power | 150 |
| cacheprof | 151 |
| hidden | 160 |
| scs | 168 |
| lcss | 175 |
| wheel-sieve1 | 199 |
| VS | 204 |
| pidigits | 216 |
| FS | 222 |
| last-piece | 230 |
| fasta | 263 |
| cryptarithm1 | 264 |
| CSD | 272 |
| digits-of-e1 | 273 |
| binary-trees | 342 |
| digits-of-e2 | 381 |
| circsim | 418 |
| S | 451 |
| VSM | 497 |
| lambda | 617 |
| integer | 905 |
| n-body | 1043 |
| linear | 1070 |
| spectral-norm | 1132 |
| constraints | 1171 |
| exact-reals | 1289 |
| mate | 1955 |
| fannkuch-redux | 2185 |
| k-nucleotide | 2989 |