Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • GHC GHC
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 5,248
    • Issues 5,248
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 562
    • Merge requests 562
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Releases
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Glasgow Haskell CompilerGlasgow Haskell Compiler
  • GHCGHC
  • Issues
  • #15357
Closed
Open
Issue created Jul 09, 2018 by Andreas Klebinger@AndreasKDeveloper

Make nofib suitable for runtime measurements.

Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless.

This comes with a bunch of issues:

  • It takes up build/runtime while not actually improving the accuracy of runtime measurements.
  • It skews results as the jump from 1ms to 2ms registers as a 200% change.
  • It can hide regression which fall outside of the measured granularity.
  • Higher number of runs per benchmark spend far too much time on the few slower benchmarks.

At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms.

For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data.

Below are the runtimes of a recent default nofib run.

benchmark Time (ms)
ansi 0
awards 0
banner 0
calendar 0
expert 0
gen_regexps 0
grep 0
pretty 0
scc 0
cse 1
eliza 1
lift 1
mkhprog 1
prolog 1
sorting 1
veritas 1
knights 2
minimax 2
x2n1 2
mandel2 3
parstof 3
fluid 4
pic 4
boyer2 5
bspt 5
cryptarithm2 5
fish 6
gg 6
reptile 6
symalg 6
tak 6
VSD 7
rfib 7
queens 9
rewrite 9
sched 11
fem 12
fibheaps 13
parser 14
rsa 14
genfft 15
clausify 17
gamteb 17
fft 18
boyer 20
gcd 20
sphere 21
fft2 23
infer 25
maillist 25
mandel 27
nucleic2 30
primes 34
cichelli 35
ida 39
listcompr 41
listcopy 43
multiplier 43
wang 43
hpg 45
anna 46
reverse-complem 46
integrate 48
primetest 50
paraffins 54
solid 54
puzzle 55
treejoin 61
compress2 64
compress 65
event 65
bernouilli 74
comp_lab_zift 78
simple 86
wheel-sieve2 91
life 100
exp3_8 102
wave4main 103
typecheck 106
atom 110
fulsom 112
CS 116
para 120
kahan 138
transform 145
power 150
cacheprof 151
hidden 160
scs 168
lcss 175
wheel-sieve1 199
VS 204
pidigits 216
FS 222
last-piece 230
fasta 263
cryptarithm1 264
CSD 272
digits-of-e1 273
binary-trees 342
digits-of-e2 381
circsim 418
S 451
VSM 497
lambda 617
integer 905
n-body 1043
linear 1070
spectral-norm 1132
constraints 1171
exact-reals 1289
mate 1955
fannkuch-redux 2185
k-nucleotide 2989
Edited Mar 10, 2019 by Andreas Klebinger
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking