Increased input size and added argument for iteration count.
Run-times were: 0.095s/0.651s/3.257s
Are now: 0,145s/1,551s/8,064s

real/smallpt's .slowstdout file was missing.
The value in spectral/dom-lt's was wrong, also runtime was lower
than 5 sek on my machine (should be between 5-10s),
so I increased the argument and updated dom-lt.slowstdout.

bc. program invocation on shake based nofib-run was broken.

Also added a few arguments to input to make single iteration longer
and adjusted iteration counts to get desired execution times.

Alex D authored 3 years ago and Matthew Pickering committed 3 years ago

Current test does some floating-point manipulations which can be non-deterministic
due to different platforms/architectures. This change truncates the result to
avoid potential stdout mismatch.

Ben Gamari authored 3 years ago and Andreas Klebinger committed 3 years ago

By reducing the size of the raytraced image.

* We adjust for the specialisation of Data.List by hiding the generic
  version from the Prelude.

This should make this change backwards compatible with older base
versions.

* Adjust the gc benchmarks. I've mostly made them the same as their
  non-gc parts now.

This adds the port of the smallpt raytracer by David Scies and bollu.

https://github.com/bollu/smallpt-opt/

This introduces a Shake-based build system, building on Neil Mitchell's
prototype from several years ago [1].

This build system has a number of features relative to the existing
make-based build system:

 * results are produced in a common CSV structure, making it easy to use
   the included analysis tool nofib-compare, to compare results from tests
   outside of nofib.
 * Cachegrind runs can be run in parallel, greatly speeding up such
   measurements
 * native support for (micro-)architectural performance counters on
   Linux via `perf stat`

[1] https://gist.github.com/ndmitchell/4733855

There are also a few more changes to tests to make this build system
work well with nofib.

In particular:

* The Main module for some benchmarks has been renamed.
* The boot phase of benchmarks is now done by a boot.sh script.
* Replace some uses of perl with ghci.
* Some benchmarks that we didn't use to run by default had
  missing/wrong(!) stdout files. I added/updated these.

Take note that shake will run *more* benchmarks by default than make
used to. So you might want to select a subset of benchmarks to run when
using it.

Simon Jakobi authored 5 years ago and Ben Gamari committed 4 years ago

Previously I would get an error during `make boot`:

    EffBench.hs:10:18: error:
        Could not load module ‘Control.Monad.State.Strict’
        It is a member of the hidden package ‘mtl-2.2.2’.
        You can run ‘:set -package mtl’ to expose it.
        (Note: this unloads all the modules in the current scope.)
        Use -v (or `:set -v` in ghci) to see a list of the files searched for.
       |
    10 | import qualified Control.Monad.State.Strict as S
       |

Since morally these flags are for the program and not runtest

Remove both unnecessary SRC_DEPS depenencies and -fglasgow-exts flags.

Instead of passing package dependencies via `SRC_HC_OPTS=-package ...`
we rather pass them via `SRC_DEPS`, allowing the build system to
know about the dependency.

Required due to an amendment that disallows as-patterns with leading
whitespace:

  f x@ pat = rhs  -- now disallowed

Vladislav Zavialov authored 5 years ago and Sebastian Graf committed 5 years ago

GHC Proposal #229 changes the lexical rules of Haskell, which may
require slight whitespace adjustments in certain cases.

This patch changes formatting in nofib in a way that enables it to
compile under the proposed rules.

Andreas Klebinger authored 6 years ago and Andreas Klebinger committed 6 years ago

Of interest is in particular that the code uses -XStrict.

Sebastian Graf authored 6 years ago and Sebastian Graf committed 6 years ago

Also check that invariant in CI: After a `make distclean`
`git clean -nxd` should find no files.

The hope is that this catches cases were a `make clean` would forget to
list a `.gitignore`d file that should either be registered as a
`(DIST_)CLEAN_FILES` or should be unignored and committed into the
repository.

Summary:
This is due to #15999, a follow-up on #5793 and #15357 and changes all
benchmarks, some of them (i.e. `wheel-sieve1`, `awards`) rather drastically.

The general plan is outlined in #15999: Identify GC-sensitive benchmarks by
looking at how productivity rates change over different nursery sizes and
iterate `main` of these benchmarks often enough for the non-monotony and
discontinuities to go away.

I was paying attention that the benchmarked logic is actually run $n times more
often, rather than just benchmarking IO operations printing the result of CAFs.

When I found benchmarks with insignificant runtime (#15357), I made sure that
parameters/input files were adjusted so that runtime of the different modes
fall within the ranges proposed in
https://ghc.haskell.org/trac/ghc/ticket/15357#comment:4

- fast: 0.1-0.2s
- norm: 1-2s
- slow: 5-10s

This is what I did:

- Stabilise bernoulli
- Stabilise digits-of-e1
- Stabilise digits-of-e2
- Stabilise gen_regexp
- Adjust running time of integrate
- Adjust running time of kahan
- Stabilise paraffins
- Stabilise primes
- Adjust running time of rfib
- Adjust running time of tak
- Stabilise wheel-sieve1
- Stabilise wheel-sieve2
- Adjust running time of x2n1
- Adjust running time of ansi
- Adjust running time of atom
- Make awards benchmark something other than IO
- Adjust running time of banner
- Stabilise boyer
- Adjust running time of boyer2
- Adjust running time of queens
- Adjust running time of calendar
- Adjust runtime of cichelli
- Stabilise circsim
- Stabilise clausify
- Stabilise constraints with moderate success
- Adjust running time of cryptarithm1
- Adjust running time of cryptarythm2
- Adjust running time of cse
- Adjust running time of eliza
- Adjust running time of exact-reals
- Adjust running time of expert
- Stabilise fft2
- Stabilise fibheaps
- Stabilise fish
- Adjust running time for gcd
- Stabilise comp_lab_zift
- Stabilise event
- Stabilise fft
- Stabilise genfft
- Stabilise ida
- Adjust running time for listcompr
- Adjust running time for listcopy
- Adjust running time of nucleic2
- Attempt to stabilise parstof
- Stabilise sched
- Stabilise solid
- Adjust running time of transform
- Adjust running time of typecheck
- Stabilise wang
- Stabilise wave4main
- Adjust running time of integer
- Adjust running time of knights
- Stabilise lambda
- Stabilise lcss
- Stabilise life
- Stabilise mandel
- Stabilise mandel2
- Adjust running time of mate
- Stabilise minimax
- Adjust running time of multiplier
- Adjust running time of para
- Stabilise power
- Adjust running time of primetest
- Stabilise puzzle with mild success
- Adjust running time for rewrite
- Stabilise simple with mild success
- Stabilise sorting
- Stabilise sphere
- Stabilise treejoin
- Stabilise anna
- Stabilise bspt
- Stabilise cacheprof
- Stablise compress
- Stablise compress2
- Stabilise fem
- Adjust running time of fluid
- Stabilise fulsom
- Stabilise gamteb
- Stabilise gg
- Stabilise grep
- Adjust running time of hidden
- Stabilise hpg
- Stabilise infer
- Stabilise lift
- Stabilise linear
- Attempt to stabilise maillist
- Stabilise mkhprog
- Stabilise parser
- Stabilise pic
- Stabilise prolog
- Attempt to stabilise reptile
- Adjust running time of rsa
- Adjust running time of scs
- Stabilise symalg
- Stabilise veritas
- Stabilise binary-trees
- Adjust running time of fasta
- Adjust running time of k-nucleotide
- Adjust running time of pidigits
- Adjust running time of reverse-complement
- Adjust running time of spectral-norm
- Adjust running time of fannkuch-redux
- Adjust running time for n-body

Problematic benchmarks:

- `last-piece`: Unclear how to stabilise. Runs for 300ms and I can't make up smaller inputs because I don't understand what it does.
- `pretty`: It's just much too small to be relevant at all. Maybe we want to get rid of this one?
- `scc`: Same as `pretty`. The input graph for which SCC analysis is done is much too small and I can't find good directed example graphs on the internet.
- `secretary`: Apparently this needs `-package random` and consequently hasn't been run for a long time.
- `simple`: Same as `last-piece`. Decent runtime (70ms), but it's unstable and I see no way to iterate it ~100 times in fast mode.
- `eff`: Every benchmark is problematic here. Not from the point of view of allocations, but because the actual logic is vacuous. IMO, these should be performance tests, not actual benchmarks. Alternatively, write an actual application that makes use of algebraic effects.
- `maillist`: Too trivial. It's just String/list manipulation, not representative of any Haskell code we would write today (no use of base library functions which could be fused, uses String instead of Text). It's only 75 loc according to `cloc`, that's not a `real` application.

Reviewers: simonpj, simonmar, bgamari, AndreasK, osa1, alpmestan, O26 nofib

GHC Trac Issues: #15999

Differential Revision: https://phabricator.haskell.org/D5438

The pragma hasn't been recognised for a long time.
Replacing it with bangs will change program semantics to how it was
intended. The perf CI build is currently broken anyway, so this
shouldn't skew numbers too much.

This should fix the build in the same way as !2.

Summary:
We don't want the result in the repo as it's a sizeable binary file that
doesn't compress well.

Storing the output file in the repository becomes infeasible for large
inputs. There are two possible remedies:

1. Generate the result files during `make boot` (Phab:D5426). We
   discovered some drawbacks (like missing dependency files to build
   `compress` during boot) to this approach which make it infeasible.
2. Shrink the output files, for example by hashing the string that we would
   normally output and compare that instead.

This patch implements the second alternative. This somewhat distorts the
runtime profile, so we might want to consider doing hashing within the
benchmark runner in the future.

Test Plan: make boot

Reviewers: AndreasK, nomeata, O26 nofib, osa1

Reviewed By: AndreasK, osa1

Subscribers: osa1

Differential Revision: https://phabricator.haskell.org/D5469

Summary:
When we don't remove the generated files, `make boot` won't regenerate
them. That in turn leads to expected output failures after bumping
`{FAST,NORM,SLOW}_OPTS`.

Test Plan: make clean && make boot && make

Reviewers: AndreasK, mpickering, osa1, alpmestan, O26 nofib

Reviewed By: alpmestan

Differential Revision: https://phabricator.haskell.org/D5365

Summary:
The `make boot` phase of `real/compress` generates files that are
cleaned up by `make clean` with `rm`. Doing `make clean` without a
prior `make boot` leads to a failure.

This just supplies `-f` to `rm` so that plain `make clean` no longer
fails.

Reviewers: O26 nofib

Differential Revision: https://phabricator.haskell.org/D5359

Runtime for nofib benchmarks was all over the place.
This patch adjusts runtime for most benchmarks such
that it falls into the 0.2-2s range.

This means that:
* A default run will take longer
* Time spent will be better distributed among benchmarks.
* More benchmarks have runtimes long enough to be used
  for runtime analysis.

Some more changes were done which go hand in hand
with changing runtimes.
* Some benchmarks now create their input files during boot.
* Moved input files for anna in their own directory.
* Remove printing of output for some of the floating
  point heavy benchmarks.
* Added a comment about desired runtimes to README.
* Made grep actually benchmark something.
* Throw cachgrind out of the default benchmarks.
  The nondeterministic behaviour has been an issue for a
  while and it doesn't seem like an essential benchmark.

Test Plan: run nofib in modes slow/normal/fast

Reviewers: O26 nofib, alpmestan

Reviewed By: alpmestan

Subscribers: sgraf, alpmestan

Differential Revision: https://phabricator.haskell.org/D4989