The nofib build system

This documents usage of nofib's Shake-based build system. The build system, nofib-run, automates running of the nofib benchmarks and the collection of a variety of compile- and run-time metrics. These metrics are recorded in a common semi-structured CSV file format, which can be compared and analysed with the included nofib-compare utility.

Usage

The nofib-run executable is used to run the benchmark suite and accepts a variety of arguments. By default the results of the run will be placed in _make/{compiler version}. The compiler version component of this path can be overridden with the --output flag.

For instance (run from the root nofib directory),

$ cabal v2-run -- nofib-run --compiler=/path/to/ghc --output=test

will produce a number of files in _make/:

$ ls -R _make

_make:
8.11.0.20200205

_make/8.11.0.20200205:
gc  imaginary  parallel  real  spectral

_make/8.11.0.20200205/gc:
cacheprof  circsim  constraints  fibheaps  gc_bench  happy  hash  lcss  linear  mutstore1  mutstore2  spellcheck  treejoin

_make/8.11.0.20200205/gc/cacheprof:
Arch_x86.hi  Arch_x86.o.result        Arch_x86.o.stats  Generics.hi  Generics.o.result        Generics.o.stats  Main.deps  Main.o         Main.o.results.csv   Main.result
Arch_x86.o   Arch_x86.o.results.csv   config.txt        Generics.o   Generics.o.results.csv   Main              Main.hi    Main.o.result  Main.o.stats         Main.results.csv

...

In the case of the gc/cacheprof benchmark we see the following were produced:

• build artifacts from the build of the benchmark (e.g. *.o, *.hi)
• for each object file a .results.csv file which contains the metrics for the object size
• Main.results.csv, which aggregates all of the above *.results.csv files

A number of classes of metrics are collected:

• object code size for each module
• runtime system statistics (e.g. mutator time, GC time, GC counts, allocations) from the execution of ghc while compiling the testcase
• runtime system statistics from the execution of the testcase

Cachegrind

The benchmarks can also be run under valgrind's cachegrind tool with the --cachegrind, which simulates a simple cache hierarchy to allow (mostly) deterministic modelling of instruction counts, memory, and cache effects. When running with --cachegrind tests can be safely parallelised with the -j<n> flag.

Performance counters with perf_events

The benchmarks can also be run under the Linux perf tool for collection of (micro-)architectural event counts. This mode is enabled with the --perf flag. Arguments to perf can be passed via --perf-arg.

Data format

The output of nofib-run is a simple CSV format (defined by the Measurements type in src/Measurements.hs) represents semi-structured string keys (known as Labels) and floating-point values. In particular, labels have a path-like structure with / being the component delimiter.

Comparing results

$ cabal v2-run nofib-compare -- approach-b.results.csv approach-a.results.csv

nofib-compare supports several different output formats, selectable with the -f flag, namely: ascii (default), markdown, csv, latex

Broken tests

Occasionally we have to deal with broken tests. The --keep-going flags tries to run as many tests as possible even if some of them are broken. This currently only allows for ignoring missmatched outputs, not crashes/compilation failures.

There is still the option of listing the non-broken tests explicitly if --keep-going doesn't work.

head.hackage

It's often necessary to use the head.hackage patchset to compile the dependencies with development GHC versions. nofib-run supports this conveniently via the --head flag.

First, update the head.hackage package index with:

cabal v2-update --project-file=cabal.project.head-hackage head.hackage.ghc.haskell.org

Now run nofib-run as usual, adding the --head flag to your command-line.