README.md 2.41 KB
Newer Older
1 2 3
# NoFib: Haskell Benchmark Suite

This is the root directory of the "NoFib Haskell benchmark suite". It
4 5
should be part of a GHC source tree, that is the 'nofib' directory
should be at the same level in the tree as 'compiler' and 'libraries'.
6

7 8 9 10 11 12 13 14 15 16
## Package Depedencies

Please make sure you have the following packages installed for your
system GHC:
 * html
 * regex-compat (will install: mtl, regex-base, regex-posix)

## Using

Then, to run the tests, execute:
17

tibbe's avatar
tibbe committed
18 19 20
    make clean
    make boot
    make 2>&1 | tee nofib-log
21

tibbe's avatar
tibbe committed
22 23
This will put the results in the file `nofib-log`. You can pass extra
options to a nofib run using the `EXTRA_HC_OPTS` variable like this:
24

tibbe's avatar
tibbe committed
25 26 27
    make clean
    make boot
    make EXTRA_HC_OPTS="-fllvm" >&1 | tee nofib-llvm-log
28

29 30 31 32 33 34 35 36
Likewise, you can pass additional arguments (e.g. RTS flags) to the command
itself by using the `EXTRA_RUNTEST_OPTS` variable like this:

    make EXTRA_RUNTEST_OPTS="-- +RTS -A2M -RTS"

The `--` here ensures that `runtest` doesn't attempt to interpret any of the
given flags as its own.

37
To compare the results of multiple runs, use the program in
tibbe's avatar
tibbe committed
38
`../utils/nofib-analyse`, for example:
39

tibbe's avatar
tibbe committed
40
    nofib-analyse nofib-log-6.4.2 nofib-log-6.6
41

tibbe's avatar
tibbe committed
42 43
to generate a comparison of the runs in captured in `nofib-log-6.4.2`
and `nofib-log-6.6`. When making comparisons, be careful to ensure
44
that the things that changed between the builds are only the things
45
that you _wanted_ to change. There are lots of variables: machine,
46
GHC version, GCC version, C libraries, static vs. dynamic GMP library,
47
build options, run options, and probably lots more. To be on the safe
48
side, make both runs on the same unloaded machine.
49

50 51 52
## Configuration

There are some options you might want to tweak; search for nofib in
tibbe's avatar
tibbe committed
53
`../mk/config.mk`, and override settings in `../mk/build.mk` as usual.
54 55 56

## Extra Metrics: Valgrind

57
To get instruction counts, memory reads/writes, and "cache misses",
tibbe's avatar
tibbe committed
58 59
you'll need to get hold of Cachegrind, which is part of
[Valgrind](http://valgrind.org).
60

61
## Extra Packages
dterei's avatar
dterei committed
62

63 64 65
Some benchmarks aren't run by default and require extra packages are
installed for the GHC compiler being tested. These packages include:
 * stm - for smp benchmarks
dterei's avatar
dterei committed
66

67 68 69 70 71 72 73 74 75 76 77
## Adding benchmarks

If you add a benchmark try to set the problem sizes for
fast/normal/slow reasonably.

Runtimes for normal should be above 0.3s if that can be reasonably
achieved. Less than that and there is a chance
nofib-analyse will ignore the result if it falls below 0.2s.