Performance Tests

The test suite contains a number of performance tests included as part of a normal run of the test suite. Performance tests measure some metric (e.g. number of bytes allocated) of ghc or generated code, then compares that metric to some baseline value. Tests will fail if the measured metric is not within some tolerance percentage of the baseline value. If you'd like to add your own test, then see adding performance Tests. To simply run the test suite, see Running the Testsuite and specifically here for establishing performance test baselines.

See here for notes on the original proposal.

Quick Start / Example

You want to run the performance tests locally on 2 commits and compare the results. Do this:

# Checkout the first commit
git checkout a12b34c56 && git submodule update --init
# Check that the working tree is clean (else metrics will not be saved automatically).
git status
# Run the test suite (optionally use --only-perf to only run performance tests).
# Performance metrics are automatically saved using git notes.
./hadrian/build.sh test

# Checkout the second commit
git checkout x98y76z54 && git submodule update --init
# Check that the working tree is clean (else metrics will not be saved automatically).
git status
# Run the test suite (optionally use --only-perf to only run performance tests).
# Performance metrics are automatically saved using git notes.
./hadrian/build.sh test

# Generate a chart of results for the 2 commits.
python3 testsuite/driver/perf_notes.py --chart ./chart.html --test-env local a12b34c56 x98y76z54
firefox ./chart.html

Note the final arguments of the python script are the commit hashes. You may use more that 2 hashes if you wish, or you can use a commit range e.g. a12b34c56..HEAD. See python3 testsuite/driver/perf_notes.py --help to seem more options for generating the chart.

Performance Metrics are Logged

Whenever a performance test is run, the resulting metrics are stored using git notes under the "perf" ref space on the current commit. Git notes are generally stored locally, and not shared between git repositories (e.g. when pushing/pulling branches). This is desirable as performance is largely dependent on the machine on which the tests are run. Each metric saved has the following data:

• Environment usually just 'local' unless run on CI.
• Test the name of the test.
• Way the way used to run the test.
• Metric what metric was recorded e.g. max bytes allocated.
• Value the observed value of the metric.

While you're free to delete notes manually via the git notes --ref=perf command, the test runner will never delete results, so multiple values for the same test may be recorded.

CI Performance Metrics

Gitlab CI is setup to collect performance metrics and push them (again as git notes) to a separate repo: https://gitlab.haskell.org/ghc/ghc-performance-notes.git. This will only contain performance metrics for commits on master, and CI jobs that pass successfully (keep this in mind when analyzing CI results).

You can fetch these results to the "ci/perf" ref space using the following command:

$ git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf

This allows the test runner to use CI results to derive baselines where local results are not available.

How Baselines are Calculated

While a tolerance percentage is specified manually in *.T, the baseline (i.e. expected) value of performance tests are recovered from previous runs of the performance tests (logged in git notes). Results fetched from CI may also be used, in which case a CI environment is chosen based on the local machine architecture and os. Baselines are derived per (Test, Way, Metric) independently, by searching git notes as follows:

1. If the current commit message specifies an expected change for the metric, then stop. The Baseline is undefined.
2. Move to the parent commit.
3. If metrics for the given (Test, Way, Metric) exist in a git note for this commit (in ref space perf), then the baseline is the average value of those metrics.
4. Else if metrics for the given (Chosen CI Environment, Test, Way, Metric) exist in a git note for this commit (in ref space ci/perf), then the baseline is the average value of those metrics.
5. If the maximum search depth is reached then stop, the baseline is undefined. Else continue to step 1.

Expected Performance Changes

In many cases, a new commit has expected performance changes. In order to allow the test suite to pass, these changes must be documented in the commit message in the format

Metric (In|De)crease <metric(s)> <options>:
    <tests>

where metrics and options are optional and allow you to specify a metric or list of metrics, the way, and test environment. Here are some examples:

Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] (test_env='linux_x86', way='default'):
    Test012
    Test345
Metric Decrease 'bytes allocated':
    Test678
Metric Increase:
    Test711

Upon failing some performance tests, the test runner will output the string required to accept all changes. First double check that you really do expect those changes! If so, you can simply copy the text into the commit message and rerun the tests to ensure they pass. (Careful: there should be no whitespace before Metric, otherwise the regex used to extract it from the commit will not match.)

CAUTION: make sure you maintain the correct expected changes in your commit messages when squashing commits.

Comparing Commits

There exists a comparison tool located at testsuite/driver/perf_notes.py to help analyze the performance metrics commits. Run the commandline testsuite/driver/perf_notes.py --help to see the available options. E.g. to see a chart of the last 100 commits as a standalone html file (omit --chart to get simple text output to stdout):

$ python3 testsuite/driver/perf_notes.py --chart chart.html HEAD~100..HEAD
$ firefox ./chart.html

This will show results of your local runs of performance tests (see above). You can also view results from CI using the --ci option. Make sure to fetch CI results first. There are a lot of results, so you'll likely want to filter for a specific test/environment e.g.:

chart.html

$ git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf
$ python3 testsuite/driver/perf_notes.py --chart chart.html --ci --test-name "T9630" --test-env x86_64-linux-deb9  master~100..master
$ firefox ./chart.html