Status June 2021
There are 4 different performance tests which we want to import statistics for:
- GHC Performance Tests
- Cabal test
- Head.hackage
- Nofib
Metric | Table Name | Pipeline Job | Import Method | Status |
---|---|---|---|---|
GHC Perf | results | master/per environment | Git Notes | Import is not idempotent or using provenance table, import does not use webhook |
Cabal test | cabal_results | perf | Webhook/Artifacts | Table populating |
Head.hackage | head_hackage_results | nightly-hackage | Webhook/Artifacts | Table populating |
Nofib | - | nofib/nofib | Webhook/Artifacts | Build is currently broken, no import taking place |
Ultimately we want to make all four of these methods use the webhook in order to populate the database.
TODO List
GHC Perf
- Fiddly: Migrate the results table to add a provenance column
- Modify the import script to use provenance column and to avoid using ON CONFLICT
- Trigger the import based on suitable completed jobs rather than scheduled nightly(?)
Cabal Test
- Create grafana dashboard once the table is populated
- Create a (materialised) view which joins commit_id on master commits
Head.hackage
- Think about how to best alert about bad regressions
Nofib
- Fix the CI build and then work out what needs to be done, at least there needs to be a new table definition and the scripts modified to use provenance ID.
Tools for working with ghc-speed history
perf-import
takes validate logs, picks out the juicy performance number from
them, and plops them into the database.
perf-import-git
traverses the database for commits lacking metadata (e.g.
commit message and date), looks them up in the given repository, and plops the
resulting metadata to the database.
$ git clone git@github.com:bgamari/ghc-perf-import
$ cd ghc-perf-import
$ git clone git@github.com:nomeata/ghc-speed-logs
$ git clone git@github.com:ghc/ghc
$ cabal install
$ psql postgres < schema.sql
$ perf-import -e nomeata ghc-perf-import/*.log
$ perf-import-git
$ echo 'SELECT count(1) FROM commits;' | psql ghc_perf postgres
$ cat <<EOF | psql ghc_perf postgres
COPY (
SELECT commits.commit_sha, commits.commit_date, results.result_value
FROM results, commits
WHERE results.commit_id = commits.commit_id AND
results.test_id = 36
ORDER BY (commits.commit_date)
) TO STDOUT CSV;
EOF
Export all results for a given test,
\copy (
SELECT *
FROM results_view
WHERE test_env = 'nomeata'
AND branch_name = 'master'
AND test_name='compile-allocs/AbsConc3'
ORDER BY commit_date
) TO 'hi' CSV;
Find commits which strongly affect compiler allocations,
WITH deltas(commit_sha, commit_date, commit_title, test_name, delta) AS (
SELECT x.commit_sha,
x.commit_date,
x.commit_title,
x.test_name,
(x.result_value - x.prev_value) / NULLIF(x.prev_value, 0) AS delta
FROM (
SELECT commit_sha,
commit_date,
commit_title,
test_name,
result_value,
lag(result_value) OVER (PARTITION BY test_name
ORDER BY commit_date) AS prev_value
FROM results_view
WHERE branch_name = 'master'
AND test_env = 'nomeata'
ORDER BY commit_date
) AS x
)
SELECT *
FROM deltas
WHERE abs(delta) > 0.10
AND test_name LIKE 'compile-allocs/%'
ORDER BY commit_sha, abs(delta) DESC;
To produce terribly-formatted plots,
$ i=0; while IFS= read -r tests; do echo $tests; ./plot.py -o $i.svg $tests; let i=i+1; done < tests.grouped
To compute a comparison between two commits,
SELECT test_envs.test_env_name, tests.test_name,
array_agg(old.result_value) as old,
array_agg(new.result_value) as new,
(avg(new.result_value) - avg(old.result_value)) / NULLIF(avg(old.result_value), 0) as change
FROM (
SELECT *
FROM results
JOIN commits ON (commits.commit_id = results.commit_id)
WHERE commits.commit_sha ~ '567dbd9bcb603*'
) AS old, (
SELECT *
FROM results
JOIN commits ON (commits.commit_id = results.commit_id)
WHERE commits.commit_sha ~ 'ffe4660510a*'
) AS new
JOIN test_envs ON (test_envs.test_env_id = new.test_env_id)
JOIN tests ON (tests.test_id = new.test_id)
WHERE old.test_env_id = new.test_env_id
AND old.test_id = new.test_id
GROUP BY (new.test_env_id, test_envs.test_env_name, tests.test_name, new.test_id);
or
SELECT avg(rel_delta)
FROM deltas2
WHERE commit_1 ~ '567dbd9bcb603*'
AND commit_2 ~ 'ffe4660510a*'
AND test_name ~ 'compile-alloc*';
To get a summary of the largest changes in the compile-allocs/*
tests,
SELECT *
FROM deltas
WHERE abs(delta) > 0
AND test_name ~ 'compile-allocs/'
ORDER BY rel_delta DESC;
Or to see the effect of a single commit,
SELECT test_name, delta, rel_delta
FROM deltas
WHERE commit_sha = '673efccb3b348e9daf23d9e65460691bbea8586e'
ORDER BY rel_delta DESC;