Skip to content
Snippets Groups Projects

Status June 2021

There are 4 different performance tests which we want to import statistics for:

  • GHC Performance Tests
  • Cabal test
  • Head.hackage
  • Nofib
Metric Table Name Pipeline Job Import Method Status
GHC Perf results master/per environment Git Notes Import is not idempotent or using provenance table, import does not use webhook
Cabal test cabal_results perf Webhook/Artifacts Table populating
Head.hackage head_hackage_results nightly-hackage Webhook/Artifacts Table populating
Nofib - nofib/nofib Webhook/Artifacts Build is currently broken, no import taking place

Ultimately we want to make all four of these methods use the webhook in order to populate the database.

TODO List

GHC Perf

  • Fiddly: Migrate the results table to add a provenance column
  • Modify the import script to use provenance column and to avoid using ON CONFLICT
  • Trigger the import based on suitable completed jobs rather than scheduled nightly(?)

Cabal Test

  • Create grafana dashboard once the table is populated
  • Create a (materialised) view which joins commit_id on master commits

Head.hackage

  • Think about how to best alert about bad regressions

Nofib

  • Fix the CI build and then work out what needs to be done, at least there needs to be a new table definition and the scripts modified to use provenance ID.

Tools for working with ghc-speed history

perf-import takes validate logs, picks out the juicy performance number from them, and plops them into the database.

perf-import-git traverses the database for commits lacking metadata (e.g. commit message and date), looks them up in the given repository, and plops the resulting metadata to the database.

$ git clone git@github.com:bgamari/ghc-perf-import
$ cd ghc-perf-import
$ git clone git@github.com:nomeata/ghc-speed-logs
$ git clone git@github.com:ghc/ghc
$ cabal install
$ psql postgres < schema.sql
$ perf-import -e nomeata ghc-perf-import/*.log
$ perf-import-git
$ echo 'SELECT count(1) FROM commits;' | psql ghc_perf postgres
$ cat <<EOF | psql ghc_perf postgres
COPY (
    SELECT commits.commit_sha, commits.commit_date, results.result_value
    FROM results, commits
    WHERE results.commit_id = commits.commit_id AND
        results.test_id = 36
    ORDER BY (commits.commit_date)
) TO STDOUT CSV;
EOF

Export all results for a given test,

\copy (
    SELECT *
    FROM results_view
    WHERE test_env = 'nomeata'
      AND branch_name = 'master'
      AND test_name='compile-allocs/AbsConc3'
    ORDER BY commit_date
) TO 'hi' CSV;

Find commits which strongly affect compiler allocations,

WITH deltas(commit_sha, commit_date, commit_title, test_name, delta) AS (
    SELECT x.commit_sha,
           x.commit_date,
           x.commit_title,
           x.test_name,
           (x.result_value - x.prev_value) / NULLIF(x.prev_value, 0) AS delta
    FROM (
        SELECT commit_sha,
               commit_date,
               commit_title,
               test_name,
               result_value,
               lag(result_value) OVER (PARTITION BY test_name
                                       ORDER BY commit_date) AS prev_value
        FROM results_view
        WHERE branch_name = 'master'
          AND test_env = 'nomeata'
        ORDER BY commit_date
    ) AS x
)
SELECT *
FROM deltas
WHERE abs(delta) > 0.10
  AND test_name LIKE 'compile-allocs/%'
ORDER BY commit_sha, abs(delta) DESC;

To produce terribly-formatted plots,

$ i=0; while IFS= read -r tests; do echo $tests; ./plot.py -o $i.svg $tests; let i=i+1; done < tests.grouped

To compute a comparison between two commits,

SELECT test_envs.test_env_name, tests.test_name,
       array_agg(old.result_value) as old,
       array_agg(new.result_value) as new,
       (avg(new.result_value) - avg(old.result_value)) / NULLIF(avg(old.result_value), 0) as change
FROM (
    SELECT *
    FROM results
    JOIN commits ON (commits.commit_id = results.commit_id)
    WHERE commits.commit_sha ~ '567dbd9bcb603*'
) AS old, (
    SELECT * 
    FROM results
    JOIN commits ON (commits.commit_id = results.commit_id)
    WHERE commits.commit_sha ~ 'ffe4660510a*'
) AS new
JOIN test_envs ON (test_envs.test_env_id = new.test_env_id)
JOIN tests ON (tests.test_id = new.test_id)
WHERE old.test_env_id = new.test_env_id
  AND old.test_id = new.test_id
GROUP BY (new.test_env_id, test_envs.test_env_name, tests.test_name, new.test_id);

or

SELECT avg(rel_delta)
FROM deltas2
WHERE commit_1 ~ '567dbd9bcb603*'
  AND commit_2 ~ 'ffe4660510a*'
  AND test_name ~ 'compile-alloc*';

To get a summary of the largest changes in the compile-allocs/* tests,

SELECT *
FROM deltas
WHERE abs(delta) > 0
  AND test_name ~ 'compile-allocs/'
ORDER BY rel_delta DESC;

Or to see the effect of a single commit,

SELECT test_name, delta, rel_delta
FROM deltas
WHERE commit_sha = '673efccb3b348e9daf23d9e65460691bbea8586e'
ORDER BY rel_delta DESC;