Perf tests can be silently skipped
Sometimes, performance tests are silently skipped because of a missing baseline. For example, in !3988 (closed) we've changed the name of an environment, which made past results unavailable. (To see this, compare the performance metrics section for the first and second commit in
validate-x86_64-linux-deb9-unreg-hadrian. Both builds are green.)
I think this can bite us in the future - we should require indicating that the metric is missing in the commit message, just like we require indicating a metric increase or decrease. Alternatively, instead of doing this for every test, we might require a blanket indication "This commit has missing baseline values".