Performance test failures can generate misleading results
I've just wasted a fair bit of time because I assumed that if a performance test was reported as an "unexpected stat failure" but not an "unexpected failure", that meant the test case generated the right output. However this seems not to be the case, at least for compile_fail
tests. If changes to GHC make a compile_fail
performance test start crashing, say, it will get reported as an unexpected stat decrease (which is normally good!) with no indication that the output is no longer correct.
I think that an unexpected output or exit code from a performance test should be reported as an "unexpected failure", and perhaps the "unexpected stat failure" line should give some indication that the result may be completely misleading.