testdriver: Accepting test output accepts output of tests with missing libraries
Often, my workflow when working on a patch is to run the testsuite with --test-accept and then review all the testsuite changes in git diff
. When my patch is final, I do a regular test run to validate I actually caught every testsuite change.
Here's an example where that workflow doesn't quite work:
I don't have QuickCheck installed, which is a prerequisite for ghci014
:
test('ghci014', reqlib('QuickCheck'), ghci_script, ['ghci014.script'])
If I run TEST=ghci014 hadrian/build.sh test
, I get
SUMMARY for test run started at Thu Jun 17 12:23:16 2021
0:00:01.125260 spent to go through
1 total tests, which gave rise to
1 test cases, of which
0 were skipped
1 had missing libraries
0 expected passes
0 expected failures
0 caused framework failures
0 caused framework warnings
0 unexpected passes
0 unexpected failures
0 unexpected stat failures
0 fragile tests
As expected. But if I run TEST=ghci014 hadrian/build.sh test --test-accept
, I get
=====> 1 of 1 [0, 0, 0]
Accepting new output.
*** unexpected pass for ghci014(ghci)
...
SUMMARY for test run started at Thu Jun 17 12:23:33 2021
0:00:01.108247 spent to go through
1 total tests, which gave rise to
1 test cases, of which
0 were skipped
0 had missing libraries
0 expected passes
0 expected failures
0 caused framework failures
0 caused framework warnings
1 unexpected passes
0 unexpected failures
0 unexpected stat failures
0 fragile tests
And I get a whole blob of ghci014.stderr that I more often than not commit accidentally (for example, when I have to switch to a different branch mid review or simply want to rebase).
I'd rather have it that tests that are expected to fail (due to missing libs, for example) don't record any test output. I believe that already is the case for tests from the category "expected failures", but it appears it's not the case for "had missing libraries".