Make tests self-contained?
At ICFP @rae and I discussed the ups and downs of the OCaml and GHC development workflows. One point that came up was the fact that tests in the OCaml testsuite generally tend to be self-contained.
For instance, this test consists of one file containing:
- a test header describing that the file contains a test (roughly analogous to the
.T
file declarations in GHC testsuite) - the program(s) to compile
- the expected output from the compiler, interleaved with the programs
Apparently the testsuite driver even has sufficient cleverness to be able to surgically update the expected output.
See this test for another example.
Adapting this to GHC
Obviously introducing all of this machinery in GHC's testsuite would be a non-trivial undertaking. However, I do think that a good first step would be to introduce the idea of a "test header", eliminating the need for .T
declarations. This would be quite worthwhile since the majority of merge conflicts I encounter (both while rebasing and backporting patches) are due to .T
files.
For instance, we might simply allow tests to begin with specially-formed comments containing the existing .T
definitions, e.g.:
{- ghc-test test('T12345', normal, compile_run, ['']) -}
Naturally, there are cases where this won't work (e.g. when we want to share logic across tests) and in these cases we would likely still use the current .T
approach. However, this small change would allow us to reduce our dependence on monolithic .T
files. I suspect in the majority of cases it wouldn't even be that hard to move the test
declarations into their respective source files via automated refactoring.
Alternatively, we might also consider instead using a less Pythonic metadata representation in the test header, taking this opportunity to start to improve the maintainability of the testsuite infrastructure.