Rethink testsuite ways
Currently, our testsuite driver has a notion of "ways", which identify a particular compile- and run-time configuration that tests can be run in. There are a multitude of ways, some of which affect only compile-time (e.g. llvm
, optasm
), some only runtime (nonmoving_thr
), and some for testing particular features of particular platforms (e.g. winio
).
By default, tests can be run in any way unless otherwise specified with the only_ways
or skip_ways
test modifiers.
On its face this seems like a reasonable design but in practice it has a few unfortunate effects:
- Many tests are added which implicitly assume that the test is run in the vanilla way; since the way space of ways is too expansive to test regularly and comprehensively, we generally only find out about such cases well after they are added. This has made stabilizing the "slow" testsuite speed (!8695), which runs a large subset of ways, very challenging. Moreover, some ways likely haven't been run in years (e.g.
compacting_gc
,g1
), further compounding the issue. - A great many tests (e.g.
makefile_test
s) are not affected by the way-specified flags at all, leading to repeated identical runs of the same test.testlib.py
has some very shady logic to catch some of these cases but it's both incomplete and hard to reason about. - Some ways will never pass on some platforms. For instance, the
prof
way specifies the-fasm
compile-time flag, which will throw a warning on any platform which doesn't have an NCG.
I don't have an immediate suggestion on how to improve this situation but I do think that the problems are deep enough that they warrant some thought.