The testsuite needs to be overhauled.
Our current test-suite driver has served us ok'ish for now, however there are a few rather annoying issues that come up over and over again.
-
The test-suite is not compatible with cross compilers.
This is annoying as it completely prohibits using the already existing tests with cross compilers. !3652 (closed) tries to address this. -
The test-suite prints copious amounts of useless data.
When running the test-suite it runs through some 7k tests, each producing 3-5 lines of output, and finally telling us that 1 to 5 tests failed. This is rather bad. Especially when looking at CI results, we don't care for succeeded tests, but for the failing ones. -
The test-suite has lots of hardcoded stdout/stderr data.
This is really annoying as it's very rigid and requires perpetual maintenance. Bump a version of some library, and you'll have to change test-suite outputs. Some tool produces some garbage to stderr/stdout on some system under some strange configuration? Well now you need to carefully clean stdout/stderr to make sure it matches the fixed output. LLVM has FileCheck which is much more flexible in this regard. It can check for specific strings, for sequences of strings, for the absence (I think) of strings, under different configurations. -
Tests are very diverse and implicit, not explicit and declarative.
Tests come in Makefiles and all.T scripts, and various other forms. Some are descriptive others are not. This is a much harder issue to solve, but maybe we can chip away at this at some point as well. Test should be declarative, such that we can interpret them as needed.