Testsuite framework failures are too easy to ignore and too hard to find
The testsuite driver really should do a better job of drawing the user's attention to framework failures. The only evidence in the testsuite summary that something went wrong is a non-zero "framework failures" count.
If you notice that this count is non-zero then you need to sift manually through the initial log output produced by the driver and find the culprit. It's just generally terrible.
Testsuite failures are failures, we should at least treat them as such and include them in the TEST=... message in the testsuite summary so they can be easily reproduced. Ideally we would also save the exception that caused the failure and show it in the summary as well.
Trac metadata
| Trac field | Value |
|---|---|
| Version | 7.10.2 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Test Suite |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | thomie |
| Operating system | |
| Architecture |