Testsuite: compile_timeout_multiplier is fragile
I'm taking a sweep through the testsuite, trying to eliminate failures in a DEBUG compiler. But two tests are stymieing me: pmcheck/should_compile/T11{276,374}. Both of these use the compile_timout_multiplier option in their all.T entries. This option means that, instead of the normal 300s allotted to the test, these tests get only 3s. In my setup, with a DEBUG (read: unoptimized) stage2 compiler, these tests fail when I'm running the testsuite in parallel. Then, when I run the tests individually, they succeed.
This poses two problems:
- I can only see the failure when my machine is stressed, and it's disconcerting to have a failure appear and disappear depending on how I look.
- There is (sometimes) a failure in
DEBUGmode.
What is the goal of compile_timeout_multiplier? If we're really testing performance (which #11276 (closed) and #11374 (closed) indicate), then these tests should be in perf/compiler and be "stat" tests, where some wibbling in the numbers is expected (and where DEBUG compilers are exempted). I would imagine that compile_timeout_multiplier should only ever get values greater than 1, when a non-performance test is so expensive to run that it takes more than 300s.
As a local solution here, I think these two tests should move to perf/compiler. Other pattern-match tests also have compile_timeout_multiplier, too, and I can move these as well. Any objections?
Trac metadata
| Trac field | Value |
|---|---|
| Version | 8.4.3 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Compiler |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture |