Skip to content

Introduce a new test failure reason for timeouts

Ömer Sinan Ağacan requested to merge osa1/ghc:timeout_tests into master

With this patch we introduce a new failures reason "timeout" and classify timed out tests as that. Previously timed out tests were classified as "bad exit code".

This relies on the fact that the timeout program (testsuite/timeout) returns 99 on timeout.

Fixes #15963

Before:

  • Build timeout:

    =====> hpc_raytrace(normal) 1 of 1 [0, 0, 0]
    ...
    Timeout happened...killed process "..."
    
    Compile failed (exit code 99) errors were:
    [ 1 of 13] Compiling Geometry         ( Geometry.hs, Geometry.o )
    
    Geometry.hs:164:1: warning: [-Wtabs (in -Wdefault)]
        Tab character found here, and in one further location.
        Please use spaces instead.
    
    *** unexpected failure for hpc_raytrace(normal)
    
    Unexpected results from:
    TEST="hpc_raytrace"
    
    SUMMARY for test run started at Tue Mar 19 13:18:16 2019 +03
     0:00:03 spent to go through
           1 total tests, which gave rise to
           7 test cases, of which
           6 were skipped
    
           0 had missing libraries
           0 expected passes
           0 expected failures
    
           0 caused framework failures
           0 caused framework warnings
           0 unexpected passes
           1 unexpected failures
           0 unexpected stat failures
    
    unexpected failures:
       libraries/hpc/tests/raytrace/hpc_raytrace.run  hpc_raytrace [exit code non-0] (normal)
  • Run timeout:

    =====> seward-space-leak(normal) 1 of 1 [0, 0, 0]
    ...
    Timeout happened...killed process "cd "programs/seward-space-leak/seward-space-leak.run" && ./seward-space-leak  "...
    
    Wrong exit code for seward-space-leak(normal)(expected 0 , actual 99 )
    *** unexpected failure for seward-space-leak(normal)
    
    Unexpected results from:
    TEST="seward-space-leak"
    
    SUMMARY for test run started at Tue Mar 19 13:19:09 2019 +03
     0:00:04 spent to go through
           1 total tests, which gave rise to
           7 test cases, of which
           6 were skipped
    
           0 had missing libraries
           0 expected passes
           0 expected failures
    
           0 caused framework failures
           0 caused framework warnings
           0 unexpected passes
           1 unexpected failures
           0 unexpected stat failures
    
    Unexpected failures:
       programs/seward-space-leak/seward-space-leak.run  seward-space-leak [bad exit code] (normal)

After:

  • Build timeout:

    =====> hpc_raytrace(normal) 1 of 1 [0, 0, 0]
    ...
    Timeout happened...killed process "..."
    
    Compile timed out, errors were:
    [ 1 of 13] Compiling Geometry         ( Geometry.hs, Geometry.o )
    
    Geometry.hs:164:1: warning: [-Wtabs (in -Wdefault)]
        Tab character found here, and in one further location.
        Please use spaces instead.
    
    *** unexpected failure for hpc_raytrace(normal)
    
    Unexpected results from:
    TEST="hpc_raytrace"
    
    SUMMARY for test run started at Tue Mar 19 13:17:33 2019 +03
     0:00:03 spent to go through
           1 total tests, which gave rise to
           7 test cases, of which
           6 were skipped
    
           0 had missing libraries
           0 expected passes
           0 expected failures
    
           0 caused framework failures
           0 caused framework warnings
           0 unexpected passes
           1 unexpected failures
           0 unexpected stat failures
    
    Unexpected failures:
       libraries/hpc/tests/raytrace/hpc_raytrace.run  hpc_raytrace [timeout] (normal)
  • Run timeout:

    =====> seward-space-leak(normal) 1 of 1 [0, 0, 0]
    ...
    Timeout happened...killed process "cd "programs/seward-space-leak/seward-space-leak.run" && ./seward-space-leak  "...
    
    seward-space-leak(normal) timed out
    *** unexpected failure for seward-space-leak(normal)
    
    Unexpected results from:
    TEST="seward-space-leak"
    
    SUMMARY for test run started at Tue Mar 19 13:16:47 2019 +03
     0:00:04 spent to go through
           1 total tests, which gave rise to
           7 test cases, of which
           6 were skipped
    
           0 had missing libraries
           0 expected passes
           0 expected failures
    
           0 caused framework failures
           0 caused framework warnings
           0 unexpected passes
           1 unexpected failures
           0 unexpected stat failures
    
    Unexpected failures:
       programs/seward-space-leak/seward-space-leak.run  seward-space-leak [timeout] (normal)
Edited by Ömer Sinan Ağacan

Merge request reports