Skip to content

Hadrian: various improvements around the 'test' rule

Alp Mestanogullari requested to merge alp/ghc:wip/alp/hadrian-test-settings into master
  • introduce a -k/--keep-test-files flag to prevent cleanup
  • add -dstg-lint to the options that are always passed to tests
  • infer library ways from the compiler to be tested instead of getting them from the flavour (like make)
  • likewise for figuring out whether the compiler to be tested is "debugged"
  • specify config.exeext
  • correctly specify config.in_tree_compiler, instead of always passing True
  • fix formatting of how we pass a few test options
  • add (potential) extensions to check-* program names
  • build check-* programs with the compiler to be tested
  • set TEST_HC_OPTS_INTERACTIVE and PYTHON env vars when running tests

For the record, while this branch allows more tests to run, we get the exact same number of unexpected failures as on master, and only one additional unexpected pass. I ran rm -rf _build; ./boot; ./configure; hadrian/build.sh -j4 test on both branches, from a clean state.

On master:

Unexpected results from:
TEST="T1288_ghci T15633a T15633b T15858 T16104 T16260 T2014 T2276_ghci T2386 T3807 T7022 T7040_ghci T7445 T9208 annth_compunits haddockHoogleTest haddockHtmlTest haddockHypsrcTest haddockLatexTest path_with_commas"

SUMMARY for test run started at Tue Mar  5 15:31:21 2019 CET
 0:23:43 spent to go through
    6863 total tests, which gave rise to
   26454 test cases, of which
   19613 were skipped

      29 had missing libraries
    6701 expected passes
      91 expected failures

       0 caused framework failures
       0 caused framework warnings
       1 unexpected passes
      19 unexpected failures
       0 unexpected stat failures

Unexpected passes:
   testsuite/tests/stranal/should_compile/T9208.run  T9208 [unexpected] (optasm)

Unexpected failures:
   testsuite/tests/annotations/should_compile/th/annth_compunits.run  annth_compunits [bad exit code] (normal)
   testsuite/tests/dynlibs/T3807.run                                  T3807 [bad exit code] (normal)
   testsuite/tests/ffi/should_run/T1288_ghci.run                      T1288_ghci [bad exit code] (ghci)
   testsuite/tests/ffi/should_run/T2276_ghci.run                      T2276_ghci [bad exit code] (ghci)
   testsuite/tests/ghci/should_run/T15633a.run                        T15633a [bad exit code] (ghci)
   testsuite/tests/ghci/should_run/T15633b.run                        T15633b [bad stderr] (ghci)
   testsuite/tests/haddock/haddock_testsuite/haddockHtmlTest.run      haddockHtmlTest [bad exit code] (normal)
   testsuite/tests/haddock/haddock_testsuite/haddockLatexTest.run     haddockLatexTest [bad exit code] (normal)
   testsuite/tests/haddock/haddock_testsuite/haddockHoogleTest.run    haddockHoogleTest [bad exit code] (normal)
   testsuite/tests/haddock/haddock_testsuite/haddockHypsrcTest.run    haddockHypsrcTest [bad exit code] (normal)
   testsuite/tests/plugins/T15858.run                                 T15858 [bad stderr] (ghci)
   testsuite/tests/plugins/T16104.run                                 T16104 [bad exit code] (normal)
   testsuite/tests/plugins/T16260.run                                 T16260 [bad exit code] (normal)
   testsuite/tests/polykinds/T7022.run                                T7022 [bad exit code] (normal)
   testsuite/tests/rts/T7040_ghci.run                                 T7040_ghci [bad exit code] (ghci)
   testsuite/tests/th/T2386.run                                       T2386 [bad exit code] (normal)
   testsuite/tests/th/T7445.run                                       T7445 [bad exit code] (normal)
   testsuite/tests/th/T2014/T2014.run                                 T2014 [bad exit code] (normal)
   testsuite/tests/th/TH_linker/path_with_commas.run                  path_with_commas [bad exit code] (normal)

On this branch:

Unexpected results from:
TEST="T1288_ghci T15633a T15633b T15858 T16104 T16260 T2014 T2276_ghci T2386 T3807 T7022 T7040_ghci T7445 T9208 annth_compunits haddockHoogleTest haddockHtmlTest haddockHypsrcTest haddockLatexTest path_with_commas recomp007"

SUMMARY for test run started at Tue Mar  5 15:29:11 2019 CET
 0:25:28 spent to go through
    6863 total tests, which gave rise to
   26594 test cases, of which
   19307 were skipped

      42 had missing libraries
    7132 expected passes
      92 expected failures

       0 caused framework failures
       0 caused framework warnings
       2 unexpected passes
      19 unexpected failures
       0 unexpected stat failures

Unexpected passes:
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/driver/recomp007/recomp007.run    recomp007 [unexpected] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/stranal/should_compile/T9208.run  T9208 [unexpected] (optasm)

Unexpected failures:
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/annotations/should_compile/th/annth_compunits.run  annth_compunits [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/dynlibs/T3807.run                                  T3807 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/ffi/should_run/T1288_ghci.run                      T1288_ghci [bad exit code] (ghci)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/ffi/should_run/T2276_ghci.run                      T2276_ghci [bad exit code] (ghci)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/ghci/should_run/T15633a.run                        T15633a [bad exit code] (ghci)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/ghci/should_run/T15633b.run                        T15633b [bad stderr] (ghci)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/haddock/haddock_testsuite/haddockHtmlTest.run      haddockHtmlTest [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/haddock/haddock_testsuite/haddockLatexTest.run     haddockLatexTest [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/haddock/haddock_testsuite/haddockHoogleTest.run    haddockHoogleTest [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/haddock/haddock_testsuite/haddockHypsrcTest.run    haddockHypsrcTest [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/plugins/T15858.run                                 T15858 [bad stderr] (ghci)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/plugins/T16104.run                                 T16104 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/plugins/T16260.run                                 T16260 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/polykinds/T7022.run                                T7022 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/rts/T7040_ghci.run                                 T7040_ghci [bad exit code] (ghci)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/th/T2386.run                                       T2386 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/th/T7445.run                                       T7445 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/th/T2014/T2014.run                                 T2014 [bad exit code] (normal)
   /run/user/1001/ghctest-ypo1pf6n/test   spaces/testsuite/tests/th/TH_linker/path_with_commas.run                  path_with_commas [bad exit code] (normal)
Edited by Alp Mestanogullari

Merge request reports