!326: --hadrian "mode" for ./validate · Merge requests · Glasgow Haskell Compiler / GHC

Alp Mestanogullari requested to merge alp/ghc:wip/alp/hadrian-validate into master Feb 08, 2019

This required quite some work, so as to be able to test an external (not in the source tree) GHC install, to make sure all the necessary programs and what not are produced, to properly detect what lib ways a given GHC install has (whether in-tree or not), and so on. Note that we have more test failures than we used to a month ago or so, but that will be the subject of future MRs. This one focuses on implementing ./validate --hadrian.

$ ./validate --hadrian
[...]
Unexpected results from:
TEST="Ppr027 Ppr040 Ppr043 T11462 T11525 T13168 T13199 T13550 T13702 T13942 T14289 T14289b T14289c T3807 T5373 T9208 T9293 TH_spliceE5_prof annth_compunits annth_make dynHelloWorld frontend01 ghci024 ghci057 path_with_commas plugin-recomp-impure plugins09 plugins10 plugins11"

SUMMARY for test run started at Fri Feb  8 18:00:16 2019 CET
 0:26:33 spent to go through
    6796 total tests, which gave rise to
   26431 test cases, of which
   19124 were skipped

      42 had missing libraries
    7145 expected passes
      91 expected failures

       0 caused framework failures
       0 caused framework warnings
       1 unexpected passes
      28 unexpected failures
       0 unexpected stat failures

Unexpected passes:
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/stranal/should_compile/T9208.run  T9208 [unexpected] (optasm)

Unexpected failures:
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/annotations/should_compile/th/annth_make.run       annth_make [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/annotations/should_compile/th/annth_compunits.run  annth_compunits [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/driver/dynHelloWorld.run                           dynHelloWorld [exit code non-0] (dyn)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/dynlibs/T5373.run                                  T5373 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/dynlibs/T3807.run                                  T3807 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/dynlibs/T13702.run                                 T13702 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/ghci/scripts/ghci024.run                           ghci024 [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/ghci/scripts/ghci057.run                           ghci057 [bad stdout] (ghci)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/ghci/scripts/T9293.run                             T9293 [bad stdout] (ghci)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/plugins/plugins09.run                              plugins09 [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/plugins/plugins10.run                              plugins10 [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/plugins/plugins11.run                              plugins11 [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/plugins/frontend01.run                             frontend01 [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/plugins/plugin-recomp-impure.run                   plugin-recomp-impure [bad stdout] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/Ppr027.run                                 Ppr027 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/Ppr040.run                                 Ppr040 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/Ppr043.run                                 Ppr043 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/T13199.run                                 T13199 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/T13550.run                                 T13550 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/T14289.run                                 T14289 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/T13942.run                                 T13942 [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/T14289b.run                                T14289b [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/printer/T14289c.run                                T14289c [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/th/TH_spliceE5_prof.run                            TH_spliceE5_prof [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/th/TH_linker/path_with_commas.run                  path_with_commas [bad exit code] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/typecheck/T13168/T13168.run                        T13168 [bad stderr] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/typecheck/should_compile/T11462.run                T11462 [exit code non-0] (normal)
   /run/user/1001/ghctest-ksvtb23z/test   spaces/testsuite/tests/typecheck/should_compile/T11525.run                T11525 [exit code non-0] (normal)

$ hadrian/build.sh -j4 test
[...]
Unexpected results from:
TEST="Ppr027 Ppr040 Ppr043 T11462 T11525 T13168 T13199 T13550 T13702 T13942 T14289 T14289b T14289c T3807 T5373 T9208 T9293 TH_spliceE5_prof annth_compunits annth_make dynHelloWorld frontend01 ghci024 ghci057 haddock.Cabal haddock.base haddock.compiler path_with_commas plugin-recomp-impure plugins09 plugins10 plugins11"

SUMMARY for test run started at Fri Feb  8 17:59:59 2019 CET
 0:26:21 spent to go through
    6796 total tests, which gave rise to
   26431 test cases, of which
   19120 were skipped

      42 had missing libraries
    7146 expected passes
      91 expected failures

       3 caused framework failures
       0 caused framework warnings
       1 unexpected passes
      31 unexpected failures
       0 unexpected stat failures

Unexpected passes:
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/stranal/should_compile/T9208.run  T9208 [unexpected] (optasm)

Unexpected failures:
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/annotations/should_compile/th/annth_make.run       annth_make [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/annotations/should_compile/th/annth_compunits.run  annth_compunits [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/driver/dynHelloWorld.run                           dynHelloWorld [exit code non-0] (dyn)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/dynlibs/T5373.run                                  T5373 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/dynlibs/T3807.run                                  T3807 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/dynlibs/T13702.run                                 T13702 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/ghci/scripts/ghci024.run                           ghci024 [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/ghci/scripts/ghci057.run                           ghci057 [bad stdout] (ghci)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/ghci/scripts/T9293.run                             T9293 [bad stdout] (ghci)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.base.run                      haddock.base [[Errno 2] No such file or directory: '/run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.Cabal.run                     haddock.Cabal [[Errno 2] No such file or directory: '/run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.compiler.run                  haddock.compiler [[Errno 2] No such file or directory: '/run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/plugins/plugins09.run                              plugins09 [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/plugins/plugins10.run                              plugins10 [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/plugins/plugins11.run                              plugins11 [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/plugins/frontend01.run                             frontend01 [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/plugins/plugin-recomp-impure.run                   plugin-recomp-impure [bad stdout] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/Ppr027.run                                 Ppr027 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/Ppr040.run                                 Ppr040 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/Ppr043.run                                 Ppr043 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/T13199.run                                 T13199 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/T13550.run                                 T13550 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/T14289.run                                 T14289 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/T13942.run                                 T13942 [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/T14289b.run                                T14289b [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/printer/T14289c.run                                T14289c [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/th/TH_spliceE5_prof.run                            TH_spliceE5_prof [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/th/TH_linker/path_with_commas.run                  path_with_commas [bad exit code] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/typecheck/T13168/T13168.run                        T13168 [bad stderr] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/typecheck/should_compile/T11462.run                T11462 [exit code non-0] (normal)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/typecheck/should_compile/T11525.run                T11525 [exit code non-0] (normal)

Framework failures:
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.base.run      haddock.base [normal] (extra_file does not exist: ../../../../libraries/base/dist-install/haddock.t)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.Cabal.run     haddock.Cabal [normal] (extra_file does not exist: ../../../../libraries/Cabal/Cabal/dist-install/haddock.t)
   /run/user/1001/ghctest-x2ytmh64/test   spaces/testsuite/tests/perf/haddock/haddock.compiler.run  haddock.compiler [normal] (extra_file does not exist: ../../../../compiler/stage2/haddock.t)

We don't get the haddock.xxx failures with ./validate --hadrian because those tests are only executed when we're testing an in-tree compiler, they're skipped when we test through ./validate. And we'll have to change them to make them pass with hadrian I suspect, since we don't produce files in the source tree...

Aside from that, the results are identical. I can't believe it myself, but I'll sure get used to it. Whack-a-mole isn't fun except when it's over.

Edited Feb 08, 2019 by Alp Mestanogullari

--hadrian "mode" for ./validate

Merge request reports