Testrunner's `grep_errmsg` doesn't produce diffs
I regularly use the grep_errmsg
... mode of the testrunner. But it severely lacks in reporting ergonomics. For example, when there's a failing test, it will just report (even with --test-verbose=5
)
=====> T18903(optasm) 1 of 1 [0, 0, 0]
cd "/tmp/ghctest-golcpn46/test spaces/testsuite/tests/stranal/should_compile/T18903.run" && "/home/sgraf-local/code/hs/ghc/misc4/_validate/stage1/bin/ghc" -c T18903.hs -dcore-lint -dstg-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -Werror=compat -dno-debug-output -O -fasm -ddump-simpl<
Actual stderr output differs from expected:
diff -uw "/tmp/ghctest-golcpn46/test spaces/testsuite/tests/stranal/should_compile/T18903.run/T18903.stderr.normalised" "/tmp/ghctest-golcpn46/test spaces/testsuite/tests/stranal/should_compile/T18903.run/T18903.comp.stderr.normalised"<
*** unexpected failure for T18903(optasm)
Without actually reporting the difference. I'd at least expect a diff of the original files, even better would be to show the diff on grep
ped lines.
To reproduce, just pick an arbitrary grep_errmsg
based test and change a relevant line of the expected output file.