Testrunner fails to generate compile time stats when a test is set to compile_and_run
If we want to collect both compile time and runtime metrics from a single test, the test runner will currently fail.
Recreation
Modify the Conversions
test in testsuite/tests/perf/should_run/all.T
to:
test('Conversions',
[collect_stats('bytes allocated',3),
collect_compiler_stats('bytes allocated',3),
only_ways(['normal'])
],
compile_and_run, ['-O -msse2'])
then run the test:
$ ./hadrian/build.sh test --only="Conversions"
Expected results
The test should pass and collect both compile time and runtime metrics. Once !1268 (closed) is merged, you can check this by looking for both "runtime/" and "compile_time/" in your git notes after running the test (make sure to commit all changes to a new commit so you have a clean working tree else the metrics will not we saved):
$ git notes --ref perf show HEAD
Actual result
The testrunner crashes complaining about a missing stats file:
/run/user/1000/ghctest-bc3dfj9q/test spaces/testsuite/tests/perf/should_run/Conversions.run Conversions [[Errno 2] No such file or directory: '/run/user/1000/ghctest-bc3dfj9q/test spaces/testsuite/tests/perf/should_run/Conversions.run/Conversions.stats'] (normal)
Notes
In the .T file we specify compile_and_run
. I'm guessing we have some offending logic that means only the runtime stats file will be generated in this case. When using compile
instead of compile_and_run
, there is still a crash but I assume now we are only generating compile time stats and not runtime stats. In short, the fix will likely support generating compile and runtime stats when using the compile_and_run
option.