Testrunner fails to generate compile time stats when a test is set to compile_and_run

If we want to collect both compile time and runtime metrics from a single test, the test runner will currently fail.

Recreation

Modify the Conversions test in testsuite/tests/perf/should_run/all.T to:

test('Conversions',
     [collect_stats('bytes allocated',3),
      collect_compiler_stats('bytes allocated',3),
      only_ways(['normal'])
     ],
    compile_and_run, ['-O -msse2'])

then run the test:

$ ./hadrian/build.sh test --only="Conversions"

Expected results

The test should pass and collect both compile time and runtime metrics. Once !1268 (closed) is merged, you can check this by looking for both "runtime/" and "compile_time/" in your git notes after running the test (make sure to commit all changes to a new commit so you have a clean working tree else the metrics will not we saved):

$ git notes --ref perf show HEAD

Actual result

The testrunner crashes complaining about a missing stats file:

/run/user/1000/ghctest-bc3dfj9q/test   spaces/testsuite/tests/perf/should_run/Conversions.run  Conversions [[Errno 2] No such file or directory: '/run/user/1000/ghctest-bc3dfj9q/test   spaces/testsuite/tests/perf/should_run/Conversions.run/Conversions.stats'] (normal)

Notes

In the .T file we specify compile_and_run. I'm guessing we have some offending logic that means only the runtime stats file will be generated in this case. When using compile instead of compile_and_run, there is still a crash but I assume now we are only generating compile time stats and not runtime stats. In short, the fix will likely support generating compile and runtime stats when using the compile_and_run option.