Test for performance of rewriting plugins seems flaky
The TcPlugin_RewritePerf
test, which measures the performance impact of rewriting plugins on type-family reductions,
seems to exhibit large decreases in allocations in some CI runs (98% decreases):
compile_time/bytes allocated decreased from x86_64-linux-deb9-hadrian baseline @ a975583c70e57434340d9a20c976c8f06fde9beb:
Expected RewritePerfDefs.hs (normal) compile_time/bytes allocated: 2239480832.0 +/-1%
Lower bound RewritePerfDefs.hs (normal) compile_time/bytes allocated: 2217086023
Upper bound RewritePerfDefs.hs (normal) compile_time/bytes allocated: 2261875641
Actual RewritePerfDefs.hs (normal) compile_time/bytes allocated: 47791056
Deviation RewritePerfDefs.hs (normal) compile_time/bytes allocated: -97.9 %
The issue seems to be that we are not always measuring the same thing in this test: sometimes all the modules in the test are tested separately, and sometimes they are lumped together.