Improve nofib comparison output
When running nofib comparisons, I use these instructions
But "Comparison" command, namely $ cabal v2-run nofib-compare -- approach-b.results.csv approach-a.results.csv, generates output that is much less helpful than it used to be. Sample:
# bytes allocated
+-------------------------------++--+----------------------------------------------------------------+-----------+----------------------------------------+-----------+
| || | /home/simonpj/code/HEAD/nofib/_make/2022-08-19/run.results.tsv | std. err. | _make/2022-08-25/run.results.tsv (rel) | std. err. |
+===============================++==+================================================================+===========+========================================+===========+
| imaginary/bernouilli || | 2.823e9 | 0.0% | 0.00% | 0.0% |
| imaginary/digits-of-e1 || | 9.147e8 | 0.0% | 0.00% | 0.0% |
| imaginary/digits-of-e2 || | 2.279e9 | 0.0% | +0.01% | 0.0% |
| imaginary/exp3_8 || | 5.887e9 | 0.0% | 0.00% | 0.0% |
| imaginary/gen_regexps || | 9.102e8 | 0.0% | 0.00% | 0.0% |
....
| imaginary/integrate || | 3.424e9 | 0.0% | 0.00% | 0.0% |
| spectral/simple || | 1.201e9 | 0.0% | 0.00% | 0.0% |
| spectral/sorting || | 3.077e9 | 0.0% | 0.00% | 0.0% |
| spectral/sphere || | 2.236e9 | 0.0% | 0.00% | 0.0% |
| spectral/treejoin || | 2.796e9 | 0.0% | 0.00% | 0.0% |
+===============================++==+================================================================+===========+========================================+===========+
| geom mean || | | -0.03% | | |
+-------------------------------++--+----------------------------------------------------------------+-----------+----------------------------------------+-----------+Problems:
- It is far too wide. I have to resize my editor window every time, and my screen becomes mostly blank.
- There is no min and max, only geometric mean.
- It's not obvious what the geom mean is for. It sits at the bottom of the "std error column" but I'm guessing it belongs in the next column.
- I have no idea what either "std err" column is. They are always zero, diverting attention.
- The old nofib-analyse started with a table that looked like this, where I'm comparing a baseline run with three comparator runs (X1, X2, and X3, say)
This was great for getting an overview of the key changes: binary size, allocs, runtime. (A column for compiler allocs would be great too.) Could we have this back?
Benchmark Binary size change Runtime allocs change Runtime change X1 X2 X3 X1 X2 X3 X1 X2 X3 spectral/sphere +2.8% -3% 0% +1% -1% +2.7% +0.8% +2% +6%
Edited by Simon Peyton Jones