Short benchmark runtimes

I'm currently working on an optimization and trying to figure out whether it is a net-win or not with all the noise is quite hard.

According to the README, the three modes should have the following meaning:

• fast: 0.1-0.2s
• norm: 1-2s
• slow: 5-10s

Of course, as machines get faster, those numbers will drift.

However, my main issue is, that some programs have no parameters at all.

E.g., spectral/pretty has only one mode and runs in approximately 0,01061s Wall time (and mutator time is 0,0002181s).

real/eff/VSM even has a value n for iterations, but hard-coded.

Afaik spectral/scc also doesn't have different inputs? (runtime 1.07E-02)

shootout/reverse-complement does have different inputs, but it's weird and complicated. (fixed)

List of programs running in <= 0.5s in mode "norm": (ghc 9.3 or something, linear scan allocator)

program	Wall time
real/mkhprog	1.06E-02
spectral/pretty	1.06E-02
spectral/scc	1.06E-02
shootout/reverse-complement	1.07E-02
real/eff/VSD	2.06E-02
real/eff/CS	9.07E-02
real/eff/VSM	0.121
real/eff/FS	0.414
real/eff/VS	0.415
spectral/last-piece	0.488

But more interesting, short mutator time vs. std. err.:

program	mutator time	std.err.
shootout/reverse-complement	1.18E-04	7.30%
spectral/pretty	2.18E-04	16.60%
spectral/scc	2.25E-04	17.20%

So this is quite annoying, I can't even run the benchmarks in slow-mode, bc. that won't change anything. Not sure what to do here. Updating the inputs for the modes is one possibility, but of course I don't know what kind of machine this expects (I have a fairly recent Ryzen 4700 laptop). But while 0.5s on my laptop may be 1s on someone else's machine, 0.002s won't.

And the programs without inputs? Can we change them?

For my analysis I might simply have to exclude all fast running benchmarks...