Stabilise benchmarks wrt. GC

Note again that this changes the runtimes of virtually all benchmarks. Some of them are changed rather drastically by weaving runtime arguments into input data, so that meaningful runtimes can even be measured.

All benchmarks have now runtimes approximately in the following brackets:

fast: 0.1-0.2s
norm: 1-2s
slow: 5-10s

And where I found that benchmarks were sensitive to GC parameters (Trac #15999), I iterated the benchmarked logic often enough to make them stable.

I don't expect that anybody will actually look at all the changes I did, but hope that you agree that this makes Nofib vastly more useful and thus is a 'breaking change' (from the perspective of historic comparibility) we should be willing to do.

I don't expect that anybody will actually look at all the changes I did, but hope that you agree that this makes Nofib vastly more useful and thus is a 'breaking change' (from the perspective of historic comparibility) we should be willing to do.

I strongly agree with this.

Thanks for doing all this work Sebastian.

It would be helpful to have a ReadMe or wiki page or something describing the setup and the steps you have taken, so that someone in 5 years time will know what to be careful about.

added 1 commit

8632268a - Stabilise benchmarks wrt. GC

Compare with previous version

I added a section in the wiki and edited the README.md, also adding a section about GC impact.

WRT to the eff benchmarks (which I added), they have caused some problems for various people so I'm happy to move them to performance tests if you think that's better. However, I placed them in nofib as they are actual real world programs written in a style which you might find in an application today rather than most the programs which are extremely dated.

I agree that we need more up to date real benchmarks and that test-driving effect handlers is a perfect fit. But only in conjunction with an actual application they are applied to rather than a counting loop. It's hard to come up with such an example (especially with a self-contained one these days). Maybe we need to ask around in the community...

The examples in his repo are all quite self-contained but still too simple probably for what you are looking for.

https://github.com/AndrasKovacs/misc-stuff/tree/master/haskell/Eff

I don't think asking the community is going to be that helpful as now there are package managers adding dependencies isn't a problem for most people.

The examples in his repo are all quite self-contained but still too simple probably for what you are looking for.

Interesting! Indeed, these suffer from the same symptoms.

I don't think asking the community is going to be that helpful as now there are package managers adding dependencies isn't a problem for most people.

We really need some kind of vendoring tool that can just pull in the source of arbitrary packages. Then we could pipe the result through a mature version of https://github.com/nomeata/hs-all-in-one and strip dead code.

merged

mentioned in issue ghc#9374

mentioned in issue ghc#16499 (closed)

Stabilise benchmarks wrt. GC

Merge request reports

Activity