Gather ideas from Rust's crater
On this Cabal issue @rebeccat mentions Rust's crater tool.
Further on in the thread https://github.com/haskell/cabal/issues/10605#issuecomment-2524594842, she writes:
Two important differences between crater and [head.hackage or clc-stackage]:
- You can leave a comment with @craterbot check on a PR and have everything run automatically. A job will be created, scheduled, run, and the report will be uploaded automatically.
- Crater keeps track of failures so that it only notifies you of new regressions. Knowing how to exclude the "ancient stuff which won't build with modern ghc" is a critical part of this problem and it's a big part of the reason head.hackage isn't very useful in its current form. Here's a recent report from crater: https://crater-reports.s3.amazonaws.com/beta-1.74-1/index.html
Note a few things about the output:
- Errors that also errored in the previous run are in a separate section.
- Errors are grouped by error code, so you can see that e.g. that run introduced regressions which added 5 new errors with code E0599.
- Test suites are run for each package.
- The report can distinguish between build failures in dependencies and build failures in a package itself.
We can already label GHC PRs with user-facing and have head.hackage run, but it sounds like that's not as nice as (1). And of course that only applies on the GHC repo, the same wouldn't apply on other forges craterbot also just runs on the respective compiler's repo (https://github.com/rust-lang/crater/blob/master/docs/bot-usage.md.)
Similarly, we already have some support for expected failures, but these are manually set, and we only test a limited set of packages. So (2) sounds like another area where we could improve.
I think these are some great ideas for potential improvements to head.hackage (especially around how we show our output).
I'm hoping to take a look at crater at some point in the new year and see if there's anything else we can learn from it.
