Proposal: Add Property-Based Tests (PBTs) to GHC
Motivation
PBTs are an excellent way of testing, and one widely accepted in the Haskell community; they are particularly well suited for testing compilers as well. Currently, GHC does not use PBTs because (in my understanding) they would require vendoring a new library for the compiler and all the work that entails.
I instead propose we add PBTs to GHC in a slightly different way that would not require vendoring, and would allow more flexibility for future tests, should we require them.
Proposal
The basic idea is you put the PBTs in a new repo ghc-extended-tests
that imports the ghc
library. These PBTs are not run on every commit/merge. Instead, they are run each night. The failures report is sent to the GHC contributor's Matrix channel upon completion.
Justification
- Why a new repo? This decouples the tests from the compiler, and thus we can use whatever libraries we'd like without having to vendor.
- Why not run on merge/commit and instead each night? PBTs are inherently "flakey" in the sense that they are not deterministic. This is good thing (or at least some good properties follow from this). It would be annoying if you were working on some feature and some random unrelated PBT failed and stalled your merge. Having it run on a schedule would eliminate this issue; there are other possibilities such as having a merge kick off a run of the PBTs as well, but not allowing it to block merging.
- Why send the failure report to Matrix? This is important: if they PBTs are just run quietly, they will eventually start failing and nobody will know. While people might still not care, at least they will know if the tests are failing.
Issues
- There will be synchronization issues with the
ghc-extended-tests
repo and the mainghc
repo. I have a couple of ideas of how to work around these, but I haven't thought them through fully; I'd like to confirm this would be a desired direction to follow before I do so. - This will require a slight change in culture, since if the PBTs fall out of sync with GHC, then this will have been a pretty pointless endeavour. Someone would likely need to be responsible for triaging failures of PBTs at least once a week, at a minimum. This would entail going through the failures and opening issues for each related group of failure cases.
- Someone needs to accept responsibility for the work and implement it. I would nominate myself, as I have a reasonable amount of GitLab CI experience and believe I have enough free time to maintain the repo. Anyone else wanting to collaborate with me is of course welcome and encouraged to, however!
Edited by Matt Walker