Add head.hackage post

blog/20190305-head-hackage.mkd

+---
+author: Ben Gamari
+title: "Integrating head.hackage with GHC's CI infrastructure"
+date: 2019-07-08
+tags: infrastructure, testing, head.hackage
+---
+
+Hello everyone,
+
+As I mentioned in my earlier [infrastructure update](20190403-infra-status.html), 
+one of the many features that we have gained as a result of our migration to
+GitLab is the ability to incorporate testing against user code (e.g. from
+Hackage) into our routine CI testing.
+
+This post will describe the current state of that infrastructure as well as
+what you (yes, you!) can do to help us make the most of this infrastructure. No
+GHC hacking experience is required!
+
+# The goal
+
+For a long time we have discussed using our large package repositories (Hackage
+and Stackage) for testing GHC snapshots and pre-releases. Specifically,
+there are two ends which such testing might serve:
+
+1. *correctness testing*: By confirming that user code compiles we have better
+   assurance that we understand the full implications of changes made in GHC on
+   our users and that those changes are implemented correctly.
+2. *compiler performance testing*: By measuring the performance of the compiler
+   as it compilers user code we learn more about the *typical* cost centers
+   within GHC. While we have dedicated performance testsuites (e.g. `nofib`)
+   that also serve this purpose, there is plenty of evidence that suggests that
+   the programs in these testsuites are qualitatively different from modern
+   Haskell programs.
+3. *runtime performance testing*: As with (2), but measuring the performance of
+   the compiled program itself rather than that of GHC.
+
+While the potential pay-offs are large, so are the challenges:
+
+* *changes in GHC and core libraries*: Due to the tight coupling between GHC
+  and its core libraries (e.g. `base`, `template-haskell`), GHC releases are
+  typically accompanied by library changes which break user programs. These
+  need to be patched but in a way that allows package authors to respect the
+  PVP.
+* *changes in user programs*: We must take care when updating the set of test
+  programs, lest we are fooled into thinking that a change in a user program is a
+  regression in GHC.
+
+Recently I have been working on bringing up infrastructure for using
+CI-produced binary distributions to build a subset of Hackage using the
+head.hackage patchset.
+
+Happily, this effort has now converged on a usable result, embodied in
+three merge requests:
+
+[ghc/head.hackage!2](https://gitlab.haskell.org/ghc/head.hackage/merge_requests/2)
+adds CI support to head.hackage. In addition to the
+usual pass/fail status, this job produces
+(e.g. as seen in [this job](https://gitlab.haskell.org/ghc/head.hackage/-/jobs/119620))
+a few additional products:
+
+ * a JSON summary of the run, describing the dependency graph and
+   pass/fail state of each package. we can feed to an external service to
+   track newly-failing packages.
+
+ * a [dependency graph](http://ghc.gitlab.haskell.org/head.hackage/summary.dot.svg) showing the dependency structure of the
+   built packages, each node colored by its pass/fail state. This is intended
+   to facilitate easily root-causing build failures.
+
+ * a tarball of build logs, each including statistics from GHC's
+   `-ddump-timings` flag. The intent here is that we can use these statistics
+   to track compiler performance on real-world code.
+
+[ghc/head.hackage!3](https://gitlab.haskell.org/ghc/head.hackage/merge_requests/3)
+introduces the infrastructure necessary to push patched packages to a standard
+[Hackage repository](http://ghc.gitlab.haskell.org/head.hackage/) for use with
+`cabal-install`'s new project-specific repository support. I will publish a
+tutorial describing how to use this repository to build projects using GHC
+pre-releases in a future post.
+
+Finally, [ghc/ghc!465](https://gitlab.haskell.org/ghc/ghc/merge_requests/465)
+adds a job to GHC's own CI pipeline to trigger a head.hackage job using the
+binary distribution produced earlier in the pipeline. This job will run
+automatically under a variety of circumstances:
+
+ * via the scheduled nightly pipeline
+
+ * on merge requests labelled with the `user-facing` label
+
+ * when manually started using a button on the MR's Pipelines tab
+
+Currently the head.hackage MR tests only a small number of Hackage
+packages (and their dependencies). Specifically, we currently build
+`aeson`, `criterion`, `singletons`, `servant`, and `scotty`. The transitive
+closure of this set pulls in 127 packages in total. However, with more hands
+involved in maintenance we can easily grow this set. If you know of a package
+that you would like to see tested regularly with GHC pre-releases feel free to
+open a [merge
+request](https://gitlab.haskell.org/ghc/head.hackage/merge_requests) [adding a
+patch](https://gitlab.haskell.org/ghc/head.hackage#adding-a-patch) and adding
+the package to the [set of tested
+packages](https://gitlab.haskell.org/ghc/head.hackage/blob/d0112e7ec7c58b49dd448d3fd873abb4b8875a3a/scripts/build-all.nix#L59).
+
+There are a few things that remain to be done:
+
+ * Better document the existence of the job and how it is triggered
+
+ * Document how to update the list of tested packages
+
+ * Work out how to handle tracking of persistent breakage; for instance, we want
+   a responsible party to be notified when a package initially breaks (e.g.
+   when a breaking change is merged to GHC) but not in every subsequent build.
+
+Finally, I'd like to acknowledge Herbert Valerio Riedel who planned and
+implemented the groundwork for the current incarnation of this infrastructure.
+
+Cheers,
+
+- Ben
+