Skip to content

WIP: Initial implementation for GNU Make jobserver support

Ellie Hermaszewska requested to merge expipiplus1/ghc:ellie-jobserver into master

The key part of this patch is in Jobserver.hs where the complicated bit happens. The rest is just option parsing interface rubbish which I'm happy to change as necessary.

Essentially, it makes use of a pair of new acquire release functions for AbstractSem.

  • acquire will: First give out the implicit job for the GHC process, after that it'll start waiting on tokens from the read fd passed in by the --jobserver-auth=r,w argument in GHC_MAKEFLAGS (or in GHC's actual arguments). When it gets a token, it adds them to an external stack to be sent back later.
  • release will: Give back tokens from the stack, and only when these are exhausted free up the implicit job.

This strategy means that we're as frugal as possible in the jobs we take from the jobserver, and that the implicit job (which we can't give back) is idle for as little time as possible.

Sadly, (because we can't read(2) from STM!) the logic in acquire is pretty complicated. I've not gone over it yet with a fine-toothed-brain, so there may be improvements to be made.

We use GHC_MAKEFLAGS rather than MAKEFLAGS to work around https://savannah.gnu.org/bugs/?57242. In a GNU Make powered build --jobserver-auth=3,4 is always passed, regardless of whether or not files 3 and 4 are valid jobserver pipes! They are only not closed if one prepends the call to ghc in Makefile with +. Unfortunately the RTS seems to like to open them, meaning that by the time we get to option parsing, it's not possible to detect whether we are actually running in a jobserver. By reading GHC_MAKEFLAGS instead the user can opt into this by making sure to set GHC_MAKEFLAGS=$$MAKEFLAGS in her Makefile when prepending the +.

In a synthetic benchmark this change yeilds a great improvement to the total compile time (23s -> 16s). The benchmark is 16 projects of 16 modules without dependencies. Compiling with make -j calling ghc -j leads to quite a lot of oversubscription on my machine, however make -j calling ghc using a jobserver keeps things chugging along with a much healthier load.

Although this isn't necessarily useful to ghc --make when called standalone, I hope it can be of use when GHC is taking part in a larger build system.

TODO:

  • add a testcase to the testsuite.
  • updates the users guide if applicable
  • mentions new features in the release notes for the next release
  • Disable properly on Windows
  • inline documentation
Edited by Ellie Hermaszewska

Merge request reports