WIP: Initial implementation for GNU Make jobserver support
- See also: !5176 (closed)
- Closes: #19416
The key part of this patch is in Jobserver.hs
where the complicated bit happens. The rest is just option parsing interface rubbish which I'm happy to change as necessary.
Essentially, it makes use of a pair of new acquire
release
functions for AbstractSem
.
-
acquire
will: First give out the implicit job for the GHC process, after that it'll start waiting on tokens from theread
fd passed in by the--jobserver-auth=r,w
argument inGHC_MAKEFLAGS
(or in GHC's actual arguments). When it gets a token, it adds them to an external stack to be sent back later. -
release
will: Give back tokens from the stack, and only when these are exhausted free up the implicit job.
This strategy means that we're as frugal as possible in the jobs we take from the jobserver, and that the implicit job (which we can't give back) is idle for as little time as possible.
Sadly, (because we can't read(2)
from STM!) the logic in acquire
is pretty complicated. I've not gone over it yet with a fine-toothed-brain, so there may be improvements to be made.
We use GHC_MAKEFLAGS
rather than MAKEFLAGS
to work around https://savannah.gnu.org/bugs/?57242. In a GNU Make powered build --jobserver-auth=3,4
is always passed, regardless of whether or not files 3 and 4 are valid jobserver pipes! They are only not closed if one prepends the call to ghc
in Makefile with +
. Unfortunately the RTS seems to like to open them, meaning that by the time we get to option parsing, it's not possible to detect whether we are actually running in a jobserver. By reading GHC_MAKEFLAGS
instead the user can opt into this by making sure to set GHC_MAKEFLAGS=$$MAKEFLAGS
in her Makefile when prepending the +
.
In a synthetic benchmark this change yeilds a great improvement to the total compile time (23s -> 16s). The benchmark is 16 projects of 16 modules without dependencies. Compiling with make -j
calling ghc -j
leads to quite a lot of oversubscription on my machine, however make -j
calling ghc
using a jobserver keeps things chugging along with a much healthier load.
Although this isn't necessarily useful to ghc --make
when called standalone, I hope it can be of use when GHC is taking part in a larger build system.
TODO:
-
add a testcase to the testsuite. -
updates the users guide if applicable -
mentions new features in the release notes for the next release -
Disable properly on Windows -
inline documentation