GHC's Capability management is racy in the presence of multiple compiler sessions
Currently the driver (namely GHC.Driver.Make.runParPipelines
) is responsible for raising the number of capabilities when parallel compilation is requested. Specifically, it raises the capability count at the beginning of the pipeline and reverts the count to its original value at the end of the pipeline.
However, this is racy in the presence of multiple compiler sessions: one pipeline may raise the capability count, only to have another concurrent pipeline execution immediately reduce it upon completion. Moreover, it leads to the potential of oversubscription, since each session assumes that it may use N
cores.
Lastly, it's unclear how the current model should be extended to admit use of a global make-jobserver-style semaphore (see #19416).
I suspect that we should rather eliminate the capability-management logic from the driver and rather move it to the compiler's main entrypoint. Instead of taking an integer job count, GHC.Driver.Make.runPipelines
should rather take something similar to the following to determine its concurrency strategy:
data CompilationParallelism
= SerialCompilation
| ParallelCompilation AbstractSem