Pipelining for improved parallel compilation
This MR addresses ticket #14095 (improve build parallelism).
See also #26797 (closed).
Summary
This patch implements two-phase interface file generation, allowing dependent modules to start typechecking as soon as their dependencies finish typechecking, without waiting for codegen to complete.
Previously, a module would only signal completion after both typechecking and code generation were done. This created unnecessary serialization in parallel builds—dependents had to wait for codegen even though they only need the interface for typechecking.
Changes
Now we signal a partial interface immediately after typechecking:
- After typechecking completes, we create an early
ModIfaceand signal it via a newpartialIfaceVarMVar - Dependent modules wait on
partialIfaceVarinstead of the full result - Codegen proceeds in parallel while dependents typecheck
- The full result (with linkable) is signaled as before
Key implementation details:
-
BuildResultnow contains an optionalpartialIfaceVarfor modules -
BuildLoopStatetracks whether two-phase signaling is enabled - New
wait_partial_ifaceshelper waits on partial interfaces -
hscPipelineWithEarlySignaltakes a callback to signal early completion - Early
HomeModInfoincludes properModDetails(viainitModDetails) - Two-phase signaling only enabled for parallel builds (
n_jobs > 1) - Error handling ensures MVar is filled even on compilation failure
Important: Modules using Template Haskell, QuasiQuotes, or home-module plugins still wait for full dependencies, since they need to execute code from those dependencies during compilation.
Benchmarks
Tested on real libraries with -j (parallel compilation):
| Library | Modules | Before | After | Improvement |
|---|---|---|---|---|
| text | 31 | 5.2s | 3.5s | ~ 33% faster |
| xhtml | 13 | 0.89s | 0.54s | ~ 39% faster |
Fixes #26797 (closed)