Skip to content

Pipelining for improved parallel compilation

Marc Scholten requested to merge mpscholten/ghc:wip/two-phase-interface into master

This MR addresses ticket #14095 (improve build parallelism).

See also #26797 (closed).

Summary

This patch implements two-phase interface file generation, allowing dependent modules to start typechecking as soon as their dependencies finish typechecking, without waiting for codegen to complete.

Previously, a module would only signal completion after both typechecking and code generation were done. This created unnecessary serialization in parallel builds—dependents had to wait for codegen even though they only need the interface for typechecking.

Changes

Now we signal a partial interface immediately after typechecking:

  1. After typechecking completes, we create an early ModIface and signal it via a new partialIfaceVar MVar
  2. Dependent modules wait on partialIfaceVar instead of the full result
  3. Codegen proceeds in parallel while dependents typecheck
  4. The full result (with linkable) is signaled as before

Key implementation details:

  • BuildResult now contains an optional partialIfaceVar for modules
  • BuildLoopState tracks whether two-phase signaling is enabled
  • New wait_partial_ifaces helper waits on partial interfaces
  • hscPipelineWithEarlySignal takes a callback to signal early completion
  • Early HomeModInfo includes proper ModDetails (via initModDetails)
  • Two-phase signaling only enabled for parallel builds (n_jobs > 1)
  • Error handling ensures MVar is filled even on compilation failure

Important: Modules using Template Haskell, QuasiQuotes, or home-module plugins still wait for full dependencies, since they need to execute code from those dependencies during compilation.

Benchmarks

Tested on real libraries with -j (parallel compilation):

Library Modules Before After Improvement
text 31 5.2s 3.5s ~ 33% faster
xhtml 13 0.89s 0.54s ~ 39% faster

Fixes #26797 (closed)

Edited by Simon Peyton Jones

Merge request reports

Loading