Skip to content

Foreign sources are not ever compiled in parallel

Summary

Foreign sources are not compiled in parallel by GHC.

The status quo: foreign sources are compiled by GHC running in one-shot mode, with one GHC invocation per source file (see https://github.com/haskell/cabal/blob/master/Cabal/src/Distribution/Simple/GHC/Build/ExtraSources.hs#L262). This happens after all Haskell sources are compiled with --make, because C and C++ sources may #include stub header files generated by Haskell source compilation. --make apparently doesn't do the thing cabal wants when passed both foreign sources and Haskell sources.

Possible fix # 1: --make can learn to understand that foreign sources should always be built after Haskell sources, and maybe even use its existing parallelism machinery to do so. I'm not sure what work exactly is required to do this. I see that the compiler options used by cabal are different between each different type of foreign sources, though I don't know if that's an obstacle in practice. I also see that --make has special handling for foreign sources already, which appears to compile those sources before Haskell sources, the reverse of what we want (https://gitlab.haskell.org/ghc/ghc/-/blob/master/ghc/Main.hs?ref_type=heads#L764). Perhaps we just need to reverse that?

Possible fix # 2: one-shot mode can learn to compile in parallel, using the -j[n] and -jsem options the same way as --make, just without dependency analysis. I've implemented this approach because it seemed safer and potentially a good addition even if --make ends up being "fixed" (assuming it's broken). The MR is !12388.

Edited by edmundnoble
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information