Commit 6486c6e4 authored by Matthew Pickering's avatar Matthew Pickering

Hadrian: Add support for building stage3

This ticket enables the building of a `stage3` compiler by making the
build logic more consistent and predictable in Hadrian.

Two of the main changes are:

1. In order to build anything at stageN we use the package database
present at stageN. Fixing #16069
2. `haddock` and `ghc-tags` are built
as stage1 executables (with the stage1 compiler) rather than as
stage2 compiler. Fixing
[hadrian#661](https://github.com/snowleopard/hadrian/issues/661)

In order to build a stage3 compiler, you have to set the new `finalStage` hadrian option to `Stage3`.
parent 6b70cf61
Pipeline #983 failed with stages
in 603 minutes and 11 seconds
......@@ -183,17 +183,24 @@ $ ./configure [--prefix=PATH] && make install
workflow, for now.
#### Building stage3
It is possible to define a flavour which builds a `stage3` compiler, which is a
compiler built using `stage2`. This is useful for cross-compilation. Detailed
instructions can be found in the
[user settings part of the manual](doc/user-settings.md#specifying-the-final-stage-to-build).
Troubleshooting
---------------
Here are a few simple suggestions that might help you fix the build:
* Hadrian is occasionally broken by changes in GHC. If this happens, you might want to switch
to an earlier GHC commit.
* If Hadrian fails with the message `Configuration file hadrian/cfg/system.config is missing`,
you have probably forgotten to pass the `--configure` flag during the first build.
* If you need help in debugging Hadrian, read the [wiki](https://github.com/snowleopard/hadrian/wiki)
and Shake's [debugging tutorial](https://shakebuild.com/debugging).
......
......@@ -93,18 +93,22 @@ project-git-commit-id = @ProjectGitCommitId@
conf-cc-args-stage0 = @CONF_CC_OPTS_STAGE0@
conf-cc-args-stage1 = @CONF_CC_OPTS_STAGE1@
conf-cc-args-stage2 = @CONF_CC_OPTS_STAGE2@
conf-cc-args-stage3 = @CONF_CC_OPTS_STAGE3@
conf-cpp-args-stage0 = @CONF_CPP_OPTS_STAGE0@
conf-cpp-args-stage1 = @CONF_CPP_OPTS_STAGE1@
conf-cpp-args-stage2 = @CONF_CPP_OPTS_STAGE2@
conf-cpp-args-stage3 = @CONF_CPP_OPTS_STAGE3@
conf-gcc-linker-args-stage0 = @CONF_GCC_LINKER_OPTS_STAGE0@
conf-gcc-linker-args-stage1 = @CONF_GCC_LINKER_OPTS_STAGE1@
conf-gcc-linker-args-stage2 = @CONF_GCC_LINKER_OPTS_STAGE2@
conf-gcc-linker-args-stage3 = @CONF_GCC_LINKER_OPTS_STAGE3@
conf-ld-linker-args-stage0 = @CONF_LD_LINKER_OPTS_STAGE0@
conf-ld-linker-args-stage1 = @CONF_LD_LINKER_OPTS_STAGE1@
conf-ld-linker-args-stage2 = @CONF_LD_LINKER_OPTS_STAGE2@
conf-ld-linker-args-stage3 = @CONF_LD_LINKER_OPTS_STAGE3@
# Include and library directories:
#=================================
......
......@@ -16,7 +16,6 @@ After all the dependencies are in place:
- `git submodule update --init`
- `./configure --target=arm-linux-gnueabihf`
- `cd hadrian`
- Modify `src/UserSettings.hs`, set `stage1Only` to `True`.
- Build the compiler by e.g. `./build.sh --flavour=quickest --integer-simple -V -j`
After that, you should have built `inplace/bin/ghc-stage1` cross compiler. We will go to the next section to validate this.
......
......@@ -138,6 +138,22 @@ You can choose which integer library to use when builing GHC using the
userFlavour :: Flavour
userFlavour = defaultFlavour { name = "user", integerLibrary = integerSimple }
```
### Specifying the final stage to build
The `finalStage` variable can be set to indicate after which stage we should
stop the compilation pipeline. By default it is set to `Stage2` which indicates
that we will build everything which uses the `Stage1` `ghc` and then stop.
```
finalStage :: Stage
finalStage = Stage2
```
Using this mechanism we can also build a `Stage3` compiler by setting
`finalStage = Stage3` or just a `Stage1` compiler by setting
`finalStage = Stage1`.
## Build ways
Packages can be built in a number of ways, such as `vanilla`, `profiling` (with
......@@ -202,10 +218,6 @@ verboseCommand = return True
## Miscellaneous
By setting `stage1Only = True` you can disable building Stage2 GHC and Stage2
utilities such as `haddock`. Note that all Stage0 and Stage1 libraries will
still be built.
To change the default behaviour of Hadrian with respect to building split
objects, override the `splitObjects` setting of the `Flavour` record:
```haskell
......
......@@ -83,9 +83,9 @@ generatedPath :: Action FilePath
generatedPath = buildRoot <&> (-/- generatedDir)
-- | Path to the package database for a given build stage, relative to the build
-- root. Note that @StageN@, where @N > 1@, uses the 'Stage1' package database.
-- root.
relativePackageDbPath :: Stage -> FilePath
relativePackageDbPath stage = stageString (min stage Stage1) -/- "lib/package.conf.d"
relativePackageDbPath stage = stageString stage -/- "lib/package.conf.d"
-- | Path to the package database used in a given 'Stage', including
-- the build root.
......
......@@ -152,8 +152,8 @@ builderProvenance = \case
Ghc _ Stage0 -> Nothing
Ghc _ stage -> context (pred stage) ghc
GhcPkg _ Stage0 -> Nothing
GhcPkg _ _ -> context Stage0 ghcPkg
Haddock _ -> context Stage2 haddock
GhcPkg _ s -> context (pred s) ghcPkg
Haddock _ -> context Stage1 haddock
Hpc -> context Stage1 hpcBin
Hp2Ps -> context Stage0 hp2ps
Hsc2Hs _ -> context Stage0 hsc2hs
......
......@@ -103,9 +103,9 @@ pkgGhciLibraryFile context@Context {..} = do
-- | Path to the configuration file of a given 'Context'.
pkgConfFile :: Context -> Action FilePath
pkgConfFile Context {..} = do
root <- buildRoot
pid <- pkgIdentifier package
return $ root -/- relativePackageDbPath stage -/- pid <.> "conf"
dbPath <- packageDbPath stage
return $ dbPath -/- pid <.> "conf"
-- | Given a 'Context' and a 'FilePath' to a source file, compute the 'FilePath'
-- to its object file. For example:
......
......@@ -183,21 +183,23 @@ autogenPath context@Context {..}
where
autogen dir = contextPath context <&> (-/- dir -/- "autogen")
-- | RTS is considered a Stage1 package.
rtsContext :: Context
rtsContext = vanillaContext Stage1 rts
-- | RTS is considered a Stage1 package. This determines RTS build directory.
rtsContext :: Stage -> Context
rtsContext stage = vanillaContext stage rts
-- | Path to the RTS build directory.
rtsBuildPath :: Action FilePath
rtsBuildPath = buildPath rtsContext
rtsBuildPath :: Stage -> Action FilePath
rtsBuildPath stage = buildPath (rtsContext stage)
-- | The 'libffi' library is considered a 'Stage1' package.
libffiContext :: Context
libffiContext = vanillaContext Stage1 libffi
-- | Build directory for libffi
-- This probably doesn't need to be stage dependent but it is for
-- consistency for now.
libffiContext :: Stage -> Context
libffiContext stage = vanillaContext stage libffi
-- | Build directory for in-tree 'libffi' library.
libffiBuildPath :: Action FilePath
libffiBuildPath = buildPath libffiContext
libffiBuildPath :: Stage -> Action FilePath
libffiBuildPath stage = buildPath (libffiContext stage)
-- | Name of the 'libffi' library.
libffiLibraryName :: Action FilePath
......
......@@ -30,18 +30,22 @@ allStages :: [Stage]
allStages = [minBound .. maxBound]
-- | This rule calls 'need' on all top-level build targets that Hadrian builds
-- by default, respecting the 'stage1Only' flag.
-- by default, respecting the 'finalStage' flag.
topLevelTargets :: Rules ()
topLevelTargets = action $ do
verbosity <- getVerbosity
when (verbosity >= Loud) $ do
(libraries, programs) <- partition isLibrary <$> stagePackages Stage1
libNames <- mapM (name Stage1) libraries
pgmNames <- mapM (name Stage1) programs
forM_ [ Stage1 ..] $ \stage -> do
when (verbosity >= Loud) $ do
(libraries, programs) <- partition isLibrary <$> stagePackages stage
libNames <- mapM (name stage) libraries
pgmNames <- mapM (name stage) programs
let stageHeader t ps =
"| Building " ++ show stage ++ " "
++ t ++ ": " ++ intercalate ", " ps
putNormal . unlines $
[ "| Building Stage1 libraries: " ++ intercalate ", " libNames
, "| Building Stage1 programs : " ++ intercalate ", " pgmNames ]
let buildStages = [Stage0, Stage1] ++ [Stage2 | not stage1Only]
[ stageHeader "libraries" libNames
, stageHeader "programs" pgmNames ]
let buildStages = [ s | s <- [Stage0 ..], s < finalStage ]
targets <- concatForM buildStages $ \stage -> do
packages <- stagePackages stage
mapM (path stage) packages
......@@ -101,7 +105,7 @@ packageRules = do
Rules.Program.buildProgramRules readPackageDb
Rules.Register.configurePackageRules
forM_ [Stage0, Stage1] (Rules.Register.registerPackageRules writePackageDb)
forM_ [Stage0 ..] (Rules.Register.registerPackageRules writePackageDb)
-- TODO: Can we get rid of this enumeration of contexts? Since we iterate
-- over it to generate all 4 types of rules below, all the time, we
......
......@@ -118,9 +118,9 @@ bindistRules = do
need ["docs"]
copyDirectory (root -/- "docs") bindistFilesDir
-- We copy the binary (<build root>/stage2/bin/haddock) to
-- We copy the binary (<build root>/stage1/bin/haddock) to
-- the bindist's bindir (<build root>/bindist/ghc-.../bin/).
haddockPath <- programPath (vanillaContext Stage2 haddock)
haddockPath <- programPath (vanillaContext Stage1 haddock)
copyFile haddockPath (bindistFilesDir -/- "bin" -/- "haddock")
-- We then 'need' all the files necessary to configure and install
......
......@@ -58,7 +58,7 @@ compilerDependencies = do
isGmp <- (== integerGmp) <$> getIntegerPackage
ghcPath <- expr $ buildPath (vanillaContext stage compiler)
gmpPath <- expr gmpBuildPath
rtsPath <- expr rtsBuildPath
rtsPath <- expr (rtsBuildPath stage)
mconcat [ return [root -/- platformH stage]
, return ((root -/-) <$> includesDependencies)
, return ((root -/-) <$> derivedConstantsDependencies)
......@@ -84,7 +84,8 @@ compilerDependencies = do
generatedDependencies :: Expr [FilePath]
generatedDependencies = do
root <- getBuildRoot
rtsPath <- expr rtsBuildPath
stage <- getStage
rtsPath <- expr (rtsBuildPath stage)
mconcat [ package compiler ? compilerDependencies
, package ghcPrim ? ghcPrimDependencies
, package rts ? return (fmap (rtsPath -/-) libffiDependencies
......@@ -188,7 +189,7 @@ generateRules = do
-- TODO: simplify, get rid of fake rts context
root -/- generatedDir ++ "//*" %> \file -> do
withTempDir $ \dir -> build $
target rtsContext DeriveConstants [] [file, dir]
target (rtsContext Stage1) DeriveConstants [] [file, dir]
where
file <~ gen = file %> \out -> generate out emptyTarget gen
......
......@@ -13,11 +13,11 @@ libffiDependencies = ["ffi.h", "ffitarget.h"]
libffiLibrary :: FilePath
libffiLibrary = "inst/lib/libffi.a"
rtsLibffiLibrary :: Way -> Action FilePath
rtsLibffiLibrary way = do
rtsLibffiLibrary :: Stage -> Way -> Action FilePath
rtsLibffiLibrary stage way = do
name <- libffiLibraryName
suf <- libsuf way
rtsPath <- rtsBuildPath
rtsPath <- rtsBuildPath stage
return $ rtsPath -/- "lib" ++ name ++ suf
fixLibffiMakefile :: FilePath -> String -> String
......@@ -27,33 +27,35 @@ fixLibffiMakefile top =
. replace "@INSTALL@" ("$(subst ../install-sh," ++ top ++ "/install-sh,@INSTALL@)")
-- TODO: check code duplication w.r.t. ConfCcArgs
configureEnvironment :: Action [CmdOption]
configureEnvironment = do
cFlags <- interpretInContext libffiContext $ mconcat
configureEnvironment :: Stage -> Action [CmdOption]
configureEnvironment stage = do
cFlags <- interpretInContext (libffiContext stage) $ mconcat
[ cArgs
, getStagedSettingList ConfCcArgs ]
ldFlags <- interpretInContext libffiContext ldArgs
sequence [ builderEnvironment "CC" $ Cc CompileC Stage1
, builderEnvironment "CXX" $ Cc CompileC Stage1
, builderEnvironment "LD" (Ld Stage1)
, builderEnvironment "AR" (Ar Unpack Stage1)
ldFlags <- interpretInContext (libffiContext stage) ldArgs
sequence [ builderEnvironment "CC" $ Cc CompileC stage
, builderEnvironment "CXX" $ Cc CompileC stage
, builderEnvironment "LD" (Ld stage)
, builderEnvironment "AR" (Ar Unpack stage)
, builderEnvironment "NM" Nm
, builderEnvironment "RANLIB" Ranlib
, return . AddEnv "CFLAGS" $ unwords cFlags ++ " -w"
, return . AddEnv "LDFLAGS" $ unwords ldFlags ++ " -w" ]
libffiRules :: Rules ()
libffiRules = do
root <- buildRootRules
fmap ((root <//> "rts/build") -/-) libffiDependencies &%> \_ -> do
libffiPath <- libffiBuildPath
need [libffiPath -/- libffiLibrary]
libffiRules =
forM_ [Stage1 ..] $ \stage -> do
root <- buildRootRules
fmap ((root -/- stageString stage -/- "rts/build") -/-) libffiDependencies
&%> \_ -> do
libffiPath <- libffiBuildPath stage
need [libffiPath -/- libffiLibrary]
-- we set a higher priority because this overlaps
-- with the static lib rule from Rules.Library.libraryRules.
priority 2.0 $ root <//> libffiLibrary %> \_ -> do
priority 2.0 $ root -/- stageString stage <//> libffiLibrary %> \_ -> do
useSystemFfi <- flag UseSystemFfi
rtsPath <- rtsBuildPath
rtsPath <- rtsBuildPath stage
if useSystemFfi
then do
ffiIncludeDir <- setting FfiIncludeDir
......@@ -62,22 +64,23 @@ libffiRules = do
copyFile (ffiIncludeDir -/- file) (rtsPath -/- file)
putSuccess "| Successfully copied system FFI library header files"
else do
libffiPath <- libffiBuildPath
build $ target libffiContext (Make libffiPath) [] []
libffiPath <- libffiBuildPath stage
build $ target (libffiContext stage) (Make libffiPath) [] []
hs <- getDirectoryFiles "" [libffiPath -/- "inst/include/*"]
forM_ hs $ \header ->
copyFile header (rtsPath -/- takeFileName header)
ways <- interpretInContext libffiContext (getLibraryWays <> getRtsWays)
ways <- interpretInContext (libffiContext stage)
(getLibraryWays <> getRtsWays)
forM_ (nubOrd ways) $ \way -> do
rtsLib <- rtsLibffiLibrary way
rtsLib <- rtsLibffiLibrary stage way
copyFileUntracked (libffiPath -/- libffiLibrary) rtsLib
putSuccess "| Successfully built custom library 'libffi'"
root <//> "libffi/build/Makefile.in" %> \mkIn -> do
libffiPath <- libffiBuildPath
root -/- stageString stage -/- "libffi/build/Makefile.in" %> \mkIn -> do
libffiPath <- libffiBuildPath stage
removeDirectory libffiPath
tarball <- unifyPath . fromSingleton "Exactly one LibFFI tarball is expected"
<$> getDirectoryFiles "" ["libffi-tarballs/libffi*.tar.gz"]
......@@ -90,20 +93,21 @@ libffiRules = do
removeDirectory (root -/- libname)
-- TODO: Simplify.
actionFinally (do
build $ target libffiContext (Tar Extract) [tarball] [root]
moveDirectory (root -/- libname) libffiPath) $
removeFiles root [libname <//> "*"]
build $ target (libffiContext stage) (Tar Extract)
[tarball]
[root -/- stageString stage]
moveDirectory (root -/- stageString stage -/- libname) libffiPath) $
removeFiles (root -/- stageString stage) [libname <//> "*"]
top <- topDirectory
fixFile mkIn (fixLibffiMakefile top)
-- TODO: Get rid of hard-coded @libffi@.
root <//> "libffi/build/Makefile" %> \mk -> do
-- TODO: Get rid of hard-coded @libffi@.
root -/- stageString stage -/- "libffi/build/Makefile" %> \mk -> do
need [mk <.> "in"]
libffiPath <- libffiBuildPath
forM_ ["config.guess", "config.sub"] $ \file ->
libffiPath <- libffiBuildPath stage
forM_ ["config.guess", "config.sub"] $ \file -> do
copyFile file (libffiPath -/- file)
env <- configureEnvironment
env <- configureEnvironment stage
buildWithCmdOptions env $
target libffiContext (Configure libffiPath) [mk <.> "in"] [mk]
target (libffiContext stage) (Configure libffiPath) [mk <.> "in"] [mk]
......@@ -88,7 +88,7 @@ buildBinary rs bin context@Context {..} = do
needLibrary =<< contextDependencies context
when (stage > Stage0) $ do
ways <- interpretInContext context (getLibraryWays <> getRtsWays)
needLibrary [ rtsContext { way = w } | w <- ways ]
needLibrary [ (rtsContext stage) { way = w } | w <- ways ]
cSrcs <- interpretInContext context (getContextData cSrcs)
cObjs <- mapM (objectPath context) cSrcs
hsObjs <- hsObjects context
......
......@@ -61,9 +61,8 @@ packageDatabaseArgs = do
stage <- getStage
dbPath <- expr (packageDbPath stage)
expr (need [dbPath -/- packageDbStamp])
root <- getBuildRoot
prefix <- ifM (builder Ghc) (return "-package-db ") (return "--package-db=")
arg $ prefix ++ root -/- relativePackageDbPath stage
arg $ prefix ++ dbPath
bootPackageDatabaseArgs :: Args
bootPackageDatabaseArgs = do
......
......@@ -7,7 +7,8 @@ import Settings.Builders.Common
configureBuilderArgs :: Args
configureBuilderArgs = do
gmpPath <- expr gmpBuildPath
libffiPath <- expr libffiBuildPath
stage <- getStage
libffiPath <- expr (libffiBuildPath stage)
mconcat [ builder (Configure gmpPath) ? do
hostPlatform <- getSetting HostPlatform
buildPlatform <- getSetting BuildPlatform
......
......@@ -10,11 +10,12 @@ makeBuilderArgs :: Args
makeBuilderArgs = do
threads <- shakeThreads <$> expr getShakeOptions
gmpPath <- expr gmpBuildPath
libffiPath <- expr libffiBuildPath
libffiPaths <- forM [Stage1 ..] $ \s -> expr (libffiBuildPath s)
let t = show $ max 4 (threads - 2) -- Don't use all Shake's threads
mconcat
[ builder (Make gmpPath ) ? pure ["MAKEFLAGS=-j" ++ t]
, builder (Make libffiPath) ? pure ["MAKEFLAGS=-j" ++ t, "install"] ]
mconcat $
(builder (Make gmpPath ) ? pure ["MAKEFLAGS=-j" ++ t]) :
[ builder (Make libffiPath) ? pure ["MAKEFLAGS=-j" ++ t, "install"]
| libffiPath <- libffiPaths ]
validateBuilderArgs :: Args
validateBuilderArgs = builder (Make "testsuite/tests") ? do
......
......@@ -108,7 +108,9 @@ stage1Packages = do
, stm
, time
, unlit
, xhtml ]
, xhtml
, ghcTags ]
++ [ haddock | not cross ]
++ [ hpcBin | not cross ]
++ [ iserv | not win, not cross ]
++ [ libiserv | not win, not cross ]
......@@ -119,10 +121,7 @@ stage1Packages = do
-- | Packages built in 'Stage2' by default. You can change this in "UserSettings".
stage2Packages :: Action [Package]
stage2Packages = do
cross <- flag CrossCompiling
return $ [ ghcTags ]
++ [ haddock | not cross ]
stage2Packages = stage1Packages
-- | Packages that are built only for the testsuite.
testsuitePackages :: Action [Package]
......
......@@ -4,7 +4,7 @@
-- accidentally commit them.
module UserSettings (
userFlavours, userPackages, userDefaultFlavour,
verboseCommand, buildProgressColour, successColour, stage1Only
verboseCommand, buildProgressColour, successColour, finalStage
) where
import Flavour
......@@ -50,10 +50,9 @@ buildProgressColour = mkBuildProgressColour (Dull Magenta)
successColour :: SuccessColour
successColour = mkSuccessColour (Dull Green)
-- TODO: Set this flag from the command line.
-- | Set this flag to 'True' to disable building Stage2 GHC and Stage2 utilities
-- such as @haddock@. All Stage0 and Stage1 libraries will still be built.
-- Also see Note [No stage2 packages when CrossCompiling or Stage1Only] in the
-- top-level @ghc.mk@.
stage1Only :: Bool
stage1Only = False
-- | Stop after building the StageN compiler.
-- For example, setting the 'finalStage' to 'Stage1' will just build the
-- 'Stage1' compiler. Setting it to 'Stage3' will build the 'Stage3'
-- compiler. Setting it to 'Stage0' will mean nothing gets built at all.
finalStage :: Stage
finalStage = Stage2
......@@ -30,24 +30,21 @@ askWithResources rs target = H.askWithResources rs target getArgs
-- TODO: Cache the computation.
-- | Given a 'Context' this 'Action' looks up the package dependencies and wraps
-- the results in appropriate contexts. The only subtlety here is that we never
-- depend on packages built in 'Stage2' or later, therefore the stage of the
-- resulting dependencies is bounded from above at 'Stage1'. To compute package
-- dependencies we transitively scan Cabal files using 'pkgDependencies' defined
-- in "Hadrian.Haskell.Cabal".
-- the results in appropriate contexts.
-- To compute package dependencies we transitively scan Cabal files using
-- 'pkgDependencies' defined in "Hadrian.Haskell.Cabal".
contextDependencies :: Context -> Action [Context]
contextDependencies Context {..} = do
depPkgs <- go [package]
return [ Context depStage pkg way | pkg <- depPkgs, pkg /= package ]
return [ Context stage pkg way | pkg <- depPkgs, pkg /= package ]
where
depStage = min stage Stage1
go pkgs = do
deps <- concatMapM step pkgs
let newPkgs = nubOrd $ sort (deps ++ pkgs)
if pkgs == newPkgs then return pkgs else go newPkgs
step pkg = do
deps <- pkgDependencies pkg
active <- sort <$> stagePackages depStage
active <- sort <$> stagePackages stage
return $ intersectOrd (compare . pkgName) active deps
cabalDependencies :: Context -> Action [String]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment