Hadrian: support shake's --lint-fsatrace feature and fix lint errors
--lint-fsatrace feature uses fsatrace to track file accesses of external commands. This allows for more accurate dependency linting which is important e.g. for caching.
Hadrian already accepts the
--lint-fsatrace command line option (it is passed on to shake automatically), but it currently has no effect. The shake options are not set correctly (In particular shakeLintInside must be set).
- shakeLintInside must be set
- Out of tree build dirs don't get linted.
buildRoot -/- "lint-test" %> \out -> let gen t f = cmd Shell "echo" t ">" (toNative f) :: Action () gen "x" $ out <.> "txt" need [out <.> "txt"] -- This need should set off a lint error. writeFile' out ""
- fsatrace gives absolute paths. Shake then tries and fails to convert to relative paths here. Stuck with an unexpected absolute path, shake thinks the file isn't needed even thought the relative path **is** needed.
How To Collect Stats
To collect fsatrace lint errors as reported below, you need to modify Shake to warn instead of error on fsatrace lint errors. Have a look at this commit of shake (from this branch) for how to do that. Then you'll want to do a clean build with linting enabled and save the output to a file e.g.:
$ ./boot && ./configure $ (./hadrian/build.sh -j --lint-fsatrace 2>&1) > build.out
Then you can use breakdown.hs to extract and compare lint error counts between builds:
$ runhaskell ~/breakdown.hs build_01.out build_02.out
You can also pass the same file twice to just get the counts of one run.
This uses a stage0 compiler with support for
-include-cpp-deps (see comment below)
Remaining lint errors by file extension
|Extension||Missing Need - Error Count|
|Extension||Late Need - Error Count|
I've tried to break these errors down into some specific cases as follows. Each point roughly corresponds to a single issue (though this is just an educated guess). Have a look at this build output and grep for "@@@" to find relevant errors.
(~ 83.5k errors) #16868
/include//*.hdependencies used in later stage of ghc copiler (see comment).
(~ 4k errors)
_build/generated/*.hused/late need for many
(108 errors) #16815
.hs-bootused but not depended upon in rule for
dependencies.mk. This is from dependency generation using
.hfiles used but not depended upon in rule for
includes/CodeGen.Platform.hsused but not depended upon in rule for
libraries/filepath/System/FilePath/Internal.hsused but not depended upon in rule for
- (2 errors)
(~ 72 errors) #16983 in rule for
_build/generated/platformConstants. I think this file is a bit of a special case.
libffi.soneeded after being used when linking various ways of rts .so.
libffi.soused but not depended upon in rule for linking some binaries.
_build/stage0/compiler/build/primop-commutable.hs-incllate need in rule for
libffi-tarballs/libffi-3.99999+git20171002+77e130c.tar.gzused but not depended upon in rule for
Things already finished: