Hadrian: support shake's --lint-fsatrace feature and fix lint errors

Shake's --lint-fsatrace feature uses fsatrace to track file accesses of external commands. This allows for more accurate dependency linting which is important e.g. for caching.

Hadrian already accepts the --lint-fsatrace command line option (it is passed on to shake automatically), but it currently has no effect. The shake options are not set correctly (In particular shakeLintInside must be set).

Required Changes:

1. shakeLintInside must be set
2. Out of tree build dirs don't get linted.

buildRoot -/- "lint-test" %> \out -> 
    let gen t f = cmd Shell "echo" t ">" (toNative f) :: Action ()
    gen "x" $ out <.> "txt"
    need [out <.> "txt"]     -- This need should set off a lint error.
    writeFile' out ""

• fsatrace gives absolute paths. Shake then tries and fails to convert to relative paths here. Stuck with an unexpected absolute path, shake thinks the file isn't needed even thought the relative path **is** needed.

How To Collect Stats

To collect fsatrace lint errors as reported below, you need to modify Shake to warn instead of error on fsatrace lint errors. Have a look at this commit of shake (from this branch) for how to do that. Then you'll want to do a clean build with linting enabled and save the output to a file e.g.:

$ ./boot && ./configure
$ (./hadrian/build.sh -j --lint-fsatrace 2>&1) > build.out

Then you can use breakdown.hs to extract and compare lint error counts between builds:

$ runhaskell ~/breakdown.hs build_01.out build_02.out

You can also pass the same file twice to just get the counts of one run.

Status

This uses a stage0 compiler with support for -include-cpp-deps (see comment below)

Remaining lint errors by file extension

Extension	Missing Need - Error Count
.h	84956
.hs-boot	108
.so	15
.hs	3
.gz	1
.hs-incl	1

Extension	Late Need - Error Count
.h	4012
.so	8
.hs-incl	1

I've tried to break these errors down into some specific cases as follows. Each point roughly corresponds to a single issue (though this is just an educated guess). Have a look at this build output and grep for "@@@" to find relevant errors.

• (~ 83.5k errors) #16868 /include//*.h dependencies used in later stage of ghc copiler (see comment).
• (~ 4k errors) _build/generated/*.h used/late need for many .o file rules.
• (108 errors) #16815 .hs-boot used but not depended upon in rule for dependencies.mk. This is from dependency generation using ghc -M.
  • other .hs/.h files used but not depended upon in rule for dependencies.mk.
    • (2 errors) includes/CodeGen.Platform.hs used but not depended upon in rule for dependencies.mk
    • (1 errors) libraries/filepath/System/FilePath/Internal.hs used but not depended upon in rule for _build/stage1/libraries/filepath/.dependencies.mk
• (~ 72 errors) #16983 (closed) in rule for _build/generated/platformConstants. I think this file is a bit of a special case.
• (12 errors) libffi.so needed after being used when linking various ways of rts .so.
• (9 errors) libffi.so used but not depended upon in rule for linking some binaries.
• (2 errors) _build/stage0/compiler/build/primop-commutable.hs-incl late need in rule for _build/stage0/compiler/build/PrimOp.o
• (1 error) libffi-tarballs/libffi-3.99999+git20171002+77e130c.tar.gz used but not depended upon in rule for ../stage1/libffi/build/Makefile.in ../stage1/libffi/build/configure

Things already finished:

• (~ 1.8k errors) CPP dependencies !845 (merged)