Skip to content

GitLab

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • GHC GHC
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 4,943
    • Issues 4,943
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 455
    • Merge requests 455
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Releases
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Glasgow Haskell Compiler
  • GHCGHC
  • Issues
  • #16400
Closed
Open
Created Mar 06, 2019 by davide@trac-davide2 of 12 tasks completed2/12 tasks

Hadrian: support shake's --lint-fsatrace feature and fix lint errors

Shake's --lint-fsatrace feature uses fsatrace to track file accesses of external commands. This allows for more accurate dependency linting which is important e.g. for caching.

Hadrian already accepts the --lint-fsatrace command line option (it is passed on to shake automatically), but it currently has no effect. The shake options are not set correctly (In particular shakeLintInside must be set).

Required Changes:

  1. shakeLintInside must be set
  2. Out of tree build dirs don't get linted.
buildRoot -/- "lint-test" %> \out -> 
    let gen t f = cmd Shell "echo" t ">" (toNative f) :: Action ()
    gen "x" $ out <.> "txt"
    need [out <.> "txt"]     -- This need should set off a lint error.
    writeFile' out ""
  • fsatrace gives absolute paths. Shake then tries and fails to convert to relative paths here. Stuck with an unexpected absolute path, shake thinks the file isn't needed even thought the relative path **is** needed.

How To Collect Stats

To collect fsatrace lint errors as reported below, you need to modify Shake to warn instead of error on fsatrace lint errors. Have a look at this commit of shake (from this branch) for how to do that. Then you'll want to do a clean build with linting enabled and save the output to a file e.g.:

$ ./boot && ./configure
$ (./hadrian/build.sh -j --lint-fsatrace 2>&1) > build.out

Then you can use breakdown.hs to extract and compare lint error counts between builds:

$ runhaskell ~/breakdown.hs build_01.out build_02.out

You can also pass the same file twice to just get the counts of one run.

Status

This uses a stage0 compiler with support for -include-cpp-deps (see comment below)

Remaining lint errors by file extension

Extension Missing Need - Error Count
.h 84956
.hs-boot 108
.so 15
.hs 3
.gz 1
.hs-incl 1
Extension Late Need - Error Count
.h 4012
.so 8
.hs-incl 1

I've tried to break these errors down into some specific cases as follows. Each point roughly corresponds to a single issue (though this is just an educated guess). Have a look at this build output and grep for "@@@" to find relevant errors.

  • (~ 83.5k errors) #16868 /include//*.h dependencies used in later stage of ghc copiler (see comment).
  • (~ 4k errors) _build/generated/*.h used/late need for many .o file rules.
  • (108 errors) #16815 .hs-boot used but not depended upon in rule for dependencies.mk. This is from dependency generation using ghc -M.
    • other .hs/.h files used but not depended upon in rule for dependencies.mk.
      • (2 errors) includes/CodeGen.Platform.hs used but not depended upon in rule for dependencies.mk
      • (1 errors) libraries/filepath/System/FilePath/Internal.hs used but not depended upon in rule for _build/stage1/libraries/filepath/.dependencies.mk
  • (~ 72 errors) #16983 (closed) in rule for _build/generated/platformConstants. I think this file is a bit of a special case.
  • (12 errors) libffi.so needed after being used when linking various ways of rts .so.
  • (9 errors) libffi.so used but not depended upon in rule for linking some binaries.
  • (2 errors) _build/stage0/compiler/build/primop-commutable.hs-incl late need in rule for _build/stage0/compiler/build/PrimOp.o
  • (1 error) libffi-tarballs/libffi-3.99999+git20171002+77e130c.tar.gz used but not depended upon in rule for ../stage1/libffi/build/Makefile.in ../stage1/libffi/build/configure

Things already finished:

  • (~ 1.8k errors) CPP dependencies !845 (merged)
Edited May 18, 2021 by Sylvain Henry
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking