GHC issueshttps://gitlab.haskell.org/ghc/ghc/-/issues2023-09-28T08:15:41Zhttps://gitlab.haskell.org/ghc/ghc/-/issues/23477ghc 9.6.2 fails to build on macOS because of missing docs/index.html2023-09-28T08:15:41ZSteve Smithghc 9.6.2 fails to build on macOS because of missing docs/index.html## Summary
ghc 9.6.2 fails to build on macOS because of a missing file `docs/index.html`. The file `./docs/index.html.in` exists, but apparently is not converted to `docs/index.html`. The same exact build procedure works for version `9.6...## Summary
ghc 9.6.2 fails to build on macOS because of a missing file `docs/index.html`. The file `./docs/index.html.in` exists, but apparently is not converted to `docs/index.html`. The same exact build procedure works for version `9.6.1`.
## Steps to reproduce
```sh
hadrian binary-dist-dir -j24
```
Observe successful bootstrap that fails with creation of the docs:
```sh
:info:build | Run Haddock BuildPackage: libraries/Cabal/Cabal/src/Distribution/Backpack/ComponentsGraph.hs (and 114 more) => _build/doc/html/libraries/Cabal-3.10.1.0/Cabal.haddock
:info:build | Run Haddock BuildIndex: _build/doc/html/libraries/binary-0.8.9.1/binary.haddock (and 35 more) => _build/doc/html/libraries/index.html
:info:build | Copy file (untracked): docs/index.html => _build/doc/html/index.html
:info:build docs/index.html: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openFdAt: does not exist (No such file or directory)
:info:build Build failed.
```
## Expected behavior
Bootrsap the compiler.
## Environment
* GHC version used: 9.6.2
Optional:
* Operating System: macOS
* System Architecture: `x86_64`⊥https://gitlab.haskell.org/ghc/ghc/-/issues/16956Hadrian build artifacts are sensitive to build location2021-09-21T16:58:39ZDavid EichmannHadrian build artifacts are sensitive to build locationBuild artifacts contain absolute paths to other build artifacts. This is makes the artifacts non-relocatable. In particular, cloud builds can largely cache misses, fail, and produce stale build artifact. I noticed this when attempting a ...Build artifacts contain absolute paths to other build artifacts. This is makes the artifacts non-relocatable. In particular, cloud builds can largely cache misses, fail, and produce stale build artifact. I noticed this when attempting a cloud build (after !1436).
## Recreation
You can grep the build directory for the absolute build dir path:
```bash
./boot && ./configure && ./hadrian -j
grep -rn _build -e "$(pwd)"
```
Here is a breakdown of what I see:
* `Makefile`s
* I think we insert some paths based on `Makefile.in` files
* `.dependencies`
* Paths to `ghcversion.h` are absolute.
* `_build/stage1/libraries/process/build/c/cbits/runProcess*`
* `_build/stage1/{utils, libraries}/*/build/{config.status, config.log}`
* `_build/stage1/{utils, libraries}/*/setup-config`
* `_build/stage1/libffi/build/.../`
* `libffi.la`
* `Makefile`
* `Makefile.in`
* `ibffi.pc`
* libffi.lai
* `_build/stage1/rts/build/c/*o`
* `_build/stage1/lib/x86_64-linux-ghc-8.9.0.20190717/libHSrts-1.0_*-ghc8.9.0.20190717.so`
* `_build/stage1/lib/x86_64-linux-ghc-8.9.0.20190717/rts-1.0/libHSrts-1.0_*.a`
You can also try a cloud build from a different directory:
```bash
# Initial build to fill cache.
./boot && ./configure && ./hadrian -j --share=../_cache
# Create a new worktree.
git worktree add ../ghc2
cd ../ghc2
git submodule update --init
# Do the same build again but from a different worktree directory.
# Assuming you have the changes from !1436 this will mostly cache miss (if you
# fix this issue it should should only take 2-3mins).
./boot && ./configure && ./hadrian -j --share=../_cache
```
<details>
<summary>This is part of a workflow to develop on different work trees and take advantage of cloud builds.</summary>
you start from the
same commit and start work on 2 separate issues. In this case, consider freezing
stage one to some base commit:
```bash
# Checkout base commit.
cd ~/ghc
git checkout 24fd2e189b
# Build 1 (25m08s): base commit in worktree ~/ghc.
# This caches build artifacts for the base commit.
./boot && ./configure && ./hadrian/build.sh -j --share=../_cache
# Make some changes (I do a revert for reproducibility).
git revert d7c6c4717c
# Build 2 (1m41s) modified worktree ~/ghc.
# Freeze stage 1 to avoid rebuilding all of stage 2.
./hadrian/build.sh -j --share=../_cache --freeze1
# Create a new work tree from the base commit.
git worktree add ~/ghc2 24fd2e189b
cd ~/ghc2
git submodule update --init
# Build 3 (xxxxx): base commit in worktree ~/ghc2.
# This is will allow us to freeze stage1 to the base commit in the next build.
# This makes good use of the cached build in build 1, so should be fast.
./boot && ./configure && ./hadrian/build.sh -j --share=../_cache
# Make some changes (I do a revert for reproducibility).
git revert 348cc8ebf1
# Build 4 (xxxxxx) modified worktree ~/ghc.
# Freeze stage 1 to avoid rebuilding all of stage 2.
./hadrian/build.sh -j --share=../_cache --freeze1
```
</details>
## Notes
In order to enable cache hits insensitive to build location, we should generally replace absolute paths with relative paths. The source of the absolute path to the ghc source directory is the configure script. It generates `hadrian/cfg/system.config` which contains `ghc-source-path = ...`. This value is used in Hadrian as `topDirectory`.
One of the issues is config files generated by cabal (e.g. `setup-config`). Investigation is needed, but this may not be an issue if these config files are only ever viewed through an oracle (is this already true?) and if the oracle's return value excludes the absolute path.⊥https://gitlab.haskell.org/ghc/ghc/-/issues/16926Hadrian: Implement cloud build artifact caching2020-01-05T18:32:18ZDavid EichmannHadrian: Implement cloud build artifact cachingShake supports cloud cache builds with it's `--share` option which caches build artifacts. Hadrian already exposes this feature, but it is unclear if it will produce correct results. Much work has already been done, but the expected bene...Shake supports cloud cache builds with it's `--share` option which caches build artifacts. Hadrian already exposes this feature, but it is unclear if it will produce correct results. Much work has already been done, but the expected benefit decreased as the cost continued to rise. Hence **this feature has been put on hold**.
If you are thinking about picking up this issue:
1. Read [developing hadrian](https://gitlab.haskell.org/ghc/ghc/wikis/Developing-Hadrian). It outlines what is needed to achieve correctness, and the lessons learned the first time around.
2. See #16400 Most remaining work is centred around fixing fsatrace lint errors. That issue roughly summarizes the remaining issues and explains how to collect such results yourself.
3. Fix #16956 to support caching independent of build and source directory location (i.e. relocatable build artifacts).
Feel free to contact me, @DavidEichmann, with any questions.⊥https://gitlab.haskell.org/ghc/ghc/-/issues/16800Hadrian: Symlink Breaks Cached Builds2019-07-09T09:40:40ZDavid EichmannHadrian: Symlink Breaks Cached Builds# Recreation
```bash
$ ./hadrian/build.sh --share=_cache -j --flavour=quickest
...
Error when running Shake build system:
at action, called at src/Rules.hs:68:19 in main:Rules
at need, called at src/Rules.hs:90:5 in main:Rules
* Dep...# Recreation
```bash
$ ./hadrian/build.sh --share=_cache -j --flavour=quickest
...
Error when running Shake build system:
at action, called at src/Rules.hs:68:19 in main:Rules
at need, called at src/Rules.hs:90:5 in main:Rules
* Depends on: _build/stage1/lib/package.conf.d/rts-1.0.conf
at need, called at src/Rules/Register.hs:101:5 in main:Rules.Register
* Depends on: _build/stage1/rts/build/libCffi_thr.a
* Raised the exception:
_cache/.shake.cache/5029bef81b20dfbd/0xF03C561F: getPermissions:getFileStatus: does not exist (No such file or directory)
$ ls -l _cache/.shake.cache/5029bef81b20dfbd/0xF03C561F
lrwxrwxrwx 2 david david 9 Jun 11 16:54 _cache/.shake.cache/5029bef81b20dfbd/0xF03C561F -> libCffi.a
```
`_cache/.shake.cache/5029bef81b20dfbd/0xF03C561F` is a broken link, but is "correct" in the sense that we want to copy the symlink as is so that `_build/stage1/rts/build/libCffi_thr.a` is a symlink to `_build/stage1/rts/build/libCffi.a`. This seems like a bug in Shake: it manages to correctly copy the symlink into the cache, but fails on the way back out.⊥David EichmannDavid Eichmann