If `withTempFileName` receives an asynchronous exception (e.g.
a canceled async), the bracket cleanup handler will attempt to
remove the temporary file. This can fail with an IO exception.

Regular bracket then throws that IO exception, swallowing the
asynchronous exception. To calling code, this appears no different
from an IO exception thrown from the body, and it won't be able
to tell that it should be exiting promptly.

This manifests concretely during temporary file clean-up of
`asyncFetchPackages` on Windows (seen during unit testing in CI),
where temporary file removal fails (on GHC 8.4), which leads
to `concurrently` failing to cancel the outstanding download
because it's handled like a regular download failure.

This fixes deadlock in asyncFetchPackages:

  withAsync action1 (\_ -> action2)

loses exceptions thrown by action1. If in addition, action2
has a dependency on data produced by action1 (as is the case
here), this will block indefinitely.

(This bug caused some of the new tests to hang, since after
the change to use async-safe try, it became easier for
fetchPackages to throw an exception.)

This is a step towards fixing interrupt handling during
asynchronous download.

The concrete bug fixed by this changes is  that `try` catches
asynchronous exceptions, preventing `withAsync` from interrupting
it when its `body` action finishes prematurely.

This manifests during `cabal build` when hitting Ctrl-C during
download, e.g.:

1. Ctrl-C interrupts a curl process
2. This interrupt is converted into a UserInterrupt exception
   via the process package's delegate_ctlc functionality.
3. UserInterrupt bubbles up through `fetchPackage`, is caught by
   the `try` (fine) and writen to the result mvar.
   Meanwhile, `fetchPackage` continues its loop, starting to
   fetch the next package.
4. Some `rebuildTarget` is waiting for the interrupted download;
  `waitAsyncFetchPackage` rethrows UserInterrupt in that thread.
5. UserInterrupt bubbles up through `collectJob`, `execute`, the
   `body` action.
6. `withAsync`'s finalizer cancels the `fetchPackages` async.
7. The `fetchPackages` async receives the AsyncCancelled exception
   while fetching the next package (see 3. above). This interrupts
   the download action (it shouldn't matter whether we happen to
   be running curl again or not), and AsyncCancelled bubbles up
   through `fetchPackage`, is caught by the `try` (not fine!) and
   written to the mvar. But no-one is reading that mvar anymore
   because the build job already aborted (step 5), and that
   AsyncCancelled exception is lost.
8. `fetchPackages` keeps doing its thing, exiting eventually.

Note that this change affects both instances of `try` in this story:
`UserInterrupt` is also an asynchronous exception, so after the
change the `UserInterrupt` takes a different path from step 3.
Followup commits fix bugs along those paths.

This primarily trigger concurrency bugs that are fixed in the
follow-up commits.

- Sort modules and tests alphabetically (they were maddeningly close to alphabetic)
- Consistently label unit test groups at top-level

* don't crash on a few stray exceptions

* try -> catch and display

* act on reviewer comments

Co-authored-by: Gershom Bazerman <gershom@arista.com>

* Avoid templating GitHub Actions workflow

GitHub Actions workflow have sufficient power to express what we need.
We don't need to maintain and additional templating solution on top.

* Add GHC 9.2, bump bounds, fix syntax

* Switch to official haskell image

* Always run cli tests for ghc 8.2 and above

* Remove step to regenerate GitHub Actions workflows

* Fix missed reference to GHC version

* Fix yaml syntax

* Fix type in the GHC version

* More CI changes

- Run bootstrap.yml on ubuntu-latest
- Use explicit matrix for linux.yml
- Drop containers in favour of haskell setup action
- Drop workaround for ancient git

* Remove unneeded package from CI setup

I belive this is only necessary to run `cabal man` which we do not in
the CI.

* Drop old GHCs from the CI

* Switch macos.yml to haskell/action/setup

Also add the same GHC versions as Linux.

* Simplify CI

- Remove cabal-plan, we actually never call it (I think)
- Remove vendored cabal-doctest
- Remove few stray allow-newer clauses no longer necessary, apparently

Originally done by @gbaz in PR #7907.

* Mark GHC 9.2.1 as experimental

* Remove reference to cabal-plan from validate.sh

* setup-haskell action already runs cabal update

* Add missing build matrix in test-windows-dogfood

* Replace cabal-plan list-bin with cabal list-bin

* Enable caching in the CI

* Fix typo

* Remove continue-on-error until I figure it out

* Keep naming consistent

* Temporarily disable 8.0.2 on macos

* Add missing step id

* Tweaks

Remove workaround for nektos/act, it accidentally sneaked in.

* More tweaks

* Tweaks

* Restore cabal-plan, temporarily mark everything experimental

cabal list-bin doesn't seem to work like cabal-plan does.

* Tweaks

* Ensure cabal-plan executable gets built

* Install automake on MacOS

* Tweaks

* Tweaks

Link experimental flags to relative GitHub issues

* Fix typo

Closes #5508

CI times appear to be dominated by the CI tests, which seems
excessive.

A typical Linux validate.sh job takes ~45 minutes, of which
15 are spent in the VCS tests. MacOS is even worse, with the
VCS tests taking >1h out of a total time of >2h.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* add changelog entries + additional release notes

* remove cruft

(cherry picked from commit 12dc5c26)

Related: #7073

Julian Ospald authored 3 years ago and andreas.abel committed 3 years ago

Also: Fix color formatting when PAGER is 'less'

Directly piping into `man -l -` does not work as BSD-`man` does not
understand option `-l`.  More standardized are the building blocks
`nroff` and `less`.

`cabal man` now should behave as pipeline
```
cabal man --raw | nroff -man /dev/stdin | less
```

Also fixed output of `cabal man --raw` so that it does not produce
warnings.

- `.R` removed.  Was warning:
  ```
  `R' is a string (producing the registered sign), not a macro.
  ```

- No quoted 'new-FOO' should appear at beginning of line.  Was
  warning:
  ```
  warning: macro `new-FOO'' not defined (probably missing space after `ne')
  ```

Added to `cabal-testsuite/PackageTests/Man/cabal.test.hs` a check that
the `stderr` output of `nroff -man /dev/stdin` is empty (no warnings).

Remaining problem:

Unfortunately, after quitting `less` with `q` the following error is
displayed:
```
fd:NNN: commitBuffer: resource vanished (Broken pipe)
```
Not sure how to fix this (my attempts failed).

* Add support for script build caching to cabal run

Enable caching of script builds by changing the location of the fake
package directory from a tmp directory to:
<cabal_dir>/scipt-builds/abs/path/to/script/

Resolves: #6354
WIP: #7842

* Add support for scripts to cabal build.

Added module Distribution.Client.ScriptUtils for code to deal with
scripts that is common between commands.

WIP: #7842

* Add script support to cabal clean.

This changes the behaviour of cabal clean to accept extra args, which it
now interprets as script files. The behaviour of cabal clean is the same
when given extra args. When given extra args it instead removes the
caches for those scripts and also any orphaned caches (caches for which
the script no longer exists)

In addition this commit changes the cache to use hashes of paths because
this significantly simplifies the implementation of clean, and more
importantly it prevents collisions when a script has the name of the
subdirectory of a previously cached script.

WIP: #7842

* Add script support to cabal repl

repl starts in the correct directory and points directly to rather than
a dummy, so that reloading works properly.

There is a downside to the current approach which is that it uses a
different fake-project.cabal file from run and build, so it cannot share
the same cache with them.

WIP: #7842
WIP: #6149

* Added changelog for pr #7851

* Fix `cabal run script.hs` issue with --builddir

Fixes tests:
cabal-testsuite/PackageTests/NewBuild/CmdRun/Script/cabal.test.hs
cabal-testsuite/PackageTests/NewBuild/CmdRun/ScriptLiterate/cabal.test.hs

* Fixes for `build script` and `repl script`

- Fix build issue introduced in 079c5f0e, where build was being passed
the wrong target filter
- Fix repl issue where script didn't work in a project context.
- Refactor code to share logic between repl and build/run
- Ensure temp directories are only created when needed

* Bug fixes relating to script support

ScriptUtils:
- Hash prefix for cache dirs was applied incorrectly.
- Overwriting fake-package files causes repeated work in some cases.
CmdClean:
- Clean distdir for script when --builddir is passed
- Always clean orphans because because there is no good way to specify
they should be cleaned. This may be bad behaviour in some obscure cases
(a cache is temporarily orphaned and an unrelated clean is run), but at
worst results in a cache rebuild.

* Add tests for improved script support

- Basic script support for build/repl/clean which checks for cached
project files
- Add check for cached project files to basic run script test
- No repeated work for build/build, build/run, run/run, and repl/repl
- Clean does not remove cache for existing scripts
- Clean does remove orphaned script caches

* Fix clean bug uncovered by 5fad1214

- clean was trying to read source-builds even if it didn't exist
- add test specific to this case with other clean tests

* Update documentation for better script support

Ready for review: #7851
May close: #7842, #6354, #6149

* Attempt to fix `repl script` on Windows

PR #7851

* Attempt to fix remote test failures

Test logs showed that the failures where because the tests depended on a
module from cabal-install that some ghc versions could not find.

Instead of depending on cabal-install, I copied the needed function into
Test.Cabal.Prelude (It seemed like an acceptable place for it)

PR #7851

* Attempt to fix `repl script` on Windows

PR #7851

* Attempt to fix tests on old ghc versions

Tests failing on pre-AMP ghcs due to unsanctioned use of (<$>)

PR #7851

* Feedback: Update docs and formatting

PR #7851

* Feedback: code style changes

- remove partial selectors
- make a constant for fake-package.cabal

PR #7851

* Feedback: make hidden control flow explicit

PR #7851

* Feedback: add expected fail script run tests

PR #7851

* Fix `repl script` when cwd is deeper than cachedir

PR #7851

* Use script in-place for build or run

- Set the hs-source-dir to the location of the script for build and run,
  the same as with repl
- This removes the need to copy the script
- repl no longer needs a separate cache because all three commands
  use identical project files
- Adds multi-module support to scripts for free (#6787)
- Add new build/repl test and run multi-module test

PR #7851

* Fix file-locking issue on Windows

PR #7851

* Fix script recompilation based on cwd

- Pass info about cwd to repl through --repl-options instead of hacking
  it into the project file.
- Improve paths output by makeRelativeCanonical, makeRelativeToDir, and
  makeRelativeToCwd.
- Script multi-module support works, but with warning in repl.
- Remove script multi-module mention support in docs.

PR #7851

* Make `repl script` respect --repl-no-load

* Feedback: minor refactor

Move argument truncation from targetStrings out of
withScriptContextAndSelectors to runAction

PR #7851

* Feedback: refactor and comments for repl options

PR #7851

* Don't use hs-source-dirs for scripts.

- instead pass absolute path to script in main-is
- resolves issue with relative paths on Windows
- simplifies code and gives prettier build output
- update tests because build output has changed
- removes ability to use multi-module scripts
  (which was never officially endorsed)
- remove test for multi-module scripts
- add checks for unsupported fields in scripts

PR #7851

* Update changelog for PR #7851