Fix recomp bug by invalidating cache on build exception

Be sure to invalidate the cache if building throws an exception!
If not, we'll abort execution with a stale recompilation cache.
See ghc#24926 for an example of how this can go wrong.

Use SetupHooks for Configure build-type

This commit implements the Configure build-type in terms of Hooks,
when build-type: Hooks is available (for Cabal >= 3.13).

This moves Configure away from an implementation in terms of UserHooks,
i.e. away from the Custom build-type.

Issue #10063 points out that cabal exits before the external command
has finished executing. This was a simple oversight to not
waitForProcess on the result of calling createProcess.

This also points out the flaw that there isn't a way for external
commands to signal failure, so we now also propagate the exit code from
the external process.

Fixes #10063

update requests per CVE-2024-35195

GHC 9.10 ships with a new wired-in package, ghc-internal, which cannot
be reinstalled. This commit prevents cabal-install from attempting it.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* add the applicable new (versions 9.2 - 9.10) GHC flags to normaliseGhcArgs

Actionable flags are:
- fdiagnostics-as-json (changes the format GHC outputs its diagnostics)
- fprint-error-index-lists (changes the way GHC displays compile time)
- fbreak-points (enables/disables break-points in GHCi)
- dipe-stats (dumps information about which info tables have IPE
  information)
- ffamily-application-cache (only changes the speed of the compiler)
- fprint-redundant-promotion-ticks
- show-error-context
- unoptimized-core-for-interpreter (only applies to GHCi)

* [chore] correct the flag names and add a FUTUREWORK

Bump dependencies and cabal-version

For `Cabal` and `cabal-install`.

base and QuickCheck.

Fix --program-suffix resulting in invalid symlink

Currently the options `--program-{prefix,suffix}` for cabal install
affects the name of the file in the install directory *and* the
executable name in the store. The installation fails:

- If using `--install-method=symlink`, the *target* of the symlink is
  not affected by the affix options and it results in an invalid symlink.
- If using `--install-method=copy`, the copy fails because the source
  is not found.

Another issue is that it affects the computation of the hash of the
build directory in the store, resulting in needless rebuild when using
successively different affix options.

Fixed by making the name of the executable in the store canonical, i.e.
always ignoring the program affix options.

Added a test for all the combinations of `--install-method` and program
affixes options.

Two fixes to multi repl command

Before this was rendered as..

```
/run/current-system/sw/bin/ghc --interactive -package-env - -j 'NumJobs (Just 8)' ...
```

Due to this place not being updated when support for semaphores was
added.

These options are configured into the program by Cabal, so we also need
to extract these and put them into the mutli-file.

Fixes #10015

Haddock recompilation avoidance

In the last commits we started re-using GHC's interface files and
objects in haddock in order to avoid recompilation.

However, if haddock is run with different flags than GHC (say, for example,
`haddock-options: -DSomethingCustom`), it will recompile the interfaces
and objects.

This commit introduces a guardrail to the process of re-using GHC's
compilation files: instead of running haddock directly on the
directories where GHC placed its output, copy the directory contents to
a temporary directory and point haddock to the objects and interfaces
there. Even if recompilation is triggered by haddock, the objects
produced by GHC will be left untouched.

With "Hi Haddock", we want to re-use GHC's compilation output (in
particular, the generated interface files) to generate documentation
from. However, haddock by default will use a temporary location to build
sources, and will therefore fail to find the interface files produced by
GHC.

To avoid recompilation, we need to instruct Cabal to pass
--no-tmp-comp-dir to Haddock so as to not use a temporary directory and
thus re-use existing interface files.

We no longer define the `__HADDOCK_VERSION__` macro when invoking GHC
through Haddock, since doing so essentially guarantees recompilation
during documentation generation.

We audited all uses of `__HADDOCK_VERSION__` in hackage, ensuring there
was a reasonable path forward to migrate away from using
`__HADDOCK_VERSION__` for each, while generating the same documentation
as it did before.

If you are a user of `__HADDOCK_VERSION__`, please take a look at the
discussion in https://github.com/haskell/cabal/pull/9177 and reach out
to us if your use case is not covered.

Reverts the version-cpp flag introduced in the previous commit to avoid
a workaround. Instead, we get rid of the problem (`__HADDOCK_VERSION__`) at its root.

See the discussion in #9177

Haddock no longer writes compilation files by default, so we do not need to pass
tmp dirs for `-hidir`, `-stubdir`, and `-odir` via `--optghc`. Indeed, we do not
*want* to do so, since it results in recompilation for every invocation of
Haddock via Cabal. This commit stops this from happening for haddock versions >=
2.28 (when Hi Haddock was introduced).

This commit also stops the default definition of the `__HADDOCK_VERSION__` macro
when invoking GHC through haddock. Since a very limited set of users may still
depend on this macro, we introduce the `--haddock-version-cpp` flag and
`haddock-version-cpp:` cabal.project field, which enable the definition of
the `__HADDOCK_VERSION__` macro when invoking GHC through Haddock. This will
almost guarantee recompilation during documentation generation due to the macro
definition.

This commit also renames the `--haddock-lib` flag to `--haddock-resources-dir`
(and `haddock-lib:` cabal.project field to `haddock-resources-dir:`), and adds
this flag to the users guide since it was missing an entry. This also allows us
to add this field to `cabal-install:test:integration-tests2`, since it is no
longer ambiguous with the `--lib` flag.

This commit also causes `documentation: true` or `--enable-documentation` to
imply `-haddock` for GHC.

Also, since Haddock >= 2.29 is renaming `--lib` to `--resources-dir`, this
commit switches the flag provided to Haddock using a backwards compatible
condition based on the Haddock version.

Adds a changelog entry.

fix: Make newSemaphoreJobControl respect verbosity

* CI: add GHC 9.10

* GHC 9.10 compat in testsuite: CPP symbols don't get passed to CC as
eagearly

Discussion: https://github.com/haskell/cabal/pull/9914#issuecomment-2077439625

Related GHC issue: ghc/ghc#21291

* GHC 9.10 compat in tests: disable tests regressing due to #9940

Fix #10042 Don't recommend deprecated/removed 'extensions:' field

Allow whitespace in target selectors

Disallowing whitespace while parsing target selectors incorrectly
disallows file targets with whitespace in the paths, which can issues
when users pass absolute paths.

fixes #8875

Add comment clarifying disabling check for scripts

closes #8963

This will avoid build problems when the GHC in the CI environment
is updated sooner than expected.
Previous breakage: https://github.com/haskell/cabal/pull/9177#issuecomment-2125549693

PackageTests/NewUpdate: fix skipping flaky tests

RejectFutureIndexStates and UpdateIndexState are marked "skip", but it's
under withRemoteRepo, which causes flakiness before skip is called.

Add docs for list-bin support of scripts

closes #7986

* generate doc/requirements.txt

As suggested in https://github.com/haskell/cabal/pull/10030#pullrequestreview-2064505535

* generate doc/requirements.txt

As suggested in https://github.com/haskell/cabal/pull/10030#pullrequestreview-2064505535

update jinja2 per CVE-2024-34064