perf: Group together packages by repo when verifying tarballs

verifyFetchedTarball has the effect of deserialising the index tarball
(see call to Sec.withIndex).

verifyFetchedTarball is called individually for each package in the
build plan (see ProjectPlanning.hs). Not once per repo.

The hackage tarball is now 880mb so it takes a non significant amount of
time to deserialise this (much better after haskell/tar#95).

This code path is important as it can add 1s with these 38 calls on to
the initial load of a project and scales linearly with the size of your
build tree.

Reproducer: Simple project with "lens" dependency deserialises the index tarball 38 times.

Solution: Refactor verifyFetchedTarball to run once per repo rather than once per package.

In future it would be much better to refactor this function so that the
items are not immediately grouped and ungrouped but I didn't want to
take that on immediately.

Fixes #10110

* Add MHS as a recognized compiler.

* Add Changelog entry

* Add comment.

* Update checksums.

* Update more checksums.

---------

Co-authored-by: Lennart Augustsson <lennart.augustsson@epicgames.com>

Be more careful in handling unconfigured programs in the program database

There was some hacky logic in place to give a "haskell-suite" compiler
a dummy location so that we would attempt to configure it.

This logic is no longer necessary since 'configureRequiredProgram' was
changed to accomodate the situation in which 'lookupKnownProgram' fails.
In fact, it is downright harmful, as this dummy location will trigger
errors when we attempt to configure all known programs, which is
a necessary step before attempting to serialise a program database.

This commit configures the unconfigured programs in the program database
when we are building a custom Setup script. This ensures that we can
find the "strip" program which might otherwise still be unconfigured.

In configureRequiredProgram, we would unconditionally configure the
program, even if it was configured already. To explain why this is
problematic:

  - The programs we try to configure in this function are those that
    arise from a "build-tool-depends" field of a package.
  - If the program is not in the "unconfigured" state, we would
    unconditionally treat it as a simpleProgram.
    This means that if such a program is builtin but is not a
    simpleProgram, we might fail to configure it properly.

This problem arises when we configure programs more eagerly: when, in
the past, we might have gone looking up an unconfigured program and
successfully configured it, now we might find the program is already
configured.

One example in which this would cause an issue is when we have

   build-tool-depends: hsc2hs

If we end up re-configuring hsc2hs in configureRequiredProgram, we would
treat it as a simple program, which would cause us to be unable to
determine its version.

We should configure unconfigured programs before serialising them
(using the Binary ProgramDb instance) in
Distribution.Client.ProjectPlanning.configureCompiler, as otherwise
we can accidentally discard information.

This isn't currently a live bug, because in practice we end up
re-running configuration steps when interfacing with the Cabal library.
However, in the bright future in which we avoid re-configuring things
when building a Cabal package that we already determined when invoking
cabal-install, this starts causing problems.

Change packages default to empty

testsuite: Pass pkgdb of store used for intree Cabal

Fix recomp bug by invalidating cache on build exception

Be sure to invalidate the cache if building throws an exception!
If not, we'll abort execution with a stale recompilation cache.
See ghc#24926 for an example of how this can go wrong.

Use SetupHooks for Configure build-type

This commit implements the Configure build-type in terms of Hooks,
when build-type: Hooks is available (for Cabal >= 3.13).

This moves Configure away from an implementation in terms of UserHooks,
i.e. away from the Custom build-type.

Issue #10063 points out that cabal exits before the external command
has finished executing. This was a simple oversight to not
waitForProcess on the result of calling createProcess.

This also points out the flaw that there isn't a way for external
commands to signal failure, so we now also propagate the exit code from
the external process.

Fixes #10063

update requests per CVE-2024-35195

GHC 9.10 ships with a new wired-in package, ghc-internal, which cannot
be reinstalled. This commit prevents cabal-install from attempting it.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* add the applicable new (versions 9.2 - 9.10) GHC flags to normaliseGhcArgs

Actionable flags are:
- fdiagnostics-as-json (changes the format GHC outputs its diagnostics)
- fprint-error-index-lists (changes the way GHC displays compile time)
- fbreak-points (enables/disables break-points in GHCi)
- dipe-stats (dumps information about which info tables have IPE
  information)
- ffamily-application-cache (only changes the speed of the compiler)
- fprint-redundant-promotion-ticks
- show-error-context
- unoptimized-core-for-interpreter (only applies to GHCi)

* [chore] correct the flag names and add a FUTUREWORK

Bump dependencies and cabal-version

For `Cabal` and `cabal-install`.

base and QuickCheck.

Fix --program-suffix resulting in invalid symlink

Currently the options `--program-{prefix,suffix}` for cabal install
affects the name of the file in the install directory *and* the
executable name in the store. The installation fails:

- If using `--install-method=symlink`, the *target* of the symlink is
  not affected by the affix options and it results in an invalid symlink.
- If using `--install-method=copy`, the copy fails because the source
  is not found.

Another issue is that it affects the computation of the hash of the
build directory in the store, resulting in needless rebuild when using
successively different affix options.

Fixed by making the name of the executable in the store canonical, i.e.
always ignoring the program affix options.

Added a test for all the combinations of `--install-method` and program
affixes options.

Two fixes to multi repl command

Before this was rendered as..

```
/run/current-system/sw/bin/ghc --interactive -package-env - -j 'NumJobs (Just 8)' ...
```

Due to this place not being updated when support for semaphores was
added.

These options are configured into the program by Cabal, so we also need
to extract these and put them into the mutli-file.

Fixes #10015

Haddock recompilation avoidance

In the last commits we started re-using GHC's interface files and
objects in haddock in order to avoid recompilation.

However, if haddock is run with different flags than GHC (say, for example,
`haddock-options: -DSomethingCustom`), it will recompile the interfaces
and objects.

This commit introduces a guardrail to the process of re-using GHC's
compilation files: instead of running haddock directly on the
directories where GHC placed its output, copy the directory contents to
a temporary directory and point haddock to the objects and interfaces
there. Even if recompilation is triggered by haddock, the objects
produced by GHC will be left untouched.

With "Hi Haddock", we want to re-use GHC's compilation output (in
particular, the generated interface files) to generate documentation
from. However, haddock by default will use a temporary location to build
sources, and will therefore fail to find the interface files produced by
GHC.

To avoid recompilation, we need to instruct Cabal to pass
--no-tmp-comp-dir to Haddock so as to not use a temporary directory and
thus re-use existing interface files.

We no longer define the `__HADDOCK_VERSION__` macro when invoking GHC
through Haddock, since doing so essentially guarantees recompilation
during documentation generation.

We audited all uses of `__HADDOCK_VERSION__` in hackage, ensuring there
was a reasonable path forward to migrate away from using
`__HADDOCK_VERSION__` for each, while generating the same documentation
as it did before.

If you are a user of `__HADDOCK_VERSION__`, please take a look at the
discussion in https://github.com/haskell/cabal/pull/9177 and reach out
to us if your use case is not covered.

Reverts the version-cpp flag introduced in the previous commit to avoid
a workaround. Instead, we get rid of the problem (`__HADDOCK_VERSION__`) at its root.

See the discussion in #9177

Haddock no longer writes compilation files by default, so we do not need to pass
tmp dirs for `-hidir`, `-stubdir`, and `-odir` via `--optghc`. Indeed, we do not
*want* to do so, since it results in recompilation for every invocation of
Haddock via Cabal. This commit stops this from happening for haddock versions >=
2.28 (when Hi Haddock was introduced).

This commit also stops the default definition of the `__HADDOCK_VERSION__` macro
when invoking GHC through haddock. Since a very limited set of users may still
depend on this macro, we introduce the `--haddock-version-cpp` flag and
`haddock-version-cpp:` cabal.project field, which enable the definition of
the `__HADDOCK_VERSION__` macro when invoking GHC through Haddock. This will
almost guarantee recompilation during documentation generation due to the macro
definition.

This commit also renames the `--haddock-lib` flag to `--haddock-resources-dir`
(and `haddock-lib:` cabal.project field to `haddock-resources-dir:`), and adds
this flag to the users guide since it was missing an entry. This also allows us
to add this field to `cabal-install:test:integration-tests2`, since it is no
longer ambiguous with the `--lib` flag.

This commit also causes `documentation: true` or `--enable-documentation` to
imply `-haddock` for GHC.

Also, since Haddock >= 2.29 is renaming `--lib` to `--resources-dir`, this
commit switches the flag provided to Haddock using a backwards compatible
condition based on the Haddock version.

Adds a changelog entry.

fix: Make newSemaphoreJobControl respect verbosity

* CI: add GHC 9.10

* GHC 9.10 compat in testsuite: CPP symbols don't get passed to CC as
eagearly

Discussion: https://github.com/haskell/cabal/pull/9914#issuecomment-2077439625

Related GHC issue: ghc/ghc#21291

* GHC 9.10 compat in tests: disable tests regressing due to #9940

Fix #10042 Don't recommend deprecated/removed 'extensions:' field