Some git files are marked as read-only. To ensure we delete the folders we are
supposed to, we first remove the read-only attribute via `CMD.exe`, then we
forcibly delete the relevant directory.

* Improve bat scripts for CCompilerOverride

* Ensure Windows tests can cleanup the temp directory

* Implement `flaky` combinator

* Remove outdated tests

* Remove broken tests

These tests were testing for messages that were removed with the `cabal check` rework.

* Make `skip` and `broken` messages uniform

* Mark flaky tests

* Re-enable DeterministicTrivial

* Fix MacOS canonical paths

* Extend cabal-testsuite readme with `flaky`

* Skip non-terminating tests in Windows CI

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Before the previous patches this test failed because an incorrect path
was passed to hsc2hs (a preprocessor), it now succeeds :)

When testing `./Setup` only, when `withDirectory` is used, instead of
changing into that directory when invoking processes, we now use the
`--working-dir` flag and keep a fixed CWD.

This will therefore passively test that `--working-dir` is working

In addition, it makes it possible to test things easily such as
`--working-dir` with a relative path as an argument. `cabal-install`
will only invoke `--working-dir` with an absolute path and hence is
isolated from any double interpretation issues.

Testing against these double interpretation issues is very important as
it also prevents over-interpretation of relative paths into absolute
paths. Passing absolute paths to tools such as hsc2hs can lead to the
build directory leaking into an interface file which leads to
non-reproducible results.

This refactoring enforces a simple property

* We use symbolic paths in Cabal in order to represent that paths to
  package databases. These paths is relative to the package root.
* We use normal filepaths in cabal-install to represent the path to a
  package database. These are relative to the current working directory.

Paths are explicitly converted from one type to the other at the
interface of `cabal-install` and `Cabal`, see `setupHsConfigureArgs` for
where this happens.

In order to achieve this `PackageDB` is abstracted over what the type of
filepaths a specific package db points to.

```
type PackageDBX fp = ... | SpecificPackageDB fp | ...
```

If you are using the Cabal library then you probably want to migrate to
use `PackageDBCWD` and `PackageDBStackCWD`.

```
type PackageDBCWD = PackageDBX FilePath
type PackageDBStackCWD = [PackageDBCWD]
```

Then at the point where you call commands in the `Cabal` library convert
these paths into paths relative to the root of the relevant package.
The easiest way to do this is convert any paths into an absolute path.

This patch fixes a double interpretation issue when the `--working-dir`
option was used and package db paths were offset incorrectly.

It is only on Cabal 3.13 that the symbolic path abstraction was
introduced.

On CI we only test with Cabal master so the incorrect bound wasn't
picked up.

In the original implementation of promised dependencies I accidentally
left over the hard coded `currentCabalId` in the `configureDependencies`
function.

This led to several errors happening later when the package name and
version would be incorrect if you looked at this field (package
arguments are not computed using it), it is used when generating cabal
macros and something in the haddock options.

The solution is to pass the package version in the
`--promised-depenency` flag so the format is now

```
NAME-VER[:COMPONENT_NAME]=CID`
```

rather than

```
NAME[:COMPONENT_NAME]=CID
```

Fixes #10166

We were passing the flag --coverage-for for every library in the package
when building the testsuites.

However, if a particular testsuite doesn't depend on one of the packages
passed with --coverage-for we would crash.

Since we look for the unit ids of all packages given in --coverage-for in the
package database (to find hpc artifacts), if the testsuite builds before
the library this lookup would fail.

The fix is easy: don't pass --coverage-for=<lib> to <testsuite> if
<testsuite> doesn't depend on <lib>. If <lib> is an indirect dependency
of <testsuite> it'll no longer be covered by HPC, but this is a weird
behaviour that I doubt anyone relies on.

Fixes #10046

The new `extra-files` field provides a way to specify extra files that
should be included in sdist, without adding any other semantics
(cf. `extra-source-files` are tracked by `cabal build`).

Resolves #8817.

In the 2decb0e7 refactor we stopped
looking for non-Haskell `main-is` files in the hs-source-dirs and other
appropriate directories. This commit fixes that oversight.

Even if it is not intuitive that main-is-C-sources are searched in the
hs-source-dirs, we don't wish to break users relying on this behaviour
as there does not exist that strong of a motivation to do so.

Fixes #10168

Co-authored-by: sheaf <sam.derbyshire@gmail.com>

This commit makes haddock-project handle sublibraries.

The commit changes how `cabal haddock` works, changing the layout
in the `dist-newstyle` folder.  With this change `haddock` subcommand
will install `package:sublib` component in a directory `package-sublib`
under `l/sublib/doc/html/`.

Add a flag to disable the hard requirement on the
build-tools-(depends) declared on the Cabal package.

When this flag is enabled (--ignore-build-tools), a build-tool which
can't be found does not block compilation.

Fixes #10061

When no pkg-config program was found, cabal would
claim that the package is not in the db, instead
of telling clearly that no pkg-config was found at
all.

New options for cabal.project and ./Setup interface:

* `profiling-shared`: Enable building profiling dynamic way
* Passing `--enable-profiling` and `--enable-executable-dynamic` builds
  profiled dynamic executables.

Support for using `profiling-shared` is guarded behind a constraint
which ensures you are using `Cabal >= 3.13`.

In the cabal file:

* `ghc-prof-shared-options`, for passing options when building in
  profiling dynamic way

Other miscellenious fixes and improvements

* Some refactoring around ways so that which
  ways we should build for a library, foreign library and executable is
  computed by the `buildWays` function (rather than ad-hoc in three
  different places).

* Improved logic for detecting whether a compiler supports compiling
  a specific way. See functions `profilingVanillaSupported`,
  `dynamicSupported`, `profilingDynamicSupported` etc
  These functions report accurate infomation after ghc-9.10.1.

* Fixed logic for determining whether to build shared libraries. (see
  #10050)
  Now, if you explicitly enable `--*-shared`, then that will always take
  effect. If it's not specified then `--enable-executable-dynamic` will
  turn on shared libraries IF `--enable-profiling` is not enabled.

* Remove assumption that dynamically linked compilers can build dynamic
  libraries (they might be cross compilers.

* Query the build compiler to determine which library way is necessary
  to be built for TH support to work.
  (rather than assume all compilers are dynamically linked)

* An extensive test which checks how options for `./Setup` and
  `cabal-install` are translated into build ways.

Fixes #4816, #10049, #10050

We should configure unconfigured programs before serialising them
(using the Binary ProgramDb instance) in
Distribution.Client.ProjectPlanning.configureCompiler, as otherwise
we can accidentally discard information.

This isn't currently a live bug, because in practice we end up
re-running configuration steps when interfacing with the Cabal library.
However, in the bright future in which we avoid re-configuring things
when building a Cabal package that we already determined when invoking
cabal-install, this starts causing problems.

Be sure to invalidate the cache if building throws an exception!
If not, we'll abort execution with a stale recompilation cache.
See ghc#24926 for an example of how this can go wrong.

Issue #10063 points out that cabal exits before the external command
has finished executing. This was a simple oversight to not
waitForProcess on the result of calling createProcess.

This also points out the flaw that there isn't a way for external
commands to signal failure, so we now also propagate the exit code from
the external process.

Fixes #10063

Currently the options `--program-{prefix,suffix}` for cabal install
affects the name of the file in the install directory *and* the
executable name in the store. The installation fails:

- If using `--install-method=symlink`, the *target* of the symlink is
  not affected by the affix options and it results in an invalid symlink.
- If using `--install-method=copy`, the copy fails because the source
  is not found.

Another issue is that it affects the computation of the hash of the
build directory in the store, resulting in needless rebuild when using
successively different affix options.

Fixed by making the name of the executable in the store canonical, i.e.
always ignoring the program affix options.

Added a test for all the combinations of `--install-method` and program
affixes options.

These options are configured into the program by Cabal, so we also need
to extract these and put them into the mutli-file.

Fixes #10015

Haddock no longer writes compilation files by default, so we do not need to pass
tmp dirs for `-hidir`, `-stubdir`, and `-odir` via `--optghc`. Indeed, we do not
*want* to do so, since it results in recompilation for every invocation of
Haddock via Cabal. This commit stops this from happening for haddock versions >=
2.28 (when Hi Haddock was introduced).

This commit also stops the default definition of the `__HADDOCK_VERSION__` macro
when invoking GHC through haddock. Since a very limited set of users may still
depend on this macro, we introduce the `--haddock-version-cpp` flag and
`haddock-version-cpp:` cabal.project field, which enable the definition of
the `__HADDOCK_VERSION__` macro when invoking GHC through Haddock. This will
almost guarantee recompilation during documentation generation due to the macro
definition.

This commit also renames the `--haddock-lib` flag to `--haddock-resources-dir`
(and `haddock-lib:` cabal.project field to `haddock-resources-dir:`), and adds
this flag to the users guide since it was missing an entry. This also allows us
to add this field to `cabal-install:test:integration-tests2`, since it is no
longer ambiguous with the `--lib` flag.

This commit also causes `documentation: true` or `--enable-documentation` to
imply `-haddock` for GHC.

Also, since Haddock >= 2.29 is renaming `--lib` to `--resources-dir`, this
commit switches the flag provided to Haddock using a backwards compatible
condition based on the Haddock version.

Adds a changelog entry.

* CI: add GHC 9.10

* GHC 9.10 compat in testsuite: CPP symbols don't get passed to CC as
eagearly

Discussion: https://github.com/haskell/cabal/pull/9914#issuecomment-2077439625

Related GHC issue: ghc/ghc#21291

* GHC 9.10 compat in tests: disable tests regressing due to #9940

Disallowing whitespace while parsing target selectors incorrectly
disallows file targets with whitespace in the paths, which can issues
when users pass absolute paths.

fixes #8875

RejectFutureIndexStates and UpdateIndexState are marked "skip", but it's
under withRemoteRepo, which causes flakiness before skip is called.

This commit makes Location a separate data type:

  data Location where
    Location
      :: SymbolicPath Pkg (Dir baseDir)
      -> RelativePath baseDir File
      -> Location

instead of being a type synonym for (FilePath, FilePath).

We noted during testing of the Hooks API that it was all too easy to
give an incorrect location for rule outputs, e.g. by omitting an
extension or using an absolute path.
This change allows us to improve the API documentation, as well as
clarifying the types to avoid any ambiguities about what kind of file
path is expected (FilePath vs SymbolicPath).

- Use real-life example from Cabal project in doctest

This allows us to compile Setup.hs without depending on base.
In particular, this ensures that a package with `build-type: Hooks`
and a custom setup stanza that does not depend on base successfully
compiles.

Tested in PackageTests/SetupHooks/SetupHooksNoBase.

This commit initialises the Haddock flag haddockKeepTempFiles with the
value of buildSettingKeepTempFiles.

This addresses an 8 year old TODO in the code, from Duncan Coutts.

This makes Setup copy/install succeed if there's
nothing to do because the package doesn't contain
a library or executable.

This allows downstream users of Cabal to avoid having to
add workarounds for this edge case.

Resolves #6750

Previously `--package-db` was only passed to test which used the `v2-`
prefix.

Now we pass `--package-db` to things which use the `v2-` prefix, not
things which use the `v1-` prefix and by default assume that unprefixed
commands are v2 commands.