Add solver Hackage benchmarks to GitHub Actions (fixes #9495)

Ignore `IntegrationTests2/config/default-config`

* Ignore invalid Unicode in pkg-config descriptions

Previously, if any of the pkg-config packages on the system had invalid
Unicode in their description fields (like the Intel vpl package has at
the time of writing, 2024-01-11, see #9608), cabal would crash because
it tried to interpret the entire `pkg-config --list-all` output as
Unicode.

This change, as suggested by gbaz in
  https://github.com/haskell/cabal/issues/9608#issuecomment-1886120831


switches to using a lazy ByteString for reading in the output, splitting
on the first space in byte land, and then parsing only the package
_name_ to a String.

For further future-proofing, package names that don't parse as valid
Unicode don't crash Cabal, but are instead ignored.

* Add changelog entry

* cabal-install-solver: Add bounds on 'text'

* No literal ASCII values, use 'ord'

* Address review comments re invalid unicode from pkg-config

* Add test for invalid unicode from pkg-config

* Compatibility with text-1.2.5.0

* Align imports

* Handle different exception type

* Use only POSIX shell syntax

* Add invalid-input handler in pkg-config shim

This is to appease shellcheck

* Actually implement all required stuff in the pkg-config shim

* Less exception dance

* Fix shebang lines

MacOS doesn't have /usr/bin/sh, and /bin/sh is the standard (for a POSIX
shell) anyway

* Don't expect a particular representation of invalid characters

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Relax `containers` upper bound in `Cabal-syntax`

Add test cases that reproduce sdist --project-file.

Solver Hackage benchmark: Explicitly initialize cabal config files

hackage-benchmark uses the environment variable CABAL_CONFIG to specify the
location of the config file. cabal started requiring the config file to exist in
that case in 6942e7e8, so this commit runs
"cabal user-config init".

Add cabal-testsuite git ignores for validate dangling files

Add extraLibDirs to runtime lib search paths of library

Runtime search paths are hard. Here's the current picture to understand
why this patch exists:

* When linking a shared library, GHC will include in the rpath entries
  of the shared library all the paths listed in the library dirs section
  of the installed package info of all packages the shared library
  depends on.
    * On darwin, GHC has special logic to inject the library dirs listed
      in the installed dependent packages info into the rpath section
      instead of passing the dirs as -rpath flags to the linker.
      However, only the dirs where used libraries are found are
      actually injected. The others are ignored. This works around
      limitations of the darwin loader.

* Cabal, in addition, passes directly to the linker (via
  -optl-Wl,-rpath,...) the library dirs of packages the
  shared library for the package being built depends on.
    * In a vanilla cabal installation, this will typically only be the
      path to the cabal store and the path to the installed GHC's boot
      libraries store.
    * When using nix there will a different library dir per installed
      package. Since these lib dirs are passed directly to the linker as
      rpaths, we bypass the darwin loader logic and, for very big
      packages, on darwin, we could end up reaching the load command
      limit and fail linking. We don't address this situation in this MR.

When we specify `extra-lib-dirs` in Cabal, these extra-lib-dirs will be
added to the library dirs listed in the installed package info of the
library they were specified for. Furthermore, when building a shared
library, extra-lib-dirs will be passed as `-L` flags to the linker
invocation. However, the same extra-lib-dirs will not be passed as
`-rpath` to the linker.

The end situation is as follows:

    1. The shared library `libA` built for a package `A` will be linked
       against some libraries `libExtra` found in extra-lib-dirs
       `extraA`.

    2. The RPATH section of `A` will NOT contain `extraA`, because we
       don't pass -rpath extra-lib-dirs when linking the library, but it
       will depend on `libExtra`.

    3. The installed package info of that package `A` will contain, in
       the library dirs section, the extra-lib-dirs `extraA` and the
       path to `libA`.

    4. When a package `B` depends on package `A`, it will include in the
       RPATH section of the shared library `libB` the lib dirs from the
       installed package info of `A`, i.e. `/path/to/libA` and `extraA`,
       and depends on `libA` and, transitively, on `libExtra`.

The conclusion is:

    5. When we load `libB`, we will load `libA`, which is found in
       `/path/to/libA`, and, transitively, load `libExtra` which is
       found in `extraA` -- they are both found because both
       `/path/to/libA` and `extraA` are listed in the RPATH entries.

    6. However, if we load `libA` directly we will /NOT/ find
       `libExtra`, because `extraA` is not included in the RPATH
       entries.

So, ultimately, what this commit fixes, is the failure described in (6),
caused by the incorrect behaviour of (2), by specifying `-rpath
extra-lib-dirs` when linking the shared library of a package, to include
the extra lib dirs in the RPATH entries of that shared library (even
though dependents of this library would already get the extra-lib-dirs
in their RPATH, the library itself didn't, resulting in cabal#7339 and
ghc#19350)

Fixes #7339
Fixes ghc#19350

- Move project sdist tests to cabal-testsuite
- Add --ignore-project test
- Duplicate tests but without cabal.project
- Add a cabal.project one folder up
- Add a package Z in the root
- Rerun --accept with more immediate parent project
- Add a readme for the tests
- Fix problems with uv package, update expected output
- Add U and V modules
- Explain what is wrong with cabal.dot-uv.test.hs
- Add a note on cabal.no-project.test.hs
- Explain what is wrong with cabal.sub-pq.test.hs
- Explain what is wrong with cabal.sub-rs.test.hs
- Explain what is wrong with cabal.dot-uv.test.hs
- Leave a note explaining cabal.no-project.test.hs
- Leave a note explaining cabal.project.test.hs
- Leave a note explaining cabal.sub-pq.test.hs
- Explain what is wrong with cabal.sub-rs.test.hs
- Patches for project respecting behaviour
- Explain root ignore-project and no-project tests
- Add *.v2.test.hs variants exercising v2-sdist
- Add v2 patches, test out not using <ROOT>

 Include the GHC "Project Unit Id" in the cabal store path

This complements the previous commit in order to fix #9326

- This allows the use of several **API incompatible builds of the same
  version of GHC** without corrupting the cabal store.
- This relies on the "Project Unit Id" which is available since GHC
  9.8.1, older versions of GHC do not benefit from this change.

[fixes #8114]

update jinja2 per CVE-2024-22195

Fix extra-prog-path propagation in the codebase.

Similarly to CmdExec and CmdTest, get paths to all dependency binaries
and add those to PATH. Unlike CmdExec, add just the explicitly required
paths.

Extra prog paths were being handled in many different ways all thorugh
the codebase. This PR introduces a unified way to look at them.

Aiming for traceability, the addition of extra paths is now traced via
`logExtraProgramSearchPath`. All appearances of
`modifyProgramSearchPath` are replaced with `appendProgramSearchPath`
which traces the added paths.

`progInvokePathEnv` was only being set by GHC for some paths to
executables in components and only under certain circumstances. Now
every `ghcInvocation` sets the extra paths directly into `pkgInvokeEnv`.

In particular this fixes PATH issues when running MinGW cabal in
PowerShell, as usually for other OSes the system path contains most of
the expected directories.

Fix haddock open on windows

Sed-replace CRLF files

Test that the last line `--help` is the name of the config file

Ensure that the last line of the help text is the name of the config file.
This invariant is used by clients such as the Haskell setup github action.

See: https://github.com/haskell-actions/setup/pull/63

Switch from criterion to tasty-bench

Makes builds faster by eliminating 10 transitive dependencies:

* base-compat-batteries
* binary-orphans
* cassava
* code-page
* criterion
* criterion-measurement
* Glob
* js-chart
* microstache
* Only

Add a format rejections function