fix #9838

* [Rel-Eng]: Add initial version of the release metadata script

Adds the initial version of the release script that generates
the metadata for a given release automatically in the correct format
which can be used by GHCup as-is which makes it way easier to add
that release to be distributed via GHCup.

This would evolve based on the feedback from the maintainers but
the basic core functionality still works.

To Resolve: #9298

* [Rel-Eng]: Change top shebang from '/bin/bash' to '/usr/bin/env bash'

Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash
as requested in the PR review

* [Rel-Eng]: Add script to download cabal-install binaries for a release

The script to generate GHCup metadata needs the artifacts already downloaded.
This script was made to be used in conjunction to the ghcup metadata generating
script.

This script:-

* Makes the directories that ./create-release-metadata-for-ghcup.sh CDs into
* Downloads the binary release files given a particular release number into
  correct directories.
* Verifies that the downloaded checksums match the expected values upstream.

* [Rel-Eng]: Integrate the release binary downloading script into GHCup metadata generating script

Checks if binary-release folders for the release passed exist.
"create-release-metadata-for-ghcup.sh" needs the release files pre-downloaded.

Echos user to run the download script first if correct binaries folder not
found.

* [Rel-Eng]: Add release-scripts binary-downloads to gitignore

We do not want to commit the downloaded binary files for a cabal release.
This prevents us from doing that, even on accident.

* [Rel-Eng]: Change from '/bin/bash' to '/usr/bin/env bash' in the download script

Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash
in the 'download-cabal-install-release-binaries.sh'

The later seems more portable as learned from the PR review comments.

* Find build-tool installed programs before programs in path (BP)

A backport of 443c8906581e4a985185a9a32de2e6c7cc63dde1 (#9762)

---------

Co-authored-by: brandon s allbery kf8nh <allbery.b@gmail.com>
Co-authored-by: Gershom Bazerman <gershom@arista.com>

Update .cabal files (backport #9761)

`extra-doc-files` and `no-autogen-paths` warnings.

(cherry picked from commit 28daab12)

* Bump version numbers to 3.10.3.0

* Update bootstrap and CI for latest minor 9.4

* Regenerate the bootstrap files

* Disable windows runner for GHC 9.0.2

This reverts commit 34b75586.

(cherry picked from commit 8674b74d)

Co-authored-by: Bodigrim <andrew.lelechenko@gmail.com>

Bump a number of dependencies (backport #9686)

(cherry picked from commit 28694733)

Cabal: Allow Win32-2.14 (backport #9679)

(cherry picked from commit ec71ed5b)

* Generate changelogs

Add extraLibDirs to runtime lib search paths of library (backport #9554)

Runtime search paths are hard. Here's the current picture to understand
why this patch exists:

* When linking a shared library, GHC will include in the rpath entries
  of the shared library all the paths listed in the library dirs section
  of the installed package info of all packages the shared library
  depends on.
    * On darwin, GHC has special logic to inject the library dirs listed
      in the installed dependent packages info into the rpath section
      instead of passing the dirs as -rpath flags to the linker.
      However, only the dirs where used libraries are found are
      actually injected. The others are ignored. This works around
      limitations of the darwin loader.

* Cabal, in addition, passes directly to the linker (via
  -optl-Wl,-rpath,...) the library dirs of packages the
  shared library for the package being built depends on.
    * In a vanilla cabal installation, this will typically only be the
      path to the cabal store and the path to the installed GHC's boot
      libraries store.
    * When using nix there will a different library dir per installed
      package. Since these lib dirs are passed directly to the linker as
      rpaths, we bypass the darwin loader logic and, for very big
      packages, on darwin, we could end up reaching the load command
      limit and fail linking. We don't address this situation in this MR.

When we specify `extra-lib-dirs` in Cabal, these extra-lib-dirs will be
added to the library dirs listed in the installed package info of the
library they were specified for. Furthermore, when building a shared
library, extra-lib-dirs will be passed as `-L` flags to the linker
invocation. However, the same extra-lib-dirs will not be passed as
`-rpath` to the linker.

The end situation is as follows:

    1. The shared library `libA` built for a package `A` will be linked
       against some libraries `libExtra` found in extra-lib-dirs
       `extraA`.

    2. The RPATH section of `A` will NOT contain `extraA`, because we
       don't pass -rpath extra-lib-dirs when linking the library, but it
       will depend on `libExtra`.

    3. The installed package info of that package `A` will contain, in
       the library dirs section, the extra-lib-dirs `extraA` and the
       path to `libA`.

    4. When a package `B` depends on package `A`, it will include in the
       RPATH section of the shared library `libB` the lib dirs from the
       installed package info of `A`, i.e. `/path/to/libA` and `extraA`,
       and depends on `libA` and, transitively, on `libExtra`.

The conclusion is:

    5. When we load `libB`, we will load `libA`, which is found in
       `/path/to/libA`, and, transitively, load `libExtra` which is
       found in `extraA` -- they are both found because both
       `/path/to/libA` and `extraA` are listed in the RPATH entries.

    6. However, if we load `libA` directly we will /NOT/ find
       `libExtra`, because `extraA` is not included in the RPATH
       entries.

So, ultimately, what this commit fixes, is the failure described in (6),
caused by the incorrect behaviour of (2), by specifying `-rpath
extra-lib-dirs` when linking the shared library of a package, to include
the extra lib dirs in the RPATH entries of that shared library (even
though dependents of this library would already get the extra-lib-dirs
in their RPATH, the library itself didn't, resulting in cabal#7339 and
ghc#19350)

Fixes #7339
Fixes ghc#19350

Ignore invalid Unicode in pkg-config descriptions (backport #9609)

Tom Smeding authored 1 year ago and Mikolaj Konarski committed 1 year ago

* Ignore invalid Unicode in pkg-config descriptions

Previously, if any of the pkg-config packages on the system had invalid
Unicode in their description fields (like the Intel vpl package has at
the time of writing, 2024-01-11, see #9608), cabal would crash because
it tried to interpret the entire `pkg-config --list-all` output as
Unicode.

This change, as suggested by gbaz in
  https://github.com/haskell/cabal/issues/9608#issuecomment-1886120831


switches to using a lazy ByteString for reading in the output, splitting
on the first space in byte land, and then parsing only the package
_name_ to a String.

For further future-proofing, package names that don't parse as valid
Unicode don't crash Cabal, but are instead ignored.

* Add changelog entry

* cabal-install-solver: Add bounds on 'text'

* No literal ASCII values, use 'ord'

* Address review comments re invalid unicode from pkg-config

* Add test for invalid unicode from pkg-config

* Compatibility with text-1.2.5.0

* Align imports

* Handle different exception type

* Use only POSIX shell syntax

* Add invalid-input handler in pkg-config shim

This is to appease shellcheck

* Actually implement all required stuff in the pkg-config shim

* Less exception dance

* Fix shebang lines

MacOS doesn't have /usr/bin/sh, and /bin/sh is the standard (for a POSIX
shell) anyway

* Don't expect a particular representation of invalid characters

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit 0b34b4ea)

Relax `containers` upper bound in `Cabal-syntax` (backport #9643)

(cherry picked from commit 09596775)

Allow containers-0.7 (backport #9297)

Closes: #9289
(cherry picked from commit 12eeb79b)

Fix extra-prog-path propagation in the codebase. (backport #9527)

Similarly to CmdExec and CmdTest, get paths to all dependency binaries
and add those to PATH. Unlike CmdExec, add just the explicitly required
paths.

(cherry picked from commit f06195d3)

# Conflicts:
#	cabal-install/src/Distribution/Client/CmdExec.hs

Extra prog paths were being handled in many different ways all thorugh
the codebase. This PR introduces a unified way to look at them.

Aiming for traceability, the addition of extra paths is now traced via
`logExtraProgramSearchPath`. All appearances of
`modifyProgramSearchPath` are replaced with `appendProgramSearchPath`
which traces the added paths.

`progInvokePathEnv` was only being set by GHC for some paths to
executables in components and only under certain circumstances. Now
every `ghcInvocation` sets the extra paths directly into `pkgInvokeEnv`.

In particular this fixes PATH issues when running MinGW cabal in
PowerShell, as usually for other OSes the system path contains most of
the expected directories.

(cherry picked from commit 46df8ba7)

# Conflicts:
#	Cabal/src/Distribution/Simple/Configure.hs
#	Cabal/src/Distribution/Simple/ConfigureScript.hs
#	Cabal/src/Distribution/Simple/GHC.hs
#	Cabal/src/Distribution/Simple/GHCJS.hs
#	Cabal/src/Distribution/Simple/Program/Db.hs
#	Cabal/src/Distribution/Simple/Program/Find.hs
#	Cabal/src/Distribution/Simple/Program/GHC.hs
#	Cabal/src/Distribution/Simple/Program/Run.hs
#	Cabal/src/Distribution/Simple/Program/Types.hs
#	cabal-install/src/Distribution/Client/CmdExec.hs
#	cabal-install/src/Distribution/Client/CmdInstall.hs
#	cabal-install/src/Distribution/Client/CmdRun.hs
#	cabal-install/src/Distribution/Client/Config.hs
#	cabal-install/src/Distribution/Client/Get.hs
#	cabal-install/src/Distribution/Client/HttpUtils.hs
#	cabal-install/src/Distribution/Client/ProjectConfig.hs
#	cabal-install/src/Distribution/Client/ProjectOrchestration.hs
#	cabal-install/src/Distribution/Client/ProjectPlanning.hs
#	cabal-install/src/Distribution/Client/SetupWrapper.hs
#	cabal-install/src/Distribution/Client/VCS.hs
#	cabal-install/tests/UnitTests/Distribution/Client/Get.hs
#	cabal-install/tests/UnitTests/Distribution/Client/VCS.hs

update jinja2 per CVE-2024-22195 (backport #9619)

(cherry picked from commit f1908283)

# Conflicts:
#	doc/requirements.in
#	doc/requirements.txt

* Account for .buildinfo in repl when build-type: Configure (#9440)

In `autoconfUserHooks` we were not updating the `preRepl` hook to read
additional build information from /package/@.buildinfo@.

Additionally updates `autoconfUserHooks` to read additional build info
information for the remaining pre-hooks that are not pre-conf.

Fixes #9401

(cherry picked from commit ee1e6b81)

# Conflicts:
#	Cabal/src/Distribution/Simple.hs

* Update Simple.hs

---------

Co-authored-by: Rodrigo Mesquita <rodrigo.m.mesquita@gmail.com>
Co-authored-by: Hécate Moonlight <Kleidukos@users.noreply.github.com>

Relax `tar` upper bound

* Add a `Compat` module to accomodate two different `tar` interfaces.
* Tweak `-Wunused-packages` conditional (thanks Phil de Joux)

Try each pkg-config query separatedly (backport #9134)

(cherry picked from commit 9f150459)

Add test for #9331

Guard Paths_* behind `cabal-version: 3.12` or higher, “fail” and
“succeed” tests.

* check: guard PackageInfo behind cabal-version ≥ 3.12

Note we do not actually check for ≥ 3.12, since it is not possible
to introduce constructors in point release.
Instead the check always fires with PackageInfo_* is present, and
suggests an upgrade path.

* Fix testsuite

* Add changelog

Co-authored-by: Mikolaj <281893+Mikolaj@users.noreply.github.com>

Relax upper bounds on filepath (backport #9523)