The refactor in f70fc980 solved an
inconsistency in `buildAndInstallUnpackedPackage`. Namely, before the
refactor, we didn't pass `hsSetupBuildArgs` to the build invocations,
and afterwards we did.

The result is that build targets of (legacy) package `B`, that a package
`A` depends on, are now passed to the ./Setup build invocation of
package `B` (e.g. `./Setup build lib:openapi3` rather than just `./Setup
build`).

This means we no longer /build always all targets of a legacy-package/
(for example, the lib, the testsuites and the executables).
Instead only the required target (just the lib) will be built.

However, despite now passing the build args to `./Setup build`, we
weren't passing the same arguments to the `./Setup copy` invocation.
Therefore, `./Setup copy` would also try to copy targets of the package
that weren't built, resulting in a failure.

This fixes that failure by correctly passing the build targets to the
`./Setup copy` invocation, just like we do for the `./Setup build` one.

Fixes #9640

This change removes support for building sub-component targets in
cabal-install, since no versions of Cabal support this feature.

The test RunMainBad is also dropped because without the concept of a
file target, RunMainBad is the same test as ScriptBad.

* Ignore invalid Unicode in pkg-config descriptions

Previously, if any of the pkg-config packages on the system had invalid
Unicode in their description fields (like the Intel vpl package has at
the time of writing, 2024-01-11, see #9608), cabal would crash because
it tried to interpret the entire `pkg-config --list-all` output as
Unicode.

This change, as suggested by gbaz in
  https://github.com/haskell/cabal/issues/9608#issuecomment-1886120831


switches to using a lazy ByteString for reading in the output, splitting
on the first space in byte land, and then parsing only the package
_name_ to a String.

For further future-proofing, package names that don't parse as valid
Unicode don't crash Cabal, but are instead ignored.

* Add changelog entry

* cabal-install-solver: Add bounds on 'text'

* No literal ASCII values, use 'ord'

* Address review comments re invalid unicode from pkg-config

* Add test for invalid unicode from pkg-config

* Compatibility with text-1.2.5.0

* Align imports

* Handle different exception type

* Use only POSIX shell syntax

* Add invalid-input handler in pkg-config shim

This is to appease shellcheck

* Actually implement all required stuff in the pkg-config shim

* Less exception dance

* Fix shebang lines

MacOS doesn't have /usr/bin/sh, and /bin/sh is the standard (for a POSIX
shell) anyway

* Don't expect a particular representation of invalid characters

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Runtime search paths are hard. Here's the current picture to understand
why this patch exists:

* When linking a shared library, GHC will include in the rpath entries
  of the shared library all the paths listed in the library dirs section
  of the installed package info of all packages the shared library
  depends on.
    * On darwin, GHC has special logic to inject the library dirs listed
      in the installed dependent packages info into the rpath section
      instead of passing the dirs as -rpath flags to the linker.
      However, only the dirs where used libraries are found are
      actually injected. The others are ignored. This works around
      limitations of the darwin loader.

* Cabal, in addition, passes directly to the linker (via
  -optl-Wl,-rpath,...) the library dirs of packages the
  shared library for the package being built depends on.
    * In a vanilla cabal installation, this will typically only be the
      path to the cabal store and the path to the installed GHC's boot
      libraries store.
    * When using nix there will a different library dir per installed
      package. Since these lib dirs are passed directly to the linker as
      rpaths, we bypass the darwin loader logic and, for very big
      packages, on darwin, we could end up reaching the load command
      limit and fail linking. We don't address this situation in this MR.

When we specify `extra-lib-dirs` in Cabal, these extra-lib-dirs will be
added to the library dirs listed in the installed package info of the
library they were specified for. Furthermore, when building a shared
library, extra-lib-dirs will be passed as `-L` flags to the linker
invocation. However, the same extra-lib-dirs will not be passed as
`-rpath` to the linker.

The end situation is as follows:

    1. The shared library `libA` built for a package `A` will be linked
       against some libraries `libExtra` found in extra-lib-dirs
       `extraA`.

    2. The RPATH section of `A` will NOT contain `extraA`, because we
       don't pass -rpath extra-lib-dirs when linking the library, but it
       will depend on `libExtra`.

    3. The installed package info of that package `A` will contain, in
       the library dirs section, the extra-lib-dirs `extraA` and the
       path to `libA`.

    4. When a package `B` depends on package `A`, it will include in the
       RPATH section of the shared library `libB` the lib dirs from the
       installed package info of `A`, i.e. `/path/to/libA` and `extraA`,
       and depends on `libA` and, transitively, on `libExtra`.

The conclusion is:

    5. When we load `libB`, we will load `libA`, which is found in
       `/path/to/libA`, and, transitively, load `libExtra` which is
       found in `extraA` -- they are both found because both
       `/path/to/libA` and `extraA` are listed in the RPATH entries.

    6. However, if we load `libA` directly we will /NOT/ find
       `libExtra`, because `extraA` is not included in the RPATH
       entries.

So, ultimately, what this commit fixes, is the failure described in (6),
caused by the incorrect behaviour of (2), by specifying `-rpath
extra-lib-dirs` when linking the shared library of a package, to include
the extra lib dirs in the RPATH entries of that shared library (even
though dependents of this library would already get the extra-lib-dirs
in their RPATH, the library itself didn't, resulting in cabal#7339 and
ghc#19350)

Fixes #7339
Fixes ghc#19350

- Move project sdist tests to cabal-testsuite
- Add --ignore-project test
- Duplicate tests but without cabal.project
- Add a cabal.project one folder up
- Add a package Z in the root
- Rerun --accept with more immediate parent project
- Add a readme for the tests
- Fix problems with uv package, update expected output
- Add U and V modules
- Explain what is wrong with cabal.dot-uv.test.hs
- Add a note on cabal.no-project.test.hs
- Explain what is wrong with cabal.sub-pq.test.hs
- Explain what is wrong with cabal.sub-rs.test.hs
- Explain what is wrong with cabal.dot-uv.test.hs
- Leave a note explaining cabal.no-project.test.hs
- Leave a note explaining cabal.project.test.hs
- Leave a note explaining cabal.sub-pq.test.hs
- Explain what is wrong with cabal.sub-rs.test.hs
- Patches for project respecting behaviour
- Explain root ignore-project and no-project tests
- Add *.v2.test.hs variants exercising v2-sdist
- Add v2 patches, test out not using <ROOT>

This complements the previous commit in order to fix #9326

Ensure that the last line of the help text is the name of the config file.
This invariant is used by clients such as the Haskell setup github action.

See: https://github.com/haskell-actions/setup/pull/63

The earlier output was from cabal-install-3.10.2.0 compiled using Cabal-3.10.2.1, from ~/.ghcup/bin/cabal

Add conflicting versions of pandoc to repo but not all of them as there are so many

* Introduce `-i/--ignore` command-line option.
    e.g.  cabal check --ignore=missing-upper-bounds  will not display
    “Missing upper bounds” warnings.

* Additionally, a filterPackageChecksById function is now exported
  in Distribution.PackageDescription.Check.Warning; this can be
  used by third party tools.

* Add CheckExplanationIDString to `cabal check` messages.
    e.g. from
      Error: The 'license' field is missing or is NONE.
    to
      Error: [license-none] The 'license' field is missing or is NONE.

    This makes using the cabal check `--ignore` option more immediate.

* Spool `MissingField` into separate constructors.
  Introducing five new constructors for `CheckExplanation`
    MissingFieldCategory
    MissingFieldMaintainer
    MissingFieldSynopsis
    MissingFieldDescription
    MissingFieldSynOrDesc
  This provides better ergonomic for `cabal check --ignore` and makes
  it easier to update the manual if the need arises.

* Add tests for desiderable `--ignore` string qualities (not too long,
  without too many '-', unique).

* Warn when `--ignore` string is not recognised.
    Also add a test for this.

* Add documentation.

* Add changelog.

When using the multi components repl, the last unit given to ghc
is the active unit. This change uses the cabal repl argument
order instead of relying on the 'listDirectory' ordering.

Tracking ticket https://github.com/haskell/cabal/issues/9530

This commit re-designs the mechanism by which we make the .mix files of
libraries available to produce the Haskell Program Coverage report after
running testsuites.

The idea, for the Cabal library, is:

* Cabal builds libraries with -fhpc, and store the hpc artifacts in
  build </> `extraCompilationArtifacts`
* At Cabal install time, `extraCompilationArtifacts` is copied into the
  package database
* At Cabal configure time, we both
    - receive as --coverage-for flags unit-ids of library components
      from the same package (ultimately, when #9493 is resolved, we will
      receive unit ids of libraries in other packages in the same
      project too),
    - and, when configuring a whole package instead of just a testsuite
      component, we determine the unit-ids of libraries in the package
  these unit-ids are written into `configCoverageFor` in `ConfigFlags`
* At Cabal test time, for each library to cover (stored in
  `configCoverageFor`), we look in the package database for the hpc
  dirs, which we eventually pass along to the `hpc markup` call as
  `--hpcdir` flags

As for cabal-install:

* After a plan has been elaborated, we select the packages which can be
  covered and pass them to Cabal's ./Setup configure as
  --coverage-for=<unit-id> flags.
    - Notably, valid libraries are non-indefinite and
      non-instantiations, since HPC does not support backpack.
    - Furthermore, we only include libraries in the same package as the
      component being configured, despite possibly there being
      more library components in other packages of the same project.
      When #9493 is resolved, we could lift this restriction and pass
      all libraries local to the package as --coverage-for. See
      `determineCoverageFor` and `shouldCoverPkg` in Distribution.Client.ProjectPlanning.

Detail:
    We no longer pass the path to the testsuite's mix dirs to `hpc
    markup` because we only ever include modules in libraries, which
    means they were previously unused.

Fixes #6440 (internal libs coverage), #6397 (backpack breaks coverage),
doesn't yet fix #8609 (multi-package coverage report) which is tracked
in #9493, and fixes in a new way the previously fixed #4798, #5213.

This commits re-enables per-component builds when coverage checking is
enabled. This restriction was previously added in #5004 to fix #4798.

- #4798 was subsequently fixed "again" with the fix for #5213, in #7493 by
fixing the paths of the testsuite `.mix` files to the same location as
that of the main library component.

Therefore the restriction to treat testsuites per-package
(legacy-fallback) is no longer needed.

We went further and fixed coverage for internal sublibraries, packages
with backpack (but without generating coverage information for
indefinite and instantiated units -- it is not clear what it would mean
for HPC to support this), and coverage for multi-package projects.

1. We allow hpc in per-component builds

2. To generate hpc files in the appropriate component directories in the
distribution tree, we remove the hack from #7493 and instead determine
the `.mix` directories that are included in the call to `hpc markup` by
passing the list of components in the project from the cabal-install
invocation of test.
We also drop an unnecessary directory in the hpc file hierarchy.

3. To account for internal (non-backpack) libraries, we include the mix
   dirs and modules of all (non-indefinite and non-instantiations)
   libraries in the project

   Indefinite libraries and instantiations are ignored as it is not
   obvious what it means for HPC to support backpack, e.g. covering a
   library function that two different instantiations

4. We now only reject coverage if there are no libraries at all in the
   project, rather than if there are no libraries in the package.

This allows us to drop the coverage masking logic in
cabal.project.coverage while still having coverage of cabal-install
(i.e. cabal test --enable-coverage cabal-install now works without the
workaround)

Even though we allow multi-package project coverage, we still cover each
package independently -- the tix files resulting from all packages are
not combined for the time being.

Multi-package project coverage is fixed in Cabal, however, the
paths to the source files listed in the `.mix` files will be incorrect
because package sources will no longer be in the root of the project
tree, but rather under the subdir with the package. We add an error for
multi-package projects when coverage is enabled, and track lifting this
error in #9493.

Includes tests for #6440, #6397, #8609, and #4798 (the test for #5213 already exists)

Fixes #6440 (internal libs coverage), #6397 (backpack breaks coverage)
, doesn't yet fix #8609 (multi-package coverage report) and fixes in a new way the
previously fixed #4798, #5213.

* Add `cabal-version` 3.12

* Add test for #9331

- `cabal check`: Guard Paths_* behind `cabal-version: 3.12` or higher,
  “fail” and “succeed” tests.
- `cabal build`: Guard Paths_* behind `cabal-version: 3.12` or higher,
  “fail” test.

* Guard PackageInfo behind cabal-version ≥ 3.12

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add project to avoid solver errors
* SkipIfWindows for warn early overwrite tests
* Windows does not natively include a touch command
* Skip symlink install on windows
* Add install by copy test and skip symlink install for Windows

I noticed that Distribution.Utils.TempTestDir was only used in the
testsuite but defined in the Cabal library. Rather than expose this in
the public interface of the `Cabal` library, it is cleaner to refactor
it into a separate library (`Cabal-tests`) which can be used by any
testsuite component.

Also, in future it gives a clearer place to put utility functions which
need to be shared across the testsuite but not exposed in Cabal.
Cabal-tests can also freely add dependencies (such as exceptions) which
we might want to avoid adding to the Cabal library.

Fixes #9453

Issue #9334 shows that `%` characters on Windows result in invalid
paths, also `/` characters on Linux create invalid paths.

This changes from using base64 to using base16 with the same length
we use for unit-ids.

* Remove 'CommandDelegate' in favour of abstracting the fallback in
  'commandsRun', there is a new variant 'commdandRunWithFallback' which
  takes a continuation
  - This restores the modularity between the `Cabal` library and
    `cabal-install` as now `Cabal` doesn't need to know anything about
    the external command interface.
  - Fixes #9403
* Set the $CABAL environment variable to the current executable path
  - This allows external commands to be implemented by calling $CABAL,
    which is strongly preferred to linking against the Cabal library as
    there is no easy way to guantee your tool and `cabal-install` link
    against the same `Cabal` library.
  - Fixes #9402
* Pass the name of the argument
  - This allows external commands to be implemented as symlinks to an
    executable, and multiple commands can be interpreted by the same
    executable.
  - Fixes #9405
* `cabal help <cmd>` is interpreted as `cabal-<cmd> --help` for external
  commands.
  - This allows the `help` command to also work for external
  commands and hence they are better integrated into cabal-install.
  - Fixes #9404

The tests are updated to test all these additions.

These features bring the external command interface up to par with the
cargo external command interface.

This adds 4 tests which test the new external commands feature:

* ExternalCommand - Tests the expected usage of external command invoked
  via cabal-install
* ExternalCommandSetup - Tests that the ./Setup interface does not
  support external commands (#9403)
* ExternalCommandEnv - Tests that environment variables are set and
  preserved appropiately  (#9402)
* ExternalCommandHelp - Test that `cabal help <cmd>` is interpreted appropiately (#9404)

The `setDocumentation` function was modifying the elaborated package
after the hash was computed. This led to the assertion failing as the
computed hash was different to what was computed in the initial install
plan.

Therefore in order to fix this we either needed to:

1. Set elabBuildHaddocks = False at the point where the hash is
   initially computed.
2. Verify that elabBuildHaddocks = True will not lead to unexpected
   results.

The latter has been implemented.

The elabBuildHaddocks option is only consulted in
`hasValidHaddockTargets`, at which point documentation building the
executable component is disabled because elabHaddockExecutables is
False.

In the added test we ensure this by checking that we didn't build
documentation for the executable which is built because of
build-tool-depends.

Fixes #6006 #8313

Issue #8356 reports occasional errors from running the testsuite about
multiple package versions available. This stems from the invokation of
`runghc` not being explicit about all dependencies of the testsuite.

The solution is provide a component in the cabal file which is explicit
about which packages the tests can depend on. This component has a
build-depends section which lists all the dependencies that the tests
require.

It would be better if this component was a library component but we
can't do this with a Custom setup because of limitations to do with
per-component builds.

Then we also enable `-hide-all-packages`, so the dependency will not be
available if it is not explicitly listed as a dependency.

You could also imagine a future where the Setup.hs script found the test
files and compiled a single executable which would run all the tests,
rather than invoking runghc on each one individually.

Fixes #8356

There were two failing tests:

1. CCompilerOverride, was attempting to use gcc.exe rather than
   clang.exe without also overriding the C options which led to
   incorrect options being passed to gcc.exe. The fix is to override to
   clang.exe on ghc-9.4 or newer.
2. ForeignLibs exposes a bug in GHC
   (ghc/ghc#24185) and hence is
   skipped for GHCs newer than 9.4 where it was first introduced.

Towards fixing #8451, we just need to fix the shared library issue now.

* Fix Semigroup target instance

When two target names are the same, `mappend`ing them should not
error but just pick the first name.

* Add `desugarBuildToolSimple`

* Reimplement cabal check

* Reorder test output

* Fix autogen modules tests .cabal files

* Add a number of tests

* Add test for #7423

i.e. Do not warn on -O2 if under off-by-default package configuration
flag conditional.

* Add a regression for:

    * Add another -WErrr test
        This is to make sure we do *not* report it if it is under
        a user, off-by-default flag.
    * Add test for non manual user flags.
    * Add “absolute path in extra-lib-dirs” test
    * Add if/else test
    * Add “dircheck on abspath” check
    * Add Package version internal test
    * Add PackageVersionsStraddle test

* Add changelog for #8427

* Integrate various reviews

* Integrate Artem’s review

(review) Clarify `combineNames` documentation

By explaining the way it operates (working if the two names are equal
or one is empty) and renaming the function from `combineName` to
`combineNames`.

(review) Use guards instead of if/then/else

(review) Match inside argument list

(review) Replace “white” with “allow”

(review) Fix typo in comment

(review) Fix typo in Check module documentation

(review) Harmonise indentation for `data` decls

First field goes in a new line than the data constructor, so we
have more space.

(review) Rename `Prim` module to `Types`

(review) Add checkPackageFilesGPD

`checkPackageFiles` — which works on PD — was used to perform IO. We
introduce a function that does the same thing but works on GPD (which
is more principled).

`checkPackageFiles` cannot just be removed, since it is part of the
interface of Distribution.PackageDescription.Check. Deprecation can
be planned once “new check” is up and running.

* Integrate Andreas’ review

(review) Add named section to missing upper bound check

“miss upper bound” checks will now list target type and name (“On
executable 'myexe', these packages miss upper bounds”) for easier
fixing by the user.

(review) remove `cabal gen-bounds` suggestion

Reasonable as `cabal gen-bounds` is stricter than `cabal check`, see
https://github.com/haskell/cabal/pull/8427#issuecomment-1446712486


Once `gen-bounds` behaves in line with `check` we can readd the
suggestion.

(review) Do not warn on shared bounds

When a target which depends on an internal library shares some
dependencies with the latter, do not warn on upper bounds.

An example is clearer

    library
     build-depends: text < 5
    ⁝
     build-depends: myPackage,        ← no warning, internal
                    text,             ← no warning, shared bound
                    monadacme         ← warning!

* Integrate Artem’s review /II

(review) Split Check.hs

Check.hs has been split in multiple file, each une sub 1000 lines:

Check              857 lines
Check.Common       147 lines
Check.Conditional  204 lines
Check.Monad        352 lines
Check.Paths        387 lines
Check.Target       765 lines
Check.Warning      865 lines

Migration guide:
- Check              GPD/PD checks plus work-tree checks.
- Check.Common       common types and functions that are
                     *not* part of monadic checking setup.
- Check.Conditional  checks on CondTree and related matter
                     (variables, duplicate modules).
- Check.Monad        Backbone of the checks, monadic inter-
                     face and related functions.
- Check.Paths        Checks on files, directories, globs.
- Check.Target       Checks on realised targets (libraries,
                     executables, benchmarks, testsuites).
- Check.Warning      Datatypes and strings for warnings
                     and severities.

(review) remove useless section header

(review) Fix typo

(review) Add warnings documentation (list)

For each warning, we document constructor/brief description
in the manual.  This might not be much useful as not but it
will come handy when introducing `--ignore=WARN` and similar
flags.

* (review Andreas) Clarify CheckExplanation comment

Whoever modifies `CheckExplanation` data constructors needs to be
aware that the documentation in  doc/cabal-commands.rst  has to be
updated too.

* Move internal Check modules to `other-modules`

No need to expose Distribution.PackageDescription.Check.*
to the world. API for checking, for cabal-install and other
tools, should be in Distribution.PackageDescription.Check.

* Make fourmolu happy

Cabal codebase has now a formatter/style standard (see #8950).

“Ravioli ravioli, give me the formuoli”

* Do not check for OptO in scripts

See #8963 for reason and clarification requests.

* Remove useless PackageId parameter

It is now in the Reader part of CheckM monad.

* Do not check PVP on internal targets

Internal: testsuite, benchmark.
See #8361.

* Make hlint happy

* Fix #9122

When checking internal version ranges, we need to make sure we
are not mistaking a libraries with the same name but from different
packages. See #9132.

* Fix grammar

neither…nor, completing what done in #9162

* Integrate Brandon’s review: grammar

* Remove unnecessary `-fvia-C` check

Brandon’s review/II.

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add a 'cabal path' command.

* Formatting fix.

* Another formatting fix.

* Categorise "cabal path" as global command.

* Allow individual paths to be printed.

* Less duplication.

* Add config-file to "cabal path".

* Use sum type instead of strings.

* cabal path: support --installdir.

* Add documentation.

* Better text.

* Formatting.

* Add some tests.

* Improve tests.

* Add changelog entry.

* Mention "cabal path" in directory documentation.

---------

Co-authored-by: Artem Pelenitsyn <a.pelenitsyn@gmail.com>

Hannes Siebenhandl authored 1 year ago and Mikolaj Konarski committed 1 year ago

The runner allows the tests to use extra dependencies and the custom Prelude
from 'cabal-testsuite'.
However, if the tests use a dependency, say 'directory', and there are two
packages with the same unit id available in the store, the test fails since
it doesn't know which one to pick.
By including an extra dependency to directory, we force the test runner to
use a specific version directory, fixing the test failure.

By converting this to a setupTest we use the in-tree Cabal library
rather than relying on a proxy of the GHC version to provide the right
Cabal library version.

Supersedes #9398

Co-authored-by: Javier Sagredo <jasataco@gmail.com>
Co-authored-by: Andrea Bedini <andrea.bedini@tweag.io>
Co-authored-by: Andrea Bedini <andrea@andreabedini.com>