Updates to LLVM 14, hopefully fixing #21964.

(cherry picked from commit b2bb3e62)

* Allow filepath to be reinstalled
* Bump some version bounds to allow newer versions of libraries
* Rework testing logic to avoid "install --lib" and package env files

Fixes #22344

(cherry picked from commit 7cbdaad0)

Ben Gamari authored 2 years ago and Matthew Pickering committed 2 years ago

Teach the `configure` script to create the `VERSION` file.
This will serve as the stable interface to allow the user to determine
the version number of a working tree.

Fixes #22322.

(cherry picked from commit a2d814dc)

This adds release jobs for ubuntu18_04 which uses glibc 2.27 which is
older than the 2.28 which is used by Rocky8 bindists.

Ticket #22268

(cherry picked from commit 0a022207)

1. A python script in .gitlab/rel_eng/mk-ghcup-metadata which generates
   suitable metadata for consumption by GHCUp for the relevant
   pipelines.

  - The script generates the metadata just as the ghcup maintainers
    want, without taking into account platform/library combinations. It
    is updated manually when the mapping changes.

  - The script downloads the bindists which ghcup wants to distribute,
    calculates the hash and generates the yaml in the correct structure.

  - The script is documented in the .gitlab/rel_eng/mk-ghcup-metadata/README.mk file

1a. The script requires us to understand the mapping from platform ->
    job. To choose the preferred bindist for each platform the
    .gitlab/gen_ci.hs script is modified to allow outputting a metadata
    file which answers the question about which job produces the
    bindist which we want to distribute to users for a specific
    platform.

2. Pipelines to run on nightly and release jobs to generate metadata

  - ghcup-metadata-nightly: Generates metadata which points directly to
    artifacts in the nightly job.

  - ghcup-metadata-release: Generates metadata suitable for inclusion
    directly in ghcup by pointing to the downloads folder where the
    bindist will be uploaded to.

2a. Trigger jobs which test the generated metadata in the downstream
    `ghccup-ci` repo. See that repo for documentation about what is
    tested and how but essentially we test in a variety of clean images
    that ghcup can download and install the bindists we say exist in our
    metadata.

(cherry picked from commit ef4ffd9d)

This job exists only for the meta-reason of not allowing nightly
pipelines to be cancelled. It was taking two minutes to run as in order
to run "true" we would also clone the whole GHC repo.

(cherry picked from commit e9cfc723)

It is better to keep these scripts in the tree as they depend on the CI
configuration and so on. By keeping them in tree we can keep them
up-to-date as the CI config changes and also makes it easier to backport
changes to the release script between release branches in future.

The final motivation is that it makes generating GHCUp metadata
possible.

(cherry picked from commit c41aeb30)

(cherry picked from commit 00981dc9)

Matthew Pickering authored 2 years ago and Ben Gamari committed 2 years ago

This fixes errors of the form:

```
fatal: detected dubious ownership in repository at '/builds/ghc/ghc'
To add an exception for this directory, call:
	git config --global --add safe.directory /builds/ghc/ghc
inferred 9.7.20230113
checking for GHC Git commit id... fatal: detected dubious ownership in repository at '/builds/ghc/ghc'
To add an exception for this directory, call:
	git config --global --add safe.directory /builds/ghc/ghc
```

(cherry picked from commit 4285ce8a)

- There is one regular wasm job run in validate pipelines
- Additionally, int-native/unreg wasm jobs run in nightly/release pipelines

Also, remove the legacy handwritten wasm ci jobs in .gitlab-ci.yml.

(cherry picked from commit 87194df0)

- Add a ghc environment including prebuilt dependencies to the
  nix-shell. Get rid of the ad hoc cabal cache and all dependencies
  are now downloaded from the nixos binary cache.
- Make gen_ci.hs a cabal package with HLS integration, to make future
  hacking of gen_ci.hs easier.
- Fix some ghc/hlint warnings after I got HLS to work.
- For the lint-ci-config job, do a shallow clone to save a few minutes
  of unnecessary git checkout time.

(cherry picked from commit 93b9bbc1)

(cherry picked from commit 84ae6573)

In 8f71d958 the make build system was made to use split-sections on
linux systems but it appears this logic never made it to hadrian.
There is the split_sections flavour transformer but this doesn't appear
to be used for perf builds on linux.

This is disbled on deb9 and windows due to #21670

Closes #21135

(cherry picked from commit be9dd9b0)

This change allows less priviledged users to trigger head.hackage jobs
because less permissions are needed to trigger jobs on the
upstream-testing branch, which is not protected.

There is a CI job which updates upstream-testing each hour to the state
of the master branch so it should always be relatively up-to-date.

(cherry picked from commit c264b06b)

The test is not configured to get the correct dependencies for the
release pipelines (and indeed stops the release pipeline being run at
all)

(cherry picked from commit f4850f36)

This enables the SLOW_VALIDATE env var for the linting head.hackage
jobs, namely the jobs enabled manually, by the label or on the nightly
build now use the deb10-numa-slow-validate bindist which has assertions
enabled.

See #22623 for a ticket which was found by using this configuration
already!

The head.hackage jobs triggered by upstream CI are now thusly:

hackage-lint: Can be triggered on any MR, normal validate pipeline or nightly build.
              Runs head.hackage with -dlint and a slow-validate bindist

hackage-label-lint: Trigged on MRs with "user-facing" label, runs the slow-validate
                    head.hackage build with -dlint.

nightly-hackage-lint: Runs automatically on nightly pipelines with slow-validate + dlint config.

nightly-hackage-perf: Runs automaticaly on nightly pipelines with release build and eventlogging enabled.

release-hackage-lint: Runs automatically on release pipelines with -dlint on a release bindist.
(cherry picked from commit cbaebfb9)

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

Also updates the bootstrap jobs to test booting 9.2 and 9.4.

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

I am not sure this job is good because it requires booting HEAD with
HEAD, but it should be fine.

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

The idea is that both the bindists should product libraries with the
same ABI and interface hash.
So the job checks with ghc-pkg to make sure the computed ABI
is the same.

In future this job can be extended to check for the other facets of
interface determinism.

Fixes #22180

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

In this job we can run on every commit we add a test which builds the
Cabal library twice and checks that the ABI hash and interface hash is
stable across the two builds.

* We run the test 20 times to try to weed out any race conditions due to
  `-j`
* We run the builds in different temporary directories to try to weed
  out anything related to build directory affecting ABI or interface
  file hash.

Fixes #22180

Add JS backend adapted from the GHCJS project by Luite Stegeman.

Some features haven't been ported or implemented yet. Tests for these
features have been disabled with an associated gitlab ticket.

Bump array submodule

Work funded by IOG.

Co-authored-by: Jeffrey Young <jeffrey.young@iohk.io>
Co-authored-by: Luite Stegeman <stegeman@gmail.com>
Co-authored-by: Josh Meredith <joshmeredith2008@gmail.com>

ci-images has recently been updated, including changes needed for wasm32-wasi CI.

Since lint-submods doesn't run in nightlies, hadrian-ghc-in-ghci needs
to mark it as "optional" so it can run if the job doesn't exist.

Fixes #22396.

Fixes #22325

Ben Gamari authored 2 years ago and Marge Bot committed 2 years ago

Previously the lint-ci job attempted to use cabal-install (specifically
`cabal update`) without a GHC in PATH. However, cabal-install-3.8
appears to want GHC, even for `cabal update`.

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

We do not use any recursively cloned submodules, and this protects us
from flaky upstream remotes.

Fixes #22121

Ben Gamari authored 3 years ago and Marge Bot committed 2 years ago

In preparation for removal of the `make`-based build system

Ben Gamari authored 2 years ago and Marge Bot committed 2 years ago

Here we add a simple qemu-based test for cross-compilers.

Ben Gamari authored 2 years ago and Marge Bot committed 2 years ago

To give the ARMv7 job access to lld, fixing #21875.

Ben Gamari authored 2 years ago and Marge Bot committed 2 years ago

Previously we would attempt to test hadrian bootstrapping in the
`validate` build flavour. However, `ci.sh` refuses to run validation
builds during release pipelines, resulting in job failures. Fix this by
testing bootstrapping in the `release` flavour during release pipelines.

We also attempted to record perf notes for these builds, which is
redundant work and undesirable now since we no longer build in a
consistent flavour.

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

The job has been failing because we don't bundle haddock docs anymore in
the docs dist created by hadrian.

Fixes #21789

Bryan R authored 2 years ago and Marge Bot committed 2 years ago

See #21859

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

We were attempting to boot a src-tarball which doesn't work as ./boot is
not included in the source tarball. This slipped through as the job is
only run on nightly.

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

These executables (such as hsc2hs) are built using the boot compiler and
crucially, most libraries from the global package database.

We also move other build-time executables to be built in this stage such
as linters which also cleans up which libraries end up in the global
package database. This allows us to remove hacks where linters-common is
removed from the package database when a bindist is created.

This fixes issues caused by infinite recursion due to bytestring adding
a dependency on template-haskell.

Fixes #21634