Alp Mestanogullari authored 5 years ago and Marge Bot committed 5 years ago

When the '--hadrian' flag is passed to the validate script, we use hadrian
to build GHC, package it up in a binary distribution and later on run GHC's
testsuite against the said bindist, which gets installed locally in the process.

Along the way, this commit fixes a typo, an omission (build iserv binaries
before producing the bindist archive) and moves the Makefile that enables
'make install' on those bindists from being a list of strings in the code to
an actual file (it was becoming increasingly annoying to work with).

Finally, the Settings.Builders.Ghc part of this patch is necessary for being
able to use the installed binary distribution, in 'validate'.

Takenobu Tani authored 6 years ago and Marge Bot committed 6 years ago

This moves all URL references to Trac Wiki to their corresponding
GitLab counterparts.

This substitution is classified as follows:

1. Automated substitution using sed with Ben's mapping rule [1]
    Old: ghc.haskell.org/trac/ghc/wiki/XxxYyy...
    New: gitlab.haskell.org/ghc/ghc/wikis/xxx-yyy...

2. Manual substitution for URLs containing `#` index
    Old: ghc.haskell.org/trac/ghc/wiki/XxxYyy...#Zzz
    New: gitlab.haskell.org/ghc/ghc/wikis/xxx-yyy...#zzz

3. Manual substitution for strings starting with `Commentary`
    Old: Commentary/XxxYyy...
    New: commentary/xxx-yyy...

See also !539

[1]: https://gitlab.haskell.org/bgamari/gitlab-migration/blob/master/wiki-mapping.json

- Remove dph from validate; dph was removed
- The required-tag argument to boot was used only for dph, remove
- check_boot_packages() was not called at all, and didn't work.
  I fixed it based on previous Perl version.

Test Plan: Harbormaster

Reviewers: bgamari, thomie

Reviewed By: bgamari

Subscribers: rwbarton, carter

Differential Revision: https://phabricator.haskell.org/D5129

Ben Gamari authored 6 years ago and Ben Gamari committed 6 years ago

Test Plan: Try `./validate`, CircleCI build; make sure core count
detection works in both cases.

Reviewers: alpmestan

Reviewed By: alpmestan

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14470

Differential Revision: https://phabricator.haskell.org/D4897

Ben Gamari authored 6 years ago and Ben Gamari committed 6 years ago

Poor DPH and its vectoriser have long been languishing; sadly it seems there is
little chance that the effort will be rekindled. Every few years we discuss
what to do with this mass of code and at least once we have agreed that it
should be archived on a branch and removed from `master`. Here we do just that,
eliminating heaps of dead code in the process.

Here we drop the ParallelArrays extension, the vectoriser, and the `vector` and
`primitive` submodules.

Test Plan: Validate

Reviewers: simonpj, simonmar, hvr, goldfire, alanz

Reviewed By: simonmar

Subscribers: goldfire, rwbarton, thomie, mpickering, carter

Differential Revision: https://phabricator.haskell.org/D4761

Ben Gamari authored 7 years ago and Ben Gamari committed 7 years ago

One step closer to being able to drop the Windows Perl tarball. We
previously attempted to do this in D3567 but were forced to revert due
to Windows problems.

Acknowledgements:
 * @Phyx kindly contributed the codepath allowing this to work on
Windows.

Test Plan: Validate

Reviewers: hvr, austin, Phyx

Subscribers: erikd, thomie, rwbarton

Differential Revision: https://phabricator.haskell.org/D3918

Ben Gamari authored 7 years ago and Ben Gamari committed 7 years ago

Test Plan: Validate, try ingesting into Jenkins.

Reviewers: austin

Subscribers: rwbarton, thomie

GHC Trac Issues: #13716

Differential Revision: https://phabricator.haskell.org/D3796

This reverts commit 0440af6a.
Unfortunately this breaks on Windows for tiresome reasons. I'll need to
reevaluate this.

Ben Gamari authored 7 years ago and Ben Gamari committed 7 years ago

Test Plan: Validate

Reviewers: hvr, austin

Subscribers: rwbarton, thomie, erikd

Differential Revision: https://phabricator.haskell.org/D3567

Ben Gamari authored 7 years ago and Ben Gamari committed 7 years ago

For reasons unknown `validate` passed `NO_CLEAN_GMP=YES` to
`maintainer-clean`, leaving a stale `gmp.h` which causes the build to
fail in the event that the uses a tree for validating for two different
target platforms. This is quite unexpected, don't do it.

Reviewers: hvr, austin, rwbarton, dfeuer

Reviewed By: dfeuer

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D3409

Ben Gamari authored 8 years ago and Ben Gamari committed 8 years ago

Due to #12554 and #12661 we must be quite picky about our choice of
Python interpreter on Windows. Allow the user to override it.

Test Plan: `PYTHON=/usr/bin/python2 ./validate` on Windows

Reviewers: austin, Phyx

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2603

GHC Trac Issues: #12554, #12661

Ben Gamari authored 8 years ago and Ben Gamari committed 8 years ago

This will allow us to split up Harbormaster output for the build and
test stages of validation.

Test Plan: `./validate --build-only && ./validate --testsuite-only`

Reviewers: thomie, hvr, austin

Differential Revision: https://phabricator.haskell.org/D2553

* See `Note [Why is there no stage1 setup function?]`.
* Move T2632 to the tests/stage1 directory (#10382).

Reviewed by: ezyang, nomeata, bgamari

Differential Revision: https://phabricator.haskell.org/D2341

GHC Trac Issues: #12197

As discussed in Phab:D1187, this approach makes it a bit easier to
inspect the test directory while working on a new test.

The only tests that needed changes are the ones that refer to files in
ancestor directories. Those files are now copied directly into the test
directory.

validate still runs the tests in a temporary directory in /tmp, see
`Note [Running tests in /tmp]` in testsuite/driver/runtests.py.

Update submodule hpc.

Reviewed by: simonmar

Differential Revision: https://phabricator.haskell.org/D2333

GHC Trac Issues: #11980

Ben Gamari authored 9 years ago and Ben Gamari committed 9 years ago

Test Plan: validate, check that gz is used

Reviewers: hvr, austin, thomie

Reviewed By: thomie

Differential Revision: https://phabricator.haskell.org/D1788

GHC Trac Issues: #11434

Thomas Miedema authored 9 years ago and Ben Gamari committed 9 years ago

On Windows, with msys2, `pwd` works (as can be seen by the use of `pwd`
that slipped into the validate script), so there is really no need for
`ghc-pwd` anymore.

Test Plan: try it

Reviewers: austin, bgamari, Phyx

Reviewed By: Phyx

Subscribers: Phyx, erikd

Differential Revision: https://phabricator.haskell.org/D1731

Ben Gamari authored 9 years ago and Ben Gamari committed 9 years ago

This should catch duplicate uniques in the future before Bad Things
happen.

Test Plan: Introduce a duplicate unique and validate

Reviewers: austin, hvr, thomie

Reviewed By: hvr, thomie

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1521

Summary:
This reverts commit aecf4a5f.

It turns out the Simons are relying on 'mk/are-validating.mk', see
D1307.

The workflow they are using is:
  * run ./validate
  * find a bug in the compiler
  * try to fix the bug, running 'make 1' (or 'make 2') repeatedly. Because
    of 'mk/are-validating.mk', this uses the same build settings as validate.
  * continue ./validate (--no-clean)

I suggested two alternatives:

  A. run 'make 1 Validating=YES' instead of 'make 1'

     Problem: when running `./validate --fast` or `./validate --hpc`
     instead of a normal `./validate`, validate sets ValidateSpeed and
     ValdateHpc in mk/are-validating.mk. You would for example have to run
     'make 1 Validating=YES ValidateSpeed=FAST' instead of 'make 1' to get the
     same build settings as `./validate --fast`, which is entirely too long and
     error prone.

  B. uncomment `#BuildFlavour=validate` in mk/build.mk, and include
     'mk/validate.mk'.

     Problems:
      * any other settings you have in build.mk will also get used.
      * the distinction between 'mk/validate.mk' and 'mk/build.mk' becomes less
        clear.
      * it is easy to forget to include 'mk/validate.mk'.
      * the build system again doesn't have access to the ValidateSpeed and
        ValdateHpc settings set by validate.

Neither of these two options is entirely satisfactory.

Reviewers: austin, bgamari

Differential Revision: https://phabricator.haskell.org/D1383

A `make` build after running `./validate` earlier should use the normal
mk/build.mk settings, without having to manually delete the leftover
file mk/are-validating or run `make clean` first.

Reviewed by: austin

Differential Revision: https://phabricator.haskell.org/D1307

Just install all packages that are built. Don't make an exception for
the dph and extra packages.

You can control whether the dph and extra packages should be build using
the variables BUILD_DPH and BUILD_EXTRA_PKGS. These variables didn't
exist before. But now that they do, InstallExtraPackages isn't really
needed anymore.

Reviewed by: austin

Differential Revision: https://phabricator.haskell.org/D1227

`make test` now runs all tests for a single way only. Use `make slowtest` to
get the previous behaviour (i.e. run all tests for all ways).

The intention is to use this new `make test` setting for Phabricator, as
a reasonable compromise between `make fasttest` (what it previously
used) and a fullblown `make slowtest` (which runs all tests for all
ways).

See Note [validate and testsuite speed] in toplevel Makefile.

Differential Revision: https://phabricator.haskell.org/D1178

We only support building GHC on mys2 nowadays, see
https://ghc.haskell.org/trac/ghc/wiki/Building/Preparation/Windows.

This (partially) reverts various commits from a few years ago, among which:
  * 5775d514
    "Add OSTYPE build-system variable, and use it"
  * 3fb8c431
    "Fix building libgmp on cygwin"
  * cdbb4720
    "Fix cmd invocation by libffi cuild system on Windows 7 cygwin"
  * e8121501
    "Fix dblatex and xml* tool detection on Windows"

Reviewed by: austin, Phyx

Differential Revision: https://phabricator.haskell.org/D1155

Show commands being run by default, so you can copy/paste and rerun them
on build failures/warnings.

See Note [Default build system verbosity].

stm and parallel have an 'extra' tag in the ./packages file, so would get
added to PACKAGES_STAGE2 by default, and subsequently build by the stage2
compiler.

With this patch, this happens only when you set BUILD_EXTRA_PKGS=YES in
build.mk. A normal validate still builds (and tests) the 'extra'
packages, but they are skipped for `validate --fast`. Maybe this brings
us closer to finishing within the 50 minute Travis limit as well.

We can later try to give random, primitive and vector an 'extra' tag as
well (now they have a 'dph' tag), but some tests will probably fail at
first.

Differential Revision: https://phabricator.haskell.org/D1065

  * By default use V=0, and call the testsuite with VERBOSE=2, which we
    did before only with validate --quiet. This disables printing the
    test commands it runs.

  * When --quiet is used, call the testsuite with VERBOSE=1. This
    disables printing the '====> Scanning' lines, and doesn't print
    which test is being run. So it only prints something when a test
    accidentally prints to stdout or when it fails.

    Don't set this option on Travis, as Travis will cancel a build if it
    doesn't see any output for more than 10 minutes.

  * When --quiet is used, set the new test option NO_PRINT_SUMMARY,
    which skips printing the test summary. Only the list of unexpected
    failures is printed, if there are any. Note that the full summary
    can still be found in testsuite_summary.txt

  * When --quiet is used, don't pass the `-v` flag to `ghc-pkg check`

  * When --quiet is used, don't print the Oops! header. It shoud be
    clear from the list of failing tests that something is wrong.

This is all done to get the most out of 30 lines of logfile. These changes can
be disabled later by simply not passing the --quiet flag to validate.

Differential Revision: https://phabricator.haskell.org/D942

And mention more prominently that `-DDEBUG` and tests that call
`compiler_stats_num_fields` don't play well together

Reviewed By: austin

Differential Revision: https://phabricator.haskell.org/D799

Reviewed By: austin

Differential Revision: https://phabricator.haskell.org/D783

Summary:
Currently, running `make test` in:
  /                   runs fast testsuite
  /testsuite          doesn't work
  /testsuite/tests    runs slow testsuite

This commit:
  * changes `make test` in the ghc root directory to run the slow testsuite,
    just like it already does in `testsuite/tests`
  * adds some simple targets to `/testsuite`, that all delegate to
    `/testsuite/tests`
  * adds a new target `fasttest` to the toplevel Makefile, with a shorthand
    name `fast` (for consistency with /testsuite and /testsuite/tests)
  * declares some more targets PHONY for safety

Wiki pages that need updating:
  * Building/StandardTargets
  * Buliding/RunningTests

Reviewers: austin

Reviewed By: austin

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D685

Summary: This will *not* work on NixOS for example.

Reviewers: austin

Reviewed By: austin

Subscribers: thomie, carter

Differential Revision: https://phabricator.haskell.org/D479

GHC Trac Issues: #9057

This should have gone in with the update in testlib.py to split
off stat test failures into a separate category in the summary.

Reviewers: austin

Reviewed By: austin

Subscribers: thomie, carter

Differential Revision: https://phabricator.haskell.org/D451

Summary:
When we run `./validate`, we are typically given an incredibly large
heap of information, a large majority of which isn't really
necessary. In particular, we don't really care about what `make` is
doing, nor `ghc` itself most of the time.

This reduces some of the output by making `./validate` quietier. By
running:

  $ ./validate --quiet

you'll enable `V=0` in the build, suppressing compiler messages, and
you will suppress `make` commands by running `make` in 'silent
mode'. It also runs the testsuite with `VERBOSE=2` to avoid extra
lines. This alone makes quite a difference for build log sizes.

Furthermore, by making the build logs less verbose, life is easier for
systems like Harbormaster and Travis-CI, which dislike dealing with
logs that are 10k lines or more.

Signed-off-by: Austin Seipp <austin@well-typed.com>

Test Plan: iiam

Reviewers: hvr, nomeata, ezyang

Reviewed By: ezyang

Subscribers: simonmar, ezyang, carter, thomie

Projects: #ghc

Differential Revision: https://phabricator.haskell.org/D298

Summary: Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

Test Plan: ran ./validate on linux

Reviewers: austin

Reviewed By: austin

Subscribers: phaskell, simonmar, relrod, ezyang, carter

Differential Revision: https://phabricator.haskell.org/D146

This was removed in bb003086
and caused some buildbots to stop working. So this commit just
re-adds it as a no-op (wrt the current default).

Signed-off-by: Austin Seipp <austin@well-typed.com>

Now that the `libraries/dph` submodule is checked out always we need
a different way to disable building DPH to save compile-time while
developing GHC.

This commit adds a new YES/NO Make variable `BUILD_DPH` that can be used
inside mk/build.mk to control whether to build libraries/dph or not.
The default setting is `BUILD_DPH=YES` (via `mk/config.mk.in`).

This also changes `validate`'s flag `--no-dph` to explicitly disable DPH
for the current validation run.

Signed-off-by: Herbert Valerio Riedel <hvr@gnu.org>

Test Plan: successful validates with `--fast --no-dph`

Differential Revision: https://phabricator.haskell.org/D31