GitLab

Convert hadrian buildscripts to unsuffixed, dashed form

final cleanups

The generated headers are now generated per stage, which means we can
skip hacks like `ghc_boot_platform.h` and just have that be the stage 0
header as proper. In general, stages are to be embraced: freely generate
everything in each stage but then just build what you depend on, and
everything is symmetrical and efficient. Trying to avoid stages because
bootstrapping is a mind bender just creates tons of bespoke
mini-mind-benders that add up to something far crazier.

Hadrian was pretty close to this "stage-major" approach already, and so
was fairly easy to fix. Make needed more work, however: it did know
about stages so at least there was a scaffold, but few packages except
for the compiler cared, and the compiler used its own counting system.
That said, make and Hadrian now work more similarly, which is good for
the transition to Hadrian. The merits of embracing stage aside, the
change may be worthy for easing that transition alone.

Currently, if you change these ^ flavour parameters, rebuilding is not
triggered, since `programContext` doesn't set up a dependency on
those values.

Exposing these values via an oracle does set the dependency and
properly triggers a rebuild of binaries.

Several attempts to factor out these actions ended up in cyclic
dependency here or there. I'm not absolutely happy with this variant
either, but at least it works.

====

Issue repro:

In UserSettings.hs:
```
dbgDynamic = defaultFlavour { name = "dbg-dynamic"
                            , dynamicGhcPrograms = pure True,
                            ... }

dbgStatic = defaultFlavour { name = "dbg-static"
                           , dynamicGhcPrograms = pure False
                           ... }
```

Then in console:

```
$ hadrian/build.sh -j --flavour=dbg-dynamic
... does the build
$ hadrian/build.sh -j --flavour=dbg-static
... does nothing, considers binaries up to date
```

07ee1591 started the transition, but the
job was never finished.

Shake will be moving from its current implementation of ?== to one from
System.FilePattern. Support for `//` is being dropped, leaving only `*`
and `**` as special forms. This commit converts the existing file
patterns in Hadrian to the new format. It also removes all occurances
of <//> and changes the user-settings docs to remove references to //
and add **.

The conversion is as follows:

- //a ==> **/a

- a// ==> a/**

- a//b ==> a/**/b

By not building anything in the dynamic way on Windows, where we don't
have a working story for DLLs yet.

Also the ghcid command needs to call bash on the hadrian/ghci.sh script
explicitly as the path gets interpreted differently otherwise.

They take the general form `foo.bar.baz [+]= some values`, where
`=` completely overrides the arguments for a builder and `+=` extends
them. We currenly only support settings for updating the GHC and C
compiler options, of the form:

```
  {stage0, ..., stage3 or *}.{package name or *}
                            .ghc.{c, hs, link, deps, toolargs or *}.opts

  {stage0, ..., stage3 or *}.{package name or *}
                            .cc.{c, deps or *}.opts
```

The supported settings and their use is covered in the new section
of `hadrian/doc/user-settings.md`, while the implementation is explained
in a new Note [Hadrian settings].

Most of the logic is implemented in a new module, `Settings.Parser`, which
contains key-value assignment/extension parsers as well as utilities for
specifying allowed settings at a high-level, generating a `Predicate` from
such a description or generating the list of possible completions for a given
string.

The additions to the `Settings` module make use of this to describe the
settings that Hadrian currently supports, and apply all such
key-value settings (from the command line and `<root>/hadrian.settings`)
to the flavour that Hadrian is going to proceed with.

This new setting system comes with support for generating Bash completions,
implemented in `hadrian/completion.sh` and Hadrian's `autocomplete` target:

> source hadrian/completion.sh
> hadrian/build.sh stage1.base.ghc.<TAB>
stage1.base.ghc.c.opts     stage1.base.ghc.hs.opts
stage1.base.ghc.*.opts     stage1.base.ghc.deps.opts
stage1.base.ghc.link.opts  stage1.base.ghc.toolargs.opts

This removes a hack that copies libffi files to the rts
build directory. This was done in a libffi rule, but now
an rts rule correctly needs and copies the relevant
files from the libffi build dir to the rts build dir.

Issues: #16272 #16304

This is a temporary workaround for #16534. We generate wrapper scripts
<build-root>/ghc-stage1 and <build-root>/ghc-stage2 that can be used to
run Stage1 and Stage2 GHCs with the right arguments.

See ghc/ghc#16534.

Fixes test T3807 when building with Hadrian.

Trac #16370

Running the `./hadrian/ghci` target will load the main compiler into
a ghci session. This is intended for fast development feedback, modules are only
typechecked so it isn't possible to run any functions in the repl.

You can also use this target with `ghcid`.

The first time this command is run hadrian will need to compile a few dependencies
which will take 1-2 minutes. Loading GHC into GHCi itself takes about 30 seconds.

Internally this works by calling a new hadrian target called `tool-args`.
This target prints out the package and include flags which are necessary
to load files into ghci. The same target is intended to be used by other
tooling which uses the GHC API in order to set up the correct GHC API
session. For example, using this target it is also possible to use HIE
when developing on GHC.

This is a preparation for #16295: https://ghc.haskell.org/trac/ghc/ticket/16295

This commit mostly focuses on getting rid of untracked dependencies,
which prevent Shake's new `--shared` feature from appropriately caching
build rules.

There are three different solutions to untracked dependencies:

* Track them! This is the obvious and the best approach, but in some
  situations we cannot use it, for example, because a build rule creates
  files whose names are not known statically and hence cannot be
  specified as the rule's outputs.

* Use Shake's `produces` to record outputs dynamically, within the rule.

* Use Shake's `historyDisable` to disable caching for a particular build
  rule. We currently use this approach only for `ghc-pkg` which mutates
  the package database and the file `package.cache`.

These two tickets are fixed as the result:

Ticket #16271: ​https://ghc.haskell.org/trac/ghc/ticket/16271

Ticket #16272: ​https://ghc.haskell.org/trac/ghc/ticket/16272 (this one
is fixed only partially: we correctly record the dependency, but we
still copy files into the RTS build tree).

This ticket enables the building of a `stage3` compiler by making the
build logic more consistent and predictable in Hadrian.

Two of the main changes are:

1. In order to build anything at stageN we use the package database
present at stageN. Fixing #16069
2. `haddock` and `ghc-tags` are built
as stage1 executables (with the stage1 compiler) rather than as
stage2 compiler. Fixing
[hadrian#661](https://github.com/snowleopard/hadrian/issues/661)

In order to build a stage3 compiler, you have to set the new `finalStage` hadrian option to `Stage3`.

This patch introduces (phony) build targets of the form

    (1) stage<N>:<lib>:<name>   (e.g: stage1:lib:Cabal)
    (2) stage<N>:<exe>:<name>   (e.g: stage2:exe:ghc-bin)

where (1) builds the given library with the stage N compiler and (2)
builds the given executable with the stage N-1 compiler. This patch may
be generating too many such targets but it's a first stab that we can
refine.

This fixes #15949.

Test Plan: hadrian/build.sh stage1:exe:ghc-bin

Reviewers: bgamari, snowleopard

Reviewed By: bgamari

Subscribers: rwbarton, carter

GHC Trac Issues: #15949

Differential Revision: https://phabricator.haskell.org/D5434

Following what was done to Rules.Library some time ago and to
Rules.Compile recently (D5412), this patch moves more rules away from
the "enumerate a lot of contexts and generate one rule for each" style
and instead uses the "parse data from file path to recover context"
approach. In fact, the only rules left to convert seem to be the ones
from Rules.Generate.

This effectively decreases the pauses described in #15938 further as
well as the amount of allocations and GC that we do, unsurprisingly.
Nowhere as drastically as D5412, though.

Test Plan: perform full build and generate docs

Reviewers: snowleopard, bgamari

Reviewed By: snowleopard

Subscribers: rwbarton, carter

GHC Trac Issues: #15938

Differential Revision: https://phabricator.haskell.org/D5422

Previously, as reported in #15938, resuming a build "in the middle",
e.g when building _build/stage1/libraries/base/, hadrian would take up
to a whole minute to get started doing actual work, building code.

This was mostly due to a big enumeration that we do in Rules.hs, to
generate all the possible patterns for object files for 1) all ways, 2)
all packages and 3) all stages. Since rule enumeration is always
performed, whatever the target, we were always paying this cost, which
seemed to grow bigger the farther in the build we stopped and were
resuming from.

Instead, this patch borrows the approach that we took for Rules.Library
in https://github.com/snowleopard/hadrian/pull/571, which exposes all the
relevant object files under as few catch-all rules as possible (8 here),
and parses all the information we need out of the object's path.

The concrete effect of this patch that I have observed is to reduce the
45-60 seconds pause to <5 seconds. Along with the Shake performance
improvements that Neil mentions in #15938, most of the pause should
effectively disappear.

Reviewers: snowleopard, bgamari, goldfire

Reviewed By: snowleopard

Subscribers: rwbarton, carter

GHC Trac Issues: #15938

Differential Revision: https://phabricator.haskell.org/D5412

Sadly subtrees haven't worked quite as well as we would have liked for
developers. See Hadrian #440.