GitLab

This vastly reduces memory usage when compiling with `--make` mode, from
about 900M when compiling Cabal to about 300M.

As a matter of uniformity, it also ensures that reading from an
interface performs the same as using the in-memory cache. We can also
delete all the horrible knot-tying in updateIdInfos.

Goes some way to fixing #13586

Accept new output of tests fixing some bugs along the way

-------------------------
Metric Decrease:
    T12545
-------------------------

This commit further expand on the design for #18516 by getting rid of
the `defaultReasonSeverity` in favour of a function called
`diagReasonSeverity` which correctly takes the `DynFlags` as input. The
idea is to compute the `Severity` and the `DiagnosticReason` of each
message "at birth", without doing any later re-classifications, which
are potentially error prone, as the `DynFlags` might evolve during the
course of the program.

In preparation for a proper refactoring, now `pprWarning` from the
Parser.Ppr module has been renamed to `mkParserWarn`, which now takes a
`DynFlags` as input.

We also get rid of the reclassification we were performing inside `printOrThrowWarnings`.

Last but not least, this commit removes the need for reclassify inside GHC.Tc.Errors,
and also simplifies the implementation of `maybeReportError`.

Update Haddock submodule

Also make the HomeUnit optional to keep the field strict and prepare for
UnitEnvs without a HomeUnit (e.g. in Plugins envs, cf #14335).

Follow-up from !2418, see #19579

Updates haddock submodule

Other than that:

* Fix T16167,json,json2,T7478,T10637 tests to reflect the introduction of
  the `MessageClass` type

* Remove `makeIntoWarning`

* Remove `warningsToMessages`

* Refactor GHC.Tc.Errors

  1. Refactors GHC.Tc.Errors so that we use `DiagnosticReason` for "choices"
     (defer types errors, holes, etc);

  2. We get rid of `reportWarning` and `reportError` in favour of a general
     `reportDiagnostic`.

* Introduce `DiagnosticReason`, `Severity` is an enum: This big commit makes
  `Severity` a simple enumeration, and introduces the concept of `DiagnosticReason`,
  which classifies the /reason/ why we are emitting a particular diagnostic.
  It also adds a monomorphic `DiagnosticMessage` type which is used for
  generic messages.

* The `Severity` is computed (for now) from the reason, statically.
  Later improvement will add a `diagReasonSeverity` function to compute
  the `Severity` taking `DynFlags` into account.

* Rename `logWarnings` into `logDiagnostics`

* Add note and expand description of the `mkHoleError` function

As suggested by @alexbiehl, this patch replaces the always updated
UniqSupply in NameCache with a fixed Char and use it with `uniqFromMask`
to generate uniques.

This required some refactoring because getting a new unique from the
NameCache can't be done in pure code anymore, in particular not in an
atomic update function for `atomicModifyIORef`. So we use an MVar
instead to store the OrigNameCache field.

For some reason, T12545 increases (+1%) on i386 while it decreases on
other CI runners.

T9630 ghc/peak increases only with the dwarf build on CI (+16%).

Metric Decrease:
    T12425
    T12545
    T9198
    T12234

Metric Increase:
    T12545
    T9630

Update haddock submodule

* Make NameCache the mutable one and replace NameCacheUpdater with it

* Remove NameCache related code duplicated into haddock

Bump haddock submodule

All the comments are now captured in the AST, there is no need for a
side-channel structure for them.

This commit adds the `lint:compiler` Hadrian target to the CI runner.
It does also fixes hints in the compiler/ and libraries/base/ codebases.

The loader state was stored into HscEnv. As we need to have two
interpreters and one loader state per interpreter in #14335, it's
natural to make the loader state a field of the Interp type.

As a side effect, many functions now only require a Interp parameter
instead of HscEnv. Sadly we can't fully free GHC.Linker.Loader of HscEnv
yet because the loader is initialised lazily from the HscEnv the first
time it is used. This is left as future work.

HscEnv may not contain an Interp value (i.e. hsc_interp :: Maybe Interp).
So a side effect of the previous side effect is that callers of the
modified functions now have to provide an Interp. It is satisfying as it
pushes upstream the handling of the case where HscEnv doesn't contain an
Interpreter. It is better than raising a panic (less partial functions,
"parse, don't validate", etc.).

tuples and sums.

fixes #1257

Metric Increase:
       T10370
       parsing001

Updates haddock submodule

When we use `withTiming` we need to force the results of each timed pass
to better represent the time spent in each phase. This patch forces
some results that weren't before.

It also retrieve timings for the CoreToStg and WriteIface passes.

This new flag embeds a lookup table from the address of an info table
to information about that info table.

The main interface for consulting the map is the `lookupIPE` C function

> InfoProvEnt * lookupIPE(StgInfoTable *info)

The `InfoProvEnt` has the following structure:

> typedef struct InfoProv_{
>     char * table_name;
>     char * closure_desc;
>     char * ty_desc;
>     char * label;
>     char * module;
>     char * srcloc;
> } InfoProv;
>
> typedef struct InfoProvEnt_ {
>     StgInfoTable * info;
>     InfoProv prov;
>     struct InfoProvEnt_ *link;
> } InfoProvEnt;

The source positions are approximated in a similar way to the source
positions for DWARF debugging information. They are only approximate but
in our experience provide a good enough hint about where the problem
might be. It is therefore recommended to use this flag in conjunction
with `-g<n>` for more accurate locations.

The lookup table is also emitted into the eventlog when it is available
as it is intended to be used with the `-hi` profiling mode.

Using this flag will significantly increase the size of the resulting
object file but only by a factor of 2-3x in our experience.

It makes the external package state independent of the home unit which
is needed to make several home units share the EPS.

In GHC.Core.SimpleOpt, I found that its inlining could duplicate
an arbitary redex inside a lambda!  Consider (\xyz. x+y).  The
occurrence-analysis treats the lamdda as a group, and says that
both x and y occur once, even though the occur under the lambda-z.
See Note [Occurrence analysis for lambda binders] in OccurAnal.

When the lambda is under-applied in a call, the Simplifier is
careful to zap the occ-info on x,y, because they appear under the \z.
(See the call to zapLamBndrs in simplExprF1.)  But SimpleOpt
missed this test, resulting in #19347.

So this patch
* commons up the binder-zapping in GHC.Core.Utils.zapLamBndrs.
* Calls this new function from GHC.Core.Opt.Simplify
* Adds a call to zapLamBndrs to GHC.Core.SimpleOpt.simple_app

This change makes test T12990 regress somewhat, but it was always
very delicate, so I'm going to put up with that.

In this voyage I also discovered a small, rather unrelated infelicity
in the Simplifier:

* In GHC.Core.Opt.Simplify.simplNonRecX we should apply isStrictId
  to the OutId not the InId. See Note [Dark corner with levity polymorphism]

It may never "bite", because SimpleOpt should have inlined all
the levity-polymorphic compulsory inlnings already, but somehow
it bit me at one point and it's generally a more solid thing
to do.

Fixing the main bug increases runtime allocation in test
perf/should_run/T12990, for (acceptable) reasons explained in a
comement on

Metric Increase:
    T12990

Before this patch, the only way to override GHC's default logging
behavior was to set `log_action`, `dump_action` and `trace_action`
fields in DynFlags. This patch introduces a new Logger abstraction and
stores it in HscEnv instead.

This is part of #17957 (avoid storing state in DynFlags). DynFlags are
duplicated and updated per-module (because of OPTIONS_GHC pragma), so
we shouldn't store global state in them.

This patch also fixes a race in parallel "--make" mode which updated
the `generatedDumps` IORef concurrently.

Bump haddock submodule

The increase in MultilayerModules is tracked in #19293.

Metric Increase:
    MultiLayerModules

This commit introduces a DecoratedSDoc type which replaces the old
ErrDoc, and hopefully better reflects the intent.

Updates Haddock submodule

This commit boldly removes the ErrDoc and the MsgDoc from the codebase.
The former was introduced with the only purpose of classifying errors
according to their importance, but a similar result can be obtained just
by having a simple [SDoc], and placing bullets after each of them.

On top of that I have taken the perhaps controversial decision to also
banish MsgDoc, as it was merely a type alias over an SDoc and as such it wasn't
offering any extra type safety. Granted, it was perhaps making type
signatures slightly more "focused", but at the expense of cognitive
burden: if it's really just an SDoc, let's call it with its proper name.

This commit paves the way to a richer and more structured representation
of GHC error messages, as per GHC proposal #306. More specifically
'Messages' from 'GHC.Types.Error' now gains an extra type parameter,
that we instantiate to 'ErrDoc' for now. Later, this will allow us to
replace ErrDoc with something more structure (for example messages
coming from the parser, the typechecker etc).

This commit removes the errShortString field from the ErrMsg type,
allowing us to cleanup a lot of dynflag-dependent error functions, and
move them in a more specialised 'GHC.Driver.Errors' closer to the
driver, where they are actually used.

Metric Increase:
  T4801
  T9961

Backpack instantiations need to be typechecked to make sure that the
arguments fit the parameters. `tcRnInstantiateSignature` checks
instantiations with concrete modules, while `tcRnCheckUnit` checks
instantiations with free holes (signatures in the current modules).

Before this change, it worked that `tcRnInstantiateSignature` was called
after typechecking the argument module, see `HscMain.hsc_typecheck`,
while `tcRnCheckUnit` was called in `unsweep'` where-bound in
`GhcMake.upsweep`. `tcRnCheckUnit` was called once per each
instantiation once all the argument sigs were processed. This was done
with simple "to do" and "already done" accumulators in the fold.
`parUpsweep` did not implement the change.

With this change, `tcRnCheckUnit` instead is associated with its own
node in the `ModuleGraph`. Nodes are now:
```haskell
data ModuleGraphNode
  -- | Instantiation nodes track the instantiation of other units
  -- (backpack dependencies) with the holes (signatures) of the current package.
  = InstantiationNode InstantiatedUnit
  -- | There is a module summary node for each module, signature, and boot module being built.
  | ModuleNode ExtendedModSummary
```
instead of just `ModSummary`; the `InstantiationNode` case is the
instantiation of a unit to be checked. The dependencies of such nodes
are the same "free holes" as was checked with the accumulator before.
Both versions of upsweep on such a node call `tcRnCheckUnit`.

There previously was an `implicitRequirements` function which would
crawl through every non-current-unit module dep to look for all free
holes (signatures) to add as dependencies in `GHC.Driver.Make`. But this
is no good: we shouldn't be looking for transitive anything when
building the graph: the graph should only have immediate edges and the
scheduler takes care that all transitive requirements are met.

So `GHC.Driver.Make` stopped using `implicitRequirements`, and instead
uses a new `implicitRequirementsShallow`, which just returns the
outermost instantiation node (or module name if the immediate dependency
is itself a signature). The signature dependencies are just treated like
any other imported module, but the module ones then go in a list stored
in the `ModuleNode` next to the `ModSummary` as the "extra backpack
dependencies". When `downsweep` creates the mod summaries, it adds this
information too.

------

There is one code quality, and possible correctness thing left: In
addition to `implicitRequirements` there is `findExtraSigImports`, which
says something like "if you are an instantiation argument (you are
substituted or a signature), you need to import its things too".  This
is a little non-local so I am not quite sure how to get rid of it in
`GHC.Driver.Make`, but we probably should eventually.

First though, let's try to make a test case that observes that we don't
do this, lest it actually be unneeded. Until then, I'm happy to leave it
as is.

------

Beside the ability to use `-j`, the other major user-visibile side
effect of this change is that that the --make progress log now includes
"Instantiating" messages for these new nodes. Those also are numbered
like module nodes and count towards the total.

------

Fixes #17188

Updates hackage submomdule

Metric Increase:
    T12425
    T13035

In eb629fab I accidentally got rid of it when inlining tons of helpers.

Closes #19004

This commit renames parser's Error and Warning types (and their
constructors) to have a 'Ps' prefix, so that this would play nicely
when more errors and warnings for other phases of the pipeline will
be added. This will make more explicit which is the particular type
of error and warning we are dealing with, and will be more informative
for users to see in the generated Haddock.

The unit database cache, the home unit and the unit state were stored in
DynFlags while they ought to be stored in the compiler session state
(HscEnv). This patch fixes this.

It introduces a new UnitEnv type that should be used in the future to
handle separate unit environments (especially host vs target units).

Related to #17957

Bump haddock submodule

Some plugins can be added via TH (cf addCorePlugin). Initialize them in
the driver instead of in the Core2Core pipeline.

Loaded plugins have nothing to do in DynFlags so this patch moves them
into HscEnv (session state).

"DynFlags plugins" become "Driver plugins" to still be able to register
static plugins.

Bump haddock submodule

1) Don't modify DynFlags (too much) for -dynamic-too: now when we
   generate dynamic outputs for "-dynamic-too", we only set "dynamicNow"
   boolean field in DynFlags instead of modifying several other fields.
   These fields now have accessors that take dynamicNow into account.

2) Use DynamicTooState ADT to represent -dynamic-too state. It's much
   clearer than the undocumented "DynamicTooConditional" that was used
   before.

As a result, we can finally remove the hscs_iface_dflags field in
HscRecomp. There was a comment on this field saying:

   "FIXME (osa): I don't understand why this is necessary, but I spent
   almost two days trying to figure this out and I couldn't .. perhaps
   someone who understands this code better will remove this later."

I don't fully understand the details, but it was needed because of the
changes made to the DynFlags for -dynamic-too.

There is still something very dubious in GHC.Iface.Recomp: we have to
disable the "dynamicNow" flag at some point for some Backpack's "heinous
hack" to continue to work. It may be because interfaces for indefinite
units are always non-dynamic, or because we mix and match dynamic and
non-dynamic interfaces (#9176), or something else, who knows?

Move linker related code into GHC.Linker. Previously it was scattered
into GHC.Unit.State, GHC.Driver.Pipeline, GHC.Runtime.Linker, etc.

Add documentation in GHC.Linker

Previously, the functions in `GHC.Core.Lint` used a patchwork of
different ways to display Core Lint errors:

* `lintPassResult` (which is the source of most Core Lint errors) renders
  Core Lint errors with a distinctive banner (e.g.,
  `*** Core Lint errors : in result of ... ***`) that sets them apart
  from ordinary GHC error messages.
* `lintAxioms`, in contrast, uses a completely different code path that
  displays Core Lint errors in a rather confusing manner. For example,
  the program in #18770 would give these results:

  ```
  Bug.hs:1:1: error:
      Bug.hs:12:1: warning:
          Non-*-like kind when *-like expected: RuntimeRep
          when checking the body of forall: 'TupleRep '[r]
          In the coercion axiom Bug.N:T :: []. Bug.T ~_R Any
          Substitution: [TCvSubst
                           In scope: InScope {r}
                           Type env: [axl :-> r]
                           Co env: []]
    |
  1 | {-# LANGUAGE DataKinds #-}
    | ^
  ```
* Further digging reveals that `GHC.IfaceToCore` displays Core Lint
  errors for iface unfoldings as though they were a GHC panic. See, for
  example, this excerpt from #17723:

  ```
  ghc: panic! (the 'impossible' happened)
    (GHC version 8.8.2 for x86_64-unknown-linux):
          Iface Lint failure
    In interface for Lib
    ...
  ```

This patch makes all of these code paths display Core Lint errors and
warnings consistently. I decided to adopt the conventions that
`lintPassResult` currently uses, as they appear to have been around the
longest (and look the best, in my subjective opinion). We now use the
`displayLintResult` function for all three scenarios mentioned above.
For example, here is what the Core Lint error for the program in #18770 looks
like after this patch:

```
[1 of 1] Compiling Bug              ( Bug.hs, Bug.o )
*** Core Lint errors : in result of TcGblEnv axioms ***
Bug.hs:12:1: warning:
    Non-*-like kind when *-like expected: RuntimeRep
    when checking the body of forall: 'TupleRep '[r_axn]
    In the coercion axiom N:T :: []. T ~_R Any
    Substitution: [TCvSubst
                     In scope: InScope {r_axn}
                     Type env: [axn :-> r_axn]
                     Co env: []]
*** Offending Program ***
axiom N:T :: T = Any -- Defined at Bug.hs:12:1
*** End of Offense ***

<no location info>: error:
Compilation had errors
```

Fixes #18770.