Sylvain Henry authored 1 year ago and Marge Bot committed 1 year ago

Replace #22356 with #22349 for these tests because #22356 has been fixed
but now these tests fail because of #22349.

Rodrigo Mesquita authored 2 years ago and Marge Bot committed 1 year ago

This commit adds support for computing an inputs hash for packages
compiled by hadrian. The result is that ABI incompatible packages should
be given different hashes and therefore be distinct in a cabal store.

Hashing is enabled by the `--flag`, and is off by default as the hash
contains a hash of the source files. We enable it when we produce
release builds so that the artifacts we distribute have the right unit
ids.

Sylvain Henry authored 1 year ago and Marge Bot committed 1 year ago

Recent egrep displays the following message, breaking golden tests:

  egrep: warning: egrep is obsolescent; using grep -E

Switch to using "grep -E" instead

Matthew Pickering authored 2 years ago and Marge Bot committed 1 year ago

This patch converts all the errors to do with loading interface files
into proper structured diagnostics.

* DriverMessage: Sometimes in the driver we attempt to load an interface
  file so we embed the IfaceMessage into the DriverMessage.
* TcRnMessage: Most the time we are loading interface files during
  typechecking, so we embed the IfaceMessage

This patch also removes the TcRnInterfaceLookupError constructor which
is superceded by the IfaceMessage, which is now structured compared to
just storing an SDoc before.

jeffrey young authored 2 years ago and Marge Bot committed 2 years ago

This MR runs the testsuite for the JS backend. Note that this is a
temporary solution until !9515 is merged.

Key point: The CI runs hadrian on the built cross compiler _but not_ on
the bindist.

Other Highlights:

 - stm submodule gets a bump to mark tests as broken
 - several tests are marked as broken or are fixed by adding more
 - conditions to their test runner instance.

List of working commit messages:

CI: test cross target _and_ emulator

CI: JS: Try run testsuite with hadrian

JS.CI: cleanup and simplify hadrian invocation

use single bracket, print info

JS CI: remove call to test_compiler from hadrian

don't build haddock

JS: mark more tests as broken

Tracked in #22576

JS testsuite: don't skip sum_mod test

Its expected to fail, yet we skipped it which automatically makes it
succeed leading to an unexpected success,

JS testsuite: don't mark T12035j as skip

leads to an unexpected pass

JS testsuite: remove broken on T14075

leads to unexpected pass

JS testsuite: mark more tests as broken

JS testsuite: mark T11760 in base as broken

JS testsuite: mark ManyUnbSums broken

submodules: bump process and hpc for JS tests

Both submodules has needed tests skipped or marked broken for th JS
backend. This commit now adds these changes to GHC.

See:

HPC: hpc/hpc!21

Process: https://github.com/haskell/process/pull/268

remove js_broken on now passing tests

separate wasm and js backend ci

test: T11760: add threaded, non-moving only_ways

test: T10296a add req_c

T13894: skip for JS backend

tests: jspace, T22333: mark as js_broken(22573)

test: T22513i mark as req_th

stm submodule: mark stm055, T16707 broken for JS

tests: js_broken(22374) on unpack_sums_6, T12010

dont run diff on JS CI, cleanup

fixup: More CI cleanup

fix: align text to master

fix: align exceptions submodule to master

CI: Bump DOCKER_REV

Bump to ci-images commit that has a deb11 build with node. Required for
!9552

testsuite: mark T22669 as js_skip

See #22669

This test tests that .o-boot files aren't created when run in using the
interpreter backend. Thus this is not relevant for the JS backend.

testsuite: mark T22671 as broken on JS

See #22835

base.testsuite: mark Chan002 fragile for JS

see #22836

revert: submodule process bump

bump stm submodule

New hash includes skips for the JS backend.

testsuite: mark RnPatternSynonymFail broken on JS

Requires TH:
 - see !9779
 - and #22261

compiler: GHC.hs ifdef import Utils.Panic.Plain

Krzysztof Gogolewski authored 2 years ago and Marge Bot committed 2 years ago

- Remove unused mkWildEvBinder
- Use typeTypeOrConstraint - more symmetric and asserts that
  that the type is Type or Constraint
- Fix escape sequences in Python; they raise a deprecation warning
  with -Wdefault

Requires various submodule bumps.

Matthew Pickering authored 2 years ago and Marge Bot committed 2 years ago

Previously we were just looking at the direct imports to try and work
out what a package qualifier could apply to but #22333 pointed out we
also needed to look for reexported modules.

Fixes #22333

Add JS backend adapted from the GHCJS project by Luite Stegeman.

Some features haven't been ported or implemented yet. Tests for these
features have been disabled with an associated gitlab ticket.

Bump array submodule

Work funded by IOG.

Co-authored-by: Jeffrey Young <jeffrey.young@iohk.io>
Co-authored-by: Luite Stegeman <stegeman@gmail.com>
Co-authored-by: Josh Meredith <joshmeredith2008@gmail.com>

sheaf authored 2 years ago and Marge Bot committed 2 years ago

The testsuite output now contains diagnostic codes, so many tests need
to be updated at once.
We decided it was best to keep the diagnostic codes in the testsuite
output, so that contributors don't inadvertently make changes to the
diagnostic codes.

Ben Gamari authored 2 years ago and Marge Bot committed 2 years ago

To 0.9.0 and 4.17.0 respectively.

Bumps array, deepseq, directory, filepath, haskeline, hpc, parsec, stm,
terminfo, text, unix, haddock, and hsc2hs submodules.

(cherry picked from commit ba47b951)

In the past I improved the part of -Wunused-packages which found which
packages were used. Now I improve the part which detects which ones were
specified. The key innovation is to use the explicitUnits field from
UnitState which has the result of resolving the package flags, so we
don't need to mess about with the flag arguments from DynFlags anymore.

The output now always includes the package name and version (and the
flag which exposed it).

```
    The following packages were specified via -package or -package-id flags,
    but were not needed for compilation:
      - bytestring-0.11.2.0 (exposed by flag -package bytestring)
      - ghc-9.3 (exposed by flag -package ghc)
      - process-1.6.13.2 (exposed by flag -package process)
```

Fixes #21307

Multiple home units allows you to load different packages which may depend on
each other into one GHC session. This will allow both GHCi and HLS to support
multi component projects more naturally.

Public Interface
~~~~~~~~~~~~~~~~

In order to specify multiple units, the -unit @⟨filename⟩ flag
is given multiple times with a response file containing the arguments for each unit.
The response file contains a newline separated list of arguments.

```
ghc -unit @unitLibCore -unit @unitLib
```

where the `unitLibCore` response file contains the normal arguments that cabal would pass to `--make` mode.

```
-this-unit-id lib-core-0.1.0.0
-i
-isrc
LibCore.Utils
LibCore.Types
```

The response file for lib, can specify a dependency on lib-core, so then modules in lib can use modules from lib-core.

```
-this-unit-id lib-0.1.0.0
-package-id lib-core-0.1.0.0
-i
-isrc
Lib.Parse
Lib.Render
```

Then when the compiler starts in --make mode it will compile both units lib and lib-core.

There is also very basic support for multiple home units in GHCi, at the
moment you can start a GHCi session with multiple units but only the
:reload is supported. Most commands in GHCi assume a single home unit,
and so it is additional work to work out how to modify the interface to
support multiple loaded home units.

Options used when working with Multiple Home Units

There are a few extra flags which have been introduced specifically for
working with multiple home units. The flags allow a home unit to pretend
it’s more like an installed package, for example, specifying the package
name, module visibility and reexported modules.

-working-dir ⟨dir⟩

    It is common to assume that a package is compiled in the directory
    where its cabal file resides. Thus, all paths used in the compiler
    are assumed to be relative to this directory. When there are
    multiple home units the compiler is often not operating in the
    standard directory and instead where the cabal.project file is
    located. In this case the -working-dir option can be passed which
    specifies the path from the current directory to the directory the
    unit assumes to be it’s root, normally the directory which contains
    the cabal file.

    When the flag is passed, any relative paths used by the compiler are
    offset by the working directory. Notably this includes -i and
    -I⟨dir⟩ flags.

-this-package-name ⟨name⟩

    This flag papers over the awkward interaction of the PackageImports
    and multiple home units. When using PackageImports you can specify
    the name of the package in an import to disambiguate between modules
    which appear in multiple packages with the same name.

    This flag allows a home unit to be given a package name so that you
    can also disambiguate between multiple home units which provide
    modules with the same name.

-hidden-module ⟨module name⟩

    This flag can be supplied multiple times in order to specify which
    modules in a home unit should not be visible outside of the unit it
    belongs to.

    The main use of this flag is to be able to recreate the difference
    between an exposed and hidden module for installed packages.

-reexported-module ⟨module name⟩

    This flag can be supplied multiple times in order to specify which
    modules are not defined in a unit but should be reexported. The
    effect is that other units will see this module as if it was defined
    in this unit.

    The use of this flag is to be able to replicate the reexported
    modules feature of packages with multiple home units.

Offsetting Paths in Template Haskell splices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When using Template Haskell to embed files into your program,
traditionally the paths have been interpreted relative to the directory
where the .cabal file resides. This causes problems for multiple home
units as we are compiling many different libraries at once which have
.cabal files in different directories.

For this purpose we have introduced a way to query the value of the
-working-dir flag to the Template Haskell API. By using this function we
can implement a makeRelativeToProject function which offsets a path
which is relative to the original project root by the value of
-working-dir.

```
import Language.Haskell.TH.Syntax ( makeRelativeToProject )

foo = $(makeRelativeToProject "./relative/path" >>= embedFile)
```

> If you write a relative path in a Template Haskell splice you should use the makeRelativeToProject function so that your library works correctly with multiple home units.

A similar function already exists in the file-embed library. The
function in template-haskell implements this function in a more robust
manner by honouring the -working-dir flag rather than searching the file
system.

Closure Property for Home Units
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For tools or libraries using the API there is one very important closure
property which must be adhered to:

> Any dependency which is not a home unit must not (transitively) depend
  on a home unit.

For example, if you have three packages p, q and r, then if p depends on
q which depends on r then it is illegal to load both p and r as home
units but not q, because q is a dependency of the home unit p which
depends on another home unit r.

If you are using GHC by the command line then this property is checked,
but if you are using the API then you need to check this property
yourself. If you get it wrong you will probably get some very confusing
errors about overlapping instances.

Limitations of Multiple Home Units
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are a few limitations of the initial implementation which will be smoothed out on user demand.

    * Package thinning/renaming syntax is not supported
    * More complicated reexports/renaming are not yet supported.
    * It’s more common to run into existing linker bugs when loading a
      large number of packages in a session (for example #20674, #20689)
    * Backpack is not yet supported when using multiple home units.
    * Dependency chasing can be quite slow with a large number of
      modules and packages.
    * Loading wired-in packages as home units is currently not supported
      (this only really affects GHC developers attempting to load
      template-haskell).
    * Barely any normal GHCi features are supported, it would be good to
      support enough for ghcid to work correctly.

Despite these limitations, the implementation works already for nearly
all packages. It has been testing on large dependency closures,
including the whole of head.hackage which is a total of 4784 modules
from 452 packages.

Internal Changes
~~~~~~~~~~~~~~~~

* The biggest change is that the HomePackageTable is replaced with the
  HomeUnitGraph. The HomeUnitGraph is a map from UnitId to HomeUnitEnv,
  which contains information specific to each home unit.
* The HomeUnitEnv contains:
    - A unit state, each home unit can have different package db flags
    - A set of dynflags, each home unit can have different flags
    - A HomePackageTable
* LinkNode: A new node type is added to the ModuleGraph, this is used to
  place the linking step into the build plan so linking can proceed in
  parralel with other packages being built.
* New invariant: Dependencies of a ModuleGraphNode can be completely
  determined by looking at the value of the node. In order to achieve
  this, downsweep now performs a more complete job of downsweeping and
  then the dependenices are recorded forever in the node rather than
  being computed again from the ModSummary.
* Some transitive module calculations are rewritten to use the
  ModuleGraph which is more efficient.
* There is always an active home unit, which simplifies modifying a lot
  of the existing API code which is unit agnostic (for example, in the
  driver).

The road may be bumpy for a little while after this change but the
basics are well-tested.

One small metric increase, which we accept and also submodule update to
haddock which removes ExtendedModSummary.

Closes #10827

-------------------------
Metric Increase:
    MultiLayerModules
-------------------------

Co-authored-by: Fendor <power.walross@gmail.com>

Ben Gamari authored 3 years ago and Marge Bot committed 3 years ago

Previously it was unclear whether req_shared_libs should require:

 * that the platform supports dynamic library loading,
 * that GHC supports dynamic linking of Haskell code, or
 * that the dyn way libraries were built

Clarify by splitting the predicate into two:

 * `req_dynamic_lib_support` demands that the platform support dynamic
   linking
 * `req_dynamic_hs` demands that the GHC support dynamic linking of
   Haskell code on the target platform

Naturally `req_dynamic_hs` cannot be true unless
`req_dynamic_lib_support` is also true.

Matthew Pickering authored 3 years ago and Marge Bot committed 3 years ago

Sebastian unfortunately wrote a very long commit message in !5667 which
caused `xargs` to fail on windows because the environment was too big.
Fortunately `xargs` and `rm` don't need anything from the environment so
just run those commands in an empty environment (which is what env -i
achieves).

Matthew Pickering authored 3 years ago and Marge Bot committed 3 years ago

Before we would check for the unused package warning even if the module
graph was compromised due to an error in downsweep. This is easily
fixed by pushing warmUnusedPackages into depanalE, and then returning
the errors like the other downsweep errors.

Fixes #20242

Matthew Pickering authored 3 years ago and Marge Bot committed 3 years ago

Closes #18567

Matthew Pickering authored 3 years ago and Marge Bot committed 3 years ago

Spurious warnings were previously emitted if an import came from a
reexport due to how -Wunused-packages were implemented. Removing the
dependency would cause compilation to fail.

The fix is to reimplement the warning a bit more directly, by searching
for which package each import comes from using the normal module finding
functions rather than consulting the EPS. This has the advantage that
the check could be performed at any time after downsweep rather than
also relying on a populated EPS.

Fixes #19518 and #19777

Matthew Pickering authored 3 years ago and Marge Bot committed 3 years ago

Before we would get the incorrect error message saying that the
rexporting package was the same as the defining package.

I think this only affects error messages for now.

```
-      it is bound as p-0.1.0.0:P2 by a reexport in package p-0.1.0.0
-      it is bound as P by a reexport in package p-0.1.0.0
+      it is bound as p-0.1.0.0:P2 by a reexport in package q-0.1.0.0
+      it is bound as P by a reexport in package r-0.1.0.0
```

and the output of `-ddump-mod-map` claimed..

```
Moo moo-0.0.0.1 (hidden package, reexport by moo-0.0.0.1)
```

Reverts many of the testsuite changes

Demand cabal 2.0 syntax instead of >= 1.20 as required by newer cabal versions.

Updates a variety of tests as Cabal is now more strict about Cabal file
form.

Matthew Pickering authored 5 years ago and Marge Bot committed 5 years ago

This patch implements overloaded quotation brackets which generalise the
desugaring of all quotation forms in terms of a new minimal interface.

The main change is that a quotation, for example, [e| 5 |], will now
have type `Quote m => m Exp` rather than `Q Exp`. The `Quote` typeclass
contains a single method for generating new names which is used when
desugaring binding structures.

The return type of functions from the `Lift` type class, `lift` and `liftTyped` have
been restricted to `forall m . Quote m => m Exp` rather than returning a
result in a Q monad.

More details about the feature can be read in the GHC proposal.

https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0246-overloaded-bracket.rst

Ben Gamari authored 5 years ago and Marge Bot committed 5 years ago

Metric Increase:
    haddock.Cabal

Michael Peyton Jones authored 6 years ago and Marge Bot committed 5 years ago

Fixes #16228. The included test case is adapted from the reproduction in
the issue, and fails without this patch.

------

We compute an initial visilibity mapping for units based on what is
present in the package databases. To seed this, we compute a set of all
the package configs to add visibilities for.

However, this set was keyed off the unit's *package name*. This is
correct, since we compare packages across databases by version. However,
we would only ever consider a single, most-preferable unit from the
database in which it was found.

The effect of this was that only one of the libraries in a Cabal package
would be added to this initial set. This would cause attempts to use
modules from the omitted libraries to fail, claiming that the package
was hidden (even though `ghc-pkg` would correctly show it as visible).

A solution is to do the selection of the most preferable packages
separately, and then be sure to consider exposing all units in the
same package in the same package db. We can do this by picking a
most-preferable unit for each package name, and then considering
exposing all units that are equi-preferable with that unit.

------

Why wasn't this bug apparent to all people trying to use sub-libraries
in Cabal? The answer is that Cabal explicitly passes `-package` and
`-package-id` flags for all the packages it wants to use, rather than
relying on the state of the package database. So this bug only really
affects people who are trying to use package databases produced by Cabal
outside of Cabal itself.

One particular example of this is the way that the
Nixpkgs Haskell infrastructure provides wrapped GHCs: typically these
are equipped with a package database containing all the needed
package dependencies, and the user is not expected to pass
`-package` flags explicitly.

Ben Gamari authored 6 years ago and Marge Bot committed 6 years ago

It doesn't fail reliably.

Ben Gamari authored 6 years ago and Marge Bot committed 6 years ago

See #16386.

This eliminates most uses of run_command in the testsuite in favor of the more
structured makefile_test.

This reverts commit 76c8fd67.

Herbert Valerio Riedel authored 6 years ago and Ben Gamari committed 6 years ago

This also requires adapting `ghc-pkg` to use the new Cabal parsing API
as the old ReadP-based one has finally been evicted for good.

Hadrian bit finished by: Ben Gamari <ben@smart-cactus.org>

Zejun Wu authored 6 years ago and Ben Gamari committed 6 years ago

Summary:
We saw following errors:

```
$ cabal install --disable-library-vanilla --disable-shared --enable-library-profiling
hashable-1.2.7.0: cannot find any of
["libHShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ.a",
 "libHShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ.p_a",
 "libHShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ-ghc8.4.3.so",
 "libHShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ-ghc8.4.3.dylib",
 "HShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ-ghc8.4.3.dll"]
```

This is because ghc-pkg is looking for
`libHShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ.p_a` instead of
`libHShashable-1.2.7.0-Q2TKVDwk4GBEHmizb4teZ_p.a`.

Test Plan: ./validate

Reviewers: simonmar, bgamari

Reviewed By: simonmar

Subscribers: rwbarton, carter

Differential Revision: https://phabricator.haskell.org/D5234

Several tests were failing in DEBUG mode, but fixing this
was easy: just pass $(TEST_HC_OPTS) in the relevant
Makefiles.

Tobias Dammers authored 6 years ago and Ben Gamari committed 6 years ago

When inferring the correct abi-depends, we now look at all the package
databases in the stack, up to and including the current one, because
these are the ones that the current package can legally depend on. While
doing so, we will issue warnings:

- In verbose mode, we warn about every package that declares
  abi-depends:, whether we actually end up overriding them with the
  inferred ones or not ("possibly broken abi-depends").

- Otherwise, we only warn about packages whose declared abi-depends
  does not match what we inferred ("definitely broken abi-depends").

Reviewers: bgamari

Reviewed By: bgamari

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14381

Differential Revision: https://phabricator.haskell.org/D4729