README.md 14.5 KB
Newer Older
1 2 3
Hadrian
=======

Andrey Mokhov's avatar
Andrey Mokhov committed
4 5 6 7
Hadrian is a new build system for the [Glasgow Haskell Compiler][ghc]. It is
based on the [Shake][shake] library and we hope that it will replace the current
[Make-based build system][make] starting from GHC 8.8. If you are curious about
the rationale behind the
8 9 10 11 12 13 14 15 16 17
project and the architecture of the build system you can find more details in
this [Haskell Symposium 2016 paper](https://dl.acm.org/authorize?N41275) and this
[Haskell eXchange 2016 talk][talk].

The new build system can work side-by-side with the existing build system, since it
places all build artefacts in a dedicated directory (called `_build` by default).

Your first build
----------------

Andrey Mokhov's avatar
Andrey Mokhov committed
18
Hadrian has not entirely caught up with the Make build system. Not afraid?
19 20
Then put on the helmet and run the following commands from the root of the GHC
tree:
21 22

```
23
./boot && ./configure
24
hadrian/build -j
25 26 27 28 29
```

or on Windows:

```
30
./boot && ./configure --enable-tarballs-autodownload
31 32 33
hadrian/build.bat -j
```

34 35 36 37
Here flag `-j` enables parallelism and is optional. We will further refer to the
build script simply as `build`. Note that Hadrian can also run the `boot` and
`configure` scripts automatically for you if you pass the flag `--configure`,
or simply `-c`. See the overview of command line flags below.
38 39 40

Notes:

41
* If the default build script doesn't work, you might want to try another one,
42 43
e.g. based on Cabal sandboxes (`build-cabal.*`), Stack (`build-stack.*`) or Nix
(`build-stack-nix.*`).
44 45 46 47 48

* On Windows, if you do not want to install MSYS, you can use the Stack-based
build script (Stack provides a managed MSYS environment), as described in
[these instructions][windows-build]. If you don't mind installing MSYS yourself
or already have it, you can use the Cabal-based build script.
49

50 51 52 53 54
* Hadrian is written in Haskell and depends on `shake` (plus a few packages that
`shake` depends on), `mtl`, `quickcheck`, and GHC core libraries.

* If you have never built GHC before, start with the
[preparation guide][ghc-preparation].
55

56 57 58 59 60 61 62
Cheatsheet for Make users
-------------------------

You can find a cheatsheet-style document that shows the Hadrian equivalents of
commands that GHC users are used to run with the Make build system
[here](doc/make.md).

63 64
Using the build system
----------------------
65 66
Once your first build is successful, simply run `build` to rebuild after some
changes. Build results are placed into `_build` by default.
67 68 69 70 71 72

#### Command line flags

In addition to standard Shake flags (try `--help`), the build system
currently supports several others:

73 74 75
* `--build-root=PATH` or `-oPATH`: specify the directory in which you want to
store all build products. By default Hadrian builds everything in the `_build/`
subdirectory of the GHC source tree. Unlike the Make build system, Hadrian
Andrey Mokhov's avatar
Andrey Mokhov committed
76
doesn't have any "inplace" logic left any more. This option is therefore useful
77 78 79
for GHC developers who want to build GHC in different ways or at different
commits, from the same source directory, and have the build products sit in
different, isolated folders.
80 81

* `--configure` or `-c`: use this flag to run the `boot` and `configure` scripts
82 83
automatically, so that you don't have to remember to run them manually as you
normally do when using Make (typically only in the first build):
84 85 86 87
    ```bash
    ./boot
    ./configure # On Windows run ./configure --enable-tarballs-autodownload
    ```
Andrey Mokhov's avatar
Andrey Mokhov committed
88 89 90 91 92 93 94 95
    Beware that with this flag Hadrian may do network I/O on Windows to download
    necessary tarballs, which may sometimes be undesirable.

* `--flavour=FLAVOUR`: choose a build flavour. The following settings are
currently supported: `default`, `quick`, `quickest`, `perf`, `prof`, `devel1`
and `devel2`. As an example, the `quickest` flavour adds `-O0` flag to all GHC
invocations and builds libraries only in the `vanilla` way, which speeds up
builds by 3-4x. Build flavours are documented
96
[here](https://gitlab.haskell.org/ghc/ghc/blob/master/hadrian/doc/flavours.md).
97

Andrey Mokhov's avatar
Andrey Mokhov committed
98 99 100 101 102
* `--freeze1`: freeze Stage1 GHC, i.e. do not rebuild it even if some of its
source files are out-of-date. This allows to significantly reduce the rebuild
time when you are working on a feature that affects both Stage1 and Stage2
compilers, but may lead to incorrect build results. To unfreeze Stage1 GHC
simply drop the `--freeze1` flag and Hadrian will rebuild all out-of-date files.
103

104 105 106
* `--freeze2`: just like `--freeze1` but tell Hadrian to additionally freeze
Stage2 GHC.

107 108
* `--skip-depends`: skips rebuilding Haskell module dependency files.

Andrey Mokhov's avatar
Andrey Mokhov committed
109 110
* `--integer-simple`: build GHC using the `integer-simple` integer library
(instead of `integer-gmp`).
111

112 113
* `--color` and `--no-color`: choose whether to use colors when printing build
progress info. By default, Hadrian tries to determine if the terminal supports
114
colored output, and proceeds accordingly.
115

Andrey Mokhov's avatar
Andrey Mokhov committed
116 117 118 119
* `--progress-info=STYLE`: choose how build progress info is printed. There are
four settings: `none`, `brief` (one line per build command; this is the default
setting), `normal` (typically a box per build command), and `unicorn` (when
`normal` just won't do).
120

Andrey Mokhov's avatar
Andrey Mokhov committed
121 122
* `--verbose`: run Hadrian in verbose mode. In particular this prints diagnostic
messages by Shake oracles.
123

124 125 126
* `--lint`: run [Shake Lint](https://shakebuild.com/manual#lint) during the
build to check that the build system is well formed. Note that the Lint check
currently fails under certain circumstances, as discussed in
127
[this ticket](https://gitlab.haskell.org/ghc/ghc/issues/15971).
128

129 130 131 132 133
#### Expressions

Hadrian expressions are used extensively for specifying build settings. For an
explanation of how they work, see the [documentation](doc/expressions.md).

134 135
#### User settings

Andrey Mokhov's avatar
Andrey Mokhov committed
136 137 138
The Make-based build system uses `mk/build.mk` to specify user build settings.
Hadrian uses `hadrian/UserSettings.hs` for the same purpose, see
[documentation](doc/user-settings.md).
139

140 141
#### Building libraries and executables

Andrey Mokhov's avatar
Andrey Mokhov committed
142
You can build a specific library or executable for a given stage by running
143 144
`build stage<N>:<lib|exe>:<package name>`. Examples:

Andrey Mokhov's avatar
Andrey Mokhov committed
145 146 147
```sh
# Build the Stage1 GHC compiler, and place the binary to the directory
# _build/stage0/bin/ghc (because it is built by the Stage0 compiler).
148 149
build stage1:exe:ghc-bin

Andrey Mokhov's avatar
Andrey Mokhov committed
150 151
# Build the Stage2 GHC compiler, and place the binary to the directory
# _build/stage1/bin/ghc (because it is built by the Stage1 compiler).
152 153
build stage2:exe:ghc-bin

Andrey Mokhov's avatar
Andrey Mokhov committed
154 155
# Build the `ghc` library with the bootstrapping (Stage0) compiler, and register
# it in the package database stored in the directory _build/stage0/lib.
156 157
build stage0:lib:ghc

Andrey Mokhov's avatar
Andrey Mokhov committed
158 159
# Build the Cabal library with the Stage1 compiler and register it
# in the package database stored in the directory _build/stage1/lib.
160

Andrey Mokhov's avatar
Andrey Mokhov committed
161 162
# Build the `text` library with the Stage2 compiler and register it
# in the package database stored in the directory _build/stage2/lib.
163 164
build stage2:lib:text

Andrey Mokhov's avatar
Andrey Mokhov committed
165 166 167
# Build Haddock using the Stage1 compiler and place the binary into the
# directory _build/stage1/haddock.
build stage1:exe:haddock
168 169
```

170 171
#### Fast feedback using ghci

172
Running the `./hadrian/ghci` script will load the main compiler into
173 174 175 176
a ghci session. This is intended for fast development feedback, modules are only
typechecked so it isn't possible to run any functions in the repl.

```
177
./hadrian/ghci
178 179 180 181 182
```

You can also use this target with `ghcid`.

```
183
ghcid --command="./hadrian/ghci"
184 185 186 187 188 189 190
```

The first time this command is run hadrian will need to compile a few dependencies
which will take 1-2 minutes. Loading GHC into GHCi itself takes about 30 seconds and
reloads after that take in the region of 1-5 seconds depending on which modules
need to be recompiled.

191 192 193 194 195
You can further speed up the script by passing `-j` as an argument. This will
have the effect of passing `-j` to both hadrian and ghci so they will both
build in parallel.

```
196
./hadrian/ghci -j8
197 198
```

199 200
#### Testing

Andrey Mokhov's avatar
Andrey Mokhov committed
201 202 203 204
To run GHC testsuite, use `build test`. See
[doc/testsuite.md](doc/testsuite.md) to learn about all associated command line
flags, as well as about the equivalents of the features that the Make build
system offers.
205

Andrey Mokhov's avatar
Andrey Mokhov committed
206
`build selftest` runs tests of the build system. The current test coverage
207 208
is close to zero (see [#197][test-issue]).

209 210 211 212
#### Clean and full rebuild

* `build clean` removes all build artefacts.

Andrey Mokhov's avatar
Andrey Mokhov committed
213
* `build -B` forces Shake to rerun all rules, even if the previous build results
214 215
are still up-to-date.

216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241
#### Staged compilation

GHC is a self-hosted compiler and consequently the build proceeds in several
stages:

1. The build begins with a user-provided installation of GHC called the
   stage0 (or bootstrap) compiler which is used (via the `build.*.sh` scripts)
   to build Hadrian.
1. Hadrian uses the stage0 compiler to build a stage1 compiler (somewhat
   confusingly found in `_build/stage0/bin/ghc`), linking against the stage0
   compiler's core libraries (e.g. `base`).
1. The stage1 compiler is used to build new core libraries (found in
   `_build/stage1/lib`).
1. The stage1 compiler is used to build a stage2 compiler (found in
   `_build/stage1/bin/ghc`), linking against these new core libraries.
1. Optionally (see the [Building Stage3](#building-stage3) section below) the
   stage2 compiler can be used to build a stage3 compiler (found in
   `build/stage2/bin/ghc`) as a further smoke-test.

Note that the stage directories in the `_build` directory can be thought of as
named after the stage that was used to *build* the artifacts in each directory.

These stages can be summarized graphically:

![an overview of the stages of a Hadrian compilation](doc/staged-compilation.svg)

242 243
#### Documentation

Andrey Mokhov's avatar
Andrey Mokhov committed
244 245 246
To build GHC documentation, run `build docs`. Note that finer-grain
documentation targets (e.g. building only HTML documentation or only the GHC
User's Guide) are currently not supported.
247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262

#### Source distribution

To build a GHC source distribution tarball, run `build source-dist`.

#### Binary distribution

To build a GHC binary distribution, run `build binary-dist`. The resulting
tarball contains just enough to support the

``` sh
$ ./configure [--prefix=PATH] && make install
```

workflow, for now.

263 264 265 266 267 268 269 270 271 272 273 274 275
### Building and installing GHC

You can get Hadrian to build _and_ install a binary distribution in one go
with the following command:

``` sh
$ build install --prefix=/some/absolute/path
```

This builds everything that would be shipped in a bindist, without creating
the archive, and just runs `./configure --prefix=PATH` and `make install`
to get GHC installed installed at `/some/absolute/path`.

Andrey Mokhov's avatar
Andrey Mokhov committed
276
#### Building Stage3
277

Andrey Mokhov's avatar
Andrey Mokhov committed
278 279 280 281
It is possible to define a build flavour that builds a Stage3 compiler, which is
a compiler built using Stage2. This is useful for cross-compilation. Detailed
instructions can be found in the corresponding
[part of the user settings manual](doc/user-settings.md#specifying-the-final-stage-to-build).
282

283 284 285 286 287 288 289 290 291 292
#### Integrating Hadrian into other tooling

The `tool-args` target is designed to allow hadrian to be integrated into other
tooling which uses the GHC API.
`tool-args` prints out a list of flags which hadrian will use to compile
a module in the `compiler` directory. Using these flags you can then set up
a GHC API session with the correct environment to load a module into your own
GHC session. This is how `haskell-ide-engine` is able to support hadrian.

```
293
> ./hadrian/build tool-args
294 295 296 297
-hide-all-packages -no-user-package-db -package-db _build/stage0/lib/packag...
```


298
The `./hadrian/ghci` script is implemented using this target.
299

300 301 302 303
Troubleshooting
---------------

Here are a few simple suggestions that might help you fix the build:
304

Andrey Mokhov's avatar
Andrey Mokhov committed
305 306
* If Hadrian fails with the message
  `Configuration file hadrian/cfg/system.config is missing`, you have probably
307
  forgotten to pass the `--configure` flag during the first build.
308

309
* With the `--configure` (`-c`) flag, Hadrian sometimes fetches a wrong
310
  Happy saying something like: `HappyTemplate-arrays-coerce: openFile: does not exist`
311
  (as mentioned [here](https://github.com/haskell/cabal/issues/5867)), in
312 313
  which case you might be better off running `./configure` manually before Hadrian.

314 315 316 317
* The call to `build test` sometimes fails with
  `libCffi_p.a: copyFile: does not exist` (as noticed
  [here](https://gitlab.haskell.org/ghc/ghc/issues/15877#note_166739)).
  The workaround is to `rm _build/stage1/libffi/build/inst/lib/libffi.a` and
318 319
  start over.

320
If you need help in debugging Hadrian, read the
321 322 323
[wiki](https://github.com/snowleopard/hadrian/wiki)
and Shake's [debugging tutorial](https://shakebuild.com/debugging).
If nothing helps, don't hesitate to create a GHC issue.
324 325 326 327 328 329 330 331 332

Current limitations
-------------------
The new build system still lacks many important features:
* Dynamic linking on Windows is not supported [#343][dynamic-windows-issue].

How to contribute
-----------------

Andrey Mokhov's avatar
Andrey Mokhov committed
333 334 335 336 337 338
The best way to contribute is to try Hadrian, report the issues you found, and
attempt to fix them. Documentation patches are particularly welcome. Please
note: the codebase is still unstable and we expect some further refactoring. In
particular, we would like to separate the general-purpose "Hadrian library" for
building Haskell projects using Shake (`src/Hadrian/*`) from GHC specifics; see
[this issue](https://github.com/snowleopard/hadrian/issues/347).
339 340 341 342

Acknowledgements
----------------

Andrey Mokhov's avatar
Andrey Mokhov committed
343 344 345 346 347 348 349 350 351 352 353 354
The list of people who helped create Hadrian is long, and we hope that it will
soon become even longer! The project was first developed in a separate GitHub
repository, where you can find the list of original
[contributors][contributors]. They had to stare at Makefiles for months, so give
them all a round of applause. Simon Peyton Jones and Simon Marlow helped with
deciphering these Makefiles, and Neil Mitchell patiently explained how to
translate Makefiles to (much nicer) Shakefiles.

The initial development of Hadrian was funded by Newcastle University, EPSRC and
Microsoft Research. Other organisations that contributed at various stages of
the project are Haskell.Org and Google (both through supporting summer student
projects), as well as Well-Typed.
355 356 357

[ghc]: https://en.wikipedia.org/wiki/Glasgow_Haskell_Compiler
[shake]: https://github.com/ndmitchell/shake
358
[make]: https://gitlab.haskell.org/ghc/ghc/wikis/building/architecture
359 360
[talk]: https://skillsmatter.com/skillscasts/8722-meet-hadrian-a-new-build-system-for-ghc
[issues]: https://github.com/snowleopard/hadrian/issues
361 362
[ghc-preparation]: https://gitlab.haskell.org/ghc/ghc/wikis/building/preparation
[ghc-windows-quick-build]: https://gitlab.haskell.org/ghc/ghc/wikis/building/preparation/windows#AQuickBuild
363
[windows-build]: https://gitlab.haskell.org/ghc/ghc/blob/master/hadrian/doc/windows.md
364 365 366 367
[test-issue]: https://github.com/snowleopard/hadrian/issues/197
[dynamic-windows-issue]: https://github.com/snowleopard/hadrian/issues/343
[bin-dist-issue]: https://github.com/snowleopard/hadrian/issues/219
[contributors]: https://github.com/snowleopard/hadrian/graphs/contributors