Skip to content

Changes

Page history
Edit Design/BuildSystem authored by Simon Peyton Jones's avatar Simon Peyton Jones
Hide whitespace changes
Inline Side-by-side
design/build-system.md
View page @ 4f113f1c
......@@ -77,7 +77,10 @@ snapshot tarball:
from source to .a/.so, including preprocessing (hsc2hs et. al.).
- Use Cabal to preprocess the .cabal file and generate metadata in the
form of Makefile bindings for our build system to use.
form of Makefile bindings for our build system to use. We plan to
generate **only** Makefile bindings this way; we will not generate Makefile
**rules** or shell commands. All Makefile code and shell commands will be
in plain honest-to-goodness Makefiles.
- Use Cabal to generate the InstalledPackageInfo.
......@@ -85,8 +88,8 @@ snapshot tarball:
- Use Cabal for Haddocking, and anything else we need to do.
>
> The advantages of this are:
The advantages of this are:
- The critical parts of the build system are under our control, and are
easily modifiable.
......@@ -110,65 +113,70 @@ snapshot tarball:
- Compared to the pre-Cabal build system, we're not duplicating the
package metadata or the code that processes it, only the build rules.
## Detailed plan
## Avoiding recursive makefiles
- Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/
- The version of Cabal used by ghc-cabal.hs does not need to be an
up-to-the-minute bleeding-edge version. It should be stable and vary
slowly. We suck a new version of Cabal into the GHC build system
manually, rather than mirroring the Cabal HEAD.
Currently, with recursive make, this means we jump around between
Makefiles a lot, which isn't good for parallelism in the build. For example, the current build order looks like this:
- Rather than installing things in-place all over the build tree, we
will have a single inplace directory at the root of the tree. The
structure inside this directory will match that of the normal install,
which will simplify various things. There are two slight wrinkles:
- With bootstrapping compiler:
- The tree will not be complete; for example, the libraries will be
registered in-place in their dist directories
- Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage\[123\]
Tools like genprimopcode, genapply etc. will probably also go into
inplace/bin, in order to make the makefiles more consistent.
- Build libraries/{filepath,Cabal}
- Build utils/ghc-cabal
- With bootstrapping compiler and ghc-cabal:
- The build order looks something like:
- Build utils/hsc2hs
- Build libraries/hpc
- Build compiler (stage 1)
- With stage 1:
- With bootstrapping compiler:
- Build libraries/\*
- Build utils/\* (except haddock)
- Build compiler (stage 2)
- With stage 2:
- Build libraries/{filepath,Cabal}
- Build utils/ghc-cabal
- With bootstrapping compiler and ghc-cabal:
- Build utils/haddock
- Build compiler (stage 3)
- With haddock:
- Build utils/hsc2hs
- Build libraries/hpc
- Build compiler (stage 1)
- With stage 1:
- libraries/\*
- compiler
- Build libraries/\*
- Build utils/\* (except haddock)
- Build compiler (stage 2)
- With stage 2:
- Build utils/haddock
- Build compiler (stage 3)
- With haddock:
Instead, following [ Recursive make considered harmful](http://miller.emu.id.au/pmiller/books/rmch/?ref=DDiyet.Com), we propose to move all the logic and dependencies into the root
Makefile (or files that get included into it) so that make sees all of
it together. Advantages:
- libraries/\*
- compiler
- We get to specify dependencies between different parts of the tree much
more easily and precisely. BIG WIN. Have you noticed how often you
need to `make distclean` in a GHC tree to make sure everything is up to
date? (well I rarely do this, because I have the dependencies in my
head and I can rebuild manually, but I imagine this isn't the case for
most people!)
Currently, with recursive make, this means we jump around between
Makefiles a lot, which isn't good for parallelism in the build.
Instead, we want to move all the logic and dependencies into the root
Makefile (or files that get included into it) so that make sees all of
it together.
- We get more parallelism, more easily. I'd argue this is a big win too,
right now a validate only uses about 1.3 out of 2 cores.
> >
> > One concern is that make may take a long time thinking if it can see
> > the rules for the whole system, even when only asked to build a single
> > file. We will have to see how well it performs in practice.
- But we still want "make" to work in subdirectories, so for example the
Makefile (actually GNUmakefile, to avoid colliding with Makefile in
libraries like Cabal) in libraries/base might look like
We don't lose modularity: different parts of the build system are still in
different files, it's just that make sees them all at once. Right now
every make invocation already reads thousands of lines of boilerplate
Makefile code, and we'd be invoking make only once rather than many times.
So we will probably have to worry about efficiency. For example, it takes
tens of seconds on Windows for make to discover that compiler/ is up to
date. We don't want that happening every time you rebuild some small part
of the tree, so we plan to cut a few dependencies on purpose. But I'm
hoping it'll be necessary to do this in a few well-defined places only, and
only when building in subdirectories. For example, If you say 'make' in
libraries/base, then we won't try to rebuild the stage1 compiler, we'll
just fail if it does't exist.
We still want "make" to work in subdirectories, so for example the
Makefile (actually GNUmakefile, to avoid colliding with Makefile in
libraries like Cabal) in libraries/base might look like
```wiki
.NOTPARALLEL
......@@ -182,18 +190,38 @@ default: dist/build/libbase.a
$(MAKE) -C ../.. libraries/base/$@
```
> >
> > (ghc.mk is discussed later). In actual fact, GNUmakefile will want to
> > be more complicated, to handle "make way=v", "make way=p", "make doc",
> > etc. Where possible, the make code will be "include"d in, rather than
> > generated, so as to make it easier to deal with.
> >
> > We need the .NOTPARALLEL or if you say "make foo bar" then the two
> > recursive make calls might both make "quux" (a dependency of foo and
> > bar) at the same time. The main Makefile will be able to do work in
> > parallel when building each of foo and bar, though. The common case,
> > where you only specify 0 or 1 targets, doesn't lose any parallelism.
(ghc.mk is discussed later). In actual fact, GNUmakefile will want to
be more complicated, to handle "make way=v", "make way=p", "make doc",
etc. Where possible, the make code will be "include"d in, rather than
generated, so as to make it easier to deal with.
We need the .NOTPARALLEL or if you say "make foo bar" then the two
recursive make calls might both make "quux" (a dependency of foo and
bar) at the same time. The main Makefile will be able to do work in
parallel when building each of foo and bar, though. The common case,
where you only specify 0 or 1 targets, doesn't lose any parallelism.
## Detailed plan
- Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/
- The version of Cabal used by ghc-cabal.hs does not need to be an
up-to-the-minute bleeding-edge version. It should be stable and vary
slowly. We suck a new version of Cabal into the GHC build system
manually, rather than mirroring the Cabal HEAD.
- Rather than installing things in-place all over the build tree, we
will have a single inplace directory at the root of the tree. The
structure inside this directory will match that of the normal install,
which will simplify various things. There are two slight wrinkles:
- The tree will not be complete; for example, the libraries will be
registered in-place in their dist directories
- Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage\[123\]
Tools like genprimopcode, genapply etc. will probably also go into
inplace/bin, in order to make the makefiles more consistent.
- In e.g. utils/ghc-pkg, the default target will be
......