nix-local-build.rst 10.6 KB
Newer Older
Leonid Onokhov's avatar
Leonid Onokhov committed
1
.. highlight:: console
2 3 4 5 6

Quickstart
==========

Suppose that you are in a directory containing a single Cabal package
7
which you wish to build (if you haven't set up a package yet check
8
out `developing packages <developing-packages.html>`__ for
9
instructions). You can configure and build it using Nix-style
10 11 12 13
local builds with this command (configuring is not necessary):

::

14
    $ cabal v2-build
15 16 17 18 19

To open a GHCi shell with this package, use this command:

::

20
    $ cabal v2-repl
21

Francesco Gazzetta's avatar
Francesco Gazzetta committed
22 23 24 25
To run an executable defined in this package, use this command:

::

26
    $ cabal v2-run <executable name> [executable args]
Francesco Gazzetta's avatar
Francesco Gazzetta committed
27

28 29 30 31 32 33 34 35 36 37 38
Developing multiple packages
----------------------------

Many Cabal projects involve multiple packages which need to be built
together. To build multiple Cabal packages, you need to first create a
``cabal.project`` file which declares where all the local package
directories live. For example, in the Cabal repository, there is a root
directory with a folder per package, e.g., the folders ``Cabal`` and
``cabal-install``. The ``cabal.project`` file specifies each folder as
part of the project:

Leonid Onokhov's avatar
Leonid Onokhov committed
39
.. code-block:: cabal
40 41 42 43 44 45 46 47 48 49

    packages: Cabal/
              cabal-install/

The expectation is that a ``cabal.project`` is checked into your source
control, to be used by all developers of a project. If you need to make
local changes, they can be placed in ``cabal.project.local`` (which
should not be checked in.)

Then, to build every component of every package, from the top-level
50
directory, run the command: (using cabal-install-2.0 or greater.)
51 52 53

::

54
    $ cabal v2-build all
55

56
To build a specific package, you can either run ``v2-build`` from the
57 58 59 60
directory of the package in question:

::

Leonid Onokhov's avatar
Leonid Onokhov committed
61
    $ cd cabal-install
62
    $ cabal v2-build
63 64

or you can pass the name of the package as an argument to
65
``cabal v2-build`` (this works in any subdirectory of the project):
66 67 68

::

69
    $ cabal v2-build cabal-install
70 71 72 73 74 75

You can also specify a specific component of the package to build. For
example, to build a test suite named ``package-tests``, use the command:

::

76
    $ cabal v2-build package-tests
77 78 79 80 81 82 83

Targets can be qualified with package names. So to request
``package-tests`` *from* the ``Cabal`` package, use
``Cabal:package-tests``.

Unlike sandboxes, there is no need to setup a sandbox or ``add-source``
projects; just check in ``cabal.project`` to your repository and
84
``v2-build`` will just work.
85

86 87 88 89 90 91 92 93 94 95 96
Cookbook
========

How can I profile my library/application?
-----------------------------------------

Create or edit your ``cabal.project.local``, adding the following
line::

    profiling: True

97
Now, ``cabal v2-build`` will automatically build all libraries and
98 99 100 101 102 103
executables with profiling.  You can fine-tune the profiling settings
for each package using :cfg-field:`profiling-detail`::

    package p
        profiling-detail: toplevel-functions

104
Alternately, you can call ``cabal v2-build --enable-profiling`` to
105 106
temporarily build with profiling.

107 108 109 110 111 112 113 114 115
How it works
============

Local versus external packages
------------------------------

One of the primary innovations of Nix-style local builds is the
distinction between local packages, which users edit and recompile and
must be built per-project, versus external packages, which can be cached
116
across projects. To be more precise:
117 118 119 120 121

1. A **local package** is one that is listed explicitly in the
   ``packages``, ``optional-packages`` or ``extra-packages`` field of a
   project. Usually, these refer to packages whose source code lives
   directly in a folder in your project (although, you can list an
122
   arbitrary Hackage package in ``extra-packages`` to force it to be
123 124 125 126 127 128 129 130
   treated as local).

Local packages, as well as the external packages (below) which depend on
them, are built **inplace**, meaning that they are always built
specifically for the project and are not installed globally. Inplace
packages are not cached and not given unique hashes, which makes them
suitable for packages which you want to edit and recompile.

131
2. An **external package** is any package which is not listed in the
132 133 134 135 136 137 138 139 140 141 142 143
   ``packages`` field. The source code for external packages is usually
   retrieved from Hackage.

When an external package does not depend on an inplace package, it can
be built and installed to a **global** store, which can be shared across
projects. These build products are identified by a hash that over all of
the inputs which would influence the compilation of a package (flags,
dependency selection, etc.). Just as in Nix, these hashes uniquely
identify the result of a build; if we compute this identifier and we
find that we already have this ID built, we can just use the already
built version.

144 145
The global package store is ``~/.cabal/store`` (configurable via
global `store-dir` option); if you need to clear your store for
Alex Biehl's avatar
Alex Biehl committed
146
whatever reason (e.g., to reclaim disk space or because the global
147
store is corrupted), deleting this directory is safe (``v2-build``
Alex Biehl's avatar
Alex Biehl committed
148
will just rebuild everything it needs on its next invocation).
149 150

This split motivates some of the UI choices for Nix-style local build
151
commands. For example, flags passed to ``cabal v2-build`` are only
152
applied to *local* packages, so that adding a flag to
153
``cabal v2-build`` doesn't necessitate a rebuild of *every* transitive
154 155
dependency in the global package store.

156
In cabal-install 2.0 and above, Nix-style local builds also take advantage of a
157 158 159 160 161 162 163 164 165 166 167
new Cabal library feature, `per-component
builds <https://github.com/ezyang/ghc-proposals/blob/master/proposals/0000-componentized-cabal.rst>`__,
where each component of a package is configured and built separately.
This can massively speed up rebuilds of packages with lots of components
(e.g., a package that defines multiple executables), as only one
executable needs to be rebuilt. Packages that use Custom setup scripts
are not currently built on a per-component basis.

Where are my build products?
----------------------------

168
A major deficiency in the current implementation of v2-build is that
169 170
there is no programmatic way to access the location of build products.
The location of the build products is intended to be an internal
171
implementation detail of v2-build, but we also understand that many
172 173
unimplemented features can only be reasonably worked around by
accessing build products directly.
174 175 176 177 178 179 180 181 182

The location where build products can be found varies depending on the
version of cabal-install:

-  In cabal-install-1.24, the dist directory for a package ``p-0.1`` is
   stored in ``dist-newstyle/build/p-0.1``. For example, if you built an
   executable or test suite named ``pexe``, it would be located at
   ``dist-newstyle/build/p-0.1/build/pexe/pexe``.

183
-  In cabal-install-2.0, the dist directory for a package ``p-0.1``
184 185 186 187 188 189 190 191 192 193
   defining a library built with GHC 8.0.1 on 64-bit Linux is
   ``dist-newstyle/build/x86_64-linux/ghc-8.0.1/p-0.1``. When
   per-component builds are enabled (any non-Custom package), a
   subcomponent like an executable or test suite named ``pexe`` will be
   stored at
   ``dist-newstyle/build/x86_64-linux/ghc-8.0.1/p-0.1/c/pexe``; thus,
   the full path of the executable is
   ``dist-newstyle/build/x86_64-linux/ghc-8.0.1/p-0.1/c/pexe/build/pexe/pexe``
   (you can see why we want this to be an implementation detail!)

194 195 196 197 198 199 200 201 202 203 204
- In cabal-install-2.2 and above, the ``/c/`` part of the above path
   is replaced with one of ``/l/``, ``/x/``, ``/f/``, ``/t/``, or
   ``/b/``, depending on the type of component (sublibrary,
   executable, foreign library, test suite, or benchmark
   respectively). So the full path to an executable named ``pexe``
   compiled with GHC 8.0.1 on a 64-bit Linux is now
   ``dist-newstyle/build/x86_64-linux/ghc-8.0.1/p-0.1/x/pexe/build/pexe/pexe``;
   for a benchmark named ``pbench`` it now is
   ``dist-newstyle/build/x86_64-linux/ghc-8.0.1/p-0.1/b/pbench/build/pbench/pbench``;


205
The paths are a bit longer in 2.0 and above but the benefit is that you can
206 207 208 209 210 211 212 213 214 215 216 217 218
transparently have multiple builds with different versions of GHC. We
plan to add the ability to create aliases for certain build
configurations, and more convenient paths to access particularly useful
build products like executables.

Caching
-------

Nix-style local builds sport a robust caching system which help reduce
the time it takes to execute a rebuild cycle. While the details of how
``cabal-install`` does caching are an implementation detail and may
change in the future, knowing what gets cached is helpful for
understanding the performance characteristics of invocations to
219
``v2-build``. The cached intermediate results are stored in
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234
``dist-newstyle/cache``; this folder can be safely deleted to clear the
cache.

The following intermediate results are cached in the following files in
this folder (the most important two are first):

``solver-plan`` (binary)
    The result of calling the dependency solver, assuming that the
    Hackage index, local ``cabal.project`` file, and local ``cabal``
    files are unmodified. (Notably, we do NOT have to dependency solve
    again if new build products are stored in the global store; the
    invocation of the dependency solver is independent of what is
    already available in the store.)
``source-hashes`` (binary)
    The hashes of all local source files. When all local source files of
235
    a local package are unchanged, ``cabal v2-build`` will skip
236 237 238 239 240 241 242 243 244 245 246 247 248 249
    invoking ``setup build`` entirely (saving us from a possibly
    expensive call to ``ghc --make``). The full list of source files
    participating in compilation are determined using
    ``setup sdist --list-sources`` (thus, if you do not list all your
    source files in a Cabal file, you may fail to recompile when you
    edit them.)
``config`` (same format as ``cabal.project``)
    The full project configuration, merged from ``cabal.project`` (and
    friends) as well as the command line arguments.
``compiler`` (binary)
    The configuration of the compiler being used to build the project.
``improved-plan`` (binary)
    Like ``solver-plan``, but with all non-inplace packages improved
    into pre-existing copies from the store.
250 251 252 253 254 255 256 257 258 259 260 261
``plan.json`` (JSON)
    A JSON serialization of the computed install plan intended
    for integrating ``cabal`` with external tooling.
    The `cabal-plan <http://hackage.haskell.org/package/cabal-plan>`__
    package provides a library for parsing ``plan.json`` files into a
    Haskell data structure as well as an example tool showing possible
    applications.

    .. todo::

        Document JSON schema (including version history of schema)

262 263 264 265

Note that every package also has a local cache managed by the Cabal
build system, e.g., in ``$distdir/cache``.

266 267 268
There is another useful file in ``dist-newstyle/cache``,
``plan.json``, which is a JSON serialization of the computed install
plan and is intended for integrating with external tooling.