Skip to content
Snippets Groups Projects
Commit 215a4789 authored by Edward Z. Yang's avatar Edward Z. Yang
Browse files

More docs. [ci skip] 


Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
parent 856e0f69
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
README.md
+ 55
− 33
View file @ 215a4789
......@@ -38,14 +38,14 @@ that you have a sufficiently recent cabal-install (see above),
it is sufficient to run:
~~~~
cabal new-build cabal-install
cabal new-build cabal
~~~~
To build a local, development copy of cabal-install. The binary
will be located at
`dist-newstyle/build/cabal-install-$VERSION/build/cabal/cabal`;
you can determine the `$VERSION` of cabal-install by looking at
[cabal-install/cabal-install.cabal](cabal-install/cabal-install.cabal).
To build a local, development copy of cabal-install. The location
of your build products will vary depending on which version of
cabal-install you use to build; see the documentation section
[Where are my build products?](http://cabal.readthedocs.io/en/latest/nix-local-build.html#where-are-my-build-products)
to find the binary (or just run `find -type f -executable -name cabal`).
Here are some other useful variations on the commands:
......@@ -55,19 +55,53 @@ cabal new-build Cabal:unit-tests # build Cabal's unit test suite
cabal new-build cabal-tests # etc...
~~~~
**Dogfooding HEAD.**
Many of the core developers of Cabal dogfood `cabal-install` HEAD
when doing development on Cabal. This helps us identify bugs
which were missed by the test suite and easily experiment with new
features.
The recommended workflow in this case is slightly different: you will
maintain two Cabal source trees: your production tree (built with a
released version of Cabal) which always tracks `master` and which you
update only when you want to move to a new version of Cabal to dogfood,
and your development tree (built with your production Cabal) that you
actually do development on.
In more detail, suppose you have checkouts of Cabal at `~/cabal-prod`
and `~/cabal-dev`, and you have a release copy of cabal installed at
`/opt/cabal/1.24/bin/cabal`. First, build your production tree:
~~~~
cd ~/cabal-prod
/opt/cabal/1.24/bin/cabal new-build cabal
~~~~
This will produce a cabal binary (see also: [Where are my build products?](http://cabal.readthedocs.io/en/latest/nix-local-build.html#where-are-my-build-products)
). Add this binary to your PATH,
and then use it to build your development copy:
~~~~
cd ~/cabal-dev
cabal new-build cabal
~~~~
Running tests
-------------
**Using Travis and AppVeyor.**
The easiest way to run tests on Cabal is to make a branch on GitHub
and then open a pull request; our continuous integration service on
Travis and AppVeyor will build and test your code. Title your PR
with WIP so we know that it does not need code review. Alternately,
you can enable Travis on your fork in your own username and Travis
should build your local branches.
If you are not in a hurry, the most convenient way to run tests on Cabal
is to make a branch on GitHub and then open a pull request; our
continuous integration service on Travis and AppVeyor will build and
test your code. Title your PR with WIP so we know that it does not need
code review.
Some tips for using Travis effectively:
* Travis builds take a long time. Use them when you are pretty
sure everything is OK; otherwise, try to run relevant tests locally
first.
* Watch over your jobs on the [Travis website](http://travis-ci.org).
If you know a build of yours is going to fail (because one job has
already failed), be nice to others and cancel the rest of the jobs,
......@@ -75,14 +109,6 @@ Some tips for using Travis effectively:
* If you want realtime notification when builds of your PRs finish, we have a [Slack team](https://haskell-cabal.slack.com/). To get issued an invite, fill in your email at [this sign up page](https://haskell-cabal.herokuapp.com).
* If you enable Travis for the fork of Cabal in your local GitHub, you
can have builds done automatically for your local branch separate
from Cabal. This is an alternative to opening a PR, and has the bonus
that you don't have to wait for the main queue on Haskell repository
to finish. It is recommended that you enable Travis only on PRs,
and open a PR on your *local* repository, so that you can also use
GitHub to push and pull branches without triggering builds.
**How to debug a failing CI test.**
One of the annoying things about running tests on CI is when they
fail, there is often no easy way to further troubleshoot the broken
......@@ -107,12 +133,7 @@ failures:
In this case, if you click on "Branch", you can get access to
the precise binaries that were built by Travis that are being
tested. If you have an Ubuntu system, you can download
the binaries and run them directly. Note that the
build is not relocatable, so you must exactly reproduce
the file system layout of the Travis build (in particular,
the build products need to live in the directory
`/home/travis/build/haskell/cabal`, and the `.cabal` directory
must live in `/home/travis/.cabal`).
the binaries and run them directly.
5. Is the test failing on AppVeyor? Consider logging in via
Remote Desktop to the build VM:
......@@ -124,19 +145,18 @@ or continuous integration breakage; please file a bug.
**Running tests locally.**
To run tests locally with `new-build`, you will need to know the
name of the test suite you want. Cabal and cabal-install have
several. In general, the test executable for
`{Cabal,cabal-install}:$TESTNAME` will be stored at
`dist-newstyle/build/{Cabal,cabal-install}-$VERSION/build/$TESTNAME/$TESTNAME`.
several. Also, you'll want to read [Where are my build products?](http://cabal.readthedocs.io/en/latest/nix-local-build.html#where-are-my-build-products)
The most important test suite is `cabal-testsuite`: most user-visible
changes to Cabal should come with a test in this framework. See
[cabal-testsuite/README.md](cabal-testsuite/README.md) for more
information about how to run tests and write new ones. Quick
start: use `cabal-tests` to run `Cabal` tests, and `cabal-tests
--with-cabal=/path/to/cabal` to run `cabal-install` tests.
--with-cabal=/path/to/cabal` to run `cabal-install` tests
(don't forget `--with-cabal`! Your cabal-install tests won't
run without it).
Among the other tests, use `-p` which applies a regex filter to the test
names.
There are also other test suites:
* `Cabal:unit-tests` are small, quick-running unit tests
on small pieces of functionality in Cabal. If you are working
......@@ -154,7 +174,9 @@ names.
* `cabal-install:integration-tests2` are integration tests on some
top-level API functions inside the `cabal-install` source code.
You should also run this test suite.
For these test executables, `-p` which applies a regex filter to the test
names.
Conventions
-----------
......
cabal-testsuite/README.md
+ 87
− 18
View file @ 215a4789
......@@ -10,23 +10,31 @@ How to run
To run a specific set of tests, use `cabal-tests PATH ...`. You can
control parallelism using the `-j` flag.
There are a few useful flags which can handle some special cases:
There are a few useful flags:
* `--builddir DIR` can be used to manually specify the dist directory
that was used to build `cabal-tests`; this can be used if
the autodetection doesn't work correctly (which may be the
case for old versions of GHC.)
* `--with-cabal PATH` can be used to specify the path of a
`cabal-install` executable. IF YOU DO NOT SPECIFY THIS FLAG,
CABAL INSTALL TESTS WILL NOT RUN.
* `--with-ghc PATH` can be used to specify an alternate version of
GHC to ask the tests to compile with.
* `--with-cabal PATH` can be used to specify the path of a
`cabal-install` executable. In this case, tests involving
this executable will also get run.
* `--builddir DIR` can be used to manually specify the dist directory
that was used to build `cabal-tests`; this can be used if
the autodetection doesn't work correctly (which may be the
case for old versions of GHC.)
How to write
------------
If you learn better by example, just look at the tests that live
in `cabal-testsuite/PackageTests`; if you `git log -p`, you can
see the full contents of various commits which added a test for
various functionality. See if you can find an existing test that
is similar to what you want to test.
Otherwise, here is a walkthrough:
1. Create the package(s) that you need for your test in a
new directory. (Currently, tests are stored in `PackageTests`
and `tests`; we might reorganize this soon.)
......@@ -39,9 +47,6 @@ How to write
-- your test code here
```
The general API is that the test is considered to succeed if
it returns exit 0, and failed if it returned exit code 1.
Standard output/error are purely for diagnostic purposes.
`setupAndCabal` test indicates that invocations of `setup`
should work both for a raw `Setup` script, as well as
`cabal-install` (if your test works only for one or the
......@@ -50,10 +55,11 @@ How to write
Code runs in the `TestM` monad, which manages some administrative
environment (e.g., the test that is running, etc.)
`Test.Cabal.Prelude` contains a number of useful functions
for testing implemented in this monad, including ways to invoke
`Setup`, `ghc-pkg`, and other important programs. For other
ideas of how to write tests, look at existing `.test.hs`
scripts. If you don't see something anywhere, that's probably
for testing implemented in this monad, including the functions `cabal`
and `setup` which let you invoke those respective programs. You should
read through that file to get a sense for what capabilities
are possible (grep for use-sites of functions to see how they
are used). If you don't see something anywhere, that's probably
because it isn't implemented. Implement it!
3. Run your tests using `cabal-tests` (no need to rebuild when
......@@ -68,22 +74,85 @@ allow multiple tests to be defined in one file but run in parallel;
at the moment, these just indicate long running tests that should
be run early (to avoid straggling.)
Frequently asked questions
--------------------------
For all of these answers, to see examples of the functions in
question, grep the test suite.
**Why isn't some output I added to Cabal showing up in the recorded
test output?** Only "marked" output is picked up by Cabal; currently,
only `notice`, `warn` and `die` produce marked output. Use those
combinators for your output.
**How do I safely let my test modify version-controlled source files?**
Use `withSourceCopy`. Note that you MUST `git add`
all files which are relevant to the test; otherwise they will not be
available when running the test.
**How can I add a dependency on a package from Hackage in a test?**
By default, the test suite is completely independent of the contents
of Hackage, to ensure that it keeps working across all GHC versions.
If possible, define the package locally. If the package needs
to be from Hackage (e.g., you are testing the global store code
in new-build), use `withRepo "repo"` to initialize a "fake" Hackage with
the packages placed in the `repo` directory.
**How do I run an executable that my test built?** The specific
function you should use depends on how you built the executable:
* If you built it using `Setup build`, use `runExe`
* If you installed it using `Setup install` or `cabal install`, use
`runInstalledExe`.
* If you built it with `cabal new-build`, use `runPlanExe`; note
that you will need to run this inside of a `withPlan` that is
placed *after* you have invoked `new-build`. (Grep
for an example!)
**How do I turn of accept tests? My test output wobbles to much.**
Use `recordMode DoNotRecord`. This should be a last resort; consider
modifying Cabal so that the output is stable. If you must do this, make
sure you add extra, manual tests to ensure the output looks like what
you expect.
**How can I manually test for a string in output?** Use the hyphenated
variants of a command (e.g., `cabal'` rather than `cabal`) and use
`assertOutputContains`. Note that this will search over BOTH stdout
and stderr.
**How do I skip running a test in some environments?** Use the
`skipIf` and `skipUnless` combinators. Useful parameters to test
these with include `hasSharedLibraries`, `hasProfiledLibraries`,
`hasCabalShared`, `ghcVersionIs`, `isWindows`, `isLinux`, `isOSX`
and `hasCabalForGhc`.
**I programatically modified a file in my test suite, but Cabal/GHC
doesn't seem to be picking it up.** You need to sleep sufficiently
long before editing a file, in order for file system timestamp
resolution to pick it up. Use `withDelay` and `delay` prior to
making a modification.
**How do I mark a test as broken?** Use `expectBroken`, which takes
the ticket number as its first argument. Note that this does NOT
handle accept-test brokenness, so you will have to add a manual
string output test, if that is how your test is "failing."
Hermetic tests
--------------
By default, we run tests directly on the source code that is checked into the
source code repository. However, some tests require programatically
modifying source files, or interact with Cabal commands which are
not hermetic (e.g., cabal freeze). In this case, cabal-testsuite
not hermetic (e.g., `cabal freeze`). In this case, cabal-testsuite
supports opting into a hermetic test, where we first make copy of all
the relevant source code before starting the test. You can opt into
this mode using the 'withSourceCopy' combinator (search for examples!)
This mode is subject to the following limitations:
* You must be running the test inside a valid Git checkout of the test
suite (withSourceCopy uses Git to determine which files should be copied.)
suite (`withSourceCopy` uses Git to determine which files should be copied.)
* You must 'git add' all files which are relevant to the test, otherwise
* You must `git add` all files which are relevant to the test, otherwise
they will not be copied.
* The source copy is still made at a well-known location, so running
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment