Changes

Page history
update authored by Simon Marlow's avatar Simon Marlow
Show whitespace changes
Inline Side-by-side
building/running-tests.md
View page @ c4ce25b2
......@@ -119,20 +119,20 @@ The following ways are defined (for GHC, also see the file config/ghc):
normal -- no special options
optc -- -O -fvia-C
optasm -- -O -fasm
prof -- -O -prof -auto-all
profc -- -O -prof -auto-all -fvia-C
profasm -- -O -prof -auto-all -fasm
unreg -- -unreg
ghci -- (run only, not compile) run test under GHCi
extcore -- -fext-core
optextcore -- -O -fext-core
threaded -- -threaded
threaded1 -- -threaded -debug
threaded2 -- -threaded -O, and +RTS -N2 at run-time
hpc -- -fhpc
```
certain ways are enabled automatically if the GHC build in the local
tree supports them. Ways that are enabled this way are optasm, prof,
profasm, unreg, threaded, and ghci.
tree supports them. Ways that are enabled this way are optasm, profc,
profasm, threaded1, threaded2, and ghci.
# Updating tests when the output changes
......@@ -142,13 +142,13 @@ correct, you can automatically update the sample output to match the
new output like so:
```wiki
make accept TESTS=<test-name>
make accept TEST=<test-name>
```
where \<test-name\> is the name of the test. In a directory which
contains a single test, or if you want to update \*all\* the tests in
the current directory, just omit the 'TESTS=\<test-name\>' part.
the current directory, just omit the 'TEST=\<test-name\>' part.
# Adding a new test
......@@ -194,7 +194,8 @@ these steps:
> > far as the test driver is concerned).
1. Having found a suitable place for the test, give the test a name.
Follow the convention for the directory in which you place the
For regression tests, we often just name the test after the bug number (e.g. 2047).
Alternatively, follow the convention for the directory in which you place the
test: for example, in typecheck/should_compile, tests are named
tc001, tc002, and so on. Suppose you name your test T, then
you'll have the following files:
......@@ -238,26 +239,38 @@ these steps:
1. Edit all.T in the relevant directory and add a line for the test. The line is always of the form
```wiki
test(<name>, <opt-fn>, <test-fn>, <args>)
test(<name>, <setup>, <test-fn>, <args>)
```
where
> > *\<name\>* is the name of the test, in quotes (' or ").
> > *\<opt-fn\>* is a function (i.e. any callable object in Python)
> > *\<setup\>* is a function (i.e. any callable object in Python)
> > which allows the options for this test to be changed.
> > There are several pre-defined functions which can be
> > There are many pre-defined functions which can be
> > used in this field:
> > > **normal** don't change any options from the defaults
> > > **skip** skip this test
> > > **skip_if_no_ghci** skip unless GHCi is available
> > > **skip_if_fast** skip if "fast" is enabled
> > > **skip_if_platform(platform)** skip if we're on the named platform
> > > **skip_if_tag(tag)** skip if the compiler has a given tag
> > > **skip_unless_tag(tag)** skip unless the compiler has a given tag
> > > **omit_ways(ways)** skip this test for certain ways
> > > **only_ways(ways)** do this test certain ways only
> > > **extra_ways(ways)** add some ways which would normally be disabled
> > > **omit_compiler_types(compilers)** skip this test for certain compilers
> > > **only_compiler_types(compilers)** do this test for certain compilers only
......@@ -272,6 +285,8 @@ these steps:
> > > **set_stdin(file)** use a different file for stdin
> > > **no_stdin** use no stdin at all (otherwise use `/dev/null`)
> > > **exit_code(n)** expect an exit code of 'n' from the prog
> > > **exit_code_if_platform(n, plat)** as exit_code, but only for the specific platform given
......@@ -280,8 +295,37 @@ these steps:
> > > **no_clean** don't clean up after this test
> > > **extra_clean(files)** extra files to clean after the test has completed
> > > **reqlib(P)** requires package P
> > > **req_profiling** requires profiling
> > > **ignore_output** don't try to compare output
> > > **alone** don't run this test in parallel with anything else
> > > **literate** look for a `.lhs` file instead of a `.hs` file
> > > **c_src** look for a `.c` file
> > > **cmd_prefix(string)** prefix this string to the command when run
> > > **normalise_slashes** convert backslashes to forward slashes before comparing the output
> >
> > To use more than one modifier on a test, just put them in a list.
> > For example, to expect an exit code of 3 and omit way 'opt', we could use
> >
> > These functions should normally not be used; instead, use the `expect_broken*`
> > ```wiki
> > [ omit_ways(['opt']), exit_code(3) ]
> > ```
> >
> > >
> > > as the `<setup>` argument.
> >
> > The following should normally not be used; instead, use the `expect_broken*`
> > functions above so that the problem doesn't get forgotten about, and when we
> > come back to look at the test later we know whether current behaviour is why
> > we marked it as expected to fail:
......@@ -294,24 +338,6 @@ these steps:
> > > **expect_fail_if_compiler_type(compiler)** expect failure from a certain compiler
> > >
> > > You can compose two of these functions together by
> > > saying compose(f,g). For example, to expect an exit
> > > code of 3 and omit way 'opt', we could use
> > >
> > > ```wiki
> > > compose(omit_ways(['opt']), exit_code(3))
> > > ```
> > >
> > >
> > > as the \<opt-fn\> argument. Calls to compose() can of
> > > course be nested, but it is simpler to use the composes
> > > function which takes a list of functions to be composed, e.g.:
> > >
> > > ```wiki
> > > composes([omit_ways(['opt']), exit_code(3), expect_broken(123)])
> > > ```
> > *\<test-fn\>*
> > is a function which describes how the test should be
> > run, and determines the form of \<args\>. The possible
......@@ -379,19 +405,10 @@ these steps:
> > > run_command
> > >
> > > >
> > > > Just run an arbitrary command. The
> > > > output is checked against T.stdout and
> > > > T.stderr, and the stdin and expected
> > > > exit code can be changed in the same
> > > > way as for compile_and_run.
> > >
> > > run_command_ignore_output
> > >
> > > >
> > > > Same as run_command, except the output
> > > > (both stdout and stderr) from the
> > > > command is ignored.
> > > > Just run an arbitrary command. The output is checked
> > > > against `T.stdout` and `T.stderr` (unless `ignore_output`
> > > > is used), and the stdin and expected exit code can be
> > > > changed in the same way as for compile_and_run.
> > >
> > > ghci_script
......@@ -442,6 +459,31 @@ as described above.
For some examples, take a look in tests/ghc-regress/programs.
# Sample output files
Normally, the sample `stdout` and `stderr` for a test T go in the
files `T.stdout` and `T.stderr` respectively. However, sometimes a
test may generate different output depending on the platform,
compiler, compiler version, or word-size. For this reason the test
driver looks for sample output files using this pattern:
```wiki
T.stdout[-<compiler>][-<version>][-ws-<wordsize>][-<platform>]
```
Any combination of the optional extensions may be given, but they must
be in the order specified. The most specific output file that matches
the current configuration will be selected; for example if the
platform is `i386-unknown-mingw32` then `T.stderr-i386-unknown-mingw32`
will be picked in preference to `T.stderr`.
Another common example is to give different sample output for an older
compiler version. For example, the sample `stderr` for GHC 6.8.x would go in the file
`T.stderr-ghc-6.8`.
# The details
......
......