Changes

Page history
update authored by Simon Marlow's avatar Simon Marlow
Show whitespace changes
Inline Side-by-side
building/running-tests.md
View page @ c4ce25b2
...@@ -119,20 +119,20 @@ The following ways are defined (for GHC, also see the file config/ghc): ...@@ -119,20 +119,20 @@ The following ways are defined (for GHC, also see the file config/ghc):
normal -- no special options normal -- no special options
optc -- -O -fvia-C optc -- -O -fvia-C
optasm -- -O -fasm optasm -- -O -fasm
prof -- -O -prof -auto-all profc -- -O -prof -auto-all -fvia-C
profasm -- -O -prof -auto-all -fasm profasm -- -O -prof -auto-all -fasm
unreg -- -unreg
ghci -- (run only, not compile) run test under GHCi ghci -- (run only, not compile) run test under GHCi
extcore -- -fext-core extcore -- -fext-core
optextcore -- -O -fext-core optextcore -- -O -fext-core
threaded -- -threaded threaded1 -- -threaded -debug
threaded2 -- -threaded -O, and +RTS -N2 at run-time
hpc -- -fhpc hpc -- -fhpc
``` ```
certain ways are enabled automatically if the GHC build in the local certain ways are enabled automatically if the GHC build in the local
tree supports them. Ways that are enabled this way are optasm, prof, tree supports them. Ways that are enabled this way are optasm, profc,
profasm, unreg, threaded, and ghci. profasm, threaded1, threaded2, and ghci.
# Updating tests when the output changes # Updating tests when the output changes
...@@ -142,13 +142,13 @@ correct, you can automatically update the sample output to match the ...@@ -142,13 +142,13 @@ correct, you can automatically update the sample output to match the
new output like so: new output like so:
```wiki ```wiki
make accept TESTS=<test-name> make accept TEST=<test-name>
``` ```
where \<test-name\> is the name of the test. In a directory which where \<test-name\> is the name of the test. In a directory which
contains a single test, or if you want to update \*all\* the tests in contains a single test, or if you want to update \*all\* the tests in
the current directory, just omit the 'TESTS=\<test-name\>' part. the current directory, just omit the 'TEST=\<test-name\>' part.
# Adding a new test # Adding a new test
...@@ -194,7 +194,8 @@ these steps: ...@@ -194,7 +194,8 @@ these steps:
> > far as the test driver is concerned). > > far as the test driver is concerned).
1. Having found a suitable place for the test, give the test a name. 1. Having found a suitable place for the test, give the test a name.
Follow the convention for the directory in which you place the For regression tests, we often just name the test after the bug number (e.g. 2047).
Alternatively, follow the convention for the directory in which you place the
test: for example, in typecheck/should_compile, tests are named test: for example, in typecheck/should_compile, tests are named
tc001, tc002, and so on. Suppose you name your test T, then tc001, tc002, and so on. Suppose you name your test T, then
you'll have the following files: you'll have the following files:
...@@ -238,26 +239,38 @@ these steps: ...@@ -238,26 +239,38 @@ these steps:
1. Edit all.T in the relevant directory and add a line for the test. The line is always of the form 1. Edit all.T in the relevant directory and add a line for the test. The line is always of the form
```wiki ```wiki
test(<name>, <opt-fn>, <test-fn>, <args>) test(<name>, <setup>, <test-fn>, <args>)
``` ```
where where
> > *\<name\>* is the name of the test, in quotes (' or "). > > *\<name\>* is the name of the test, in quotes (' or ").
> > *\<opt-fn\>* is a function (i.e. any callable object in Python) > > *\<setup\>* is a function (i.e. any callable object in Python)
> > which allows the options for this test to be changed. > > which allows the options for this test to be changed.
> > There are several pre-defined functions which can be > > There are many pre-defined functions which can be
> > used in this field: > > used in this field:
> > > **normal** don't change any options from the defaults > > > **normal** don't change any options from the defaults
> > > **skip** skip this test > > > **skip** skip this test
> > > **skip_if_no_ghci** skip unless GHCi is available
> > > **skip_if_fast** skip if "fast" is enabled
> > > **skip_if_platform(platform)** skip if we're on the named platform
> > > **skip_if_tag(tag)** skip if the compiler has a given tag
> > > **skip_unless_tag(tag)** skip unless the compiler has a given tag
> > > **omit_ways(ways)** skip this test for certain ways > > > **omit_ways(ways)** skip this test for certain ways
> > > **only_ways(ways)** do this test certain ways only > > > **only_ways(ways)** do this test certain ways only
> > > **extra_ways(ways)** add some ways which would normally be disabled
> > > **omit_compiler_types(compilers)** skip this test for certain compilers > > > **omit_compiler_types(compilers)** skip this test for certain compilers
> > > **only_compiler_types(compilers)** do this test for certain compilers only > > > **only_compiler_types(compilers)** do this test for certain compilers only
...@@ -272,6 +285,8 @@ these steps: ...@@ -272,6 +285,8 @@ these steps:
> > > **set_stdin(file)** use a different file for stdin > > > **set_stdin(file)** use a different file for stdin
> > > **no_stdin** use no stdin at all (otherwise use `/dev/null`)
> > > **exit_code(n)** expect an exit code of 'n' from the prog > > > **exit_code(n)** expect an exit code of 'n' from the prog
> > > **exit_code_if_platform(n, plat)** as exit_code, but only for the specific platform given > > > **exit_code_if_platform(n, plat)** as exit_code, but only for the specific platform given
...@@ -280,8 +295,37 @@ these steps: ...@@ -280,8 +295,37 @@ these steps:
> > > **no_clean** don't clean up after this test > > > **no_clean** don't clean up after this test
> > > **extra_clean(files)** extra files to clean after the test has completed
> > > **reqlib(P)** requires package P
> > > **req_profiling** requires profiling
> > > **ignore_output** don't try to compare output
> > > **alone** don't run this test in parallel with anything else
> > > **literate** look for a `.lhs` file instead of a `.hs` file
> > > **c_src** look for a `.c` file
> > > **cmd_prefix(string)** prefix this string to the command when run
> > > **normalise_slashes** convert backslashes to forward slashes before comparing the output
> >
> > To use more than one modifier on a test, just put them in a list.
> > For example, to expect an exit code of 3 and omit way 'opt', we could use
> > > >
> > These functions should normally not be used; instead, use the `expect_broken*` > > ```wiki
> > [ omit_ways(['opt']), exit_code(3) ]
> > ```
> >
> > >
> > > as the `<setup>` argument.
> >
> > The following should normally not be used; instead, use the `expect_broken*`
> > functions above so that the problem doesn't get forgotten about, and when we > > functions above so that the problem doesn't get forgotten about, and when we
> > come back to look at the test later we know whether current behaviour is why > > come back to look at the test later we know whether current behaviour is why
> > we marked it as expected to fail: > > we marked it as expected to fail:
...@@ -294,24 +338,6 @@ these steps: ...@@ -294,24 +338,6 @@ these steps:
> > > **expect_fail_if_compiler_type(compiler)** expect failure from a certain compiler > > > **expect_fail_if_compiler_type(compiler)** expect failure from a certain compiler
> > >
> > > You can compose two of these functions together by
> > > saying compose(f,g). For example, to expect an exit
> > > code of 3 and omit way 'opt', we could use
> > >
> > > ```wiki
> > > compose(omit_ways(['opt']), exit_code(3))
> > > ```
> > >
> > >
> > > as the \<opt-fn\> argument. Calls to compose() can of
> > > course be nested, but it is simpler to use the composes
> > > function which takes a list of functions to be composed, e.g.:
> > >
> > > ```wiki
> > > composes([omit_ways(['opt']), exit_code(3), expect_broken(123)])
> > > ```
> > *\<test-fn\>* > > *\<test-fn\>*
> > is a function which describes how the test should be > > is a function which describes how the test should be
> > run, and determines the form of \<args\>. The possible > > run, and determines the form of \<args\>. The possible
...@@ -379,19 +405,10 @@ these steps: ...@@ -379,19 +405,10 @@ these steps:
> > > run_command > > > run_command
> > > > > >
> > > > > > > >
> > > > Just run an arbitrary command. The > > > > Just run an arbitrary command. The output is checked
> > > > output is checked against T.stdout and > > > > against `T.stdout` and `T.stderr` (unless `ignore_output`
> > > > T.stderr, and the stdin and expected > > > > is used), and the stdin and expected exit code can be
> > > > exit code can be changed in the same > > > > changed in the same way as for compile_and_run.
> > > > way as for compile_and_run.
> > >
> > > run_command_ignore_output
> > >
> > > >
> > > > Same as run_command, except the output
> > > > (both stdout and stderr) from the
> > > > command is ignored.
> > > > > >
> > > ghci_script > > > ghci_script
...@@ -442,6 +459,31 @@ as described above. ...@@ -442,6 +459,31 @@ as described above.
For some examples, take a look in tests/ghc-regress/programs. For some examples, take a look in tests/ghc-regress/programs.
# Sample output files
Normally, the sample `stdout` and `stderr` for a test T go in the
files `T.stdout` and `T.stderr` respectively. However, sometimes a
test may generate different output depending on the platform,
compiler, compiler version, or word-size. For this reason the test
driver looks for sample output files using this pattern:
```wiki
T.stdout[-<compiler>][-<version>][-ws-<wordsize>][-<platform>]
```
Any combination of the optional extensions may be given, but they must
be in the order specified. The most specific output file that matches
the current configuration will be selected; for example if the
platform is `i386-unknown-mingw32` then `T.stderr-i386-unknown-mingw32`
will be picked in preference to `T.stderr`.
Another common example is to give different sample output for an older
compiler version. For example, the sample `stderr` for GHC 6.8.x would go in the file
`T.stderr-ghc-6.8`.
# The details # The details
... ...
......