Changes

Page history
Edit Building/RunningTests authored by Simon Peyton Jones's avatar Simon Peyton Jones
Show whitespace changes
Inline Side-by-side
building/running-tests.md
View page @ b954a841
......@@ -177,38 +177,35 @@ these steps:
regression tests go in the typechecker/ directory, parser tests
go in parser/, and so on.
> >
> > It's not always possible to find a single best place for a test;
> > in those cases just pick one which seems reasonable.
>
> It's not always possible to find a single best place for a test;
> in those cases just pick one which seems reasonable.
>
> Under each main directory may be up to three subdirectories:
>
> > **should_compile**:
> >
> > Under each main directory may be up to three subdirectories:
> > >
> > > should_compile:
> > >
> > > >
> > > > tests which need to compile only
> > >
> > > should_fail:
> > >
> > > >
> > > > tests which should fail to compile and generate a particular error message
> > > tests which need to compile only
> >
> > **should_fail**:
> >
> > >
> > > should_run:
> > > tests which should fail to compile and generate a particular error message
> >
> > **should_run**:
> >
> > >
> > > >
> > > > tests which should compile, run with some specific input, and generate a particular output.
> > > tests which should compile, run with some specific input, and generate a particular output.
> >
> > We don't always divide the tests up like this, and it's not
> > essential to do so (the directory names have no meaning as
> > far as the test driver is concerned).
>
> We don't always divide the tests up like this, and it's not
> essential to do so (the directory names have no meaning as
> far as the test driver is concerned).
1. Having found a suitable place for the test, give the test a name.
For regression tests, we often just name the test after the bug number (e.g. 2047).
For regression tests, we often just name the test after the bug number (e.g. T2047).
Alternatively, follow the convention for the directory in which you place the
test: for example, in typecheck/should_compile, tests are named
tc001, tc002, and so on. Suppose you name your test T, then
......@@ -256,214 +253,197 @@ these steps:
test(<name>, <setup>, <test-fn>, <args>)
```
where
The format of these fields is described in the [next section](building/running-tests#format-of-the-test-entries).
> > *\<name\>* is the name of the test, in quotes (' or ").
> > *\<setup\>* is a function (i.e. any callable object in Python)
> > which allows the options for this test to be changed.
> > There are many pre-defined functions which can be
> > used in this field:
A multi-module test is straightforward. It usually goes in a
directory of its own (although this isn't essential), and the source
files can be named anything you like. The test must have a name, in
the same way as a single-module test; and the stdin/stdout/stderr
files follow the name of the test as before. In the same directory,
place a file 'test.T' containing a line like
> > > **normal** don't change any options from the defaults
```wiki
test(multimod001, normal, multimod_compile_and_run, \
[ 'Main', '-fglasgow-exts', '', 0 ])
```
> > > **skip** skip this test
> > > **skip_if_no_ghci** skip unless GHCi is available
as described above.
> > > **skip_if_fast** skip if "fast" is enabled
> > > **omit_ways(ways)** skip this test for certain ways
For some examples, take a look in tests/ghc-regress/programs.
> > > **only_ways(ways)** do this test certain ways only
# Format of the test entries
> > > **extra_ways(ways)** add some ways which would normally be disabled
> > > **omit_compiler_types(compilers)** skip this test for certain compilers
Each test in a `test.T` file is specified by a line the form
> > > **only_compiler_types(compilers)** do this test for certain compilers only
```wiki
test(<name>, <setup>, <test-fn>, <args>)
```
> > > **expect_broken(bug)** this test is a expected not to work due to the indicated trac bug number
## The \<name\> field
> > > **expect_broken_for(bug, ways)** as expect_broken, but only for the indicated ways
*\<name\>* is the name of the test, in quotes (' or ").
> > > **if_compiler_type(compiler_type, f)** Do `f`, but only for the given compiler type
## The \<setup\> field
> > > **if_platform(plat, f)** Do `f`, but only if we are on the specific platform given
*\<setup\>* is a function (i.e. any callable object in Python)
which allows the options for this test to be changed.
There are many pre-defined functions which can be
used in this field:
> > > **if_tag(tag, f)** do `f` if the compiler has a given tag
- **normal** don't change any options from the defaults
- **skip** skip this test
- **skip_if_no_ghci** skip unless GHCi is available
> > > **unless_tag(tag, f)** do `f` unless the compiler has a given tag
- **skip_if_fast** skip if "fast" is enabled
> > > **set_stdin(file)** use a different file for stdin
- **omit_ways(ways)** skip this test for certain ways
> > > **no_stdin** use no stdin at all (otherwise use `/dev/null`)
- **only_ways(ways)** do this test certain ways only
> > > **exit_code(n)** expect an exit code of 'n' from the prog
- **extra_ways(ways)** add some ways which would normally be disabled
> > > **extra_run_opts(opts)** pass some extra opts to the prog
- **omit_compiler_types(compilers)** skip this test for certain compilers
> > > **no_clean** don't clean up after this test
- **only_compiler_types(compilers)** do this test for certain compilers only
> > > **extra_clean(files)** extra files to clean after the test has completed
- **expect_broken(bug)** this test is a expected not to work due to the indicated trac bug number
> > > **reqlib(P)** requires package P
- **expect_broken_for(bug, ways)** as expect_broken, but only for the indicated ways
> > > **req_profiling** requires profiling
- **if_compiler_type(compiler_type, f)** Do `f`, but only for the given compiler type
> > > **ignore_output** don't try to compare output
- **if_platform(plat, f)** Do `f`, but only if we are on the specific platform given
> > > **alone** don't run this test in parallel with anything else
- **if_tag(tag, f)** do `f` if the compiler has a given tag
> > > **literate** look for a `.lhs` file instead of a `.hs` file
- **unless_tag(tag, f)** do `f` unless the compiler has a given tag
> > > **c_src** look for a `.c` file
- **set_stdin(file)** use a different file for stdin
> > > **cmd_prefix(string)** prefix this string to the command when run
- **no_stdin** use no stdin at all (otherwise use `/dev/null`)
> > > **normalise_slashes** convert backslashes to forward slashes before comparing the output
- **exit_code(n)** expect an exit code of 'n' from the prog
> >
> > To use more than one modifier on a test, just put them in a list.
> > For example, to expect an exit code of 3 and omit way 'opt', we could use
> >
> > ```wiki
> > [ omit_ways(['opt']), exit_code(3) ]
> > ```
> >
> > >
> > > as the `<setup>` argument.
- **extra_run_opts(opts)** pass some extra opts to the prog
> >
> > The following should normally not be used; instead, use the `expect_broken*`
> > functions above so that the problem doesn't get forgotten about, and when we
> > come back to look at the test later we know whether current behaviour is why
> > we marked it as expected to fail:
- **no_clean** don't clean up after this test
> > > **expect_fail** this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
- **extra_clean(files)** extra files to clean after the test has completed
> > > **expect_fail_for(ways)** expect failure for certain ways
- **reqlib(P)** requires package P
> > *\<test-fn\>*
> > is a function which describes how the test should be
> > run, and determines the form of \<args\>. The possible
> > values are:
- **req_profiling** requires profiling
> > >
> > > compile
> > >
> > > >
> > > > Just compile the program, the compilation should succeed.
- **ignore_output** don't try to compare output
> > >
> > > compile_fail
> > >
> > > >
> > > > Just compile the program, the
> > > > compilation should fail (error
> > > > messages will be in T.stderr).
> > > > This kind of failure is mandated by the language definition - it does **not** indicate any bug in the compiler.
- **alone** don't run this test in parallel with anything else
> > >
> > > compile_and_run
> > >
> > > >
> > > > Compile the program and run it,
> > > > comparing the output against the
> > > > relevant files.
- **literate** look for a `.lhs` file instead of a `.hs` file
> > >
> > > multimod_compile
> > >
> > > >
> > > > Compile a multi-module program
> > > > (more about multi-module programs
> > > > below).
- **c_src** look for a `.c` file
> > >
> > > multimod_compile_fail
> > >
> > > >
> > > > Compile a multi-module program,
> > > > and expect the compilation to fail
> > > > with error messages in T.stderr. This kind of failure does **not** indicate a bug in the compiler.
- **cmd_prefix(string)** prefix this string to the command when run
> > >
> > > multimod_compile_and_run
> > >
> > > >
> > > > Compile and run a multi-module
> > > > program.
- **normalise_slashes** convert backslashes to forward slashes before comparing the output
> > >
> > > compile_and_run_with_prefix
> > >
> > > >
> > > > Same as compile_and_run, but with command to use to run the execution of the result binary.
> > >
> > > multimod_compile_and_run_with_prefix
> > >
> > > >
> > > > Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
The following should normally not be used; instead, use the `expect_broken*`
functions above so that the problem doesn't get forgotten about, and when we
come back to look at the test later we know whether current behaviour is why
we marked it as expected to fail:
> > >
> > > run_command
> > >
> > > >
> > > > Just run an arbitrary command. The output is checked
> > > > against `T.stdout` and `T.stderr` (unless `ignore_output`
> > > > is used), and the stdin and expected exit code can be
> > > > changed in the same way as for compile_and_run.
- **expect_fail** this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
> > >
> > > ghci_script
> > >
> > > >
> > > > Runs the current compiler, passing
> > > > --interactive and using the specified
> > > > script as standard input.
- **expect_fail_for(ways)** expect failure for certain ways
> > *\<args\>*
> > is a list of arguments to be passed to \<test-fn\>.
> >
> > For compile, compile_fail and compile_and_run, \<args\>
> > is a list with a single string which contains extra
> > compiler options with which to run the test. eg.
> >
> > ```wiki
> > test('tc001', normal, compile, ['-fglasgow-exts'])
> > ```
> >
> >
> > would pass the flag -fglasgow-exts to the compiler
> > when compiling tc001.
To use more than one modifier on a test, just put them in a list.
For example, to expect an exit code of 3 and omit way 'opt', we could use
> >
> > The multimod_ versions of compile and compile_and_run
> > expect an extra argument on the front of the list: the
> > name of the top module in the program to be compiled
> > (usually this will be 'Main').
```wiki
[ omit_ways(['opt']), exit_code(3) ]
```
A multi-module test is straightforward. It usually goes in a
directory of its own (although this isn't essential), and the source
files can be named anything you like. The test must have a name, in
the same way as a single-module test; and the stdin/stdout/stderr
files follow the name of the test as before. In the same directory,
place a file 'test.T' containing a line like
as the `<setup>` argument.
## The \<test-fn\> field
*\<test-fn\>*
is a function which describes how the test should be
run, and determines the form of \<args\>. The possible
values are:
- **compile** Just compile the program, the compilation should succeed.
- **compile_fail**
Just compile the program, the
compilation should fail (error
messages will be in T.stderr).
This kind of failure is mandated by the language definition - it does **not** indicate any bug in the compiler.
- **compile_and_run**
Compile the program and run it,
comparing the output against the
relevant files.
- **multimod_compile**
Compile a multi-module program
(more about multi-module programs
below).
- **multimod_compile_fail**
Compile a multi-module program,
and expect the compilation to fail
with error messages in T.stderr. This kind of failure does **not** indicate a bug in the compiler.
- **multimod_compile_and_run**
Compile and run a multi-module
program.
- **compile_and_run_with_prefix**
Same as compile_and_run, but with command to use to run the execution of the result binary.
- **multimod_compile_and_run_with_prefix**
Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
- **run_command**
Just run an arbitrary command. The output is checked
against `T.stdout` and `T.stderr` (unless `ignore_output`
is used), and the stdin and expected exit code can be
changed in the same way as for compile_and_run.
- **ghci_script**
Runs the current compiler, passing
--interactive and using the specified
script as standard input.
## The \<args\> field
*\<args\>* is a list of arguments to be passed to \<test-fn\>.
For compile, compile_fail and compile_and_run, \<args\>
is a list with a single string which contains extra
compiler options with which to run the test. eg.
```wiki
test(multimod001, normal, multimod_compile_and_run, \
[ 'Main', '-fglasgow-exts', '', 0 ])
test('tc001', normal, compile, ['-fglasgow-exts'])
```
as described above.
would pass the flag -fglasgow-exts to the compiler
when compiling tc001.
For some examples, take a look in tests/ghc-regress/programs.
The multimod_ versions of compile and compile_and_run
expect an extra argument on the front of the list: the
name of the top module in the program to be compiled
(usually this will be 'Main').
# Sample output files
......
......