... | ... | @@ -177,38 +177,35 @@ these steps: |
|
|
regression tests go in the typechecker/ directory, parser tests
|
|
|
go in parser/, and so on.
|
|
|
|
|
|
> >
|
|
|
> > It's not always possible to find a single best place for a test;
|
|
|
> > in those cases just pick one which seems reasonable.
|
|
|
>
|
|
|
> It's not always possible to find a single best place for a test;
|
|
|
> in those cases just pick one which seems reasonable.
|
|
|
|
|
|
>
|
|
|
> Under each main directory may be up to three subdirectories:
|
|
|
>
|
|
|
> > **should_compile**:
|
|
|
> >
|
|
|
> > Under each main directory may be up to three subdirectories:
|
|
|
|
|
|
> > >
|
|
|
> > > should_compile:
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > tests which need to compile only
|
|
|
|
|
|
> > >
|
|
|
> > > should_fail:
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > tests which should fail to compile and generate a particular error message
|
|
|
|
|
|
> > > tests which need to compile only
|
|
|
> >
|
|
|
> > **should_fail**:
|
|
|
> >
|
|
|
> > >
|
|
|
> > > should_run:
|
|
|
> > > tests which should fail to compile and generate a particular error message
|
|
|
> >
|
|
|
> > **should_run**:
|
|
|
> >
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > tests which should compile, run with some specific input, and generate a particular output.
|
|
|
> > > tests which should compile, run with some specific input, and generate a particular output.
|
|
|
|
|
|
> >
|
|
|
> > We don't always divide the tests up like this, and it's not
|
|
|
> > essential to do so (the directory names have no meaning as
|
|
|
> > far as the test driver is concerned).
|
|
|
>
|
|
|
> We don't always divide the tests up like this, and it's not
|
|
|
> essential to do so (the directory names have no meaning as
|
|
|
> far as the test driver is concerned).
|
|
|
|
|
|
1. Having found a suitable place for the test, give the test a name.
|
|
|
For regression tests, we often just name the test after the bug number (e.g. 2047).
|
|
|
For regression tests, we often just name the test after the bug number (e.g. T2047).
|
|
|
Alternatively, follow the convention for the directory in which you place the
|
|
|
test: for example, in typecheck/should_compile, tests are named
|
|
|
tc001, tc002, and so on. Suppose you name your test T, then
|
... | ... | @@ -256,214 +253,197 @@ these steps: |
|
|
test(<name>, <setup>, <test-fn>, <args>)
|
|
|
```
|
|
|
|
|
|
where
|
|
|
The format of these fields is described in the [next section](building/running-tests#format-of-the-test-entries).
|
|
|
|
|
|
> > *\<name\>* is the name of the test, in quotes (' or ").
|
|
|
|
|
|
> > *\<setup\>* is a function (i.e. any callable object in Python)
|
|
|
> > which allows the options for this test to be changed.
|
|
|
> > There are many pre-defined functions which can be
|
|
|
> > used in this field:
|
|
|
A multi-module test is straightforward. It usually goes in a
|
|
|
directory of its own (although this isn't essential), and the source
|
|
|
files can be named anything you like. The test must have a name, in
|
|
|
the same way as a single-module test; and the stdin/stdout/stderr
|
|
|
files follow the name of the test as before. In the same directory,
|
|
|
place a file 'test.T' containing a line like
|
|
|
|
|
|
> > > **normal** don't change any options from the defaults
|
|
|
```wiki
|
|
|
test(multimod001, normal, multimod_compile_and_run, \
|
|
|
[ 'Main', '-fglasgow-exts', '', 0 ])
|
|
|
```
|
|
|
|
|
|
> > > **skip** skip this test
|
|
|
|
|
|
> > > **skip_if_no_ghci** skip unless GHCi is available
|
|
|
as described above.
|
|
|
|
|
|
> > > **skip_if_fast** skip if "fast" is enabled
|
|
|
|
|
|
> > > **omit_ways(ways)** skip this test for certain ways
|
|
|
For some examples, take a look in tests/ghc-regress/programs.
|
|
|
|
|
|
> > > **only_ways(ways)** do this test certain ways only
|
|
|
# Format of the test entries
|
|
|
|
|
|
> > > **extra_ways(ways)** add some ways which would normally be disabled
|
|
|
|
|
|
> > > **omit_compiler_types(compilers)** skip this test for certain compilers
|
|
|
Each test in a `test.T` file is specified by a line the form
|
|
|
|
|
|
> > > **only_compiler_types(compilers)** do this test for certain compilers only
|
|
|
```wiki
|
|
|
test(<name>, <setup>, <test-fn>, <args>)
|
|
|
```
|
|
|
|
|
|
> > > **expect_broken(bug)** this test is a expected not to work due to the indicated trac bug number
|
|
|
## The \<name\> field
|
|
|
|
|
|
> > > **expect_broken_for(bug, ways)** as expect_broken, but only for the indicated ways
|
|
|
*\<name\>* is the name of the test, in quotes (' or ").
|
|
|
|
|
|
> > > **if_compiler_type(compiler_type, f)** Do `f`, but only for the given compiler type
|
|
|
## The \<setup\> field
|
|
|
|
|
|
> > > **if_platform(plat, f)** Do `f`, but only if we are on the specific platform given
|
|
|
*\<setup\>* is a function (i.e. any callable object in Python)
|
|
|
which allows the options for this test to be changed.
|
|
|
There are many pre-defined functions which can be
|
|
|
used in this field:
|
|
|
|
|
|
> > > **if_tag(tag, f)** do `f` if the compiler has a given tag
|
|
|
- **normal** don't change any options from the defaults
|
|
|
- **skip** skip this test
|
|
|
- **skip_if_no_ghci** skip unless GHCi is available
|
|
|
|
|
|
> > > **unless_tag(tag, f)** do `f` unless the compiler has a given tag
|
|
|
- **skip_if_fast** skip if "fast" is enabled
|
|
|
|
|
|
> > > **set_stdin(file)** use a different file for stdin
|
|
|
- **omit_ways(ways)** skip this test for certain ways
|
|
|
|
|
|
> > > **no_stdin** use no stdin at all (otherwise use `/dev/null`)
|
|
|
- **only_ways(ways)** do this test certain ways only
|
|
|
|
|
|
> > > **exit_code(n)** expect an exit code of 'n' from the prog
|
|
|
- **extra_ways(ways)** add some ways which would normally be disabled
|
|
|
|
|
|
> > > **extra_run_opts(opts)** pass some extra opts to the prog
|
|
|
- **omit_compiler_types(compilers)** skip this test for certain compilers
|
|
|
|
|
|
> > > **no_clean** don't clean up after this test
|
|
|
- **only_compiler_types(compilers)** do this test for certain compilers only
|
|
|
|
|
|
> > > **extra_clean(files)** extra files to clean after the test has completed
|
|
|
- **expect_broken(bug)** this test is a expected not to work due to the indicated trac bug number
|
|
|
|
|
|
> > > **reqlib(P)** requires package P
|
|
|
- **expect_broken_for(bug, ways)** as expect_broken, but only for the indicated ways
|
|
|
|
|
|
> > > **req_profiling** requires profiling
|
|
|
- **if_compiler_type(compiler_type, f)** Do `f`, but only for the given compiler type
|
|
|
|
|
|
> > > **ignore_output** don't try to compare output
|
|
|
- **if_platform(plat, f)** Do `f`, but only if we are on the specific platform given
|
|
|
|
|
|
> > > **alone** don't run this test in parallel with anything else
|
|
|
- **if_tag(tag, f)** do `f` if the compiler has a given tag
|
|
|
|
|
|
> > > **literate** look for a `.lhs` file instead of a `.hs` file
|
|
|
- **unless_tag(tag, f)** do `f` unless the compiler has a given tag
|
|
|
|
|
|
> > > **c_src** look for a `.c` file
|
|
|
- **set_stdin(file)** use a different file for stdin
|
|
|
|
|
|
> > > **cmd_prefix(string)** prefix this string to the command when run
|
|
|
- **no_stdin** use no stdin at all (otherwise use `/dev/null`)
|
|
|
|
|
|
> > > **normalise_slashes** convert backslashes to forward slashes before comparing the output
|
|
|
- **exit_code(n)** expect an exit code of 'n' from the prog
|
|
|
|
|
|
> >
|
|
|
> > To use more than one modifier on a test, just put them in a list.
|
|
|
> > For example, to expect an exit code of 3 and omit way 'opt', we could use
|
|
|
> >
|
|
|
> > ```wiki
|
|
|
> > [ omit_ways(['opt']), exit_code(3) ]
|
|
|
> > ```
|
|
|
> >
|
|
|
> > >
|
|
|
> > > as the `<setup>` argument.
|
|
|
- **extra_run_opts(opts)** pass some extra opts to the prog
|
|
|
|
|
|
> >
|
|
|
> > The following should normally not be used; instead, use the `expect_broken*`
|
|
|
> > functions above so that the problem doesn't get forgotten about, and when we
|
|
|
> > come back to look at the test later we know whether current behaviour is why
|
|
|
> > we marked it as expected to fail:
|
|
|
- **no_clean** don't clean up after this test
|
|
|
|
|
|
> > > **expect_fail** this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
|
|
|
- **extra_clean(files)** extra files to clean after the test has completed
|
|
|
|
|
|
> > > **expect_fail_for(ways)** expect failure for certain ways
|
|
|
- **reqlib(P)** requires package P
|
|
|
|
|
|
> > *\<test-fn\>*
|
|
|
> > is a function which describes how the test should be
|
|
|
> > run, and determines the form of \<args\>. The possible
|
|
|
> > values are:
|
|
|
- **req_profiling** requires profiling
|
|
|
|
|
|
> > >
|
|
|
> > > compile
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Just compile the program, the compilation should succeed.
|
|
|
- **ignore_output** don't try to compare output
|
|
|
|
|
|
> > >
|
|
|
> > > compile_fail
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Just compile the program, the
|
|
|
> > > > compilation should fail (error
|
|
|
> > > > messages will be in T.stderr).
|
|
|
> > > > This kind of failure is mandated by the language definition - it does **not** indicate any bug in the compiler.
|
|
|
- **alone** don't run this test in parallel with anything else
|
|
|
|
|
|
> > >
|
|
|
> > > compile_and_run
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Compile the program and run it,
|
|
|
> > > > comparing the output against the
|
|
|
> > > > relevant files.
|
|
|
- **literate** look for a `.lhs` file instead of a `.hs` file
|
|
|
|
|
|
> > >
|
|
|
> > > multimod_compile
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Compile a multi-module program
|
|
|
> > > > (more about multi-module programs
|
|
|
> > > > below).
|
|
|
- **c_src** look for a `.c` file
|
|
|
|
|
|
> > >
|
|
|
> > > multimod_compile_fail
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Compile a multi-module program,
|
|
|
> > > > and expect the compilation to fail
|
|
|
> > > > with error messages in T.stderr. This kind of failure does **not** indicate a bug in the compiler.
|
|
|
- **cmd_prefix(string)** prefix this string to the command when run
|
|
|
|
|
|
> > >
|
|
|
> > > multimod_compile_and_run
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Compile and run a multi-module
|
|
|
> > > > program.
|
|
|
- **normalise_slashes** convert backslashes to forward slashes before comparing the output
|
|
|
|
|
|
> > >
|
|
|
> > > compile_and_run_with_prefix
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Same as compile_and_run, but with command to use to run the execution of the result binary.
|
|
|
|
|
|
> > >
|
|
|
> > > multimod_compile_and_run_with_prefix
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
|
|
|
The following should normally not be used; instead, use the `expect_broken*`
|
|
|
functions above so that the problem doesn't get forgotten about, and when we
|
|
|
come back to look at the test later we know whether current behaviour is why
|
|
|
we marked it as expected to fail:
|
|
|
|
|
|
> > >
|
|
|
> > > run_command
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Just run an arbitrary command. The output is checked
|
|
|
> > > > against `T.stdout` and `T.stderr` (unless `ignore_output`
|
|
|
> > > > is used), and the stdin and expected exit code can be
|
|
|
> > > > changed in the same way as for compile_and_run.
|
|
|
- **expect_fail** this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
|
|
|
|
|
|
> > >
|
|
|
> > > ghci_script
|
|
|
> > >
|
|
|
> > > >
|
|
|
> > > > Runs the current compiler, passing
|
|
|
> > > > --interactive and using the specified
|
|
|
> > > > script as standard input.
|
|
|
- **expect_fail_for(ways)** expect failure for certain ways
|
|
|
|
|
|
> > *\<args\>*
|
|
|
> > is a list of arguments to be passed to \<test-fn\>.
|
|
|
|
|
|
> >
|
|
|
> > For compile, compile_fail and compile_and_run, \<args\>
|
|
|
> > is a list with a single string which contains extra
|
|
|
> > compiler options with which to run the test. eg.
|
|
|
> >
|
|
|
> > ```wiki
|
|
|
> > test('tc001', normal, compile, ['-fglasgow-exts'])
|
|
|
> > ```
|
|
|
> >
|
|
|
> >
|
|
|
> > would pass the flag -fglasgow-exts to the compiler
|
|
|
> > when compiling tc001.
|
|
|
To use more than one modifier on a test, just put them in a list.
|
|
|
For example, to expect an exit code of 3 and omit way 'opt', we could use
|
|
|
|
|
|
> >
|
|
|
> > The multimod_ versions of compile and compile_and_run
|
|
|
> > expect an extra argument on the front of the list: the
|
|
|
> > name of the top module in the program to be compiled
|
|
|
> > (usually this will be 'Main').
|
|
|
```wiki
|
|
|
[ omit_ways(['opt']), exit_code(3) ]
|
|
|
```
|
|
|
|
|
|
|
|
|
A multi-module test is straightforward. It usually goes in a
|
|
|
directory of its own (although this isn't essential), and the source
|
|
|
files can be named anything you like. The test must have a name, in
|
|
|
the same way as a single-module test; and the stdin/stdout/stderr
|
|
|
files follow the name of the test as before. In the same directory,
|
|
|
place a file 'test.T' containing a line like
|
|
|
as the `<setup>` argument.
|
|
|
|
|
|
## The \<test-fn\> field
|
|
|
|
|
|
*\<test-fn\>*
|
|
|
is a function which describes how the test should be
|
|
|
run, and determines the form of \<args\>. The possible
|
|
|
values are:
|
|
|
|
|
|
- **compile** Just compile the program, the compilation should succeed.
|
|
|
|
|
|
- **compile_fail**
|
|
|
Just compile the program, the
|
|
|
compilation should fail (error
|
|
|
messages will be in T.stderr).
|
|
|
This kind of failure is mandated by the language definition - it does **not** indicate any bug in the compiler.
|
|
|
|
|
|
- **compile_and_run**
|
|
|
Compile the program and run it,
|
|
|
comparing the output against the
|
|
|
relevant files.
|
|
|
|
|
|
- **multimod_compile**
|
|
|
Compile a multi-module program
|
|
|
(more about multi-module programs
|
|
|
below).
|
|
|
|
|
|
- **multimod_compile_fail**
|
|
|
Compile a multi-module program,
|
|
|
and expect the compilation to fail
|
|
|
with error messages in T.stderr. This kind of failure does **not** indicate a bug in the compiler.
|
|
|
|
|
|
- **multimod_compile_and_run**
|
|
|
Compile and run a multi-module
|
|
|
program.
|
|
|
|
|
|
- **compile_and_run_with_prefix**
|
|
|
Same as compile_and_run, but with command to use to run the execution of the result binary.
|
|
|
|
|
|
- **multimod_compile_and_run_with_prefix**
|
|
|
Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
|
|
|
|
|
|
- **run_command**
|
|
|
Just run an arbitrary command. The output is checked
|
|
|
against `T.stdout` and `T.stderr` (unless `ignore_output`
|
|
|
is used), and the stdin and expected exit code can be
|
|
|
changed in the same way as for compile_and_run.
|
|
|
|
|
|
- **ghci_script**
|
|
|
Runs the current compiler, passing
|
|
|
--interactive and using the specified
|
|
|
script as standard input.
|
|
|
|
|
|
## The \<args\> field
|
|
|
|
|
|
*\<args\>* is a list of arguments to be passed to \<test-fn\>.
|
|
|
|
|
|
|
|
|
For compile, compile_fail and compile_and_run, \<args\>
|
|
|
is a list with a single string which contains extra
|
|
|
compiler options with which to run the test. eg.
|
|
|
|
|
|
```wiki
|
|
|
test(multimod001, normal, multimod_compile_and_run, \
|
|
|
[ 'Main', '-fglasgow-exts', '', 0 ])
|
|
|
test('tc001', normal, compile, ['-fglasgow-exts'])
|
|
|
```
|
|
|
|
|
|
|
|
|
as described above.
|
|
|
would pass the flag -fglasgow-exts to the compiler
|
|
|
when compiling tc001.
|
|
|
|
|
|
|
|
|
For some examples, take a look in tests/ghc-regress/programs.
|
|
|
The multimod_ versions of compile and compile_and_run
|
|
|
expect an extra argument on the front of the list: the
|
|
|
name of the top module in the program to be compiled
|
|
|
(usually this will be 'Main').
|
|
|
|
|
|
# Sample output files
|
|
|
|
... | ... | |