... | ... | @@ -10,21 +10,40 @@ For adding any test case, follow these guide lines and then refer to the more sp |
|
|
regression test cases go in the typechecker/ directory, parser test
|
|
|
cases go in parser/, and so on.
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> It's not always possible to find a single best place for a test case;
|
|
|
> in those cases just pick one which seems reasonable.
|
|
|
>
|
|
|
>
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> Under each main directory there are usually up to three subdirectories:
|
|
|
>
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > **should_compile**: test cases which need to compile only
|
|
|
> >
|
|
|
> > **should_fail**: test cases which should fail to compile and generate a particular error message
|
|
|
> >
|
|
|
> > **should_run**: test cases which should compile, run with some specific input, and generate a particular output.
|
|
|
> >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> We don't always divide the test cases up like this, and it's not
|
|
|
> essential to do so. The directory names have no meaning as
|
|
|
> far as the test driver is concerned, it is simply a convention.
|
|
|
>
|
|
|
>
|
|
|
|
|
|
1. Having found a suitable place for the test case, give the test case a name.
|
|
|
For regression test cases, we often just name the test case after the bug
|
... | ... | @@ -33,46 +52,97 @@ For adding any test case, follow these guide lines and then refer to the more sp |
|
|
test cases are named tc001, tc002, and so on. Suppose you name your test
|
|
|
case T, then you'll have the following files:
|
|
|
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > `T.hs`
|
|
|
> >
|
|
|
> >
|
|
|
> > >
|
|
|
> > >
|
|
|
> > > The source file(s) containing the test case. Details on
|
|
|
> > > how to handle single Vs multi source test cases are
|
|
|
> > > explained below.
|
|
|
> > >
|
|
|
> > >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
>
|
|
|
> >
|
|
|
> > >
|
|
|
> > >
|
|
|
> > > **If your test depends on source files that don't start with name of the test, you have to specify them using the `extra_files` setup function (see below).**
|
|
|
> > >
|
|
|
> > >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > `T.stdin` (for test cases that run, and optional)
|
|
|
> >
|
|
|
> >
|
|
|
> > >
|
|
|
> > >
|
|
|
> > > A file to feed the test case as standard input when it
|
|
|
> > > runs.
|
|
|
> > >
|
|
|
> > >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > `T.stdout` (for test cases that run, and optional)
|
|
|
> >
|
|
|
> >
|
|
|
> > >
|
|
|
> > >
|
|
|
> > > For test cases that run, this file is compared against
|
|
|
> > > the standard output generated by the program. If
|
|
|
> > > T.stdout does not exist, then the program must not
|
|
|
> > > generate anything on stdout.
|
|
|
> > >
|
|
|
> > >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > `T.stderr` (optional)
|
|
|
> >
|
|
|
> >
|
|
|
> > >
|
|
|
> > >
|
|
|
> > > For test cases that run, this file is compared
|
|
|
> > > against the standard error generated by the program.
|
|
|
> > >
|
|
|
> > >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
>
|
|
|
> >
|
|
|
> > >
|
|
|
> > >
|
|
|
> > > For test cases that compile only, this file is compared
|
|
|
> > > against the standard error output of the compiler,
|
|
|
> > > which is normalised to eliminate bogus differences
|
|
|
> > > (e.g. absolute pathnames are removed, whitespace
|
|
|
> > > differences are ignored, etc.)
|
|
|
> > >
|
|
|
> > >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
1. Edit all.T in the relevant directory and add a line for the test case. The line is always of the form
|
|
|
|
|
|
```
|
|
|
test(<name>,<setup>,<test-fn>,<args...>)
|
|
|
test(<name>, <setup>, <test-fn>, <args...>)
|
|
|
```
|
|
|
|
|
|
The format of this line is explained in more detail below as it differs for test case types.
|
... | ... | @@ -82,12 +152,22 @@ For adding any test case, follow these guide lines and then refer to the more sp |
|
|
dependent on how complex it is to build your test case. The \<test-fn\> specifies a build method
|
|
|
more then anything else.
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> Note also that the all.T file is simply a python source file that gets executed by the test
|
|
|
> framework. Hence any Python code in it is valid.
|
|
|
>
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> Below we will look at some of the more common test case setups.
|
|
|
>
|
|
|
>
|
|
|
|
|
|
## A single module test case
|
|
|
|
... | ... | @@ -96,24 +176,28 @@ A single module test case is very easy. Simply name the Haskell source files the |
|
|
name (so T.hs in our running example).
|
|
|
|
|
|
|
|
|
|
|
|
Then for a test case that should compile and run fine we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('cgrun001', normal, compile_and_run,[''])
|
|
|
test('cgrun001', normal, compile_and_run, [''])
|
|
|
```
|
|
|
|
|
|
|
|
|
For a test case that should compile but you don't want run, we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('cg002', normal,compile,[''])
|
|
|
test('cg002', normal, compile, [''])
|
|
|
```
|
|
|
|
|
|
|
|
|
For a test case that should fail during compilation we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('drvfail001', normal, compile_fail,[''])
|
|
|
test('drvfail001', normal, compile_fail, [''])
|
|
|
```
|
|
|
|
|
|
|
... | ... | @@ -125,32 +209,40 @@ For more detailed control of a test case, see below. \\REF |
|
|
A multiple module test case is slightly more complex then a single module one. Firstly we a concerned with how to handle the simplest form of a multiple module test case, that is one where the whole test case can be built in one go using the `--make` command of GHC. If you have more complex needs (like compiling source files that `--make` can't handle, and/or need to compile different modules with different GHC arguments, then see below)
|
|
|
|
|
|
|
|
|
|
|
|
Then for a test case that should compile and run fine we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test(multimod001, normal, multimod_compile_and_run, \
|
|
|
['Main','-fglasgow-exts',''])
|
|
|
['Main', '-fglasgow-exts', ''])
|
|
|
```
|
|
|
|
|
|
|
|
|
This example would compile a multiple module test case where the top module is Main.hs and `-fglasgow-exts` is passed to GHC when compiling.
|
|
|
|
|
|
|
|
|
|
|
|
For a test case that should compile but you don't want run, we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('T3286',[], multimod_compile,['T3286','-v0'])
|
|
|
test('T3286', [], multimod_compile, ['T3286', '-v0'])
|
|
|
```
|
|
|
|
|
|
|
|
|
This example would compile a multiple module test case where the top module is T3286 and before compiling the test the files T3286b.o and T3286b.hi are removed.
|
|
|
|
|
|
|
|
|
|
|
|
For a test case that should fail during compilation we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('Over',[],
|
|
|
multimod_compile_fail,['OverD','-no-hs-main -c -v0'])
|
|
|
test('Over',
|
|
|
[],
|
|
|
multimod_compile_fail,
|
|
|
['OverD', '-no-hs-main -c -v0'])
|
|
|
```
|
|
|
|
|
|
## Advanced multiple module test case
|
... | ... | @@ -159,27 +251,40 @@ test('Over',[], |
|
|
If you have a test case that can't be built with the simpler two methods described above then you should try the method described below. The build method below allows you to explicitly provide a list of `(source file, GHC flags)` tuples. GHC then builds those in the order you specify. This is useful for test cases say that use a .cmm source file or .c source file, these are files that GHC can build but aren't picked up by `--make`.
|
|
|
|
|
|
|
|
|
|
|
|
Then for a test case that should compile and run fine we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('cgrun069', omit_ways(['ghci']), multi_compile_and_run,['cgrun069',[('cgrun069_cmm.cmm','')],''])
|
|
|
test('cgrun069', omit_ways(['ghci']), multi_compile_and_run,
|
|
|
['cgrun069', [('cgrun069_cmm.cmm', '')], ''])
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
This test case relies on a .cmm file, hence it can't use the simpler `multimod_compile_and_run` \<test-fn\>. We also see here how we can stop a test case running in a certain WAY.
|
|
|
|
|
|
|
|
|
|
|
|
For a test case that should compile but you don't want run, we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('Check02', normal, multi_compile,['Check02',[('Check02_A.hs',''),('Check02_B.hs','')],'-trust base'])
|
|
|
test('Check02', normal, multi_compile, ['Check02', [
|
|
|
('Check02_A.hs', ''),
|
|
|
('Check02_B.hs', '')
|
|
|
], '-trust base'])
|
|
|
```
|
|
|
|
|
|
|
|
|
For a test case that should fail during compilation we would put this line in all.T:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('Check01', normal, multi_compile_fail,['Check01',[('Check01_A.hs',''),('Check01_B.hs','-trust base')],''])
|
|
|
test('Check01', normal, multi_compile_fail, ['Check01', [
|
|
|
('Check01_A.hs', ''),
|
|
|
('Check01_B.hs', '-trust base')
|
|
|
], ''])
|
|
|
```
|
|
|
|
|
|
|
... | ... | @@ -188,10 +293,12 @@ This test case must use the `multi_compile_fail` method as it relies on being ab |
|
|
## Format of the test entries in all.T
|
|
|
|
|
|
|
|
|
|
|
|
Each test in a `test.T` file is specified by a line the form
|
|
|
|
|
|
|
|
|
```
|
|
|
test(<name>,<setup>,<test-fn>,<args...>)
|
|
|
test(<name>, <setup>, <test-fn>, <args...>)
|
|
|
```
|
|
|
|
|
|
|
... | ... | @@ -227,7 +334,7 @@ There are many pre-defined functions which can be used in this field: |
|
|
- **extra_files(files)** extra files that the test depends on. By default the testsuite driver assumes tests only depend on files that start with the name of the test `i.e. (<testname>*)`. For the time being, `extra_files` can also be specified in the file `testsuite/driver/extra_files.py`.
|
|
|
|
|
|
```
|
|
|
test('prog013', extra_files(['Bad.hs','Good.hs']), ghci_script,['prog013.script'])`
|
|
|
test('prog013', extra_files(['Bad.hs', 'Good.hs']), ghci_script, ['prog013.script'])`
|
|
|
```
|
|
|
|
|
|
- **reqlib(P)** requires package P
|
... | ... | @@ -309,7 +416,8 @@ There are a number of predefined lists of the ways meeting various criteria: |
|
|
|
|
|
In some cases you may want to re-use the same stdout file for multiple tests. You can accomplish this using the `use_specs` function.
|
|
|
|
|
|
- **use_specs** allows one to override files based on suffixes. e.g. `'stdout'`, `'stderr'`, `'asm'`, `'prof.sample'`, etc. Example `use_specs({'stdout' : 'prof002.stdout'})` to make the test re-use `prof002.stdout`.
|
|
|
|
|
|
- **use_specs** allows one to override files based on suffixes. e.g. `'stdout'`, `'stderr'`, `'asm'`, `'prof.sample'`, etc. You must also include this file as `extra_files`. Example `use_specs({'stdout' : 'prof002.stdout'})` to make the test re-use `prof002.stdout`.
|
|
|
|
|
|
|
|
|
To use more than one modifier on a test, just put them in a list.
|
... | ... | @@ -333,12 +441,17 @@ exception that the `collect_compiler_stats()` function measures the performance |
|
|
More documentation can be found in the `driver/README.md` file, or the comments in `driver/perf_notes.py`.
|
|
|
|
|
|
|
|
|
|
|
|
Here's an example test:
|
|
|
|
|
|
|
|
|
```
|
|
|
test('perf001',[ collect_compiler_stats('bytes allocated',10)],compile,[''])
|
|
|
test('ticketNumber',[ collect_stats()],
|
|
|
compile_and_run,[''])
|
|
|
test('perf001',
|
|
|
[ collect_compiler_stats('bytes allocated',10) ],
|
|
|
compile, [''])
|
|
|
test('ticketNumber',
|
|
|
[ collect_stats() ],
|
|
|
compile_and_run, [''])
|
|
|
```
|
|
|
|
|
|
|
... | ... | @@ -378,10 +491,9 @@ In summary: |
|
|
|
|
|
- Tests which measure the performance of the *program*, not the compiler, should be used in conjunction with a **compile_and_run** test.
|
|
|
|
|
|
### The \<test-fn\> field
|
|
|
|
|
|
See [Running Performance Tests](building/running-tests/performance-tests) on how to run these tests.
|
|
|
|
|
|
### The \<test-fn\> field
|
|
|
|
|
|
*\<test-fn\>* is a function which describes how the test should be
|
|
|
built and maybe run. It also determines the number of arguments for
|
... | ... | @@ -402,24 +514,47 @@ file. The possible test functions are: |
|
|
This is the simplest test function and can only handle compiling a single module test case.
|
|
|
The source file to compile must correspond to the \<name\> of the test.
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> *\<args...\> = \[\<extra_hc_opts\>\]*
|
|
|
>
|
|
|
>
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> Where:
|
|
|
>
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > *\<extra_hc_opts\>*: arguments to pass to GHC when it compiles your test case.
|
|
|
> >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
- **multimod_compile**, **multimod_compile_fail**, **multimod_compile_and_run**:
|
|
|
|
|
|
Compile a multi-module program using the GHC `--make` build system.
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> *\<args...\> = \[\<topmod\>, \<extra_hc_opts\>\]*
|
|
|
>
|
|
|
>
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> Where:
|
|
|
>
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > *\<topmod\>*: The top level source file for your test case.
|
|
|
> >
|
|
|
> > *\<extra_hc_opts\>*: arguments to pass to GHC when it compiles your test case.
|
|
|
> >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
- **multi_compile**, **multi_compile_fail**, **multi_compile_and_run**:
|
|
|
|
... | ... | @@ -427,14 +562,27 @@ file. The possible test functions are: |
|
|
not enough, such as when you first need to compile a .c or .cmm file before compiling the
|
|
|
Haskell top level module.
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> *\<args...\> = \[\<topmod\>, \[(\<extra_mod\>, \<hc_opts\>)\], \<extra_hc_opts\>\]*
|
|
|
>
|
|
|
>
|
|
|
|
|
|
>
|
|
|
>
|
|
|
> Where:
|
|
|
>
|
|
|
>
|
|
|
> >
|
|
|
> >
|
|
|
> > *\<topmod\>*: The top level source file for your test case.
|
|
|
> >
|
|
|
> > *\[(\<extra_mod\>, \<hc_opts\>)\]*: A list of tuples where the first element is a source file for GHC to compile and the second element are arguments GHC should use to compile that particular source file.
|
|
|
> >
|
|
|
> > *\<extra_hc_opts\>*: arguments to pass to GHC when it compiles your test case (applied to all source files).
|
|
|
> >
|
|
|
> >
|
|
|
>
|
|
|
|
|
|
- **run_command**
|
|
|
Just run an arbitrary command. The output is checked
|
... | ... | |