... | ... | @@ -10,84 +10,70 @@ For adding any test case, follow these guide lines and then refer to the more sp |
|
|
regression test cases go in the typechecker/ directory, parser test
|
|
|
cases go in parser/, and so on.
|
|
|
|
|
|
>
|
|
|
> It's not always possible to find a single best place for a test case;
|
|
|
> in those cases just pick one which seems reasonable.
|
|
|
It's not always possible to find a single best place for a test case; in those cases just pick one which seems reasonable.
|
|
|
|
|
|
>
|
|
|
> Under each main directory there are usually up to three subdirectories:
|
|
|
>
|
|
|
> > **should_compile**: test cases which need to compile only
|
|
|
> > **should_fail**: test cases which should fail to compile and generate a particular error message
|
|
|
> > **should_run**: test cases which should compile, run with some specific input, and generate a particular output.
|
|
|
Under each main directory there are usually up to three subdirectories:
|
|
|
|
|
|
>
|
|
|
> We don't always divide the test cases up like this, and it's not
|
|
|
> essential to do so. The directory names have no meaning as
|
|
|
> far as the test driver is concerned, it is simply a convention.
|
|
|
- **should_compile**: test cases which need to compile only
|
|
|
- **should_fail**: test cases which should fail to compile and generate a particular error message
|
|
|
- **should_run**: test cases which should compile, run with some specific input, and generate a particular output.
|
|
|
|
|
|
We don't always divide the test cases up like this, and it's not essential to do so. The directory names have no meaning as far as the test driver is concerned, it is simply a convention.
|
|
|
|
|
|
1. Having found a suitable place for the test case, give the test case a name.
|
|
|
2. Having found a suitable place for the test case, give the test case a name.
|
|
|
For regression test cases, we often just name the test case after the bug
|
|
|
number (e.g. T2047). Alternatively, follow the convention for the directory
|
|
|
in which you place the test case: for example, in typecheck/should_compile,
|
|
|
test cases are named tc001, tc002, and so on. Suppose you name your test
|
|
|
case T, then you'll have the following files:
|
|
|
|
|
|
> > `T.hs`
|
|
|
> >
|
|
|
> > >
|
|
|
> > > The source file(s) containing the test case. Details on
|
|
|
> > > how to handle single Vs multi source test cases are
|
|
|
> > > explained below.
|
|
|
|
|
|
> > > **If your test depends on source files that don't start with name of the test, you have to specify them using the `extra_files` setup function (see below).**
|
|
|
|
|
|
> > `T.stdin` (for test cases that run, and optional)
|
|
|
> >
|
|
|
> > >
|
|
|
> > > A file to feed the test case as standard input when it
|
|
|
> > > runs.
|
|
|
|
|
|
> > `T.stdout` (for test cases that run, and optional)
|
|
|
> >
|
|
|
> > >
|
|
|
> > > For test cases that run, this file is compared against
|
|
|
> > > the standard output generated by the program. If
|
|
|
> > > T.stdout does not exist, then the program must not
|
|
|
> > > generate anything on stdout.
|
|
|
|
|
|
> > `T.stderr` (optional)
|
|
|
> >
|
|
|
> > >
|
|
|
> > > For test cases that run, this file is compared
|
|
|
> > > against the standard error generated by the program.
|
|
|
|
|
|
> > >
|
|
|
> > > For test cases that compile only, this file is compared
|
|
|
> > > against the standard error output of the compiler,
|
|
|
> > > which is normalised to eliminate bogus differences
|
|
|
> > > (e.g. absolute pathnames are removed, whitespace
|
|
|
> > > differences are ignored, etc.)
|
|
|
|
|
|
1. Edit all.T in the relevant directory and add a line for the test case. The line is always of the form
|
|
|
|
|
|
```
|
|
|
test(<name>,<setup>,<test-fn>,<args...>)
|
|
|
```
|
|
|
|
|
|
The format of this line is explained in more detail below as it differs for test case types.
|
|
|
- **`T.hs`**
|
|
|
|
|
|
The source file(s) containing the test case. Details on
|
|
|
how to handle single Vs multi source test cases are
|
|
|
explained below.
|
|
|
|
|
|
**If your test depends on source files that don't start with name of the test, you have to specify them using the `extra_files` setup function (see below).**
|
|
|
|
|
|
- **`T.stdin` (for test cases that run, and optional)**
|
|
|
|
|
|
A file to feed the test case as standard input when it runs.
|
|
|
|
|
|
- **`T.stdout` (for test cases that run, and optional)**
|
|
|
|
|
|
For test cases that run, this file is compared against
|
|
|
the standard output generated by the program.
|
|
|
If T.stdout does not exist, then the program must not
|
|
|
generate anything on stdout.
|
|
|
|
|
|
- **`T.stderr` (optional)**
|
|
|
|
|
|
For test cases that run, this file is compared
|
|
|
against the standard error generated by the program.
|
|
|
|
|
|
For test cases that compile only, this file is compared
|
|
|
against the standard error output of the compiler,
|
|
|
which is normalised to eliminate bogus differences
|
|
|
(e.g. absolute pathnames are removed, whitespace
|
|
|
differences are ignored, etc.)
|
|
|
|
|
|
3. Edit all.T in the relevant directory and add a line for the test case. The line is always of the form
|
|
|
|
|
|
```
|
|
|
test(<name>,<setup>,<test-fn>,<args...>)
|
|
|
```
|
|
|
|
|
|
The format of this line is explained in more detail below as it differs for test case types.
|
|
|
It allows you to say if the test case should fail to compile, run fine, run but terminate
|
|
|
with a certain exit code... ect. The \<args...\> argument is a list argument, where the length
|
|
|
and format of the list depends on the \<test-fn\> you use. The choice of \<test-fn\> is largely
|
|
|
dependent on how complex it is to build your test case. The \<test-fn\> specifies a build method
|
|
|
more then anything else.
|
|
|
|
|
|
>
|
|
|
> Note also that the all.T file is simply a python source file that gets executed by the test
|
|
|
> framework. Hence any Python code in it is valid.
|
|
|
Note also that the all.T file is simply a python source file that gets executed by the test
|
|
|
framework. Hence any Python code in it is valid.
|
|
|
|
|
|
>
|
|
|
> Below we will look at some of the more common test case setups.
|
|
|
Below we will look at some of the more common test case setups.
|
|
|
|
|
|
## A single module test case
|
|
|
|
... | ... | |