... | @@ -10,84 +10,70 @@ For adding any test case, follow these guide lines and then refer to the more sp |
... | @@ -10,84 +10,70 @@ For adding any test case, follow these guide lines and then refer to the more sp |
|
regression test cases go in the typechecker/ directory, parser test
|
|
regression test cases go in the typechecker/ directory, parser test
|
|
cases go in parser/, and so on.
|
|
cases go in parser/, and so on.
|
|
|
|
|
|
>
|
|
It's not always possible to find a single best place for a test case; in those cases just pick one which seems reasonable.
|
|
> It's not always possible to find a single best place for a test case;
|
|
|
|
> in those cases just pick one which seems reasonable.
|
|
|
|
|
|
|
|
>
|
|
Under each main directory there are usually up to three subdirectories:
|
|
> Under each main directory there are usually up to three subdirectories:
|
|
|
|
>
|
|
|
|
> > **should_compile**: test cases which need to compile only
|
|
|
|
> > **should_fail**: test cases which should fail to compile and generate a particular error message
|
|
|
|
> > **should_run**: test cases which should compile, run with some specific input, and generate a particular output.
|
|
|
|
|
|
|
|
>
|
|
- **should_compile**: test cases which need to compile only
|
|
> We don't always divide the test cases up like this, and it's not
|
|
- **should_fail**: test cases which should fail to compile and generate a particular error message
|
|
> essential to do so. The directory names have no meaning as
|
|
- **should_run**: test cases which should compile, run with some specific input, and generate a particular output.
|
|
> far as the test driver is concerned, it is simply a convention.
|
|
|
|
|
|
We don't always divide the test cases up like this, and it's not essential to do so. The directory names have no meaning as far as the test driver is concerned, it is simply a convention.
|
|
|
|
|
|
1. Having found a suitable place for the test case, give the test case a name.
|
|
2. Having found a suitable place for the test case, give the test case a name.
|
|
For regression test cases, we often just name the test case after the bug
|
|
For regression test cases, we often just name the test case after the bug
|
|
number (e.g. T2047). Alternatively, follow the convention for the directory
|
|
number (e.g. T2047). Alternatively, follow the convention for the directory
|
|
in which you place the test case: for example, in typecheck/should_compile,
|
|
in which you place the test case: for example, in typecheck/should_compile,
|
|
test cases are named tc001, tc002, and so on. Suppose you name your test
|
|
test cases are named tc001, tc002, and so on. Suppose you name your test
|
|
case T, then you'll have the following files:
|
|
case T, then you'll have the following files:
|
|
|
|
|
|
> > `T.hs`
|
|
- **`T.hs`**
|
|
> >
|
|
|
|
> > >
|
|
The source file(s) containing the test case. Details on
|
|
> > > The source file(s) containing the test case. Details on
|
|
how to handle single Vs multi source test cases are
|
|
> > > how to handle single Vs multi source test cases are
|
|
explained below.
|
|
> > > explained below.
|
|
|
|
|
|
**If your test depends on source files that don't start with name of the test, you have to specify them using the `extra_files` setup function (see below).**
|
|
> > > **If your test depends on source files that don't start with name of the test, you have to specify them using the `extra_files` setup function (see below).**
|
|
|
|
|
|
- **`T.stdin` (for test cases that run, and optional)**
|
|
> > `T.stdin` (for test cases that run, and optional)
|
|
|
|
> >
|
|
A file to feed the test case as standard input when it runs.
|
|
> > >
|
|
|
|
> > > A file to feed the test case as standard input when it
|
|
- **`T.stdout` (for test cases that run, and optional)**
|
|
> > > runs.
|
|
|
|
|
|
For test cases that run, this file is compared against
|
|
> > `T.stdout` (for test cases that run, and optional)
|
|
the standard output generated by the program.
|
|
> >
|
|
If T.stdout does not exist, then the program must not
|
|
> > >
|
|
generate anything on stdout.
|
|
> > > For test cases that run, this file is compared against
|
|
|
|
> > > the standard output generated by the program. If
|
|
- **`T.stderr` (optional)**
|
|
> > > T.stdout does not exist, then the program must not
|
|
|
|
> > > generate anything on stdout.
|
|
For test cases that run, this file is compared
|
|
|
|
against the standard error generated by the program.
|
|
> > `T.stderr` (optional)
|
|
|
|
> >
|
|
For test cases that compile only, this file is compared
|
|
> > >
|
|
against the standard error output of the compiler,
|
|
> > > For test cases that run, this file is compared
|
|
which is normalised to eliminate bogus differences
|
|
> > > against the standard error generated by the program.
|
|
(e.g. absolute pathnames are removed, whitespace
|
|
|
|
differences are ignored, etc.)
|
|
> > >
|
|
|
|
> > > For test cases that compile only, this file is compared
|
|
3. Edit all.T in the relevant directory and add a line for the test case. The line is always of the form
|
|
> > > against the standard error output of the compiler,
|
|
|
|
> > > which is normalised to eliminate bogus differences
|
|
```
|
|
> > > (e.g. absolute pathnames are removed, whitespace
|
|
test(<name>,<setup>,<test-fn>,<args...>)
|
|
> > > differences are ignored, etc.)
|
|
```
|
|
|
|
|
|
1. Edit all.T in the relevant directory and add a line for the test case. The line is always of the form
|
|
The format of this line is explained in more detail below as it differs for test case types.
|
|
|
|
|
|
```
|
|
|
|
test(<name>,<setup>,<test-fn>,<args...>)
|
|
|
|
```
|
|
|
|
|
|
|
|
The format of this line is explained in more detail below as it differs for test case types.
|
|
|
|
It allows you to say if the test case should fail to compile, run fine, run but terminate
|
|
It allows you to say if the test case should fail to compile, run fine, run but terminate
|
|
with a certain exit code... ect. The \<args...\> argument is a list argument, where the length
|
|
with a certain exit code... ect. The \<args...\> argument is a list argument, where the length
|
|
and format of the list depends on the \<test-fn\> you use. The choice of \<test-fn\> is largely
|
|
and format of the list depends on the \<test-fn\> you use. The choice of \<test-fn\> is largely
|
|
dependent on how complex it is to build your test case. The \<test-fn\> specifies a build method
|
|
dependent on how complex it is to build your test case. The \<test-fn\> specifies a build method
|
|
more then anything else.
|
|
more then anything else.
|
|
|
|
|
|
>
|
|
Note also that the all.T file is simply a python source file that gets executed by the test
|
|
> Note also that the all.T file is simply a python source file that gets executed by the test
|
|
framework. Hence any Python code in it is valid.
|
|
> framework. Hence any Python code in it is valid.
|
|
|
|
|
|
|
|
>
|
|
Below we will look at some of the more common test case setups.
|
|
> Below we will look at some of the more common test case setups.
|
|
|
|
|
|
|
|
## A single module test case
|
|
## A single module test case
|
|
|
|
|
... | | ... | |