(StatusCodeException (Response {responseStatus = Status {statusCode = 403, statusMessage = "Forbidden"}, responseVersion = HTTP/1.1, responseHeaders = [("Date","Sun, 10 Mar 2019 06:58:16 GMT"),("Server","Apache/2.2.22 (Debian)"),("Strict-Transport-Security","max-age=63072000; includeSubDomains"),("Vary","Accept-Encoding"),("Content-Encoding","gzip"),("Content-Length","259"),("Content-Type","text/html; charset=iso-8859-1")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access /trac/ghc/wiki/Building/RunningTests\non this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at ghc.haskell.org Port 443</address>\n</body></html>\n"))
Original source:
```trac
[[PageOutline]]
= GHC Test framework =
NOTE: you need Python (any version >= 1.5 will probably do) in order
# GHC Test framework
NOTE: you need Python (any version \>= 1.5 will probably do) in order
to use the testsuite.
To run the test suite against stage 1 of a GHC build in the same
source tree:
{{{
```wiki
cd tests/ghc-regress
make
}}}
```
(from now on, we'll assume that you're in the tests/ghc-regress
directory).
To run a fast version of the testsuite, which should complete in under
5 minutes on a fast machine with an optimised GHC build:
{{{
```wiki
make fast
}}}
```
To run the testsuite with the stage2 compiler (this is often what you
want, because GHCi tests will fail with stage1):
{{{
```wiki
make stage=2
}}}
```
To run the test suite against a different GHC, say ghc-5.04:
{{{
```wiki
make TEST_HC=ghc-5.04
}}}
```
To run an individual test or tests (eg. tc054):
{{{
```wiki
make TEST=tc054
}}}
```
(you can also go straight to the directory containing the test and say
'make TEST=tc054' from there, which will save some time).
To run the tests one particular way only (eg. GHCi):
{{{
```wiki
make WAY=ghci
}}}
```
To add specific options to the compiler:
```wiki
make EXTRA_HC_OPTS='+RTS -K32M -RTS'
```
For more details, see below.
= Running the testsuite with a compiler other than GHC =
# Running the testsuite with a compiler other than GHC
(to be written. The plan is something like:
{{{
```wiki
cvs checkout fpconfig
cd fptools
cvs checkout testsuite
...
...
@@ -72,20 +81,27 @@ For more details, see below.
./configure
cd testsuite
make TEST_HC=nhc98 COMPILER=nhc98
}}}
```
)
= Running individual tests or subdirectories of the testsuite =
# Running individual tests or subdirectories of the testsuite
Most of the subdirectories in the testsuite have a Makefile. In these
subdirectories you can use 'make' to run the test driver in two
ways:
{{{
```wiki
make -- run all the tests in the current directory
make accept -- run the tests, accepting the current output
}}}
```
The following variables may be set on the make command line:
{{{
```wiki
TESTS -- specific tests to run
TEST_HC -- compiler to use
EXTRA_HC_OPTS -- extra flags to send to the Haskell compiler
...
...
@@ -94,9 +110,12 @@ The following variables may be set on the make command line:
COMPILER -- stem of a different configuration file
-- from the config directory [default: ghc]
WAY -- just this way
}}}
```
The following ways are defined (for GHC, also see the file config/ghc):
{{{
```wiki
normal -- no special options
opt -- -O
optasm -- -O -fasm
...
...
@@ -107,24 +126,32 @@ The following ways are defined (for GHC, also see the file config/ghc):
extcore -- -fext-core
optextcore -- -O -fext-core
threaded -- -threaded
}}}
hpc -- -fhpc
```
certain ways are enabled automatically if the GHC build in the local
tree supports them. Ways that are enabled this way are optasm, prof,
profasm, unreg, threaded, and ghci.
= Updating tests when the output changes =
# Updating tests when the output changes
If the output of a test has changed, but the new output is still
correct, you can automatically update the sample output to match the
new output like so:
{{{
```wiki
make accept TESTS=<test-name>
}}}
where <test-name> is the name of the test. In a directory which
contains a single test, or if you want to update *all* the tests in
the current directory, just omit the 'TESTS=<test-name>' part.
```
where \<test-name\> is the name of the test. In a directory which
contains a single test, or if you want to update \*all\* the tests in
the current directory, just omit the 'TESTS=\<test-name\>' part.
# Adding a new test
= Adding a new test =
For a test which can be encapsulated in a single source file, follow
these steps:
...
...
@@ -136,171 +163,232 @@ these steps:
regression tests go in the typechecker/ directory, parser tests
go in parser/, and so on.
It's not always possible to find a single best place for a test;
in those cases just pick one which seems reasonable.
Under each main directory may be up to three subdirectories:
should_compile:
tests which need to compile only
should_fail:
tests which should fail to compile and generate a particular error message
should_run:
tests which should compile, run with some specific input, and generate a particular output.
We don't always divide the tests up like this, and it's not
essential to do so (the directory names have no meaning as
far as the test driver is concerned).
2. Having found a suitable place for the test, give the test a name.
> >
> > It's not always possible to find a single best place for a test;
> > in those cases just pick one which seems reasonable.
> >
> > Under each main directory may be up to three subdirectories:
> > >
> > > should_compile:
> > >
> > > >
> > > > tests which need to compile only
> > >
> > > should_fail:
> > >
> > > >
> > > > tests which should fail to compile and generate a particular error message
> > >
> > > should_run:
> > >
> > > >
> > > > tests which should compile, run with some specific input, and generate a particular output.
> >
> > We don't always divide the tests up like this, and it's not
> > essential to do so (the directory names have no meaning as
> > far as the test driver is concerned).
1. Having found a suitable place for the test, give the test a name.
Follow the convention for the directory in which you place the
test: for example, in typecheck/should_compile, tests are named
tc001, tc002, and so on. Suppose you name your test T, then
you'll have the following files:
T.hs
The source file containing the test
T.stdin (for tests that run, and optional)
A file to feed the test as standard input when it
runs.
T.stdout (for tests that run, and optional)
For tests that run, this file is compared against
the standard output generated by the program. If
T.stdout does not exist, then the program must not
generate anything on stdout.
T.stderr (optional)
For tests that run, this file is compared
against the standard error generated by the program.
For tests that compile only, this file is compared
against the standard error output of the compiler,
which is normalised to eliminate bogus differences
(eg. absolute pathnames are removed, whitespace
differences are ignored, etc.)
> >
> > T.hs
> >
> > >
> > > The source file containing the test
> >
> > T.stdin (for tests that run, and optional)
> >
> > >
> > > A file to feed the test as standard input when it
> > > runs.
> >
> > T.stdout (for tests that run, and optional)
> >
> > >
> > > For tests that run, this file is compared against
> > > the standard output generated by the program. If
> > > T.stdout does not exist, then the program must not
> > > generate anything on stdout.
> >
> > T.stderr (optional)
> >
> > >
> > > For tests that run, this file is compared
> > > against the standard error generated by the program.
> > >
> > > For tests that compile only, this file is compared
> > > against the standard error output of the compiler,
> > > which is normalised to eliminate bogus differences
> > > (eg. absolute pathnames are removed, whitespace
> > > differences are ignored, etc.)
1. Edit all.T in the relevant directory and add a line for the test. The line is always of the form
{{{
test(<name>, <opt-fn>, <test-fn>, <args>)
}}}
where
''<name>'' is the name of the test, in quotes (' or ").
''<opt-fn>'' is a function (i.e. any callable object in Python)
which allows the options for this test to be changed.
There are several pre-defined functions which can be
used in this field:
'''normal''' don't change any options from the defaults
'''skip''' skip this test
'''omit_ways(ways)''' skip this test for certain ways
'''only_ways(ways)''' do this test certain ways only
'''omit_compiler_types(compilers)''' skip this test for certain compilers
'''only_compiler_types(compilers)''' do this test for certain compilers only
'''expect_fail''' this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
'''expect_fail_for(ways)''' expect failure for certain ways
'''expect_fail_if_platform(plat)''' expect failure on a certain platform
'''expect_fail_if_compiler_type(compiler)''' expect failure from a certain compiler
'''set_stdin(file)''' use a different file for stdin
'''exit_code(n)''' expect an exit code of 'n' from the prog
'''extra_run_opts(opts)''' pass some extra opts to the prog
'''no_clean''' don't clean up after this test
You can compose two of these functions together by
saying compose(f,g). For example, to expect an exit
code of 3 and omit way 'opt', we could use
{{{
compose(omit_ways(['opt']), exit_code(3))
}}}
as the <opt-fn> argument. Calls to compose() can of
course be nested.
''<test-fn>''
is a function which describes how the test should be
run, and determines the form of <args>. The possible
values are:
compile
Just compile the program, the compilation should succeed.
compile_fail
Just compile the program, the
compilation should fail (error
messages will be in T.stderr).
This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler.
compile_and_run
Compile the program and run it,
comparing the output against the
relevant files.
multimod_compile
Compile a multi-module program
(more about multi-module programs
below).
multimod_compile_fail
Compile a multi-module program,
and expect the compilation to fail
with error messages in T.stderr. This kind of failure does '''not''' indicate a bug in the compiler.
multimod_compile_and_run
Compile and run a multi-module
program.
run_command
Just run an arbitrary command. The
output is checked against T.stdout and
T.stderr, and the stdin and expected
exit code can be changed in the same
way as for compile_and_run.
run_command_ignore_output
Same as run_command, except the output
(both stdout and stderr) from the
command is ignored.
ghci_script
Runs the current compiler, passing
--interactive and using the specified
script as standard input.
```wiki
test(<name>, <opt-fn>, <test-fn>, <args>)
```
''<args>''
is a list of arguments to be passed to <test-fn>.
where
For compile, compile_fail and compile_and_run, <args>
is a list with a single string which contains extra
compiler options with which to run the test. eg.
{{{
test(tc001, normal, compile, ['-fglasgow-exts'])
}}}
would pass the flag -fglasgow-exts to the compiler
when compiling tc001.
> > *\<name\>* is the name of the test, in quotes (' or ").
The multimod_ versions of compile and compile_and_run
expect an extra argument on the front of the list: the
name of the top module in the program to be compiled
(usually this will be 'Main').
> > *\<opt-fn\>* is a function (i.e. any callable object in Python)
> > which allows the options for this test to be changed.
> > There are several pre-defined functions which can be
> > used in this field:
> > > **normal** don't change any options from the defaults
> > > **skip** skip this test
> > > **omit_ways(ways)** skip this test for certain ways
> > > **only_ways(ways)** do this test certain ways only
> > > **omit_compiler_types(compilers)** skip this test for certain compilers
> > > **only_compiler_types(compilers)** do this test for certain compilers only
> > > **expect_fail** this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
> > > **expect_fail_for(ways)** expect failure for certain ways
> > > **expect_fail_if_platform(plat)** expect failure on a certain platform
> > > **expect_fail_if_compiler_type(compiler)** expect failure from a certain compiler
> > > **set_stdin(file)** use a different file for stdin
> > > **exit_code(n)** expect an exit code of 'n' from the prog
> > > **extra_run_opts(opts)** pass some extra opts to the prog
> > > **no_clean** don't clean up after this test
> > >
> > > You can compose two of these functions together by
> > > saying compose(f,g). For example, to expect an exit
> > > code of 3 and omit way 'opt', we could use
> > >
> > > ```wiki
> > > compose(omit_ways(['opt']), exit_code(3))
> > > ```
> > >
> > >
> > > as the \<opt-fn\> argument. Calls to compose() can of
> > > course be nested.
> > *\<test-fn\>*
> > is a function which describes how the test should be
> > run, and determines the form of \<args\>. The possible
> > values are:
> > >
> > > compile
> > >
> > > >
> > > > Just compile the program, the compilation should succeed.
> > >
> > > compile_fail
> > >
> > > >
> > > > Just compile the program, the
> > > > compilation should fail (error
> > > > messages will be in T.stderr).
> > > > This kind of failure is mandated by the language definition - it does **not** indicate any bug in the compiler.
> > >
> > > compile_and_run
> > >
> > > >
> > > > Compile the program and run it,
> > > > comparing the output against the
> > > > relevant files.
> > >
> > > multimod_compile
> > >
> > > >
> > > > Compile a multi-module program
> > > > (more about multi-module programs
> > > > below).
> > >
> > > multimod_compile_fail
> > >
> > > >
> > > > Compile a multi-module program,
> > > > and expect the compilation to fail
> > > > with error messages in T.stderr. This kind of failure does **not** indicate a bug in the compiler.
> > >
> > > multimod_compile_and_run
> > >
> > > >
> > > > Compile and run a multi-module
> > > > program.
> > >
> > > run_command
> > >
> > > >
> > > > Just run an arbitrary command. The
> > > > output is checked against T.stdout and
> > > > T.stderr, and the stdin and expected
> > > > exit code can be changed in the same
> > > > way as for compile_and_run.
> > >
> > > run_command_ignore_output
> > >
> > > >
> > > > Same as run_command, except the output
> > > > (both stdout and stderr) from the
> > > > command is ignored.
> > >
> > > ghci_script
> > >
> > > >
> > > > Runs the current compiler, passing
> > > > --interactive and using the specified
> > > > script as standard input.
> > *\<args\>*
> > is a list of arguments to be passed to \<test-fn\>.
> >
> > For compile, compile_fail and compile_and_run, \<args\>
> > is a list with a single string which contains extra
> > compiler options with which to run the test. eg.
