... | @@ -228,10 +228,8 @@ Where \<args...\> is a list of arguments. |
... | @@ -228,10 +228,8 @@ Where \<args...\> is a list of arguments. |
|
|
|
|
|
### The \<setup\> field
|
|
### The \<setup\> field
|
|
|
|
|
|
*\<setup\>* is a function (i.e. any callable object in Python)
|
|
*\<setup\>* is a function (i.e. any callable object in Python) which allows the options for this test to be changed.
|
|
which allows the options for this test to be changed.
|
|
There are many pre-defined functions which can be used in this field:
|
|
There are many pre-defined functions which can be
|
|
|
|
used in this field:
|
|
|
|
|
|
|
|
- **normal** don't change any options from the defaults
|
|
- **normal** don't change any options from the defaults
|
|
- **skip** skip this test
|
|
- **skip** skip this test
|
... | @@ -310,6 +308,31 @@ For example, to expect an exit code of 3 and omit way 'opt', we could use |
... | @@ -310,6 +308,31 @@ For example, to expect an exit code of 3 and omit way 'opt', we could use |
|
|
|
|
|
as the `<setup>` argument.
|
|
as the `<setup>` argument.
|
|
|
|
|
|
|
|
### Performance tests
|
|
|
|
|
|
|
|
|
|
|
|
Performance tests can specify ranges for certain statistics in the `<setup>` field. Here's an example test:
|
|
|
|
|
|
|
|
```wiki
|
|
|
|
test('perf001',
|
|
|
|
[ if_wordsize(32,
|
|
|
|
compiler_stats_range_field('bytes allocated', 40000000, 10)),
|
|
|
|
if_wordsize(64,
|
|
|
|
compiler_stats_range_field('bytes allocated', 79110184, 10))
|
|
|
|
],
|
|
|
|
compile,[''])
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
This is testing the performance of GHC itself, and requiring that the statistic 'bytes allocated' for the compiler when compiling the module `perf001.hs` is +/- 10% of 40000000 bytes (on a 32-bit machine; there is a different baseline for 64-bit machines).
|
|
|
|
|
|
|
|
|
|
|
|
The kinds of constraint that can be used are:
|
|
|
|
|
|
|
|
- **compiler_stats_range_field(stat, baseline, deviation)** tests the performance of GHC, and should be used with **compile** or **compile_fail** tests. **stat** is one of the following: `'bytes allocated'`, `'peak_megabytes_allocated'`, or `'max_bytes_used'`; **baseline** is the baseline value obtained by running the benchmark, and **deviation** is the percentage deviation from the baseline that the framework will allow for the test to pass.
|
|
|
|
|
|
|
|
- **stats_range_field(stat, baseline, deviation)** is the same, but tests the performance of the *program*, not the compiler. It should be used in conjunction with a **compile_and_run** test.
|
|
|
|
|
|
### The \<test-fn\> field
|
|
### The \<test-fn\> field
|
|
|
|
|
|
*\<test-fn\>* is a function which describes how the test should be
|
|
*\<test-fn\>* is a function which describes how the test should be
|
... | | ... | |