... | ... | @@ -336,6 +336,9 @@ as the `<setup>` argument. |
|
|
|
|
|
Performance tests have recently been revamped significantly and are now much easier to use.
|
|
|
In order to dignify a test as a performance test, it is sufficient to use the `collect_stats()` function.
|
|
|
The `collect_compiler_stats()` function and the `collect_stats()` function are exactly equivalent with the
|
|
|
exception that the `collect_compiler_stats()` function measures the performance of the compiler and the
|
|
|
`collect_stats()` function measures the performance of the code generated by the compiler.
|
|
|
More documentation can be found in the `driver/README.md` file, or the comments in `driver/perf_notes.py`.
|
|
|
|
|
|
|
... | ... | @@ -343,43 +346,41 @@ Here's an example test: |
|
|
|
|
|
```wiki
|
|
|
test('perf001',
|
|
|
[ collect_stats('bytes allocated',10,True) ],
|
|
|
[ collect_compiler_stats('bytes allocated',10) ],
|
|
|
compile, [''])
|
|
|
test('ticketNumber',
|
|
|
[ collect_stats() ],
|
|
|
compile_and_run, [''])
|
|
|
```
|
|
|
|
|
|
|
|
|
This is testing the performance of GHC itself, and requiring that the statistic 'bytes allocated' for the compiler when compiling the module `perf001.hs` is +/- 10% of the value recorded in the previous commit.
|
|
|
The third parameter, True, denotes this as a test which measures the performance of GHC itself. If it is omitted, the test is a test which measures the performance of the 'program' itself.
|
|
|
The first test is testing the performance of GHC itself, and requiring that the statistic 'bytes allocated' for the compiler when compiling the module `perf001.hs` is +/- 10% of the value recorded in the previous commit. The second test is testing the performance of the program contained inside the *ticketNumber* test and requires that the metrics 'bytes allocated', 'peak_megabytes_allocated', and 'max_bytes_used' vary no more than +/- 20% each of the values recorded in the previous commit.
|
|
|
|
|
|
|
|
|
The collect_stats function takes 3 arguments:
|
|
|
The collect_compiler_stats function takes 2 arguments:
|
|
|
|
|
|
```wiki
|
|
|
v- defaults to 'all' v- defaults to 20% v- defaults to False
|
|
|
collect_stats( metrics_to_measure, max_deviation_allowed, is_compiler_performance_test)
|
|
|
v- defaults to 'all' v- defaults to 20%
|
|
|
collect_compiler_stats( metrics_to_measure, max_deviation_allowed)
|
|
|
```
|
|
|
|
|
|
|
|
|
The possible metrics that can be measured are `'bytes allocated'`, `'peak_megabytes_allocated'`, or `'max_bytes_used'`.
|
|
|
|
|
|
|
|
|
For its first parameter, the collect_stats function will take either
|
|
|
For its first parameter, the collect_compiler_stats function will take either
|
|
|
|
|
|
- One of those strings. eg: `collect_stats('bytes allocated')`
|
|
|
- A list of those strings. eg: `collect_stats(['bytes allocated', 'max_bytes_used'])`
|
|
|
- One of those strings. eg: `collect_compiler_stats('bytes allocated')`
|
|
|
- A list of those strings. eg: `collect_compiler_stats(['bytes allocated', 'max_bytes_used'])`
|
|
|
- A string 'all' which is a shorthand for the list containing all 3 possible measurements.
|
|
|
|
|
|
|
|
|
For its second parameter, it takes a non-negative integer as its % of maximum deviation allowed.
|
|
|
A deviation of 5 means that the difference between the expected value and the actual value measured by the test driver can differ by no more than +/- 5%.
|
|
|
This value defaults to 20% as correctness is prioritized over speed and a test should not "fail" when it's "correct" even if it introduces a semi-severe regression.
|
|
|
This value defaults to 20% as correctness is prioritized over speed and a test should not "fail" when it's "correct" even if a potential performance regression is introduced.
|
|
|
|
|
|
|
|
|
For its third parameter, collect_stats takes a boolean. If the boolean is True, the test is considered one that measures the performance of the compiler itself, if it's false, it's a test measuring the performance of the 'program' itself;
|
|
|
this defaults to False as the vast majority of tests in the testsuite are "regular" performance tests.
|
|
|
|
|
|
|
|
|
Since all of the parameters have defaults, the function can be called with either no arguments, just the metric to measure, just the metric to measure and the allowed deviation, or all 3 arguments.
|
|
|
Since all of the parameters have defaults, the function can be called with either no arguments, just the metric to measure, or the metric to measure and the allowed deviation.
|
|
|
(That is, if you need to specify an argument, all arguments to the left of that argument must be specified as well but all arguments to the right are optional)
|
|
|
|
|
|
|
... | ... | |