1. 09 Apr, 2014 1 commit
    • Edward Z. Yang's avatar
      Fix #5435, adding new test config check_stdout. · f8e12e2b
      Edward Z. Yang authored
      
      
      check_stdout(f) allows you to override the test framework's
      diff based output checking with another mechanism.  f is
      a function which takes two arguments: the first is the
      filename containing the observed stdout, the second is the
      normaliser that would have been applied (in case you want
      to read, normalise, and then do something.)
      
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      f8e12e2b
  2. 05 Oct, 2013 1 commit
    • Joachim Breitner's avatar
      Flag to test suite: SKIP_PERF_TESTS · 718e86b4
      Joachim Breitner authored
      More often than not the output of the performance tests is in the way,
      rather than helping. This allows the use of `make SKIP_PERF_TESTS=YES`
      to skip these tests. Fixes #8413
      718e86b4
  3. 20 Sep, 2013 2 commits
  4. 11 Feb, 2013 2 commits
  5. 07 Feb, 2013 1 commit
  6. 17 Jan, 2013 1 commit
  7. 14 Dec, 2012 1 commit
  8. 25 Oct, 2012 1 commit
  9. 10 Oct, 2012 1 commit
  10. 25 Sep, 2012 1 commit
  11. 23 Apr, 2012 1 commit
    • pcapriotti's avatar
      Add combined_output option. · ef0d0b70
      pcapriotti authored
      Added an option to combine stdout and stderr into a single file. This is
      useful for ghci scripts that produce interleaved errors and normal
      output.
      
      Also modified check_stderr_ok so that it normalizes stderr in the same
      way as compile tests.
      ef0d0b70
  12. 16 Feb, 2012 3 commits
    • Simon Marlow's avatar
      add req_smp · c64c8027
      Simon Marlow authored
      c64c8027
    • Simon Marlow's avatar
      Allow perf bounds to be specified as base + percentage deviation · ef050859
      Simon Marlow authored
      This makes it much easier to update the bounds.  Instead of coming up
      with a suitable (min,max) pair, you just give e.g. (base, 10) to allow
      10% deviation from the base figure, which can be pasted from the
      error.
      
      e.g. previously:
      
      -      # expected value: 458700632 (amd64/Linux):
      -      if_wordsize(64,
      -          compiler_stats_num_field('bytes allocated', 440000000,
      -                                                      480000000)),
      
      now:
      
      +      if_wordsize(64, # sample from amd64/Linux 15/2/2012
      +          compiler_stats_range_field('bytes allocated', 360243576, 10)),
      
      Note: use stats_range_field rather than stats_num_field.  I left
      support for the old way for now so that we can do a gradual migration.
      
      (next I suppose we should make it so that 'make accept' works for perf
      tests, but that's for another day)
      ef050859
    • Simon Marlow's avatar
      Make the compiler_always_flags modifiable on a per-test basis · eb7aacb6
      Simon Marlow authored
      Motivation: we wanted to disable -dcore-lint for the perf tests
      eb7aacb6
  13. 06 Jan, 2012 1 commit
    • Simon Marlow's avatar
      Count the number of tests that failed due to reqlib() · 2ac8fe3b
      Simon Marlow authored
      OVERALL SUMMARY for test run started at Fri Jan  6 09:46:46 GMT 2012
            82 total tests, which gave rise to
           820 test cases, of which
             0 caused framework failures
           818 were skipped
      
             0 expected passes
             2 had missing libraries  <---
             0 expected failures
             0 unexpected passes
             0 unexpected failures
      2ac8fe3b
  14. 16 Nov, 2011 1 commit
    • Simon Marlow's avatar
      Fix meaning of the only_ways field (fixed cgrun057(normal) in validate) · a97f82b8
      Simon Marlow authored
      The default setting for only_ways was [], which meant "all ways".
      However, it's really useful to have [] mean "no ways".
      
      In cgrun057 I used only_ways(prof_ways) to mean "only run this in the
      profiling ways", but if profiling is disabled then prof_ways = [] and
      this doesn't do what we want.  Profiling is normally disabled in
      validate, but I had it enabled because I was testing profiling.
      
      So now only_ways defaults to None, which means "all ways", and [] now
      means "no ways".
      a97f82b8
  15. 09 Nov, 2011 1 commit
  16. 18 Oct, 2011 1 commit
  17. 18 Sep, 2011 1 commit
    • Ian Lynagh's avatar
      Rejig how performance test stats are stored · cf379454
      Ian Lynagh authored
      In particular, this allows more specific results to be specified, e.g.
          if_wordsize(64,                    stats_num_field('foo', 50, 54)),
          if_platform('x86_64-apple-darwin', stats_num_field('foo', 62, 66)),
      cf379454
  18. 08 Aug, 2011 1 commit
  19. 07 Aug, 2011 1 commit
    • Ian Lynagh's avatar
      Add support for checking whether files are written by more than one test · 5f12b5e3
      Ian Lynagh authored
      Work in progress, but largely works. Known issues:
      * only supported when using the timeout program
      * 'test.strace' files aren't cleaned, as they end up in the root
        directory rather than the test's directory
      * Doesn't yet track what the current directory is, so finds several
        files like "A.o" being written by multiple tests (and conversely,
        may be missing writes to the same file from different directories)
      * Lots of tests write to $HOME/.ghc/ghci_history. We should probably
        be passing ghci a flag to stop this from happening.
      * Some .strace lines aren't understood yet, causing framework failures
      * One .strace file can cause muiltiple framework failures, if it
        contains lots of lines that aren't understood
      
      Performance:
      Threads       fast testsuite time     fast testsuite time with checks
      1             16:36.14                25:16.07
      5              5:33.95                 8:04.05
      5f12b5e3
  20. 06 Aug, 2011 1 commit
  21. 13 Jul, 2011 1 commit
  22. 04 Apr, 2011 1 commit
  23. 28 Apr, 2010 1 commit
  24. 19 Dec, 2009 2 commits
  25. 18 Dec, 2009 1 commit
  26. 28 Nov, 2009 1 commit
  27. 24 Nov, 2009 1 commit
  28. 04 Nov, 2009 1 commit
  29. 04 Apr, 2009 1 commit
  30. 03 Apr, 2009 1 commit
  31. 23 Nov, 2008 1 commit
  32. 12 Jun, 2008 1 commit
  33. 20 Apr, 2008 1 commit
  34. 06 Feb, 2008 1 commit
    • Simon Marlow's avatar
      Find compiler version-specific output files automatically · 3bc99c34
      Simon Marlow authored
      Also, clean up the way we find the output file. From the comment:
      
      # Finding the sample output.  The filename is of the form
      #
      #   <test>.stdout[-<compiler>][-<version>][-<wordsize>][-<platform>]
      #
      # and we pick the most specific version available.  The <version> is
      # the major version of the compiler (e.g. 6.8.2 would be "6.8").  For
      # more fine-grained control use if_compiler_lt().
      
      I'll update the wiki too.
      3bc99c34
  35. 22 Jan, 2008 1 commit