1. 12 Sep, 2015 1 commit
  2. 03 Sep, 2015 2 commits
  3. 02 Sep, 2015 3 commits
    • thomie's avatar
      Testsuite: delete dead code · bd16e0bc
      thomie authored
      bd16e0bc
    • thomie's avatar
      Testsuite: by default run all tests for a single way · c43c8e2c
      thomie authored
      `make test` now runs all tests for a single way only. Use `make slowtest` to
      get the previous behaviour (i.e. run all tests for all ways).
      
      The intention is to use this new `make test` setting for Phabricator, as
      a reasonable compromise between `make fasttest` (what it previously
      used) and a fullblown `make slowtest` (which runs all tests for all
      ways).
      
      See Note [validate and testsuite speed] in toplevel Makefile.
      
      Differential Revision: https://phabricator.haskell.org/D1178
      c43c8e2c
    • thomie's avatar
      Testsuite: refactoring only · ad26c54b
      thomie authored
      * Rename `platform_wordsize_qualify` to `find_expected_file`, and make
        it return a filename instead of an (absolute) filepath.
      
      * Replace most usages of `qualify` by `in_testdir`. Others usage sites
        will be deleted in a later commit.
      
      These changes will be useful in a later commit, when we'll distinguish
      between files in the source directory and those in a (newly created)
      test directory.
      
      Reviewed by: austin, bgamari
      
      Differential Revision: https://phabricator.haskell.org/D1186
      ad26c54b
  4. 15 Aug, 2015 1 commit
  5. 14 Jul, 2015 2 commits
  6. 13 Jul, 2015 2 commits
  7. 30 Jun, 2015 1 commit
  8. 15 Jun, 2015 1 commit
  9. 12 Jun, 2015 3 commits
    • thomie's avatar
      Testsuite: add function compile_timeout_multiplier (#10345) · 6e542a62
      thomie authored
      And rename timeout_multiplier to run_timeout_multiplier.
      
      timeout_multiplier was added in commit
      a0038979. The name suggested that it
      would affect any test, but it actually only affected tests that had a
      run component, and only that run component (as needed by test T367).
      
      Differential Revision: https://phabricator.haskell.org/D982
      6e542a62
    • thomie's avatar
      Testsuite: diff non-whitespace normalised output (#10152) · 5ddd9041
      thomie authored
      On a test failure, we show a diff between the expected and the actual
      output. The method of how we do this has changed a couple of times:
      
      * In 2007: 9951189c
        "On failure, diff the normalised test outputs"
      
      * In 2011: 3019b1e4
        "When the output files differ, present the diffs between the *actual*
        output, not the normalised output. The latter may have newlines
        removed, making the diff unreadable."
      
      * In 2015 (now): do something in between.
        - Do apply the normalisers again, to make the diff smaller (only
          showing the actual problem).
        - But don't apply normalise_whitespace, as it indeed makes the diff
          unreadable.
      
      Differential Revision: https://phabricator.haskell.org/D984
      5ddd9041
    • thomie's avatar
      Testsuite: fix the little known CHECK_FILES_WRITTEN=1 · c14bd017
      thomie authored
      The testsuite driver has a little known feature to check which files
      each test writes to, whether there are tests that write to same file,
      and whether the tests leave any files behind when CLEANUP=1. It uses
      strace under the hood.
      
      This commit fixes some bitrot, and filters out some more strace lines
      that we're not interested in (and are shown as framework failures
      otherwise).
      
      Differential Revision: https://phabricator.haskell.org/D979
      c14bd017
  10. 11 Jun, 2015 2 commits
    • thomie's avatar
      Testsuite: change some expect_fail tests to expect_broken · 5e66a698
      thomie authored
      Change the following tests from expect_fail to expect_broken: and list
      the ticket number:
      
          * driver/sigof03m/sigof03 (#9252)
          * driver/static001 (#8127)
          * partial-sigs/should_compile/EqualityConstraint (#9478)
          * partial-sigs/should_compile/ExtraNumAMROn (#9478)
          * partial-sigs/should_compile/PatBind2 (#9478)
          * partial-sigs/should_fail/TidyClash2 (#9478)
          * simplCore/should_compile/T8832 (#8832)
      
      The following tests are still marked as expect_fail, but it is not
      clearly documented why so:
      
          * gadt/lazypatok
          * indexed-types/should_fail/SkolemOccursLoop
      
      All other expect_fail tests are only expected to fail on either a
      certain platform/os or for a certain way only.
      
      Differential Revision: https://phabricator.haskell.org/D966
      5e66a698
    • thomie's avatar
      Testsuite Windows: fix T8172 (#8172) · da84fd54
      thomie authored
      Use the new function `normalise_drive_letter` to change D:\ to C:\
      before comparing outputs.
      da84fd54
  11. 09 Jun, 2015 1 commit
    • thomie's avatar
      Revert "The test runner now also works under the msys-native Python." · bb996712
      thomie authored
      To make the test runner work under msys-native Python...
      
      Commit 5258566e broke the msys testsuite
      driver (#10441). It changed the quoting of `config.compiler` from single
      quotes to double quote, which turns out to not be compatible with what
      the function `passThroughCmd` expected.
      
      We could fix `passThroughCmd` to handle the case where `config.compiler`
      is double quoted, and scatter some notes around to make sure the quoting
      done in various places of the testsuite driver stay compatible.
      
      Instead, this commit reverts 101c62e2,
      which introdced the function `passThroughCmd` in the first place
      (#9626). ezyang reports that doing this revert fixes the testsuite
      driver for him using the the following version of msys2:
      
        msys2-keyring r8.3864337-1
        msys2-runtime 2.1.0.16351.cd3184b-1
        msys2-runtime-devel 2.1.0.16351.cd3184b-1
        msys2-w32api-headers 5.0.0.4456.c8b6742-1
        msys2-w32api-runtime 5.0.0.4455.32db221-1
      
      Ideally we'd know what minimum version of msys2 we require, but for now
      this fix is better than nothing.
      
      Only gintas ever reported the original problem, and he actually
      mentioned shortly afterwards: "This may have been fixed by a recent
      release of msys2, but I am not sure."
      
      Differential Revision: https://phabricator.haskell.org/D952
      bb996712
  12. 06 Jun, 2015 1 commit
    • thomie's avatar
      Testsuite: only show output diff when test is expected to pass · ae83a81a
      thomie authored
      Don't let the output of tests that either have missing libraries or are
      expected to be broken obscure real failures.
      
      This makes it easier to analyse the testlogs.
      
      The only consequence is that when a test fails because a certain library
      isn't installed, you have to check the all.T file in which the test is
      defined to actually find out _which_ library that is. Before it would
      print something like
      
          Compile failed (status 256) errors were:
      
          stm052.hs:10:8: error:
           Could not find module ‘System.Random’
           Use -v to see a list of the files searched for.
      
      And now it doesn't. I think this is an acceptable tradeoff.
      
      Differential Revision: https://phabricator.haskell.org/D945
      ae83a81a
  13. 04 Jun, 2015 2 commits
    • thomie's avatar
      Testsuite: add/fix cleanup for certain tests · e340f6ec
      thomie authored
      * extra_clean argument should be a list
        Add an assert to prevent regressions.
      
      * properly clean package conf direcories
        They are directories now, which was causing problems.
      
      * properly clean write_interface_* tests
        We were getting these errors:
      
        [Errno 21] Is a directory: './driver/write_interface_oneshot'
        [Errno 39] Directory not empty: './driver/write_interface_oneshot'
        [Errno 21] Is a directory: './driver/write_interface_make'
        [Errno 39] Directory not empty: './driver/write_interface_make'
      
      * outputdir() is better than -outputdir, as it knows how to (pre)clean
        itself.
      e340f6ec
    • thomie's avatar
      Make validate more quiet · d0063e89
      thomie authored
        * By default use V=0, and call the testsuite with VERBOSE=2, which we
          did before only with validate --quiet. This disables printing the
          test commands it runs.
      
        * When --quiet is used, call the testsuite with VERBOSE=1. This
          disables printing the '====> Scanning' lines, and doesn't print
          which test is being run. So it only prints something when a test
          accidentally prints to stdout or when it fails.
      
          Don't set this option on Travis, as Travis will cancel a build if it
          doesn't see any output for more than 10 minutes.
      
        * When --quiet is used, set the new test option NO_PRINT_SUMMARY,
          which skips printing the test summary. Only the list of unexpected
          failures is printed, if there are any. Note that the full summary
          can still be found in testsuite_summary.txt
      
        * When --quiet is used, don't pass the `-v` flag to `ghc-pkg check`
      
        * When --quiet is used, don't print the Oops! header. It shoud be
          clear from the list of failing tests that something is wrong.
      
      This is all done to get the most out of 30 lines of logfile. These changes can
      be disabled later by simply not passing the --quiet flag to validate.
      
      Differential Revision: https://phabricator.haskell.org/D942
      d0063e89
  14. 28 May, 2015 1 commit
    • Rufflewind's avatar
      Testdriver: don't use os.popen in config/ghc · ef904660
      Rufflewind authored
      Rewrite config/ghc to use getStdout (which use subprocess.Popen) instead
      of os.popen, which is deprecated; this also avoids the use of shell
      
      Also:
      * Move getStdout to driver/testutil.py so both config/ghc and
        driver/runtests.py can use it
      
      * Remove support for Python below 2.4, which doesn't have subprocess
      
      Reviewed By: thomie
      
      Differential Revision: https://phabricator.haskell.org/D908
      ef904660
  15. 23 May, 2015 1 commit
  16. 14 Apr, 2015 1 commit
  17. 03 Apr, 2015 1 commit
  18. 01 Apr, 2015 1 commit
    • thomie's avatar
      Don't `make accept` output of expect_broken tests · d4cf7051
      thomie authored
      When running `make accept` on a directory in the testsuite, don't accept
      the output of tests that are marked as expect_broken. This makes it
      easier to review `git diff` after running `make accept`.
      
      When you change an error message in the compiler that shows up in the
      output of many tests, you can run `make accept` in the testsuite
      directory, and all expected test output will be updated. But since your
      change didn't magically fix all the other bugs in the compiler for which
      we have an expect_broken test, the output for those tests should
      probably not be updated.
      
      Before, the effect of running `make accept` could be that some tests
      would end up in the 'unexpected passes' group.
      
      [skip ci]
      
      Reviewed By: austin
      
      Differential Revision: https://phabricator.haskell.org/D781
      d4cf7051
  19. 31 Mar, 2015 2 commits
  20. 23 Mar, 2015 1 commit
  21. 17 Mar, 2015 1 commit
  22. 16 Mar, 2015 1 commit
  23. 13 Mar, 2015 1 commit
    • thomie's avatar
      Move the function strip_quotes to testutil.py · cc07a0ba
      thomie authored
      If one runs the testsuite with a profiling compiler, during the import
      of `testlib.py`, `testlib.py` sets the global variable `gs_working`. To
      do so, it executes a few statements which require the function
      `strip_quotes` to be in scope. But that function only gets defined at
      the very end of testlib.py.
      
      This patch moves the definition of `strip_quotes` to testutil.py, which
      is imported at the very top of testlib.py. This unbreaks the nightly
      builders.
      
      Reviewed By: austin
      
      Differential Revision: https://phabricator.haskell.org/D728
      cc07a0ba
  24. 11 Mar, 2015 2 commits
  25. 10 Mar, 2015 1 commit
    • Edward Z. Yang's avatar
      Refactor testsuite with normalise_version() · 8cbd7f5d
      Edward Z. Yang authored
      
      
      Summary:
      This function generalizes the normaliseBytestringPackage and other similar
      one-off functions into normalise_version() with takes a package name to
      normalize against.  This JUST manages package versions; we also could use
      a normalize for keys.
      
      In the process, I modified all the normalization functions to be accumulative;
      I don't think this makes a difference for current test cases but I think it
      makes things nicer.
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      
      Reviewers: austin
      
      Subscribers: thomie
      
      Differential Revision: https://phabricator.haskell.org/D725
      8cbd7f5d
  26. 06 Mar, 2015 1 commit
    • thomie's avatar
      testsuite: format commands using config dict · 91c11fea
      thomie authored
      Summary:
      Allow `cmd_wrapper` to return a format string that can refer to config values.
      Very useful! This allows for many tests to be defined in pure Python, instead
      of in an additional script or Makefile.
      
      Example:
      
        def Thpc(cmd):
          return(cmd + ' && {hpc} report Thpc.tix')
        test('Thpc', [cmd_wrapper(Thpc), only_ways['hpc']), compile_and_run, [''])
      
      The `{hpc}` is replaced by the value of `config.hpc`. The result is that the
      module `Thpc` first gets compiled, then the binary `Thpc` is run, and then the
      `hpc report` command is run. The output of all of this is redirected
      (and later appended) to Thpc.run.stdout/stderr as normally.
      
      Reviewed By: austin
      
      Differential Revision: https://phabricator.haskell.org/D706
      91c11fea
  27. 23 Feb, 2015 1 commit
    • thomie's avatar
      Always ignore user-package-db when running tests · 30dc59e8
      thomie authored
      Summary:
      The user package database was already ignored for systems that
      `have_subprocess`.
      
      To [wiki:Debugging/InstallingPackagesInplace install] a package inplace:
      `cabal install
        --with-compiler=<inplace-ghc>
        --with-package-db=<inplace-package-db>`
        <package>
      
      Reviewers: austin
      
      Reviewed By: austin
      
      Subscribers: thomie
      
      Differential Revision: https://phabricator.haskell.org/D668
      30dc59e8
  28. 19 Nov, 2014 1 commit
  29. 31 Oct, 2014 1 commit