1. 28 Jul, 2017 1 commit
  2. 03 Mar, 2017 1 commit
    • rwbarton's avatar
      testsuite: Move echoing commands in make invocations to VERBOSE=5 · 6421c6f2
      rwbarton authored
      D2894 added a new verbosity level VERBOSE=4 to strip -s/--silent
      flags from make invocations in test commands. This will probably
      cause the test to fail of course, but is useful for seeing what
      a test that's already failing is doing.
      However there was already an undocumented meaning of VERBOSE=4,
      added in commit cfeededf, that causes the results of performance
      tests to be printed unconditionally (even when they are within the
      expected range). nomeata's ghc builder uses these figures to
      collect historical data on performance test figures. The new
      meaning of VERBOSE=4 added in D2894 means that any test that uses
      make now fails on the builder.
      This commit moves the new behavior of D2894 to the level VERBOSE=5
      so that nomeata's ghc builder again produces useful results on
      failing tests. It also adds documentation for both settings.
      Test Plan: did some manual testing
      Reviewers: austin, bgamari, Phyx, nomeata
      Reviewed By: bgamari, Phyx
      Subscribers: nomeata, thomie, Phyx
      Differential Revision: https://phabricator.haskell.org/D3141
  3. 23 Feb, 2017 1 commit
  4. 22 Dec, 2016 1 commit
  5. 21 Dec, 2016 1 commit
    • Gabor Greif's avatar
      Suppress duplicate .T files · 9a29b65b
      Gabor Greif authored
      As per http://stackoverflow.com/questions/7961363/removing-duplicates-in-lists
      use the set() function to zap duplicates from the obtained list of .T files.
      I am using
      $ python3 --version
      Python 3.5.1
      and strangely findTFiles() returns some .T files twice:
      -- BEFORE
      Found 376 .T files...
      ====> Scanning ../../libraries/array/tests/all.T
      ====> Scanning ../../libraries/array/tests/all.T
      *** framework failure for T2120(duplicate) There are multiple tests with this name
      *** framework failure for largeArray(duplicate) There are multiple tests with this name
      *** framework failure for array001(duplicate) There are multiple tests with this name
      *** framework failure for T9220(duplicate) There are multiple tests with this name
      *** framework failure for T229(duplicate) There are multiple tests with this name
      -- AFTER
      Found 365 .T files...
      ====> Scanning ../../libraries/array/tests/all.T
      Even more strangely 'find' begs to differ:
      $ find libraries testsuite/tests -name "*.T" | sort | uniq | wc -l
  6. 30 Nov, 2016 2 commits
    • Ben Gamari's avatar
      testsuite: Use python3 by default · 605bb9b4
      Ben Gamari authored
      It turns out that Phyx's fix for #12554 (D2684) still fails with mingw-w64
      python 2.7. However, Python 3 (both msys2 and mingw-w64) work fine. Given that
      supporting Python 2 has already become rather tiresome (as @thomie warned it
      would), let's just move to python3 by default.
      Test Plan: Validate
      Reviewers: austin, Phyx
      Reviewed By: Phyx
      Subscribers: Phyx, thomie
      Differential Revision: https://phabricator.haskell.org/D2766
      GHC Trac Issues: #12554
    • Tamar Christina's avatar
      Fix testsuite threading, timeout, encoding and performance issues on Windows · 0ce59be3
      Tamar Christina authored
      In a land far far away, a project called Cygwin was born.
      Cygwin used newlib as it's standard C library implementation.
      But Cygwin wanted to emulate POSIX systems as closely as possible.
      So it implemented `execv` using the Windows function `spawnve`.
      spawnve (_P_OVERLAY, path, argv, cur_environ ())
      `_P_OVERLAY` is crucial, as it makes the function behave *sort of*
      like execv on linux. the child process replaces the original process.
      With one major difference because of the difference in process models
      on Windows: the original process signals the caller that it's done.
      this is why the file is still locked. because it's still running,
      control was returned because the parent process was destroyed,
      but the child is still running.
      I think it's just pure dumb luck, that the older runtimes are slow
      enough to give the process time to terminate before we tried deleting
      the file.  Which explains why you do have sporadic failures even on
      older runtimes like 2.5.0, of a test or two (like T7307).
      So this patch fixes a couple of things. I leverage the existing
      `timeout.exe` to implement a workaround for this issue.
      a) The old timeout used to start the process then assign it to the job.
         This is slightly faulty since child processes are only assigned to a
         job is their parent were assigned at the time they started. So this
         was a race condition. I now create the process suspended, assign it
         to the job and then resume it. Which means all child processes are
         not running under the same job.
      b) First things, Is to prevent dangling child processes. I mark the job
         with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` so when the last process in
         the job is done, it insures all processes under the job are killed.
      c) Secondly, I change the way we wait for results. Instead of waiting
         for the parent process to terminate, I wait for the job itself to
         There's a slight subtlety there, we can't wait on the job itself.
         Instead we have to create an I/O Completion port and wait for signals
         on it.  See
      This fixes the issues on all runtimes for me and makes T7307 pass
      The threading was also simplified by hiding all the locking in a single
      semaphore and a completion class. Futhermore some additional error
      reporting was added.
      For encoding the testsuite now no longer passes a file handle to the
      subprocess since on windows, sh.exe seems to acquire a lock on the file
      that is not released in a timely fashion.
      I suspect this because cygwin seems to emulate console handles by
      creating file handles and using those for std handles. So when we give
      it an existing file handle it just locks the file. I what's happening is
      that it's not releasing the handle until all shared cygwin processes are
      dead. Which explains why it worked in single threaded mode.
      So now instead we pass a pipe and do not interpret the resulting data.
      Any bytes written to stdin or read out of stdout/stderr are done so in
      binary mode and we do not interpret the data. The reason for this is
      that we have encoding tests in GHC which pass invalid utf-8. If we try
      to handle the data as text then python will throw an exception instead
      of a test comparison failing.
      Also I have fixed the ability to override `PYTHON` when calling `make
      tests`. This now works the same as with `.\validate`.
      Finally, after cleaning up the locks I was able to make the abort
      behavior work correctly as I believe it was intended: when you press
      Ctrl+C and send an interrupt signal, the testsuite finishes the active
      tests and then gracefully exits showing you a report of the progress it
      did make. So using Ctrl+C will not just *die* as it did before.
      These changes lift the restriction on which python version you use
      (msys/mingw) or which runtime or python 3 or python 2.  All combinations
      should now be supported.
      Test Plan:
      PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH &&
      PYTHON=/usr/bin/python THREADS=9 make test
      THREADS=9 make test
      PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH &&
      PYTHON=/usr/bin/python ./validate --quiet --testsuite-only
      Reviewers: erikd, RyanGlScott, bgamari, austin
      Subscribers: jrtc27, mpickering, thomie, #ghc_windows_task_force
      Differential Revision: https://phabricator.haskell.org/D2684
      GHC Trac Issues: #12725, #12554, #12661, #12004
  7. 02 Nov, 2016 1 commit
    • Ben Gamari's avatar
      testsuite: Simplify kernel32 glue logic · cc4710af
      Ben Gamari authored
      On Windows the testsuite driver calls kernel32 to set the current
      terminal codepage. The previous implementation of this was significantly
      more complex than necessary, and was wrong in the case of MSYS2, which
      requires that we explicitly load the library using the name of its
      DLL, including its file extension.
      Test Plan: Validate on Windows
      Reviewers: austin, RyanGlScott, Phyx
      Reviewed By: RyanGlScott, Phyx
      Subscribers: thomie
      Differential Revision: https://phabricator.haskell.org/D2641
      GHC Trac Issues: #12661
  8. 17 Oct, 2016 2 commits
  9. 28 Jun, 2016 2 commits
  10. 27 Jun, 2016 1 commit
  11. 18 Jun, 2016 2 commits
    • thomie's avatar
      Testsuite: write "\n" instead of "\r\n" when using mingw Python · 6f6f5154
      thomie authored
      Mingw style Python uses '\r\n' by default for newlines. This is
      annoying, because it means that when a GHC developer on Windows uses
      mingw Python to `make accept` a test, every single line of the
      .stderr file is touched. This makes it difficult to spot the real
      changes, and it leads to unnecessary git history bloat.
      Prevent this from happening by using io.open instead of open.
      See `Note [Universal newlines]`
      Reviewed by: Phyx
      Differential Revision: https://phabricator.haskell.org/D2342
    • thomie's avatar
      Testsuite: run tests in <testdir>.run instead of /tmp · f72f23f9
      thomie authored
      As discussed in Phab:D1187, this approach makes it a bit easier to
      inspect the test directory while working on a new test.
      The only tests that needed changes are the ones that refer to files in
      ancestor directories. Those files are now copied directly into the test
      validate still runs the tests in a temporary directory in /tmp, see
      `Note [Running tests in /tmp]` in testsuite/driver/runtests.py.
      Update submodule hpc.
      Reviewed by: simonmar
      Differential Revision: https://phabricator.haskell.org/D2333
      GHC Trac Issues: #11980
  12. 07 Jun, 2016 1 commit
  13. 02 Jun, 2016 1 commit
  14. 24 May, 2016 1 commit
  15. 17 May, 2016 1 commit
    • thomie's avatar
      Testsuite: run tests in /tmp after copying required files · 3f3dc23e
      thomie authored
      Major change to the testsuite driver.
      For each TEST:
         * create a directory `<testdir>` inside `/tmp`.
         * link/copy all source files that the test needs into `<testdir>`.
         * run the test inside `<testdir>`.
         * delete `<testdir>`
      Extra files are (temporarily) tracked in
      `testsuite/driver/extra_files.py`, but can also be specified using the
      `extra_files` setup function.
      Differential Revision: https://phabricator.haskell.org/D1187
      Reviewed by: Rufflewind, bgamari
      Trac: #11980
  16. 29 Oct, 2015 1 commit
    • thomie's avatar
      Testsuite: report and error out on unfound tests · 032be43b
      thomie authored
      Users are sometimes confused why their test doesn't run. It is usually
      because of a misspelled testname, for example using 'TEST=1234' instead
      of 'TEST=T1234'. After this patch it is hopefully more clear what the
      problem is, showing:
          ERROR: tests not found: ['1234']
      Instead of:
          0 total tests, which gave rise to
          0 test cases, of which
          0 were skipped
      Reviewed by: austin, bgamari
      Differential Revision: https://phabricator.haskell.org/D1388
  17. 20 Oct, 2015 1 commit
    • thomie's avatar
      Testsuite Windows: don't use forward slashes in topdir path · ae4acbd1
      thomie authored
      Changing backwards slashes to forward slashes apparently confuses
      msys2/mingw magic path handling. I don't quite understand why, but this
      fixes it.
      Test Plan: on Windows, make sure PATH does not contain
      'inplace/mingw/bin' (let the testsuite driver add it), then run: make
      TEST='ghcilink003 ghcilink006'. Before this patch, it would fail.
      Reviewed by: Phyx, bgamari, austin
      Differential Revision: https://phabricator.haskell.org/D1343
  18. 03 Oct, 2015 1 commit
    • Tamar Christina's avatar
      Make Windows linker more robust to unknown sections · 620fc6f9
      Tamar Christina authored
      The Windows Linker has 3 main parts that this patch changes.
      1) Identification and classification of sections
      2) Adding of symbols to the symbols tables
      3) Reallocation of sections
      Previously section identification used to be done on a whitelisted
      basis. It was also exclusively being done based on the names of the
      sections. This meant that there was a bit of a cat and mouse game
      between `GCC` and `GHC`. Every time `GCC` added new sections there was a
      good chance `GHC` would break. Luckily this hasn't happened much in the
      past because the `GCC` versions `GHC` used were largely unchanged.
      The new code instead treats all new section as `CODE` or `DATA`
      sections, and changes the classifications based on the `Characteristics`
      flag in the PE header. By doing so we no longer have the fragility of
      changing section names. The one exception to this is the `.ctors`
      section, which has no differentiating flag in the PE header, but we know
      we need to treat it as initialization data.
      The check to see if the sections are aligned by `4` has been removed.
      The reason is that debug sections often time are `1 aligned` but do have
      relocation symbols. In order to support relocations of `.debug` sections
      this check needs to be gone. Crucially this assumption doesn't seem to
      be in the rest of the code. We only check if there are at least 4 bytes
      to realign further down the road.
      The second loop is iterating of all the symbols in the file and trying
      to add them to the symbols table. Because the classification of the
      sections we did previously are (currently) not available in this phase
      we still have to exclude the sections by hand. If they don't we will
      load in symbols from sections we've explicitly ignored the in # 1. This
      whole part should rewritten to avoid this. But didn't want to do it in
      this commit.
      Finally the sections are relocated. But for some reason the PE files
      contain a Linux relocation constant in them `0x0011` This constant as
      far as I can tell does not come from GHC (or I couldn't find where it's
      being set). I believe this is probably a bug in GAS. But because the
      constant is in the output we have to handle it. I am thus mapping it to
      the constant I think it should be `0x0003`.
      Finally, static linking *should* work, but won't. At least not if you
      want to statically link `libgcc` with exceptions support. Doing so would
      require you to link `libgcc` and `libstd++` but also `libmingwex`. The
      problem is that `libmingwex` also defines a lot of symbols that the RTS
      automatically injects into the symbol table. Presumably because they're
      symbols that it needs. like `coshf`. The these symbols are not in a
      section that is declared with weak symbols support. So if we ever want
      to get this working, we should either a) Ask mingw to declare the
      section as such, or b) treat all a imported symbols as being weak.
      Though this doesn't seem like it's a good idea..
      Test Plan:
      Running ./validate for both x86 and x86_64
      Also running the specific test case for #10672
      make TESTS="T10672_x86 T10672_x64"
      Reviewed By: ezyang, thomie, austin
      Differential Revision: https://phabricator.haskell.org/D1244
      GHC Trac Issues: #9907, #10672, #10563
  19. 11 Jun, 2015 1 commit
  20. 04 Jun, 2015 1 commit
    • thomie's avatar
      Make validate more quiet · d0063e89
      thomie authored
        * By default use V=0, and call the testsuite with VERBOSE=2, which we
          did before only with validate --quiet. This disables printing the
          test commands it runs.
        * When --quiet is used, call the testsuite with VERBOSE=1. This
          disables printing the '====> Scanning' lines, and doesn't print
          which test is being run. So it only prints something when a test
          accidentally prints to stdout or when it fails.
          Don't set this option on Travis, as Travis will cancel a build if it
          doesn't see any output for more than 10 minutes.
        * When --quiet is used, set the new test option NO_PRINT_SUMMARY,
          which skips printing the test summary. Only the list of unexpected
          failures is printed, if there are any. Note that the full summary
          can still be found in testsuite_summary.txt
        * When --quiet is used, don't pass the `-v` flag to `ghc-pkg check`
        * When --quiet is used, don't print the Oops! header. It shoud be
          clear from the list of failing tests that something is wrong.
      This is all done to get the most out of 30 lines of logfile. These changes can
      be disabled later by simply not passing the --quiet flag to validate.
      Differential Revision: https://phabricator.haskell.org/D942
  21. 28 May, 2015 2 commits
    • thomie's avatar
      Testdriver: do not interfer with MinGW path magic (#10449) · ce166a3a
      thomie authored
      This should fix the testsuite driver on Windows using the MinGW tools
      with a native build of Python.
      MinGW automagically converts MinGW-style paths (e.g.
      '/c/programs/ghc/bin/ghc') into ordinary Windows paths (e.g.
      'C:/programs/ghc/bin/ghc') when a native Windows program is invoked. But
      it doesn't do so when those paths are wrapped with a pair of escaped
      double quotes.
      The fix is to not call `eval` on the paths in Python, which let's us use
      one less pair of quotes, and makes MinGW happy.
      Reviewers: Rufflewind, austin
      Differential Revision: https://phabricator.haskell.org/D911
    • Rufflewind's avatar
      Testdriver: don't use os.popen in config/ghc · ef904660
      Rufflewind authored
      Rewrite config/ghc to use getStdout (which use subprocess.Popen) instead
      of os.popen, which is deprecated; this also avoids the use of shell
      * Move getStdout to driver/testutil.py so both config/ghc and
        driver/runtests.py can use it
      * Remove support for Python below 2.4, which doesn't have subprocess
      Reviewed By: thomie
      Differential Revision: https://phabricator.haskell.org/D908
  22. 19 Oct, 2014 1 commit
    • Krzysztof Gogolewski's avatar
      Python 3 support, second attempt (Trac #9184) · d576fc38
      Krzysztof Gogolewski authored
      This is a fixup of https://phabricator.haskell.org/D233
      The only difference is in findTFiles (first commit), which
      previously broke Windows runner; now I translated literally
      instead attempting to improve it, and checked it works.
      Test Plan:
      I used validate under 2,3 on Linux and under 2 on msys2.
      On Windows I've seen a large number of failures, but they don't
      seem to be connected with the patch.
      Reviewers: hvr, simonmar, thomie, austin
      Reviewed By: austin
      Subscribers: thomie, carter, ezyang, simonmar
      Differential Revision: https://phabricator.haskell.org/D310
      GHC Trac Issues: #9184
  23. 11 Oct, 2014 2 commits
    • gintas's avatar
      Extend windows detection in testsuite to recognize MSYS target · 034b2035
      gintas authored
      Currently, the detection recognizes the following `uname -s` strings:
       - `CYGWIN_NT-6.3`
       - `MINGW32_NT-6.3`
       - `MINGW64_NT_6.3`
      However, MSYS2 provides an additional target, in which case `uname -s`
      returns a string such as `MSYS_NT-6.3`. In all these cases, the system
      ought to be recognized as being a `windows` os by the testsuite runner.
      See also #9604
    • gintas's avatar
      Fallback to `ctypes.cdll` if `ctypes.windll` unavailable · 1032554a
      gintas authored
      On Windows, we may be using a native build of Python or a mingw/msys build. The
      former exports `ctypes.windll`, the latter exports `cdll`. Previously the code
      threw an exception when using the msys Python because it expected `windll` to
      always be available on Windows.
      Differential Revision: https://phabricator.haskell.org/D308
  24. 03 Oct, 2014 1 commit
  25. 01 Oct, 2014 1 commit
    • Krzysztof Gogolewski's avatar
      Basic Python 3 support for testsuite driver (Trac #9184) · 084d241b
      Krzysztof Gogolewski authored
      Most of the changes is adaptation of old Python 2 only code.
      My priority was not breaking Python 2, and so I avoided bigger
      changes to the driver. In particular, under Python 3 the output
      is a str and buffering cannot be disabled.
      To test, define PYTHON=python3 in testsuite/mk/boilerplate.mk.
      Thanks to aspidites <emarshall85@gmail.com> who provided the initial patch.
      Test Plan: validate under 2 and 3
      Reviewers: hvr, simonmar, thomie, austin
      Reviewed By: thomie, austin
      Subscribers: aspidites, thomie, simonmar, ezyang, carter
      Differential Revision: https://phabricator.haskell.org/D233
      GHC Trac Issues: #9184
  26. 01 Sep, 2014 2 commits
  27. 14 Jul, 2014 1 commit
  28. 12 Nov, 2013 1 commit
  29. 05 Oct, 2013 1 commit
    • Joachim Breitner's avatar
      Flag to test suite: SKIP_PERF_TESTS · 718e86b4
      Joachim Breitner authored
      More often than not the output of the performance tests is in the way,
      rather than helping. This allows the use of `make SKIP_PERF_TESTS=YES`
      to skip these tests. Fixes #8413
  30. 20 Sep, 2013 2 commits
  31. 11 Feb, 2013 2 commits