Josh Meredith authored 2 years ago and Cheng Shao committed 2 months ago

(cherry picked from commit d69ecac2)

(cherry picked from commit 1fad5bf1)

(cherry picked from commit 280c49af)

(cherry picked from commit 94673d41)
(cherry picked from commit f13a2942)
(cherry picked from commit 97551107)

This patch removes the -fasm flag from test ways, except ways like
optasm that explicitly state they are meant to be compiled with NCG
backend. Most test ways should use the default codegen backend, and
the precense of -fasm can cause stderr mismatches like this when GHC
is configured with the unregisterised backend:

```
--- /dev/null
+++ /tmp/ghctest-3hydwldj/test   spaces/testsuite/tests/profiling/should_compile/prof-late-cc.run/prof-late-cc.comp.stderr.normalised
@@ -0,0 +1,2 @@
+when making flags consistent: warning: [GHC-74335] [-Winconsistent-flags (in -Wdefault)]
+    Target platform uses unregisterised ABI, so compiling via C
*** unexpected failure for prof-late-cc(prof_no_auto)
```

This has been breaking the wasm unreg nightly job since !12595 landed.

(cherry picked from commit 88488847)
(cherry picked from commit f603ac9c)
(cherry picked from commit 4977e2a6)

(cherry picked from commit 64fba310)
(cherry picked from commit b35707a9)
(cherry picked from commit ead1c3cc)

objc-hi/objcxx-hi should be skipped when cross compiling. The existing
opsys('darwin') predicate only asserts the host system is darwin but
tells us nothing about the target, hence the oversight.

(cherry picked from commit 595c0894)
(cherry picked from commit 5273d3d0)
(cherry picked from commit 6babf56f)

Add hie.yaml to ghc-config project directory so it can be edited using
HLS.

(cherry picked from commit 7eda4bd2)
(cherry picked from commit ed81a875)
(cherry picked from commit 9108734a)

(cherry picked from commit e17f2df9)
(cherry picked from commit 9a2f3313)
(cherry picked from commit 08acf27f)

The testsuite driver CPU feature detection logic only detects host CPU
and only makes sense when we are not testing a cross GHC.

(cherry picked from commit c4c6d714)
(cherry picked from commit 07641012)
(cherry picked from commit 68bc5d72)

T17920 was marked as fragile on wasm before; it can be trivially fixed
by avoiding calling variadic printf() in cmm.

(cherry picked from commit 7b4c1998)
(cherry picked from commit c4060ce3)
(cherry picked from commit b64e0f63)

(cherry picked from commit b1e0c313)
(cherry picked from commit 34882192)
(cherry picked from commit 94adf918)

Ever since the removal of the make build system, the in tree lndir
hasn't been actually built, so this patch removes it.

(cherry picked from commit d3a050d2)
(cherry picked from commit 04860c24)

hs_try_putmvar002 includes pthread.h and doesn't work on targets
without this header (e.g. wasm32). It doesn't need to include this
header at all. This was previously unnoticed by wasm CI, though recent
toolchain upgrade brought in upstream changes that completely removes
pthread.h in the single-threaded wasm32-wasi sysroot, therefore we
need to handle that change.

(cherry picked from commit 2ce2a493)
(cherry picked from commit f581edb0)

On posix platforms, when performing read/write on FDs, we check the
nonblocking flag first. For FDs without this flag (e.g. stdout), we
call fdReady() first, which in turn calls poll() to wait for I/O to be
available on that FD. This is problematic for wasm32-wasi: although
select()/poll() is supported via the poll_oneoff() wasi syscall, that
syscall is rather heavyweight and runtime behavior differs in
different wasi implementations. The issue is even worse when targeting
browsers, given there's no satisfactory way to implement async I/O as
a synchronous syscall, so existing JS polyfills for wasi often give up
and simply return ENOSYS.

Before we have a proper I/O manager that avoids poll_oneoff() for
async I/O on wasm, this patch improves the status quo a lot by merely
pretending all FDs are "nonblocking". Read/write on FDs will directly
invoke read()/write(), which are much more reliably handled in
existing wasi implementations, especially those in browsers.

Fixes #23275 and the following test cases: T7773 isEOF001 openFile009
T4808 cgrun025

Approved by CLC proposal #234:
https://github.com/haskell/core-libraries-committee/issues/234

(cherry picked from commit 2eca52b4)
(cherry picked from commit 288692dd)

(cherry picked from commit af771148)
(cherry picked from commit 84c875d1)

The wasm backend didn't properly make use of all Cmm global registers
due to #24347. Now that it is fixed, this patch re-enables full
register mapping for wasm32, and we can now generate smaller & faster
wasm modules that doesn't always spill arguments onto the stack. Fixes #22460 #24152.

(cherry picked from commit 0cda2b8b)
(cherry picked from commit f1f5068b398b1effb837add38ecc5303dc9a381f)
(cherry picked from commit 1e695750)

ghc-api tests for wasm32 are more likely to timeout due to the large
wasm module sizes, especially when testing with wasm native tail
calls, given wasmtime's handling of tail call opcodes are suboptimal
at the moment. It makes sense to increase timeout specifically for
these tests on wasm32. This doesn't affect other targets, and for
wasm32 we don't increase timeout for all tests, so not to risk letting
major performance regressions slip through the testsuite.

(cherry picked from commit 07ab5cc1)
(cherry picked from commit c4d1edb1)

This patch includes all wasm32-specific testsuite fixes.

(cherry picked from commit bd2bfdec)

Sylvain Henry authored 2 years ago and Cheng Shao committed 2 months ago

req_cmm is more informative than js_skip

(cherry picked from commit 239202a2)

This patch adds missing annotations (req_th, req_dynamic_lib_support,
req_rts_linker) to some tests. They were discovered when testing
wasm32, though it's better to be explicit about what features they
require, rather than simply adding when(arch('wasm32'), skip).

(cherry picked from commit b174a110)

This patch adds the req_host_target_ghc predicate to the testsuite to
assert the ghc compiler being tested can compile both host/target
code. When testing cross GHCs this is not supported yet, but it may
change in the future.

(cherry picked from commit cb933665)

This patch adds the req_ghc_with_threaded_rts predicate to the
testsuite to assert the platform has threaded RTS, and mark some tests
as req_ghc_with_threaded_rts. Also makes ghc_with_threaded_rts a
config field instead of a global variable.

(cherry picked from commit aa6afe8a)

(cherry picked from commit a580722e)
(cherry picked from commit 4f317bfd)
(cherry picked from commit d94d0176)

Sylvain Henry authored 2 years ago and Cheng Shao committed 2 months ago

See #22630 and !9552

This commit:
 - splits req_smp into req_target_smp and req_ghc_smp
 - changes the testsuite driver to calculate req_ghc_smp
 - changes a handful of tests to use req_target_smp instead of req_smp
 - changes a handful of tests to use req_host_smp when needed

The problem:
 - the problem this solves is the ambiguity surrounding req_smp
 - on master req_smp was used to express the constraint that the program
 being compiled supports smp _and_ that the host RTS (i.e., the RTS used
 to compile the program) supported smp. Normally that is fine, but in
 cross compilation this is not always the case as was discovered in #22630.

The solution:
 - Differentiate the two constraints:
   - use req_target_smp to say the RTS the compiled program is linked
   with (and the platform) supports smp
   - use req_host_smp to say the RTS the host is linked with supports smp

WIP: fix req_smp (target vs ghc)

add flag to separate bootstrapper

split req_smp -> req_target_smp and req_ghc_smp

update tests smp flags

cleanup and add some docstrings

only set ghc_with_smp to bootstrapper on S1 or CC

Only set ghc_with_smp to bootstrapperWithSMP of when testing stage 1
and cross compiling

test the RTS in config/ghc not hadrian

re-add ghc_with_smp

fix and align req names

fix T11760 to use req_host_smp

test the rts directly, avoid python 3.5 limitation

test the compiler in a try block

align out of tree and in tree withSMP flags

mark failing tests as host req smp

testsuite: req_host_smp --> req_ghc_smp

Fix ghc vs host, fix ghc_with_smp leftover

(cherry picked from commit 06036d93)

When the testsuite driver copy files instead of symlinking them, it
should also copy the permission bits, otherwise there'll be permission
denied errors. Also, enforce file copying when testing wasm32, since
wasmtime doesn't handle host symlinks quite well
(https://github.com/bytecodealliance/wasmtime/issues/6227).

(cherry picked from commit 791cce64)

This patch implements logic to automatically exclude ghci ways when
there is no rts linker. It's way better than having to annotate
individual test cases.

(cherry picked from commit e6416b10)

This patch fixes hp2ps related framework failures when testing the
wasm backend by including target exe extension in heap profile
filenames.

(cherry picked from commit 6f511c36)

This patch fixes cross prefix stripping in the testsuite driver. The
normalization logic used to only handle prefixes of the triple form
<arch>-<vendor>-<os>, now it's relaxed to allow any number of tokens
in the prefix tuple, so the cross prefix stripping logic would work
when ghc is configured with something like --target=wasm32-wasi.

(cherry picked from commit b5f00811)

This patch refactors the testsuite driver, gets rid of multi-threading
logic for running test cases concurrently, and uses asyncio &
coroutines instead. This is not yak shaving for its own sake; the
previous multi-threading logic is prone to livelock/deadlock
conditions for some reason, even if the total number of threads is
bounded to a thread pool's capacity.

The asyncify change is an internal implementation detail of the
testsuite driver and does not impact most GHC maintainers out there.
The patch does not touch the .T files, test cases can be
added/modified the exact same way as before.

(cherry picked from commit ea853ff0)

This patch changes a thread-local variable to context variable
instead, which works as intended when the testsuite transitions to use
asyncio & coroutines instead of multi-threading to concurrently run
test cases. Note that this also raises the minimum Python version to
3.7.

(cherry picked from commit 0149f32f)

This patch fixes some mypy typing errors which weren't caught in
previous linting jobs.

(cherry picked from commit 8fe8b653)

(cherry picked from commit f7478d95)

(cherry picked from commit a984a103)

(cherry picked from commit f1beee36)

(cherry picked from commit 30c84511)

Ben Gamari authored 2 years ago and Cheng Shao committed 2 months ago

(cherry picked from commit e7392b4e)

Ben Gamari authored 2 years ago and Cheng Shao committed 2 months ago

(cherry picked from commit 79d0cb32)

The list_broken make target will transitively depend on the
calibrate.out target, which used STAGE1_GHC instead of TEST_HC. It
really should be TEST_HC since that's what get passed in the gitlab CI
config.

(cherry picked from commit 4a24dbbe)