GHC issueshttps://gitlab.haskell.org/ghc/ghc/-/issues2024-02-13T15:23:02Zhttps://gitlab.haskell.org/ghc/ghc/-/issues/24420CI: Intermittent "Command failed with error code: -4" on Darwin.2024-02-13T15:23:02ZBryan Rbryan@haskell.foundationCI: Intermittent "Command failed with error code: -4" on Darwin.### Recent occurrences:
| json->>'web_url' | json->>'status' | json->>'ref' | json->>'name' | json->>'runner.description' | date(json->>'created_at') |
|------...### Recent occurrences:
| json->>'web_url' | json->>'status' | json->>'ref' | json->>'name' | json->>'runner.description' | date(json->>'created_at') |
|---------------------------------------------------|-----------------|--------------------------------|------------------------|----------------------------------------------|---------------------------|
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1772620 | failed | refs/merge-requests/11968/head | x86_64-darwin-validate | m1-delta_Mini18-Delta_f87b8a204711 | 2024-02-07 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1767573 | failed | refs/merge-requests/11576/head | x86_64-darwin-validate | aarch64-darwin-1_Mini18-Beta_f87b8a204711 | 2024-02-01 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1767305 | failed | refs/merge-requests/11976/head | x86_64-darwin-validate | aarch64-darwin-1_Mini18-Epsilon_004e0928d81c | 2024-02-01 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1764568 | failed | refs/merge-requests/11989/head | x86_64-darwin-validate | aarch64-darwin-1_Mini18-Beta_f87b8a204711 | 2024-01-29 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1753166 | failed | refs/merge-requests/11923/head | x86_64-darwin-validate | aarch64-darwin-1_Mini18-Beta_f87b8a204711 | 2024-01-16 |
### Analysis
* Signal 4 is "Illegal instruction", so maybe Rosetta is involved.
* The (truncated) command lines that failed in these occurences are:
* `/nix/store/sbfhs3rv53vgh3cn46q900fmgsl5q4x4-ghc/bin/ghc-pkg --no-user-package-db field filepath depends`
* `_build/stage0/bin/ghc -Wall -Wcompat -fdiagnostics-color=never -dynamic-too -hisuf hi -osuf o -hcsuf <SNIP>`
* `_build/stage0/bin/ghc -Wall -Wcompat -hisuf dyn_hi -osuf dyn_o -hcsuf dyn_hc -fPIC -dynamic <SNIP>`
* `/nix/store/sbfhs3rv53vgh3cn46q900fmgsl5q4x4-ghc/bin/ghc-pkg --expand-pkgroot --no-user-package-db describe stm`
* `/usr/bin/gcc --target=x86_64-apple-darwin -Qunused-arguments -E -MM -MG -MF _build/stage1/rts/build/c/sm/NonMovingScav.thr_debug_o.d <SNIP>`
So if it managed to affect nix-packaged software, our own built software, *and* `/usr/bin/gcc`, I feel it has to be a system issue. Unfortunately it also seems to affect multiple machines.
### Tracking link
https://grafana.gitlab.haskell.org/d/167r9v6nk/ci-spurious-failures?orgId=2&refresh=15m&from=now-30d&to=now&var-types=code_-4&var-runners=All&var-jobs=Allhttps://gitlab.haskell.org/ghc/ghc/-/issues/24417Intermittent "wasm32-wasi-ghc-pkg: createProcess: exec: invalid argument (Bad...2024-02-13T14:29:10ZBryan Rbryan@haskell.foundationIntermittent "wasm32-wasi-ghc-pkg: createProcess: exec: invalid argument (Bad file descriptor)"The error seems to happen when running `wasm32-wasi-ghc-pkg recache`.
Occurrences in the last 3-4 weeks:
| json->>'web_url' | json->>'status' | json->>'ref' | ...The error seems to happen when running `wasm32-wasi-ghc-pkg recache`.
Occurrences in the last 3-4 weeks:
| json->>'web_url' | json->>'status' | json->>'ref' | json->>'name' | json->>'runner.description' | date(json->>'created_at') |
|---------------------------------------------------|-----------------|--------------------------------|----------------------------------------------------------------------------------------|--------------------------------------------------|---------------------------|
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1772088 | failed | master | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-5.zw3rk_x86_64-linux-5_db038e4e1623 | 2024-02-07 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1771720 | failed | refs/merge-requests/11400/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-02-06 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1771185 | failed | refs/merge-requests/12028/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-02-06 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1767896 | failed | refs/merge-requests/11551/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-02-02 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1767533 | failed | master | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-6.zw3rk_x86_64-linux-6_db038e4e1623 | 2024-02-01 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1767370 | failed | refs/merge-requests/11551/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-5.zw3rk_x86_64-linux-5_db038e4e1623 | 2024-02-01 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1767029 | failed | master | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-02-01 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1766313 | failed | refs/merge-requests/11976/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-31 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1765681 | failed | refs/merge-requests/11551/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-3.zw3rk_x86_64-linux-3_db038e4e1623 | 2024-01-30 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1765496 | failed | refs/merge-requests/10417/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-3.zw3rk_x86_64-linux-3_db038e4e1623 | 2024-01-30 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1764875 | failed | master | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-29 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1762230 | failed | master | nightly-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-3.zw3rk_x86_64-linux-3_db038e4e1623 | 2024-01-26 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1761309 | failed | master | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-01-25 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1761061 | failed | refs/merge-requests/11978/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-24 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1756116 | failed | master | nightly-x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-19 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1755330 | failed | master | nightly-x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-01-18 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1754127 | failed | refs/merge-requests/11930/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-17 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1753168 | failed | refs/merge-requests/11923/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-01-16 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1752167 | failed | master | nightly-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-15 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1751802 | failed | refs/merge-requests/11910/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-6.zw3rk_x86_64-linux-6_db038e4e1623 | 2024-01-14 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1750663 | failed | master | nightly-x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-01-12 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1750390 | failed | master | nightly-x86_64-linux-alpine3_17-wasm-unreg-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-01-12 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1748067 | failed | refs/merge-requests/11879/head | x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static | x86_64-linux-2.zw3rk_x86_64-linux-2_d292a225d2eb | 2024-01-09 |Cheng ShaoCheng Shaohttps://gitlab.haskell.org/ghc/ghc/-/issues/24407head.hackage: ghc-debugger's test:system-test hangs on test-downstream job2024-02-06T15:33:58ZBryan Rbryan@haskell.foundationhead.hackage: ghc-debugger's test:system-test hangs on test-downstream jobThe failure affects both x86-64 and arm64.
Example 1:
https://gitlab.haskell.org/ghc/head.hackage/-/jobs/1769505
Full list:
https://gitlab.haskell.org/ghc/head.hackage/-/jobs?statuses=RUNNING
![image](/uploads/bc181b40cb8e8753603fb7...The failure affects both x86-64 and arm64.
Example 1:
https://gitlab.haskell.org/ghc/head.hackage/-/jobs/1769505
Full list:
https://gitlab.haskell.org/ghc/head.hackage/-/jobs?statuses=RUNNING
![image](/uploads/bc181b40cb8e8753603fb7344a9732cb/image.png)
(and many more)
Since a test timeout is treated as a spurious failure, these get restarted. I think there's around 10 restarts, which results in pipelines taking 80 hours to fail entirely. The result is that many head.hackage pipelines are still running.
https://gitlab.haskell.org/ghc/head.hackage/-/pipelines?page=1&scope=all&status=running
![image](/uploads/cd6673132a4e42c70488ae23434c1242/image.png)
Luckily, the test is merely zombied and doesn't take any system resources. Therefore I'll keep them running for now, so people can investigate. Example process tree:
```
489457 ? S 0:00 \_ bash /nix/store/36qkrsgvqb0scf7hcdr45vz37nbi904x-run-ci
489516 ? Sl 0:00 \_ /nix/store/jx4pn482brdqi9fp41m6faawwcibq354-head-hackage-ci-0.1.0.0/bin/head-hackage-ci test-patches --extra-cabal-fragment=/builds/ghc/head.hackage/ci/run/deps.cabal.project --patches=../../patches --with-compiler=/builds/ghc/head.hackage/ghc/bin/ghc --extra-cabal-fragment=/builds/ghc/head.hackage/ci/config.cabal.project --expect-broken=charsetdetect --expect-broken=packman --extra-package=lens==5.2.2 --extra-package=optics==0.4.2.1 --extra-package=aeson==2.2.0.0 --extra-package=criterion==1.6.2.0 --extra-package=scotty==0.12.1 --extra-package=generic-lens==2.2.2.0 --extra-package=microstache==1.0.2.3 --extra-package=singletons-base==3.1.1 --extra-package=servant==0.20 --extra-package=hgmp==0.1.2.1 --extra-package=Agda==2.6.3 --extra-package=mmark==0.0.7.6 --extra-package=doctest==0.22.0 --extra-package=tasty==1.4.3 --extra-package=pandoc==3.1.5 --extra-package=servant-conduit==0.16 --extra-package=servant-machines==0.16 --extra-package=linear-generics==0.2.2 --extra-package=futhark==0.25.2 --extra-package=generic-random==1.5.0.1 --extra-package=lame==0.2.1 --extra-package=inspection-testing==0.5.0.2 --extra-package=ghcide==2.0.0.1 --extra-package=vector-space==0.16 --build-tool-package=alex --build-tool-package=happy --build-tool-package=c2hs --only=tasty --test-package=system-test=/builds/ghc/head.hackage/ci/../tests/ghc-debug/test/ --test-package=ghc-tests=/builds/ghc/head.hackage/ci/../tests/ghc-tests --test-package=all=/builds/ghc/head.hackage/ci/../tests/text --test-package=bytestring-tests=/builds/ghc/head.hackage/ci/../tests/bytestring --test-package=all=/builds/ghc/head.hackage/ci/../tests/containers/containers-tests --cabal-option=-j5 --ghc-option=-dlint
548483 ? Sl 0:00 \_ /nix/store/gwlb8xmagw5a8wam56ja1l0xjpzd139a-cabal-install-3.10.1.0/bin/cabal new-test system-test --enable-tests -j5 -w /builds/ghc/head.hackage/ghc/bin/ghc
548511 ? Sl 0:00 \_ /nix/store/gwlb8xmagw5a8wam56ja1l0xjpzd139a-cabal-install-3.10.1.0/bin/.cabal-wrapped act-as-setup --build-type=Simple -- test --verbose=1 --builddir=/builds/ghc/head.hackage/ci/run/test-system-test/dist-newstyle/build/aarch64-linux/ghc-9.9.20240202/ghc-debugger-0.5.0.0/t/system-test --log=$pkgid-$test-suite.log --machine-log=$pkgid.log --show-details=failures system-test
548522 ? Sl 0:00 \_ /builds/ghc/head.hackage/ci/run/test-system-test/dist-newstyle/build/aarch64-linux/ghc-9.9.20240202/ghc-debugger-0.5.0.0/t/system-test/build/system-test/system-test
548717 ? Z 0:00 \_ [clock] <defunct>
```9.10.1Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/24365Intermittent hadrian exception: withFile: resource busy (file is locked)2024-01-29T07:09:22ZBryan Rbryan@haskell.foundationIntermittent hadrian exception: withFile: resource busy (file is locked)Not many in the last two weeks, though I think I've seen it before:
| json->>'id' | json->>'web_url' | json->>'runner.description' | date(json->>'created_at') |
|-------------|----...Not many in the last two weeks, though I think I've seen it before:
| json->>'id' | json->>'web_url' | json->>'runner.description' | date(json->>'created_at') |
|-------------|---------------------------------------------------|----------------------------------------------------|---------------------------|
| 1758509 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1758509 | azure-ci-1__9d97fee6dfe8 | 2024-01-23 |
| 1757881 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757881 | aarch64-darwin-2_aarch64-darwin-2_cf5fc9331e7a | 2024-01-22 |
| 1748231 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1748231 | aarch64-linux-2.zw3rk_aarch64-linux-2_379242e21d29 | 2024-01-09 |
Two of the occasions mention `stamp-ghc-bignum-1.3-inplace_dyn`; one of them mentions `stamp-ghc-9.9-inplace_dyn`. But see #20886: the filename may be a red herring. Or maybe not!
```
/---------------------------------------------------------------------------------\
| Successfully built library 'ghc-bignum' (Stage1, way v). |
| Library: _build/stage1/libraries/ghc-bignum/build/libHSghc-bignum-1.3-inplace.a |
| Library synopsis: GHC BigNum library. |
\---------------------------------------------------------------------------------/
Error when running Shake build system:
at want, called at src/Main.hs:126:44 in main:Main
* Depends on: test:all_deps
at apply1, called at src/Development/Shake/Internal/Rules/Rerun.hs:41:5 in shake-0.19.7-cd9c3e55acea68cf885a8c15ce0927a9c158686ecdfdc3cb2cfc3905a7237499:Development.Shake.Internal.Rules.Rerun
* Depends on: test:check-exact
at apply1, called at src/Development/Shake/Internal/Rules/Rerun.hs:41:5 in shake-0.19.7-cd9c3e55acea68cf885a8c15ce0927a9c158686ecdfdc3cb2cfc3905a7237499:Development.Shake.Internal.Rules.Rerun
* Depends on: _build/test/bin/check-exact
at need, called at src/Rules/Test.hs:163:17 in main:Rules.Test
* Depends on: _build/stage1/bin/check-exact
at need, called at src/Rules/Register.hs:86:5 in main:Rules.Register
* Depends on: _build/stage1/lib/i386-linux-ghc-9.9.20240123/libHSghc-bignum-1.3-inplace-ghc9.9.20240123.so
at need, called at src/Rules/Library.hs:83:5 in main:Rules.Library
* Depends on: _build/stage1/lib/package.conf.d/ghc-bignum-1.3-inplace.conf
at need, called at src/Rules/Register.hs:140:5 in main:Rules.Register
* Depends on: _build/stage1/libraries/ghc-bignum/build/stamp-ghc-bignum-1.3-inplace_dyn
* Raised the exception:
_build/stage1/libraries/ghc-bignum/build/stamp-ghc-bignum-1.3-inplace_dyn: withFile: resource busy (file is locked)
```https://gitlab.haskell.org/ghc/ghc/-/issues/24362Many unexpected test timeouts2024-01-23T15:25:15ZBryan Rbryan@haskell.foundationMany unexpected test timeoutsThe overwhelming culprit is zw3rk linux-4, so I have paused that runner and alerted @angerman.
Incidents in the last two weeks:
| json->>'id' | json->>'web_url' | json->>'runner.description' ...The overwhelming culprit is zw3rk linux-4, so I have paused that runner and alerted @angerman.
Incidents in the last two weeks:
| json->>'id' | json->>'web_url' | json->>'runner.description' | date(json->>'created_at') |
|-------------|---------------------------------------------------|--------------------------------------------------|---------------------------|
| 1758393 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1758393 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1758303 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1758303 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1758294 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1758294 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1758160 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1758160 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1757990 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757990 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1757824 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757824 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1757808 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757808 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-22 |
| 1757741 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757741 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-21 |
| 1757731 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757731 | x86_64-linux-6.zw3rk_x86_64-linux-6_db038e4e1623 | 2024-01-21 |
| 1757668 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757668 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-21 |
| 1757655 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757655 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-21 |
| 1757477 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757477 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-21 |
| 1757474 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757474 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-21 |
| 1757469 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1757469 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-21 |
| 1756592 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1756592 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-20 |
| 1756583 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1756583 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-20 |
| 1756128 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1756128 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-19 |
| 1756117 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1756117 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-19 |
| 1755896 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1755896 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-18 |
| 1755698 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1755698 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-18 |
| 1755343 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1755343 | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-18 |
| 1755328 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1755328 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-18 |
| 1755237 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1755237 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-17 |
| 1754569 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1754569 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-17 |
| 1754212 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1754212 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-17 |
| 1754209 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1754209 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-17 |
| 1754206 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1754206 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-17 |
| 1754033 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1754033 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-17 |
| 1753889 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753889 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753888 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753888 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753875 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753875 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753559 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753559 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753534 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753534 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753483 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753483 | azure-ghc-ci-linux-2 | 2024-01-16 |
| 1753289 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753289 | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-16 |
| 1753248 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753248 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753239 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753239 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753234 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753234 | x86_64-linux-5.zw3rk_x86_64-linux-5_db038e4e1623 | 2024-01-16 |
| 1753228 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753228 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753118 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753118 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1753112 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1753112 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-16 |
| 1752820 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1752820 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-15 |
| 1752817 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1752817 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-15 |
| 1752718 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1752718 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-15 |
| 1752420 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1752420 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-15 |
| 1752416 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1752416 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-15 |
| 1751858 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1751858 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-14 |
| 1751852 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1751852 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-14 |
| 1751646 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1751646 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-14 |
| 1751642 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1751642 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-14 |
| 1751181 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1751181 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-13 |
| 1751177 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1751177 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-13 |
| 1750977 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1750977 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-12 |
| 1750644 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1750644 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-12 |
| 1750643 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1750643 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-12 |
| 1750346 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1750346 | x86_64-linux-7.zw3rk_x86_64-linux-7_db038e4e1623 | 2024-01-12 |
| 1750264 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1750264 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-11 |
| 1750238 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1750238 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-11 |
| 1749516 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1749516 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-10 |
| 1749513 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1749513 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-10 |
| 1749001 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1749001 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-10 |
| 1748835 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1748835 | x86_64-linux-6.zw3rk_x86_64-linux-6_db038e4e1623 | 2024-01-10 |
| 1748634 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1748634 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-10 |
| 1748455 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1748455 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-10 |
| 1748087 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1748087 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-09 |
| 1747940 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1747940 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-09 |
| 1747936 | https://gitlab.haskell.org/ghc/ghc/-/jobs/1747936 | x86_64-linux-4.zw3rk_x86_64-linux-4_db038e4e1623 | 2024-01-09 |https://gitlab.haskell.org/ghc/ghc/-/issues/24303"MultiLayerModulesDefsGhciReload" timeouts on master2024-01-09T16:17:25ZBryan Rbryan@haskell.foundation"MultiLayerModulesDefsGhciReload" timeouts on masterSeen on https://grafana.gitlab.haskell.org/d/phJc-UyVz/ci-success-stats?orgId=2&refresh=30m&viewPanel=8
* https://gitlab.haskell.org/ghc/ghc/-/jobs/1741961
* https://gitlab.haskell.org/ghc/ghc/-/jobs/1740330
* https://gitlab.haskell.org...Seen on https://grafana.gitlab.haskell.org/d/phJc-UyVz/ci-success-stats?orgId=2&refresh=30m&viewPanel=8
* https://gitlab.haskell.org/ghc/ghc/-/jobs/1741961
* https://gitlab.haskell.org/ghc/ghc/-/jobs/1740330
* https://gitlab.haskell.org/ghc/ghc/-/jobs/1741973ZubinZubinhttps://gitlab.haskell.org/ghc/ghc/-/issues/24274Spurious failures due to intermittent stack concurrency bug2024-01-19T12:53:04ZBryan Rbryan@haskell.foundationSpurious failures due to intermittent stack concurrency bugWe have spurious failures because of https://github.com/commercialhaskell/stack/issues/6388.
What's not clear to me is why a package that required a call to `updateHackageIndex` in one run did *not* require the call in a different run. ...We have spurious failures because of https://github.com/commercialhaskell/stack/issues/6388.
What's not clear to me is why a package that required a call to `updateHackageIndex` in one run did *not* require the call in a different run. I would expect the set of packages that trigger the call to be fixed. Are we caching the Pantry database?https://gitlab.haskell.org/ghc/ghc/-/issues/24259JS: Spurious failures in CI2023-12-21T21:18:04ZLuite StegemanJS: Spurious failures in CIThe JS backend experiences occasional failures in CI, which is holding up marge batches.
This is likely due to a combination of real bugs and poorly adjusted timeouts for the tests.
I'm going to mark the tests as `js_fragile` with a re...The JS backend experiences occasional failures in CI, which is holding up marge batches.
This is likely due to a combination of real bugs and poorly adjusted timeouts for the tests.
I'm going to mark the tests as `js_fragile` with a reference to this ticket number. In the future we can track the underlying cause here, and if necessary open new tickets for them.
Hopefully in a few months we can check back on this ticket, review the CI logs and remove the `js_fragile` designation if tests aren't failing anymore.Luite StegemanLuite Stegemanhttps://gitlab.haskell.org/ghc/ghc/-/issues/24209Alpine-based CI jobs fail with out-of-date Hackage index2023-11-28T15:24:20ZBen GamariAlpine-based CI jobs fail with out-of-date Hackage indexNightly Alpine jobs are failing with:
```
Running hadrian/build-cabal --flavour=release+fully_static -j8 --broken-test= --bignum=gmp --docs=none test:all_deps binary-dist -V...
Warning: Unknown/unsupported 'ghc' version detected (Cabal 3...Nightly Alpine jobs are failing with:
```
Running hadrian/build-cabal --flavour=release+fully_static -j8 --broken-test= --bignum=gmp --docs=none test:all_deps binary-dist -V...
Warning: Unknown/unsupported 'ghc' version detected (Cabal 3.6.2.0 supports
'ghc' version < 9.4): /opt/ghc/9.4.3/bin/ghc is version 9.4.3
Warning: Requested index-state 2023-10-05T11:38:51Z is newer than
'hackage.haskell.org'! Falling back to older state (2022-06-09T19:15:55Z).
Resolving dependencies...
cabal: Could not resolve dependencies:
...
```
For instance, see Job [#1717907](https://gitlab.haskell.org/ghc/ghc/-/jobs/1717907), Job [#1717906](https://gitlab.haskell.org/ghc/ghc/-/jobs/1717906), and Job [#1717910](https://gitlab.haskell.org/ghc/ghc/-/jobs/1717910).
This is strange since we earlier had run:
```
Running /usr/local/bin/cabal update -w /opt/ghc/9.4.3/bin/ghc hackage.haskell.org,2023-10-05T11:38:51Z...
Config file path source is default config file.
Config file /builds/ghc/ghc/cabal/config not found.
Writing default configuration to /builds/ghc/ghc/cabal/config
Downloading the latest package list from hackage.haskell.org
Updated package list of hackage.haskell.org to the index-state 2023-10-05T11:38:51Z
```
My initial suspicion was that we are using a different `cabal-install` between these two invocations. However, looking at `ci.sh` it appears that we use `$CABAL` consistently. Very odd.9.10.1Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/24056aarch64-deb10 image has too old LLVM2023-11-01T00:05:01ZBen Gamariaarch64-deb10 image has too old LLVMCurrently the nightly `nightly-aarch64-linux-deb10-validate+llvm` job fails as the image has too old an LLVM toolchain (`llvm-12`)
See https://gitlab.haskell.org/ghc/ghc/-/jobs/1674986.Currently the nightly `nightly-aarch64-linux-deb10-validate+llvm` job fails as the image has too old an LLVM toolchain (`llvm-12`)
See https://gitlab.haskell.org/ghc/ghc/-/jobs/1674986.Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/2397895.5% fail rate for Validate pipelines on master in the last month2023-10-24T08:19:33ZBryan Rbryan@haskell.foundation95.5% fail rate for Validate pipelines on master in the last month[![image](/uploads/d7445438005c220886d5d74478668c42/image.png)](https://grafana.gitlab.haskell.org/d/kv1oojs4z/ci-health?orgId=2&refresh=1m&viewPanel=8)
| job | pipeline | created_at |
|----|--------|-----------|
| [aarch64-darwin-vali...[![image](/uploads/d7445438005c220886d5d74478668c42/image.png)](https://grafana.gitlab.haskell.org/d/kv1oojs4z/ci-health?orgId=2&refresh=1m&viewPanel=8)
| job | pipeline | created_at |
|----|--------|-----------|
| [aarch64-darwin-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1675871) | [Pipeline #84634](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84634) | 2023-09-26 19:26:02.182335 |
| [aarch64-linux-deb10-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1675872) | [Pipeline #84634](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84634) | 2023-09-26 19:26:02.182335 |
| [i386-linux-deb10-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1689335) | [Pipeline #85255](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85255) | 2023-10-10 23:06:03.398766 |
| [i386-linux-deb10-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1695092) | [Pipeline #85560](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85560) | 2023-10-18 23:41:53.60398 |
| [lint-ci-config](https://gitlab.haskell.org/ghc/ghc/-/jobs/1675089) | [Pipeline #84587](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84587) | 2023-09-26 01:57:12.916866 |
| [lint-ci-config](https://gitlab.haskell.org/ghc/ghc/-/jobs/1697082) | [Pipeline #85656](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85656) | 2023-10-22 06:50:42.367846 |
| [lint-submods-branch](https://gitlab.haskell.org/ghc/ghc/-/jobs/1690820) | [Pipeline #85348](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85348) | 2023-10-12 14:32:05.387576 |
| [stack-hadrian-build](https://gitlab.haskell.org/ghc/ghc/-/jobs/1675870) | [Pipeline #84634](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84634) | 2023-09-26 19:26:02.182335 |
| [test-bootstrap: [9.6.2, registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-deb10-ghc9_6:$DOCKER_REV]](https://gitlab.haskell.org/ghc/ghc/-/jobs/1687643) | [Pipeline #85186](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85186) | 2023-10-10 01:36:49.329917 |
| [typecheck-testsuite](https://gitlab.haskell.org/ghc/ghc/-/jobs/1677545) | [Pipeline #84717](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84717) | 2023-09-28 07:27:41.812681 |
| [typecheck-testsuite](https://gitlab.haskell.org/ghc/ghc/-/jobs/1690817) | [Pipeline #85348](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85348) | 2023-10-12 14:32:05.387576 |
| [x86_64-darwin-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1679545) | [Pipeline #84812](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84812) | 2023-09-30 20:11:34.546562 |
| [x86_64-darwin-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1683714) | [Pipeline #84981](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84981) | 2023-10-05 09:38:38.353623 |
| [x86_64-darwin-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1696173) | [Pipeline #85610](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85610) | 2023-10-20 08:03:44.795575 |
| [x86_64-linux-alpine3_12-validate+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1675875) | [Pipeline #84634](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84634) | 2023-09-26 19:26:02.182335 |
| [x86_64-linux-alpine3_12-validate+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1692921) | [Pipeline #85454](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85454) | 2023-10-14 23:20:02.15024 |
| [x86_64-linux-alpine3_12-validate+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1697091) | [Pipeline #85656](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85656) | 2023-10-22 06:50:42.367846 |
| [x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1677207) | [Pipeline #84703](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84703) | 2023-09-27 21:34:27.758873 |
| [x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1686166) | [Pipeline #85110](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85110) | 2023-10-07 14:01:21.41664 |
| [x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1686556) | [Pipeline #85129](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85129) | 2023-10-08 19:07:09.149642 |
| [x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1695095) | [Pipeline #85560](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85560) | 2023-10-18 23:41:53.60398 |
| [x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1695322) | [Pipeline #85560](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85560) | 2023-10-18 23:41:53.60398 |
| [x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static](https://gitlab.haskell.org/ghc/ghc/-/jobs/1696175) | [Pipeline #85610](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85610) | 2023-10-20 08:03:44.795575 |
| [x86_64-linux-deb10-int_native-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1676342) | [Pipeline #84653](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84653) | 2023-09-27 05:19:55.819081 |
| [x86_64-linux-deb10-numa-slow-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1682650) | [Pipeline #84946](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84946) | 2023-10-04 09:46:12.331567 |
| [x86_64-linux-deb10-numa-slow-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1695100) | [Pipeline #85560](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85560) | 2023-10-18 23:41:53.60398 |
| [x86_64-linux-deb10-unreg-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1690042) | [Pipeline #85309](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85309) | 2023-10-11 20:04:22.784955 |
| [x86_64-linux-deb10-validate+debug_info](https://gitlab.haskell.org/ghc/ghc/-/jobs/1690835) | [Pipeline #85348](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85348) | 2023-10-12 14:32:05.387576 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1675886) | [Pipeline #84634](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84634) | 2023-09-26 19:26:02.182335 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1677566) | [Pipeline #84717](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84717) | 2023-09-28 07:27:41.812681 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1679557) | [Pipeline #84812](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84812) | 2023-09-30 20:11:34.546562 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1683726) | [Pipeline #84981](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84981) | 2023-10-05 09:38:38.353623 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1689348) | [Pipeline #85255](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85255) | 2023-10-10 23:06:03.398766 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1690515) | [Pipeline #85332](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85332) | 2023-10-12 10:50:42.9644 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1692932) | [Pipeline #85454](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85454) | 2023-10-14 23:20:02.15024 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1695105) | [Pipeline #85560](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85560) | 2023-10-18 23:41:53.60398 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1695587) | [Pipeline #85585](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85585) | 2023-10-19 15:38:16.332596 |
| [x86_64-linux-deb11-int_native-cross_javascript-unknown-ghcjs-validate](https://gitlab.haskell.org/ghc/ghc/-/jobs/1697718) | [Pipeline #85689](https://gitlab.haskell.org/ghc/ghc/-/pipelines/85689) | 2023-10-23 13:10:40.502862 |
| [x86_64-linux-fedora33-release](https://gitlab.haskell.org/ghc/ghc/-/jobs/1677567) | [Pipeline #84717](https://gitlab.haskell.org/ghc/ghc/-/pipelines/84717) | 2023-09-28 07:27:41.812681 |
```sql
with pipe as (select *
from ci_pipelines p
where p.project_id = 1
and p.status in ('success', 'failed')
and p.created_at > now() - interval '1 month'
and p.pipeline_schedule_id is null
and p.ref = 'master')
select format('[%s](%s)', j.name, 'https://gitlab.haskell.org/ghc/ghc/-/jobs/' || j.id :: text) "job",
format('[Pipeline #%s}(%s)', pipe.id, 'https://gitlab.haskell.org/ghc/ghc/-/pipelines/' || pipe.id) :: text "pipeline",
pipe.created_at
from ci_builds j
join ci_stages s
on j.stage_id = s.id
join pipe
on s.pipeline_id = pipe.id
where j.status = 'failed'
and not j.allow_failure
order by j.name, pipe.created_at
```https://gitlab.haskell.org/ghc/ghc/-/issues/23844debian9 nightly build failing due to older sphinx version2023-08-22T14:59:16ZMatthew Pickeringdebian9 nightly build failing due to older sphinx versionSee https://gitlab.haskell.org/ghc/ghc/-/jobs/1639202
```
===> Command failed with error code: 1
Extension error:
Could not import extension flags (exception: No module named 'packaging')
Running Sphinx v1.4.9
haddock: user interrupt
Er...See https://gitlab.haskell.org/ghc/ghc/-/jobs/1639202
```
===> Command failed with error code: 1
Extension error:
Could not import extension flags (exception: No module named 'packaging')
Running Sphinx v1.4.9
haddock: user interrupt
Error when running Shake build system:
at want, called at src/Main.hs:124:44 in main:Main
* Depends on: binary-dist
at apply1, called at src/Development/Shake/Internal/Rules/Rerun.hs:41:5 in shake-0.19.7-210fc1cbce69aed26102707e3a1f9a8d35afdb700b5ceb2aa5cba89db1cc9fcb:Development.Shake.Internal.Rules.Rerun
* Depends on: binary-dist-dir
at need, called at src/Rules/BinaryDist.hs:220:24 in main:Rules.BinaryDist
* Depends on: docs
at need, called at src/Rules/Documentation.hs:129:9 in main:Rules.Documentation
* Depends on: _build/doc/users_guide.pdf
at cmd', called at src/Builder.hs:393:23 in main:Builder
at cmdArgs, called at src/Builder.hs:573:8 in main:Builder
at cmdArgs, called at src/Builder.hs:597:18 in main:Builder
at cmdArgs, called at src/Builder.hs:597:18 in main:Builder
at error, called at src/Builder.hs:642:13 in main:Builder
* Raised the exception:
Command failed
Build failed.
```Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/23641emsdk >? 3.1.40 breaks javascript build2023-08-08T15:23:04ZMatthew Pickeringemsdk >? 3.1.40 breaks javascript buildIn #23585 we discovered that new versions of emsdk break some of the javascript tests.
For example we see in this job: https://gitlab.haskell.org/ghc/ghc/-/jobs/1598987
```
Unexpected failures:
/builds/ghc/ghc/tmp/ghctest-46palsnb/t...In #23585 we discovered that new versions of emsdk break some of the javascript tests.
For example we see in this job: https://gitlab.haskell.org/ghc/ghc/-/jobs/1598987
```
Unexpected failures:
/builds/ghc/ghc/tmp/ghctest-46palsnb/test spaces/testsuite/tests/callarity/unittest/CallArity1.run CallArity1 [bad exit code (1)] (normal)
/builds/ghc/ghc/tmp/ghctest-46palsnb/test spaces/testsuite/tests/corelint/LintEtaExpand.run LintEtaExpand [bad exit code (1)] (normal)
/builds/ghc/ghc/tmp/ghctest-46palsnb/test spaces/testsuite/tests/ghc-api/T10942.run T10942 [bad exit code (1)] (normal)
/builds/ghc/ghc/tmp/ghctest-46palsnb/test spaces/testsuite/tests/ghc-api/T18522-dbg-ppr.run T18522-dbg-ppr [bad exit code (1)] (normal)
/builds/ghc/ghc/tmp/ghctest-46palsnb/test spaces/testsuite/tests/ghc-api/T9595.run T9595 [bad exit code (1)] (normal)
```
Which are caused by failures of this kind:
```
T18522-dbg-ppr: ghc no longer supports single-file style package databases (/builds/ghc/ghc/_build/stage1/lib/package.conf.d) use 'ghc-pkg init' to create the database with the correct format.
```
Perhaps this is a regression in emsdk but for now in the images we are pinning the version to 3.1.40 which works.https://gitlab.haskell.org/ghc/ghc/-/issues/23144Darwin gitlab-runner occasionally receives SIGQUIT and dies2023-03-24T08:48:37ZBryan Rbryan@haskell.foundationDarwin gitlab-runner occasionally receives SIGQUIT and diesSeems to be happening about once a month starting in August 2022. It usually happens when the runner is uploading artifacts at the end of a run. Once, though, it happened while uploading the cache at the end of a run.
I guess this is pr...Seems to be happening about once a month starting in August 2022. It usually happens when the runner is uploading artifacts at the end of a run. Once, though, it happened while uploading the cache at the end of a run.
I guess this is probably a bug in gitlab-runner. My plan is just to treat it as spurious and retry such jobs.
I am opening this ticket just so we have a discussion spot for this topic, should it ever need discussion.
NB gitlab-runner apparently uses SIGQUIT as a signal for a "graceful shutdown", which makes me sad. That's what SIGTERM is already for...?
| json->>'web_url' | date(json->>'created_at') | json->>'runner.description' |
|---------------------------------------------------|---------------------------|------------------------------------------------|
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1397503 | 2023-03-08 | aarch64-darwin-1_Mini18-Beta_f87b8a204711 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1393354 | 2023-03-03 | aarch64-darwin-2_aarch64-darwin-2_a1a3a7139d97 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1381660 | 2023-02-18 | aarch64-darwin-1_Mini18-Beta_f87b8a204711 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1348404 | 2023-01-31 | m1-delta_Mini18-Delta_f87b8a204711 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1303051 | 2023-01-12 | aarch64-darwin-1_Mini18-Epsilon_004e0928d81c |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1275162 | 2022-12-22 | aarch64-darwin-1_Mini18-Epsilon_004e0928d81c |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1261884 | 2022-12-14 | aarch64-darwin-1_Mini18-Beta_f87b8a204711 |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1175376 | 2022-09-19 | aarch64-darwin-1_Mini18-Epsilon_004e0928d81c |
| https://gitlab.haskell.org/ghc/ghc/-/jobs/1132694 | 2022-08-06 | aarch64-darwin-1_Mini18-Beta_f87b8a204711 |Bryan Rbryan@haskell.foundationBryan Rbryan@haskell.foundationhttps://gitlab.haskell.org/ghc/ghc/-/issues/22967"ghc-config: /builds/ghc/ghc/_build/stage1-test/bin/ghc: readCreateProcess: p...2023-03-07T13:32:34ZBryan Rbryan@haskell.foundation"ghc-config: /builds/ghc/ghc/_build/stage1-test/bin/ghc: readCreateProcess: posix_spawnp: resource busy (Text file busy)"Example: https://gitlab.haskell.org/ghc/ghc/-/jobs/1357237
Seems to have only started in January:
```sql
select count(job_id) 'cnt',
strftime('%Y-%m', json ->> 'created_at') 'date', json->>'$.name' 'name'
from job
where job_id in (...Example: https://gitlab.haskell.org/ghc/ghc/-/jobs/1357237
Seems to have only started in January:
```sql
select count(job_id) 'cnt',
strftime('%Y-%m', json ->> 'created_at') 'date', json->>'$.name' 'name'
from job
where job_id in (
select rowid from job_trace where trace match
'"ghc-config: /builds/ghc/ghc/_build/stage1-test/bin/ghc: readCreateProcess: posix_spawnp: resource busy (Text file busy)"'
)
group by
strftime('%Y-%m', json ->> 'created_at'), json->>'$.name'
order by strftime('%Y-%m', json ->> 'created_at') desc
-------------
cnt date name
--- ------- -------------------------------------------------
3 2023-02 i386-linux-deb9-validate
1 2023-02 nightly-i386-linux-deb9-validate
1 2023-02 nightly-x86_64-linux-deb10-no_tntc-validate
1 2023-02 release-x86_64-linux-deb10-release+debug_info
1 2023-02 release-x86_64-linux-fedora33-release
1 2023-02 x86_64-linux-alpine3_12-validate+fully_static
3 2023-02 x86_64-linux-deb10-int_native-validate
1 2023-02 x86_64-linux-deb10-numa-slow-validate
1 2023-02 x86_64-linux-deb10-unreg-validate
2 2023-02 x86_64-linux-deb10-validate+debug_info
1 2023-02 x86_64-linux-fedora33-release
1 2023-01 aarch64-linux-deb10-validate
3 2023-01 i386-linux-deb9-validate
2 2023-01 nightly-x86_64-linux-deb10-validate+debug_info
2 2023-01 nightly-x86_64-linux-deb11-validate
1 2023-01 release-i386-linux-deb9-release+no_split_sections
1 2023-01 x86_64-linux-alpine3_12-validate+fully_static
2 2023-01 x86_64-linux-deb10-int_native-validate
4 2023-01 x86_64-linux-deb10-numa-slow-validate
1 2023-01 x86_64-linux-deb10-unreg-validate
5 2023-01 x86_64-linux-deb10-validate+debug_info
1 2023-01 x86_64-linux-fedora33-release
```
Note: the match string can be reduced to simply `"posix_spawnp: resource busy"` and the same results are returned.Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/21557T16875 suspiciously improves all the time on Windows2022-05-29T01:00:52ZSebastian GrafT16875 suspiciously improves all the time on WindowsIt appears that a number of recent MRs observe metric decreases in T16875:
- !8177
- !8158
- !8096
- !7599
- !7891
- ...
It appears that none of the decreases really make it into master: https://grafana.gitlab.haskell.org/d/s3PVoM_Mz/...It appears that a number of recent MRs observe metric decreases in T16875:
- !8177
- !8158
- !8096
- !7599
- !7891
- ...
It appears that none of the decreases really make it into master: https://grafana.gitlab.haskell.org/d/s3PVoM_Mz/ghc-testsuite-performance?orgId=2&var-environment=x86_64-windows-validate&var-test_name=compile_time%2Fbytes%20allocated%2FT16875%2Fnormal
So it's just noise. Perhaps we should increase the acceptance ratio or otherwise figure out what's going on.https://gitlab.haskell.org/ghc/ghc/-/issues/21350T10529c failed spuriously2022-04-11T09:16:18ZSimon JakobiT10529c failed spuriously…in https://gitlab.haskell.org/ghc/ghc/-/jobs/998555 on `x86_64-windows-validate`:
```
--- "C:/GitLabRunner/builds/1/998555/tmp/ghctest-a2uhb1yw/test spaces/libraries/hpc/tests/simple/tixs/T10529c.run/T10529c.stderr.normalised" 2022-0...…in https://gitlab.haskell.org/ghc/ghc/-/jobs/998555 on `x86_64-windows-validate`:
```
--- "C:/GitLabRunner/builds/1/998555/tmp/ghctest-a2uhb1yw/test spaces/libraries/hpc/tests/simple/tixs/T10529c.run/T10529c.stderr.normalised" 2022-04-02 06:17:14.458016400 +0200
+++ "C:/GitLabRunner/builds/1/998555/tmp/ghctest-a2uhb1yw/test spaces/libraries/hpc/tests/simple/tixs/T10529c.run/T10529c.run.stderr.normalised" 2022-04-02 06:17:14.458016400 +0200
@@ -1,3 +0,0 @@
-hpc: can not parse ./.hpc/NoParse.mix
-CallStack (from HasCallStack):
- error, called at libraries/hpc/Trace/Hpc/Mix.hs:<line>:<column> in <package-id>:Trace.Hpc.Mix
```https://gitlab.haskell.org/ghc/ghc/-/issues/21347Spurious CI failures for the x86_64-windows-validate builds.2022-12-02T18:07:24ZAndreas KlebingerSpurious CI failures for the x86_64-windows-validate builds.I've seen this job fail twice now. Latest failure is here: https://gitlab.haskell.org/ghc/ghc/-/jobs/1000062
I assume something is wrong with the runner here. (`Runner: #141 (QX_Lxgyq) x86-64-win-1.zw3rk.com`)
@bgamari @angermanI've seen this job fail twice now. Latest failure is here: https://gitlab.haskell.org/ghc/ghc/-/jobs/1000062
I assume something is wrong with the runner here. (`Runner: #141 (QX_Lxgyq) x86-64-win-1.zw3rk.com`)
@bgamari @angermanBen GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/21184T12903 unexpectedly times out2023-02-16T11:43:57ZSimon JakobiT12903 unexpectedly times outSee https://gitlab.haskell.org/ghc/ghc/-/jobs/968679:
```
Timeout happened...killed process "cd "rts/T12903.run" && ./T12903 "...
Wrong exit code for T12903(normal)(expected 0 , actual 99 )
*** unexpected failure for T12903(normal)
`...See https://gitlab.haskell.org/ghc/ghc/-/jobs/968679:
```
Timeout happened...killed process "cd "rts/T12903.run" && ./T12903 "...
Wrong exit code for T12903(normal)(expected 0 , actual 99 )
*** unexpected failure for T12903(normal)
```
Note that this test is already marked fragile or skipped on some platforms and ways:
https://gitlab.haskell.org/ghc/ghc/-/blob/706deee0524ca6af26c8b8d5cff17a6e401a2c18/testsuite/tests/rts/all.T#L412-419Moritz AngermannMoritz Angermannhttps://gitlab.haskell.org/ghc/ghc/-/issues/20667T16525b is fragile on Windows2021-11-27T17:24:42ZBen GamariT16525b is fragile on WindowsRecently `T16525b` has been quite a source of non-deterministic CI failures on Windows. The failures range from bad stderr output, to timeouts. However, sometimes it does in fact pass.Recently `T16525b` has been quite a source of non-deterministic CI failures on Windows. The failures range from bad stderr output, to timeouts. However, sometimes it does in fact pass.