Skip to content

CI: Intermittent "Command failed with error code: -4" on Darwin.

Recent occurrences:

json->>'web_url' json->>'status' json->>'ref' json->>'name' json->>'runner.description' date(json->>'created_at')
https://gitlab.haskell.org/ghc/ghc/-/jobs/1772620 failed refs/merge-requests/11968/head x86_64-darwin-validate m1-delta_Mini18-Delta_f87b8a204711 2024-02-07
https://gitlab.haskell.org/ghc/ghc/-/jobs/1767573 failed refs/merge-requests/11576/head x86_64-darwin-validate aarch64-darwin-1_Mini18-Beta_f87b8a204711 2024-02-01
https://gitlab.haskell.org/ghc/ghc/-/jobs/1767305 failed refs/merge-requests/11976/head x86_64-darwin-validate aarch64-darwin-1_Mini18-Epsilon_004e0928d81c 2024-02-01
https://gitlab.haskell.org/ghc/ghc/-/jobs/1764568 failed refs/merge-requests/11989/head x86_64-darwin-validate aarch64-darwin-1_Mini18-Beta_f87b8a204711 2024-01-29
https://gitlab.haskell.org/ghc/ghc/-/jobs/1753166 failed refs/merge-requests/11923/head x86_64-darwin-validate aarch64-darwin-1_Mini18-Beta_f87b8a204711 2024-01-16

Analysis

  • Signal 4 is "Illegal instruction", so maybe Rosetta is involved.
  • The (truncated) command lines that failed in these occurences are:
    • /nix/store/sbfhs3rv53vgh3cn46q900fmgsl5q4x4-ghc/bin/ghc-pkg --no-user-package-db field filepath depends
    • _build/stage0/bin/ghc -Wall -Wcompat -fdiagnostics-color=never -dynamic-too -hisuf hi -osuf o -hcsuf <SNIP>
    • _build/stage0/bin/ghc -Wall -Wcompat -hisuf dyn_hi -osuf dyn_o -hcsuf dyn_hc -fPIC -dynamic <SNIP>
    • /nix/store/sbfhs3rv53vgh3cn46q900fmgsl5q4x4-ghc/bin/ghc-pkg --expand-pkgroot --no-user-package-db describe stm
    • /usr/bin/gcc --target=x86_64-apple-darwin -Qunused-arguments -E -MM -MG -MF _build/stage1/rts/build/c/sm/NonMovingScav.thr_debug_o.d <SNIP>

So if it managed to affect nix-packaged software, our own built software, and /usr/bin/gcc, I feel it has to be a system issue. Unfortunately it also seems to affect multiple machines.

Tracking link

https://grafana.gitlab.haskell.org/d/167r9v6nk/ci-spurious-failures?orgId=2&refresh=15m&from=now-30d&to=now&var-types=code_-4&var-runners=All&var-jobs=All

Edited by Bryan R
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information