Tracking issue: Docker/gitlab-runner/golang incompatibility

A breaking change in golang broke Docker (https://github.com/docker/cli/issues/4437#issuecomment-1637715017). It also broke gitlab-runner (https://gitlab.com/gitlab-org/gitlab-runner/-/issues/36051).

Until fixes for these problems flow down the distribution channels, our gitlab runners are pinned to fixed versions of nixpkgs.

Original report follows.

---

I have had to pause zw3rk linux runners 5,6 and 7.

ERROR: Job failed (system failure): prepare environment: http: invalid Host header (exec.go:71:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

See: https://gitlab.haskell.org/ghc/ghc/-/jobs/1619718

cc @chreekat @angerman

added Phighest label

https://github.com/NixOS/nixpkgs/issues/244159

Oh no, that's a terrible bug. Will you revert the machines back to a working nixpkgs version?

If I understand correctly, there is currently a fix in nixos-23.05 which should resolve the issue.

bumped past the fix that @bgamari mentioned. Deploy in progress. There used to be a fix (overriding docker with an older revision) present. I'm not sure why that didn't work.

Hope it's fixed now

marked this issue as related to ci-config#6 (closed)

I had to pause aarch64-linux-2 as well.

added CI breakage label

Also https://gitlab.haskell.org/ghc/ghc/-/jobs/1621012

Pausing aarch64-linux_ghc-arm-5_6b65c6b684af

all zw3rk runners should have been deployed past nixpkgs@3e0779ce103122aa446f480047dbe6fa43838bf8. I do not know who maintains arm-5

Thanks for update Moritz, I will continue to monitor the situatin.

There is a quite recent failure on -5, was this after the deployment?

https://gitlab.haskell.org/jhrcek/ghc/-/jobs/1621057

Yes, that would have been past the deployment.

I have paused linux-5 again.

Crap... https://github.com/NixOS/nixpkgs/issues/245365

@mpickering I'll try to follow up on this. I'm afraid I'm currently on holidays and only with limited internet availability.

Thanks for diagnosing that Moritz, it's really helpful, perhaps we can roll back to a commit which uses go-1.20.5?

https://github.com/NixOS/nixpkgs/pull/245381

apparently merged 9hs ago. Let's try another bump

I'm a bit reluctant to rollback, as I need some GitHub Runners (for which GitHub conveniently has rolling releases, and disables old releases ).

I am not sure I see how that linked commit will fix the issue, I couldn't see where they mentioned it would fix the problem in the changelog.

Fair, I might have misread a "fix" into the note

Note that the docker executor is broken since Go 1.20.6. #245365 from: https://github.com/NixOS/nixpkgs/pull/245366

I have paused 5,6,7 again.

trying:

                                (self: super: {
                                    go = super.go.overrideDerivation (old: rec {
                                        version = "1.20.5";
                                        src = super.fetchurl {
                                            url = "https://go.dev/dl/go${version}.src.tar.gz";
                                            hash = "sha256-mhXBM7os+v55ZS9IFbYufPwmf2jfG5RUxqsqPKi5aog=";
                                        };
                                    });
                                })

deployed.

Thanks @angerman, still seems an issue on aarch64-linux-2.

https://gitlab.haskell.org/ghc/ghc/-/jobs/1621141

We don't have any working aarch64 runners currently so all CI pipelines are blocked.

linux-7 also still failing: https://gitlab.haskell.org/ghc/ghc/-/jobs/1621292

I think rolling back the runners to a known good commit is a sensible option given our CI systems are blocked completely now.

As per https://github.com/NixOS/nixpkgs/issues/245365#issuecomment-1651470840, now https://github.com/angerman/nixpkgs/commit/604883e4fc2e235e3769bb41ef3f97a7f04a8de4 deployed.

https://gitlab.haskell.org/ghc/ghc/-/jobs/1621352 looks ok, so I guess it worked this time around.

Can someone summarize the current state of affairs? I take it some runners have been re-enabled -- by pinning an older version, patching, or upgrading to a fixed version? Which runners are fixed by which means? Are other runners still borked?

@angerman, I think you would be the best person to summarize here.

5,6,7 (and 8; though that serves mostly GitHub.com/haskell); as well as the aarch64-linux machine are deployed from https://github.com/angerman/nixpkgs/commit/604883e4fc2e235e3769bb41ef3f97a7f04a8de4.

@chreekat does have access to github:zw3rk/infra, which contains the relevant nixos/nix-darwin deployment.

For the record, gitlab-runner upstream is discussing how to proceed with fixing this issue here:

• https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4249
• https://gitlab.com/gitlab-org/gitlab-runner/-/issues/36051

I am blocked on this issue still at aarch64-linux_ghc-arm-5_6b65c6b684af is still paused (this is the only one we can build docker images on so ci-images CI is broken). Could we please prioritise reenabling this machine @chreekat @bgamari ?

After @mpickering messaged me on IRC yesterday, I was able to fix this runner as well.

ghc-ci-perf-1 and ghc-arm-5 are both NixOS machines running in Equinix whose configuration and documentation is in the ci-config repo. See e.g. the changes to fix ghc-arm-5 here: ci-config@49c9cf33 . There was some confusion on this fact, I gather.

mentioned in issue ci-config#6 (closed)

changed title from ghc-5,6,7 zw3rk runners all failing to Tracking issue: Docker/gitlab-runner/golang incompatibility

changed the description

removed CI breakage Phighest labels

added CI breakage Phighest labels

All upstreams have been patched to deal with this, so I think we can close this issue now.

closed