internal error: Evaluated a CAF (0xc98630) that was GC'd!

Summary

After trying to upgrade to GHC 9.0.2 from GHC 8.10.7, we are seeing a binary fail with

Internal error: Evaluated a CAF (0xc98630) that was GC'd!

This seems to happen non-deterministically but fairly reliably across a few runs. However seemingly irrelevant changes sometimes make the issue go away but I tried to minimize as far as possible while still reproducing fairly reliably.

This looks similar to #18680 but given that for us this seems to be a regression after upgrading (unable to reproduce this on 8.10.7), I’m opening a separate issue.

Steps to reproduce

1. Download the package from https://purelyfunctional.org/downloads/ghc-rts-bug.tar.gz (Note that this contains a fairly large text file used as input data, the exact contents don’t seem to matter).
2. Run cabal run inside the project a few times until it fails. There is a nix-shell environment to reproduce exactly.

Expected behavior

The binary succeeds without errors.

Environment

• GHC version used: 9.0.2

Optional:

• Operating System: Ubuntu 20.04
• System Architecture: x86_64