panic! collectNBinders

I'm getting

<no location info>: error:
    panic! (the 'impossible' happened)
  GHC version 9.6.2:
	collectNBinders
  1
  Call stack:
      CallStack (from HasCallStack):
        callStackDoc, called at compiler/GHC/Utils/Panic.hs:189:37 in ghc:GHC.Utils.Panic
        pprPanic, called at compiler/GHC/Core.hs:2050:25 in ghc:GHC.Core
  CallStack (from HasCallStack):
    panic, called at compiler/GHC/Utils/Error.hs:454:29 in ghc:GHC.Utils.Error

Repro: with GHC 9.6.2 run:

cabal test simplifiedOnlyTest -ftest_seq --enable-optimization --allow-newer=base

on commit

https://github.com/Mikolaj/horde-ad/commit/a9ea6e930f3561d90237cf8a431b8aad24387c8e

9.8.1-alpha with head.hackage compiles fine (but --allow-newer=base doesn't suffice, so I can't try the exact same commandline as with GHc 9.6.2). GHC 9.4.6 with and without --allow-newer=base compiles fine.

Similar issues: #13413 (closed) #15002 (closed) #15108 (closed)