GHC HEAD fails to build with `-fspecialize-aggressively -fexpose-all-unfoldings`

Running hadrian/build -j6 --flavour=default+no_profiled_libs "stage1.*.ghc.hs.opts += -fspecialize-aggressively -fexpose-all-unfoldings" -o_spec_trans

I get this panic:

ghc: panic! (the 'impossible' happened)
  GHC version 9.7.20230104:
        lookupIdSubst
  v_sa3N
  InScope {s_aa6X eta_aa6Z debugTerminalKeys $trModule loc_s8n5
           loc_s8n6 loc_s8n7 loc_s8n8 loc_s8n9 loc_s8na loc_s8nb $dIP_s8nc
           ...
           ...
           ...
           $dMonad_sa70}
  Call stack:
      CallStack (from HasCallStack):
        callStackDoc, called at compiler/GHC/Utils/Panic.hs:189:37 in ghc:GHC.Utils.Panic
        pprPanic, called at compiler/GHC/Core/Subst.hs:197:17 in ghc:GHC.Core.Subst
  CallStack (from HasCallStack):
    panic, called at compiler/GHC/Utils/Error.hs:454:29 in ghc:GHC.Utils.Error