Coercion optimisation: don't put a sym'd coercion in the LiftingContext when optimising InstCo
Ticket: #27374
The ticket pointed out an issue with GHC.Core.Coercion.Opt.optCoercion's handling of InstCo: it contravened (LC2) in Note [The LiftingContext in optCoercion] because it applied the ambient sym to a coercion that was then added to the lifting context substitution.
I didn't manage to reproduce this with a source Haskell program triggering the bug, although the pre-existing T15725 test case comes very close: some coercions do go through the problematic code path for that test, but because certain sub-coercions are Refl/GRefl, we get enough cancellation to narrowly avoid the bug.
Tested with a GHC API test in T27374. Both the forall-over-type and forall-over-coercion examples fail prior to this patch, the former with an assertion failure in mkTransCo and the latter with a Core Lint error:
-- forall-over-co failure
warning:
From-kind of Cast differs from kind of enclosed type
From-kind: k1
Kind of enclosed type: k2
Actual enclosed type: x
Coercion used in cast: coR
In the coercion ‘<x>_N’
-- forall-over-type failure
panic! (the 'impossible' happened)
GHC version 10.1.20260617:
ASSERT failed!
in_co1 ((,) (Maybe (Sym cv_0_p))_N (Maybe (Sym cv_0_p))_N)_N
in_co2 ((,) (Maybe cv_0_p)_N (Sym (Maybe cv_0_p)_N))_N
in_co1 kind (Maybe q_0_m, Maybe q_0_m) ~ (Maybe p_0_l, Maybe p_0_l)
in_co2 kind (Maybe p_0_l, Maybe q_0_m) ~ (Maybe q_0_m, Maybe p_0_l)
Call stack (from HasCallStack):
callStackDoc, called at compiler\GHC\Core\Coercion\Opt.hs:788:22 in ghc-10.1-inplace:GHC.Core.Coercion.Opt
opt_trans_rule, called at compiler\GHC\Core\Coercion\Opt.hs:765:16 in ghc-10.1-inplace:GHC.Core.Coercion.Opt
opt_trans2, called at compiler\GHC\Core\Coercion\Opt.hs:756:20 in ghc-10.1-inplace:GHC.Core.Coercion.Opt
opt_trans1, called at compiler\GHC\Core\Coercion\Opt.hs:749:20 in ghc-10.1-inplace:GHC.Core.Coercion.Opt
opt_trans', called at compiler\GHC\Core\Coercion\Opt.hs:731:5 in ghc-10.1-inplace:GHC.Core.Coercion.Opt
opt_trans, called at compiler\GHC\Core\Coercion\Opt.hs:438:21 in ghc-10.1-inplace:GHC.Core.Coercion.Opt
Thanks to @sjakobi for the bug report and for a reproducer.