Simon Peyton Jones authored Feb 08, 2021 and Marge Bot committed Feb 13, 2021

The roughMatchTcs function enables a quick definitely-no-match test
in lookupInstEnv.  Unfortunately, it didn't account for type families.
This didn't matter when type families were flattened away, but now
they aren't flattened it matters a lot.

The fix is very easy. See INVARIANT in GHC.Core.InstEnv
Note [ClsInst laziness and the rough-match fields]

Fixes #19336

The change makes compiler perf worse on two very-type-family-heavy
benchmarks, T9872{a,d}:
  T9872a(normal) ghc/alloc  2172536442.7  2216337648.0  +2.0%
  T9872d(normal) ghc/alloc   614584024.0   621081384.0  +1.1%
(Everything else is 0.0% or at most 0.1%.)

I think we just have to put up with this. Some cases were being
wrongly filtered out by roughMatchTcs that might actually match, which
could lead to false apartness checks.  And it only affects these very
type-family-heavy cases.

Metric Increase:
    T9872a
    T9872d