Specialiser creates bogus specialisations

Original report

I'd like to ask for help in what is most probably a case of a bad port (by yours truly) of a typing plugin (GHC.TypeLits.Normalise) that fails in GHC 9.6.1, but worked fine in GHC 9.4.4 and 9.2.7 (and works fine in 9.6.1 with -O0). The typing breakage is described below. The bad port to GHC 9.6 is in https://github.com/clash-lang/ghc-tcplugins-extra/pull/24 and https://github.com/clash-lang/ghc-typelits-natnormalise/pull/72. I tried to follow the migration guide, but I didn't understand much and just randomly wibbled until it type-checked.

To reproduce, clone https://github.com/Mikolaj/horde-ad/tree/repro-rank-mismatch and run (the linters are included only to show they don't produce any info at all for the part of the codebase that is being compiled)

cabal test simplifiedOnlyTest --test-options='-p 3concatBuild' --allow-newer --enable-optimization  --ghc-options='-dcore-lint -dstg-lint -dcmm-lint -dtag-inference-checks'

Normally it should pass fine, but instead you would be getting the following error

Exception: fromVector: rank mismatch (1,32)

where the value 32 comes from type variable n, from a different test than the one being run. If you change number 33 in this other test, you get a different error message. If you remove the other test, the error disappears. With -O0 and with older GHCs the problem does not appear. In larger versions of this reproducer, the error differed when run with profiling mode and without.

If it helps, I can try to minimize further, but the corruption is very fragile and tiny changes eliminate the error or at least change it to a form that is not as obviously wrong (does not mention a number from the code it should not depend on).

cc: @christiaanb

Edited Nov 13, 2023 by Simon Peyton Jones

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information