Performance degrades from -O1 to -O2

GHC seems to get stuck at SpecConstr stage when compiling.

I've managed to get the example down to what appears to be minimal but making a number of minor changes makes the compilation go through quickly:

• changing toIdx to const 0
• removing one of the V.forM_ layers.
• using "-fno-spec-constr" (unsuprisingly).

I can't reproduce it on 8.0.2.

In larger examples I was able to get memory and cpu to sky rocket quite quickly but I appear to have lost some of that behaviour when paring down the example.