Tag inference - take 3

Hopefully this works as is, but if seen some inconsistencies when doing the cabal test.

So let's see if this was a mistake on my side or a problem with the code.

Addresses #16970 (closed)