GHC rejects the following program:

{-# LANGUAGE TypeFamilies, FlexibleInstances #-}
module F where
type family F a
type instance F Int = Int -> Int
instance Show (F Int) where
    show _ = "fun"

But this doesn't seem fundamentally problematic to me: at the instance declaration, we know that F Int will reduce, and after that reduction there isn't any problem with the instance.

Here's a less restrictive instance head test: first, we reduce type families as much as possible. Then, if there are still leftover non-reducing type families, reject the instance.

After having typed this up, I have realized that I don't actually need this feature. But I'm filing it here for posterity.