Allow data family instances with type family applications in the data family's result kind

In #17021 (closed) we fixed the situation where an unlifted newtype instance of a data family with a type family application in its result kind led to a panic. That was fixed, but non--XUnliftedNewtypes code is still rejected with an error message:

{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE PolyKinds #-}

module Lib where

import GHC.Exts

type family Id x where
  Id x = x

data family Foo (r :: RuntimeRep) :: TYPE (Id r)

data instance Foo 'LiftedRep = Foo Int

is rejected with

test2.hs:11:44: error:
    • Type constructor ‘Id’ cannot be used here
        (perhaps you intended to use DataKinds)
    • In the first argument of ‘TYPE’, namely ‘(Id r)’
      In the kind ‘TYPE (Id r)’
      In the data family declaration for ‘Foo’
   |
11 | data family Foo (r :: RuntimeRep) :: TYPE (Id r)
   |

Note that this would work if we would make Foo a newtype instance instead, and activated -XUnliftedNewtypes. I'm not sure whether that's a bug or a feature.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information