Haddock complains about out-of-scope pattern synonym builders

While working on !4819 (merged) I noticed that Haddock throws many warnings of the form:

GHC error in desugarer lookup in Ghci1:
  Can't find interface-file declaration for variable $bSDoc
    Probable cause: bug in .hi-boot file, or inconsistent .hi file
    Use -ddump-if-trace to get an idea of which file caused the error

These symbols are pattern synonym builders. It's a bit unclear whether this is a Haddock bug or a GHC bug but nevertheless it deserves investigation.