Revert "Make the specialiser handle polymorphic specialisation"

This commit causes core lint errors see (#22358 (closed) and #22357 (closed))

Tracked by #21470 (closed)