Backport "Don't do binder-swap for GlobalIds"

This isn't a proper solution but we currently see head.hackage failures due to this (#17448 (closed)).