Clean up JS backend libffi (and other?) dependencies:

@mpickering in !13090 (closed):

It seems there is something not quite right about libffi dependencies and the javascript backend, there are quite a few ad-hoc checks to not build things if the target is javascript. I just perpetuated that for now rather than investigate deeply.

We should probably do some cleanup work.