Depend directly on the source of binary, instead of the library itself.
This is unfortunate as it complicates the cabal file, but we have to in
order to remove circular dependencies when installing dependencies of
the test-suites and benchmarks.