Revert "Enable GNUABI build on more targets #525 (#562)"

Only Linux supports the AArch64 vector ABI, so it doesn't make sense
to expose these symbols anywhere else for AArch64. Without any
compelling reason to provide them on x86_64, it is better to just
revert this patch and deal with this more carefully after the release.

This reverts commit 0e47fcc3.