Fix multi-isa build feature

* Explicitly build NEON-only files with the base architecture.
* Remove redundant compiler flags.

Signed-off-by: Viet-Hoa Do <viet-hoa.do@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Jakub Sujak <jakub.sujak@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>