To not influence anything right now, all of these are on by default, in
situations where clang is enabled.
Selectively turning a few targets off manually should work. Turning on
only one target should work too, even if that target does not correspond
to the build architecture. (In that case, LLVM_NATIVE_ARCH will not be
defined, and you can only use the resulting clang executable for
cross-compiling.)
I performed a few measurements on one of the FreeBSD.org reference
machines, building clang from scratch, with all targets enabled, and
with only the x86 target enabled. The latter was ~12% faster in real
time (on a 32-core box), and ~14% faster in user time. For a full
buildworld the difference will probably be less pronounced, though.