It is a problem if it has a security hole. Or if you have reason not to trust the Bazel developers. (Or trust people who might have illegitimate access to said developers' machines - for example the NSA.)

See https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7... for the canonical paper on why it can be important to be able to recreate your build chain from scratch in a verified way.