I think you mean that it is hard to factor the product of two primes, factoring primes is pretty easy regardless their size.

Depends on what you mean by this... There are no practical systems which will do so. But the reason why quantum computing became interesting was because of Shor's work showing that they can factor large primes quickly (in theory).

... that we know of. The intelligence community has secretly outpaced the rest of the world in computing, cryptography and cryptanalysis before.

There are very hard fundamental physics problems with quantum computing. IANAexpert, but as I understand it, the difficulty lies in closing off the system (cold atoms, what have you) you're using to do the computation from the outside world, while you're doing the computation. Any contact (up to a very, very low limit), and you smear ("decohere") the computation so that the output will be in essence noise---not what we want at all. One can approach this as (in some sense) an engineering/experiment problem and try to limit contact with the outside world, or as a theory problem and try to come up with a system that inherently resists problematic contact with the outside world (topological quantum computation, https://en.wikipedia.org/wiki/Topological_quantum_computer), comes to my mind).

So we're dealing with a physics problem: either the gloriously messy, difficult work of experiment, trying to track down and eliminate sources of decoherence, or the theoretical work of finding an appropriate system (and then figuring out how to implement it).

Both of these are far from the core competencies of the NSA, as I understand them (not that I have any real information---just an impression based on way to much time spent reading Hacker News :-) ). As you say, NSA is (perceived as) very, very good and far ahead of the community at computing, cryptography, and cryptanalysis. The problem is, this is not a computing problem, it's a physics problem.

One possibilty, though, is that the NSA has figured out some way to use something like the D-Wave quantum computer (which I am ridiculously far from understanding, but you might recall Scott Aaronson's blog post on the subject not too long ago) to factor large primes---that is to say, to transform prime factorization into the D-Wave problem. At a gut level, I find this much more plausible than that the NSA has independently built a quantum computer: that sort of problem-transformation is much more the kind of math that I feel the NSA would be good at, or be able to become good at very quickly.

Much more cheaper and practical approach is for them than to try to outpace and outspend everyone else combined is to stockpile on zero day exploits which can be used when needed.