If I'm understanding this right, someone can download and cache the decryption key (while it's available) and then decrypt the cipher text at any time they want in the future, right?

I don't think it's beyond the NSA's ability to just pre-emptively cache every decryption key, just in case they need it later.