Well, I am not as well-versed in optimization, but there is this:

https://en.wikipedia.org/wiki/Cache_timing_attack

If you can exploit the effect that cache has on latency, it is reasonable to assume that you can manipulate it in other ways.