Interesting point, but the thing about eyes is that they can look in unpredicted directions, so you need much more data available than just what the eye can process.

This wouldn't be true if we didn't have lag (then we could just stream the right images for where the eye was looking). But our eyes are extremely sensitive to lag - even done 100% locally, it's difficult to be fast enough (or used to be).

To compensate, we can download a whole environment, and track the eyes locally.