If you wear prisms that invert the world, your brain will eventually see the world inverted when you remove the prisms, and this effect will auto-correct after some duration.
Things might work differently for small children who wear VR goggles for extended durations. Their brains might get wired the wrong way, and there might be no "auto-correct" in this case.
I don't think it would be catastrophic, unless these children spend 90% of their time playing VR games. I would expect it to be similar to exposing a child to a second language. If you replace their exposure to the previous language completely with the new one, their proficiency at the first one will suffer but if you provide enough time for both, the brain will improve in both.
Adjusting to the inability to focus is different from learning to focus. Like the jellyfish who can't orient themselves properly if they were born in zero gravity [1].
