Yeah, that's a good point. Maybe Glass will be like prescription sunglasses, and people will have a separate pair? Maybe there will be a way to overtly disable the computer part or remove it entirely?
I think it's really unlikely that, aside from early adopters, people will have Glass "in place" when they're not using it, though.
