I feel about 30% confident that a "silly" piece of plastic on one side would have made it possible for Glass to deliver on its original vision of a mainstream consumer device, since people would feel way less awkward using it if they didn't look like they were part of the Borg collective. (And before you say "but cost", most likely a mainstream consumer version of Glass would be priced way below the initial explorer versions.) In that sense, it isn't "silly", and my guess is that arguments like yours are the reason it was never attempted.

Design requires fitting form to function, and that often means you add non-functional, ie "useless", parts to ensure the interface between human and device is ideal. Aesthetic appeal is a huge part of that, one could argue any component put on a device strictly for aesthetic appeal is "useless." Yet, in many cases, this "useless" component is strictly necessary for a successful design. (Ie, one that people actually use.)

For something you wear on your face, Glass screams out that it was under-designed. Humans are hardwired to prefer symmetric human faces, studies have shown this. So why would you design a device you wear on your face to be asymmetric? Seems obvious.

This + strapping a goddamn camera to your face that could be recording all the time.

People walked into bars and clubs with these. Then bouncers kicked them off.

If people use your product and they are labelled assholes, clearly you have not done market research.

Apple would have made it aesthetic as hell before a rumour even got out.

No one (that I know of) is strapping BMW motorcycles to their faces, though.