I've seen what you're talking about, but when I've seen it, it usually seems more like an acknowledgement of the amusing mild awkwardness inherent in being surrounded by people who are REALLY passionate about something and not being able to partake. I can see it having elitist intent in some cases, but in a lot of cases it's just poking fun at oneself for being so out of touch with whatever all the people around you are into.