The problem for me is the time investment required to be any good at gaming. When people say they want to play Super Smash Bros. or Halo or something, I have no idea what I'm supposed to do, no one wants to show me what to do, and so I lose interest very quickly. It's not like watching a movie, where you can spend two hours on it and come out with a decent understanding of the movie, even if you aren't a movie buff.
I think I know exactly what you mean. Super Smash Bros. is actually the exact game that made me realize I was no longer going to pretend to be a gamer, for exactly the reasons you explain.
