GitHub's "scopes" (permissions, see http://developer.github.com/v3/oauth/#scopes) are very coarse-grained. So in this case we need to occasionally (via opt-in) add an SSH key to your account, but that permission isn't covered by a fine-leveled scope, so we need the coarse level "user" scope.
We discuss it in some more detail here: https://circleci.com/docs/github-permissions