Usually the token kept in memory is one distributed by the application and is not that which the services send back. This allows greater restriction on actions and make it far easier to revoke effectively
The token in this case is for the local user, that you know, logged into the service in question... How is it insecure for my browser, with me in front of it, to be logged into facebook? I mean, yes, if you're using anything other than a session cookie or sessionStorage, there's risk from other users on the system... but with an SPA, without any hard storage, it's no less secure than using that site/app.
