Some terminals support mouse input without any special modifier keys. I’ve used vim in iterm doing exactly that.