And, if you're returning the correct error message, it doesn't matter.. the whole point of a timing attack is to determine the difference.

IMHO usability is more important. There are other ways to improve security. Rate limiting with < N failed attempts via an IP in under < X minutes.