And why is there no proper 'invalid password' response? It always seems to me like if you have the correct password the connection succeeds and if you have the wrong password the connection just mysteriously fails and the computer has to guess that maybe it was because the password was wrong? In fact I've had a Macbook tell me the password was wrong when it was definitely correct. I imagine it's because they have to guess the failure reason.
If you're using a pre-shared key, the password is verified during the 4-way handshake. The thing is, if your password is wrong, then the Message Authentication Code (MAC) of the messages your are sending is wrong. The AP will simply drop frames with a wrong MAC, and will not respond to them. The problem is that as a client you do not know whether the AP is not responding because (1) the MAC was wrong, and hence the password was wrong; or (2) the message did not arrive at the AP (or you did not receive the response of the AP).
tl;dr: can't tell the difference between dropped messages due to wrong authentication check (i.e. wrong password), or dropped messages due to bad connection.
I think this is half right. There is still an L1 ACK, so the STA doesn't have to retry sending the packet, it knows it was received.
I believe what happens is AP sends Nonce to the STA, STA uses the PSK to send Message 2 back to the AP. It will receive a '802.11 Ack' but then no Message 3 of the 4-way handshake will ever come from the AP.
Good drivers see this and flag an invalid password warning back to the user within milliseconds. But bad drivers... sure, they will just keep assuming magic dust got in the way and if they just keep retrying the handshake enough maybe they will see a Message 3.
I'm not sure why from a security hardening perspective it's better not to specify the AP should send '802.11 Disassoc' immediately after receiving an invalid Message 2 with a proper error code so that the driver can message the UI that the password is wrong instantly.
Not really. The STA may know that its message was received, but can never be sure whether the AP replied. The reply from the AP could have been missed due to noise, or maybe it didn't reply at all. You cannot be sure. There are only heuristics.
Good drivers indeed tell you whether the message arrived or not. But it's up to the client to decide what to do with that information. And again, it's just a heuristic. I've read and messed with the code of four different Wi-Fi clients, and none of them attempt to detect a bad password this way. Most simply report an error after trying to retransmit message 2 multiple times (e.g. if wpa_supplicant got message 1 from the AP, but didn't get a reply to message 2, it warns that maybe the password was wrong).