Most end users have no idea what HTTPS is. They've just been (incorrectly) taught that the padlock means it's secure. Disable the padlock for self-signed HTTPS, and disable the CA-signed HTTPS-only features, and it becomes strictly better than HTTP.
Especially because there is no way to MITM a connection with perfect-forward-secrecy only if it ends up serving a self-signed certificate, because the connection first negotiates an ephemeral key with which everything, including the certificate, will be encrypted.
This means that with eSNI and at least one CA-signed cert on the IP, any attacker runs the risk of having to spoof the CA-signed certificate.
A sophisticated attacker might know that you were going to connect to a self-signed site, though. Interestingly though, private DNS (DoH, etc.) might help further shroud this fact from the attacker.
All in all, I'd say that the browser should still throw up a full-page warning because of the implications of TOFU, but it can be one where the "continue to site" option is clearly shown even to a naïve user, and not hidden behind a spoiler.
