Increasing entropy mitigates brute force attacks but not necessarily rainbow table attacks, hence the distinction. If every user had a unique password, rainbow tables would be rendered useless. This is why it's important to reduce the likelihood that a randomly generated password is comprised of a common pattern.
> Increasing entropy mitigates brute force attacks but not necessarily rainbow table attacks
No. All time-space trade-offs need to expend the entire attack effort once (and usually it's considerably more). Their advantages are that you can do this in advance (timeliness) and that you can re-use the product (which is what salt mitigates).
You can very simply increase entropy until the attack effort cannot be deployed at all, regardless of when. For example the 'pass' tool lots of people have mentioned in this thread defaults to 24 character random passwords, far more than 128 bits of entropy. As a result it simply isn't possible to deploy the attack effort even once, you can neither do a Brute Force attack nor build the Rainbow Table to find these passwords.
The way the Rainbow Table got famous is its application to the LM Hash, an old (but already terrible when it shipped) Microsoft password hash. LM Hash uses two 56-bit values to represent a password, naively this looks like 112-bits of entropy but you can attack each independently so it isn't.
Rainbow Tables took attacking this from something that could in theory work on any password but you'd get bored waiting for your "crack" program to finish if it wasn't trivial to a few minutes on a fast laptop for every possible password. Because it's a time-space tradeoff, so somebody put all the effort in once and then you can re-use it.
Yes. For example, if a rainbow table contains a match for "a" repeating 500 times, then that password's entropy is a non-factor. Therefore entropy in of itself does not necessarily mitigate rainbow table attacks. Entropy does necessarily mitigate naive brute force attacks. We can use this same logic for more reasonably sized passwords that are comprised of known patterns but would otherwise be difficult to brute force.
Obviously you could use extremely long passwords and be fairly certain that you won't end up with something that would be found in a rainbow table or dictionary. You could also just use passwords that are long enough that they can't be brute forced and remove common patterns from the set of possible passwords. Both of these solutions are valid, and both have theoretical weaknesses that are irrelevant in practice.
You've spent a significant amount of time trying to convince me that my random password generator is flawed in some way. Why don't you just demonstrate how instead of continuously trying to argue tangentially related theoretical points? I would hope that such an effort would be motivated by a practical concern and not merely the desire to bikeshed out of boredom.
> Yes. For example, if a rainbow table contains a match for "a" repeating 500 times, then that password's entropy is a non-factor.
This muddle reflects your overall intuitive mistake.
When you roll two fair dice the "snake eyes" (two ones) outcome has only one chance in thirty six. But if you've just rolled snake eyes, the probabily it was snake eyes isn't one in thirty six it's 1.0 exactly. We do not say, as a result, that you were "certain" to roll snake eyes, you weren't, but it has happened anyway and so now it won't change. Entropy is not a measure of what did happen it's a measure of the unknown. Password generation entropy doesn't result in passwords which change constantly, it just means you can't know what the password might be within that parameter.
Constructing a rainbow table where the choice function just cycles through some number of arbitrary guesses - and then saying if those guesses are right the rainbow worked, therefore rainbow tables were a success skips the step where it's overwhelmingly likely that none match, it fails and was a total waste of resources. Nobody actually does this because it's pointless.
When we tell humans to pick randomly they behave like you, and your algorithm - trying so hard to pick a "non-obvious" answer and thus inadvertently they're predictable. Your algorithm is, as a result, needlessly predictable compared to sane "random password" code and yet also significantly more complicated.
Look at the number line. See the first number you think isn't interesting? Well the fact it's the first non-interesting number is pretty interesting, isn't it? So now maybe that's not the first one that isn't interesting after all. Next one, same argument. Likewise your removal of "common patterns" just means the set of possible passwords is made smaller, and leaves remaining passwords with the "common pattern" of not having those "common patterns". It's futile, stop it.
If you want to put your intuition about what "random" ought to be like, put it into a playlist shuffle codebase or something, where humans will appreciate how properly "random" it feels to always get a nice mix of things. In security software this is a mistake, no matter how sure you are that it's common sense to do it your way.
You missed my point again, and your tangent on randomness is unnecessary; I have no misconceptions of how randomness works, and it is precisely that understanding that has lead me to these decisions - a pure random password generator produces every word in the English language. That's not a good thing!
The point remains that if your password generator produces passwords such as "aaaaaaa" then it is a bad algorithm, end of discussion. It doesn't matter if the passwords it produces are completely random. That's not the goal and is completely irrelevant. The goal is to produce unique, unpredictable passwords that utilize randomness. We're not producing fixed-length keys. In the average case for web-based logins, uniqueness is far more important than entropy.
So, I'll say it once more. If you can demonstrate how passwords generated by the algorithm I wrote are predictable or otherwise insecure in a real-world setting as you claim, then do it. If you cannot, then any further responses are in vain.
> The point remains that if your password generator produces passwords such as "aaaaaaa" then it is a bad algorithm, end of discussion.
The correct algorithm does this, and you've managed to convince yourself that this is bad and it's time to invoke lots more complicated code. This is the end of the discussion in the sense that you've departed from reality so severely that you may be unable to recover.
For those who can't read shell that's saying to run the random data from the OS kernel through a pipeline which removes everything except $characters and then consume $length bytes of the result as the new password.
This absolutely can give you "aaaaaaa" as a password if you've unaccountably chosen to use such short passwords - but no more easily than it might choose "fuckwit" or "X3_$mwK" or any other sequence of permitted symbols.
But enough about how to do this correctly, you're very focused on your bizarre way to pick "unique, unpredictable passwords that utilize randomness", so let's look at that again:
The tight inner loop picks characters randomly using code from Sodium. Unfortunately it discards characters from the candidate list once it has picked them, and then it has a further rule which may ignore this (already discarded) character and go round again.
For the short case (size = 4) this means it can produce only about 5.2 million different passwords, whereas a better (simpler) solution gives 78 million different passwords, you've made it more than a decimal order of magnitude worse.
> For example, if a rainbow table contains a match for "a" repeating 500 times, then that password's entropy is a non-factor.
But it doesn't. You seem to be taking it as gospel that this sort of password is going to show up in the early in the list of passwords people are trying. But I don't buy it. There is no need to reject a long password that contains some repeated characters.
Why doesn't it? If that password became public knowledge, then it certainly does exist in lists and tables. Its high entropy is only protective as long as it remains secret. This is why it's important to avoid common patterns, even if those patterns are a result of a random number generator.
>Every password that becomes public knowledge ends up in credential stuffing lists, whether it matches your password policy or not.
That's right. And we don't want to produce passwords that are likely to be on those lists. A simple policy greatly reduces the chances of that happening. After a certain number of zeros, entropy is no longer a concern.
>"Common patterns" and "passwords that contain repeated characters" are not even remotely the same thing.
