Certainty has nothing to do with how many people said yes or no, it’s about how many people are in the experiment.
If you sample size of 2, and both like the new UI, then you can’t say you have 100% certainty.
I.e. 3/4 people isn’t the same certainty as 75/100
I think the point he’s trying to make is it’s easier to get the statement “95% of people prefer X” from a study then to get “99% of people”. The first requires >20 people, the later >100.
But he’s tying together the concepts of sample size, confidence, and the outcome of the experiment in a way hat doesn’t make sense
That's not at all what he's saying. Maybe understand the article first before calling it nonsense. Kind of funny how all you people are making his point for him!
Certainty has nothing to do with how many people said yes or no, it’s about how many people are in the experiment.
If you sample size of 2, and both like the new UI, then you can’t say you have 100% certainty.
I.e. 3/4 people isn’t the same certainty as 75/100
I think the point he’s trying to make is it’s easier to get the statement “95% of people prefer X” from a study then to get “99% of people”. The first requires >20 people, the later >100.
But he’s tying together the concepts of sample size, confidence, and the outcome of the experiment in a way hat doesn’t make sense