And why do you think they failed to provide an accurate probability estimate?
If I tell you that the odds of your dice roll being a 6 is only 16%, and you roll a 6, does that mean I failed to provide an accurate probability estimate?
The only way to judge the accuracy of a probabilistic model, is by quantifying the weighted error across numerous predictions. Not by looking at two yes/no outcomes. This is the exact point they are trying to get across, which you wrote off as annoying.
You do raise an interesting point about the betting markets - I too am very curious about whether they have a better track record than people like Nate Silver. Looking forward to someone analyzing their relative accuracy over a large sample of independent elections.
I saw predictit.org swing from pro Biden the morning of the election, to Trump in the afternoon, to Biden the next morning, so it seemed they were as confused as anyone.
They also didn't seem very aware that mail votes (leaning Biden) would likely be counted after in person votes (leaning Trump), which 538 had been predicting would cause a temporary pro-Trump lean for ages.
If you roll that same die 10 times and you get a 6 each time, then we can say that your probability estimate was wrong. But that's what happened with the polling of these states. The fact that the estimates consistently overestimated the Biden vote points towards a bias in the model used to estimate likely voters. If this were just a matter of the margin of error at work, each pull at the lever (each state) should be randomly distributed around the estimate. But that wasn't the case.
> If you roll that same die 10 times and you get a 6 each time, then we can say that your probability estimate was wrong. But that's what happened with the polling of these states.
The 538 model is explicitly based on the assumption that the state-polling errors are correlated to one another to some extent. Ie, if Trump performs better-than-expected in OH, he will likely perform better-than-expected in FL as well. This is why they rated Trump's chances as being 0.1, not 0.1^8. Given how polling works, this is exactly the right assumption to make.
Hence my earlier comment that if you want to evaluate the accuracy of 538 or any other model, you need to evaluate it across numerous different elections/events, over an extended period of time. Not a single day of elections in a single country.
This is kind of missing the point. The issue is the bias in the state-level polls. 538's model is to predict the winner of the election based on state polling data from other sources. The state level predictions are just averages of polls presumably weighted by quality. But this averaging cannot remove bias in the polls if that same bias is in many polls.
If I tell you that the odds of your dice roll being a 6 is only 16%, and you roll a 6, does that mean I failed to provide an accurate probability estimate?
The only way to judge the accuracy of a probabilistic model, is by quantifying the weighted error across numerous predictions. Not by looking at two yes/no outcomes. This is the exact point they are trying to get across, which you wrote off as annoying.
You do raise an interesting point about the betting markets - I too am very curious about whether they have a better track record than people like Nate Silver. Looking forward to someone analyzing their relative accuracy over a large sample of independent elections.