My two year old can read numbers up to 10. It takes a little while to go from the symbol to the amount but generally speaking she can do it. It's typically around 2 where you can start to learn letters and numbers and basic counting.
My twin 4 year olds could play the game no problem, but they cannot read numbers well so they just counted from the left. Turns out they knew 1-3 already, picked up 0 fast as it was the odd one out. I have not pushed numbers very much, they know more of the alphabet. I am not sure whether I should teach them numbers, I kind of think letting school do that might be better, but your comment has made me think a bit more.