I never said this was specific to drummers. Again, yes they count using numbers, but that's just convention. Unless you're counting crazy composite time signatures like 17/8 or some shit, the numbers are meaningless in the context of a single measure. For 99.9% of popular music using 3-6 beats, it could just as easily be letters or colors or something. Numbers are sufficient but not necessary.
There’s a reason it’s done the way it is. For one, the names of the notes are letters, so using numbers makes it easy to differentiate notes from beats. Time signatures are also numerically noted. So trying to say using A/B time makes absolutely no sense. Trying to promote using numbers is arbitrary is nonsensical.