To be fair, I think the counterargument is that ostensible experts can overstate their ability/skill/knowledge to detrimental effect just as easily, and by virtue of the label ignore the reality of the argument/scenario at hand. That is, experts can overlook mistakes they're making, or conflicts of interest, etc because of their status, and because they overestimate their own ability. There's been studies of this in group decision making in crisis situations, where hierarchies can cause failures because the "leader" becomes overconfident and fails to heed warnings by others in the group.

This all gets really murky quickly in practice because of what "low" and "high" competence means, and what constitutes the actual scope of expertise with reference to a particular scenario.