But how do you determine if someone is an authority? Our world is filled with epistemological bubbles. One person's expert is another's snake oil peddler.
Right, I don't think we really disagree about anything. "That's an appeal to authority" is not a useful objection, because appealing to authority is often a great idea. "That's not a good authority" is a good objection and often very relevant.
It's sort of like seeing a lot of arguments that rely on false or misleading evidence and deciding that "appeal to evidence" is a fallacy. The choice of authority or evidence is the issue, not the act of appealing.