This is the exact sort of question that Wittgenstein addresses and created a terrifying argument to press a wedge into. In the Investigations he formulates the Rule Following Argument which in many aspects mirrors the underdetermination argument for computational anti-realism; any physical state embodies any computational function under some arbitrary description (also known as pancomputationalism or computational trivialism). Wittgenstein intended to show-- and I think succeeded-- that meanings are underdetermined logically, and thus non-rational (not irrational) forces determine how one means something. Thus, causal structures of social bodies that holistically determine meanings.

This argument was initially somewhat ignored until revived by the great Saul Kripke in his book Wittgenstein on Rules and Private Language. I take this to be one of the great epistemological problems, joining the ranks of similar challenges like Descartes' demon. Like Descartes, Wittgenstein offered a solution, albeit a much less religious one than Descartes.