Good explanation. Which is why the success of transformers, LLMs etc. is still not the final word in Rich Sutton's "The Bitter Lesson" -- no learning method is free of inductive biases.
Inductive biases can work even if they're wrong, because they allow for simple and quick action, simpler reasoning. They don't need to be correct for that to pay off, they just need a positive expected value.
