The irony is that this phrase predates the Torricelli experiment, in which he used a glass vial and mercury to produce a vacuum, and the eventual discovery that the vast majority of nature is a vacuum (outside of the atmosphere of planets and stars).
And if it wasn‘t for gravity, everything inside the atmosphere of planets and stars would be rushing to fill that vacuum. Nature abhors a vacuum, but that‘s only tempered by the fact that there are more powerful forces than vacuums.
That's not quite right. Even if there were no gravity or other forces, there would be no incentive for matter to go in one direction vs another, regardless of the distribution of matter around it. Over time you'd get a uniform-ish distribution, but that's a question of statistical mechanics, not one of pressure differences.
On earth, matter only rushes to fill vacuums because the surrounding air or water pressure pushes it in.