That's fine when it is under your control. But, it isn't. And companies are using it for their benefit, and will avoid your defense, short of a court order.
If you think location is the only thing these companies are collecting, you're mistaken. The data is sold to unknown third parties and freely given to dangerous governments. There are any number of bad outcomes possible, though I admit many are not likely.
The point is that these folks are not on your team, will never be, and are potentially malicious to you and your family. Why support them?