No, phosphate is vastly cheaper than battery metals. I mean, yes, we're using way too much of it (for agriculture, in a manner that ends up unrecoverably flushed into the oceans). And we're going to hit a wall, and its price is going to skyrocket. But to matter to a battery producer it would have to be so expensive that we'd have all starved anyway. The price levels between batteries and fertilizers are just too different.
I repeat: if phosphate was so expensive as to make a significant portion of the material cost of a LiFePO4 battery, then we would have long since starved for lack of crops. Relative to food, batteries are an extraordinarily expensive luxury good.