That 'hit' is equivalent to reducing the carrying capacity of the planet by ~5 billion humans or using 50% of the ice free land mass for agriculture to support them.
Again, only if you believe there is no alternative to using the Haber process to make nitrogen inputs for agriculture to provide humans with food. If, like me, you believe that the Haber process was invented at the turn of the previous century, and it's time for us to grow up and master biochemistry, then the "hit" is entirely avoidable.
Your goal is to replace mining. If we had free energy (let's say "zero point energy"), then we could stop using natural gas and get the H2 needed for the Haber process directly from electricity. There would be no need to replace the Haber process.
But we don't have enough excess power for this to be economical.
The thing about 100 years of history is there's also 100 years of optimization, and 100 years to find something better. And there are many proposals of methods which are better in the lab. Just not at industrial scales that can feed a planet.
Nor is it for lack of trying. There are economic and personal incentives for anyone who can radically improve our understanding - to "grow up" as you so crudely put it. It's a certain Nobel Prize, following in Haber's footsteps. Yes, we know it can be done since bacteria have figured out how to fix soils at STP.
(To correct a point you made earlier, nitrogen fixation is certainly NOT part of the biochemical repertoire of any tree or plant.)
But as I pointed out, ammonia synthesis is only one of several very large industrial processes we use for food production. In addition to phosphates and potash, a lot of crop land is zinc deficient or in other micronutrients. Neither plants nor bacteria have figured out how to replenish those chemicals on their own or through bacteria, and depend instead on glaciers, erosion and volcanoes.
While we mine rock to make fertilizers more quickly.
It is not so easy to get rid of mining and feed the world's billions. There is obviously much profit to be had in finding something renewable which is also cheaper than mining. Most of Australia's farmers, for one, want to replenish the nutrient-poor, ancient soils under their crops.
We mine phosphate rock for fertilizer, since P is one of those elements plants need.
Mined potash provides the K.
We can do without them, but we'll take a huge hit in food crop yield.