For this we have a secret management system, called confidant (https://lyft.github.io/confidant/), that we use to distribute any necessary secrets. So, yes, you may need to have keys on every node (depending on your monitoring system), but assuming you securely distribute them, it's not a big deal.
This is, of course, a general problem that's not necessarily related to envoy.
This increases your attack surface area. Any breach to one of those machines and the attacker can start doing mitm attacks. It also limits auto scalability assuming newly provisioned machines require manual approval of priv key distribution (that stays in memory) via hsm, and the same goes if the process dies. One way to limit the key distribution is to embed the routing information you require in the SNI at a second lb layer that's shielded from public traffic. This way your public machines don't hold any keys and if they get compromised, limiting the damage.
I agree it's a general problem. But sometimes certain architectures would require more vulnerable approaches vs others.
