The biggest issue here is that you have to hire real engineers to manage the bought product, and they feel dead ended or just waiting for a layoff opportunity.
The institutional knowledge around your data never develops because the engineer whose job it is to manage datadog and observability never gets the opportunity to learn or demonstrate technical depth, and so jumps ship to another team or company literally as soon as possible.
SRE recruiters make this worse, making point after point of gaslighting new hires.
I don’t buy your take at all. Isn’t this like arguing we should be running our own Postgres bare-metal instead of relying on RDS because engineers who work with RDS are “dead ended” and a “layoff opportunity” and will never develop institutional knowledge about our database schema? In my experience working on all kinds of infra, I have 1000 things I need to do, and buying a product that lets me avoid worrying implementing 300 of those things myself is a no-brainer.
If you're spending time jumping through hoops with RDS, then it's very possible bare-metal will be easier.
I don't believe the same is true today but in the past we did exactly this because we were spending a bunch of time banging our collective heads against some RDS idiosyncrasies and it turned out running our own Postgres was a breeze.
Plenty of things can and should just simply be bought. It's not always so cut and dry, though.
All too often the integration costs are overlooked.
Do we spend two engineers to build a system or buy a system (and spend two engineers integrating and maintaining it)? I've seen this happen more times than I can count.
The institutional knowledge around your data never develops because the engineer whose job it is to manage datadog and observability never gets the opportunity to learn or demonstrate technical depth, and so jumps ship to another team or company literally as soon as possible.
SRE recruiters make this worse, making point after point of gaslighting new hires.