> Of course you have more maintenance on on-prem, but typical k8s update is maybe a few hours of work, when you know what you are doing.

You just mentioned one dimension of what I described, and "when you know what you are doing" is doing a lot of the heavy lifting in your argument.

> Also AWS is also, complex, also requires configuration and also generates alerts in the middle of the night.

I'm confused. So we are on agreement there?

I feel you might be confusing my point with an on-prem vs AWS discussion, and that's not it.

This is encouraging teams to run databases / search / cache / secrets and everything on top of k8s and assuming a magic k8s operator is doing the same job as a team of humans and automation managing all those services for you.

> assuming a magic k8s operator is doing the same job as a team of humans and automation managing all those services for you.

What do you think AWS is doing behind the scenes when you run Postgres RDS? It's their own equivalent of a "K8S operator" managing it. They make bold claims about how good/reliable/fault-tolerant it is, but the truth is that you can't actually test or predict its failure modes, and it can fail and fails badly (I've had it get into a weird state where it took 24h to recover, presumably once an AWS guy finally SSH'd in and fixed it manually - I could've done the same but without having to wait 24h).

Fair, but my point is that AWS has a full team of people that built and contributed to that magic box that is managing the database. When something goes wrong, they're the first ones to know (ideally) and they have a lot of know-how on what went wrong, what the automation is doing, how to remediate issues, etc.

When you use a k8s operator you're using an off the shelve component with very little idea of what is doing and how. When things go wrong, you don't have a team of experts to look into what failed and why.

The tradeoff here is obviously cost, but my point is those two levels of "automation" are not comparable.

Edit: well, when I write "you" I mean most people (me included)

> Fair, but my point is that AWS has a full team of people that built and contributed to that magic box that is managing the database.

You sure about that? I used to work at AWS, and although I wasn't on K8S in particular, I can tell you from experience that AWS is a revolving door of developers who mostly quit the instant their two-year sign-on bonus is paid out, because working there sucks ass. The ludicrous churn means there actually isn't very much buildup of institutional knowledge.

> Fair, but my point is that AWS has a full team of people that built and contributed to that magic box that is managing the database

You think so. The real answer is maybe maybe not. They could have all left and the actual maintainers now don't actually know the codebase. There's no way to know.

> When things go wrong, you don't have a team of experts to look into what failed and why.

I've been on both sides of consulting / managed services teams and each time the "expert" was worse than the junior. Sure, there's some luck and randomness but it's not as clear cut as you make it.

> and they have a lot of know-how on what went wrong, what the automation is doing, how to remediate issues, etc.

And to continue on the above I've also worked at SaaS/IaaS/PaaS where the person on call doesn't know much about the product (not always their fault) and so couldn't contribute much on incident.

There's just to much trust and good faith in this reply. I'm not advocating to manage everything yourself but yes, don't trust that the experts have everything either.

If you don't want complexity of operators, you'll be probably OK with DB cluster outside of k8s. They're quite easy to setup, automate and there are straightforward tools to monitor them (eg. from Percona).

If you want to fully replicate AWS it may be more expensive than just paying AWS. But for most use cases it's simply not necessary.

As with everything it's not black or white, but rather a spectrum. Sure, updating k8s is not that bad, but operating a distributed storage solution is no joke. Or really anything that requires persistence and clustering (like elastic).

You can also trade operational complexity for cash via support contracts and/or enterprise solutions (like just throwing money at Hitachi for storage rather than trying to keep Ceph alive).

If you don't need something crazy you can just grab what a lot of enterprises already had done for years, which is drop a few big storage servers and call it a day, connecting over iSCSI/NFS/whatever

If you are in Kubernetes land you probably want object storage and some kind of PVC provider. Not thaaat different from an old fashioned iSCSI/NFS setup to be honest, but in my experience different enough to cause friction in an enterprise setting. You really don't want a ticket-driven, manual, provisioning process of shares

a PVC provider is nice, sure, but depending on how much you need/want simplest cases can be "mount a subdirectory from common exported volume", and for many applications ticket-based provisioning will be enough.

That said on my todo-list is some tooling to make simple cases with linux NFS or SMI-capable servers work as PVC providers.

Sure, but it requires that your engineers are vertically capable. In my experience, about 1 in 5 developers has the required experience and does not flat out refuse to have vertical responsibility over their software stack.

And that number might be high, in larger more established companies there might be more engineers who want to stick to their comfort bubble. So many developers reject the idea of writing SQL themselves instead of having the ORM do it, let alone know how to configure replication and failover.

I'd maybe hire for the people who could and would, but the people advocating for just having the cloud take care of these things have a point. You might miss out on an excellent application engineer, if you reject them for not having any Linux skills.

Our devs are responsible for their docker image and the app. Then other team manages platform. You need some level of cooperation of course, but none of the devs cares too much about k8s internals or how the storage works.