I like One Big (virtual) Server until you come to software updates. At a current project we have one server running the website in production. It runs an old version of Centos, the web server, MySQL and Elasticsearch all on the one machine.
No network RTTs when doing too many MySQL queries on each page - great! But when you want to upgrade one part of that stack... we end up cloning the server, upgrading it, testing everything, and then repeating the upgrade in-place on the production server.
I don't like that. I'd far rather have separate web, DB and Elasticsearch servers where each can be upgraded without fear of impacting the other services.
You could just run system containers (eg. lxd) for each component, but still on one server. That gets you multiple "servers" for the purposes of upgrades, but without the rest of the paradigm shift that Docker requires.
Which is great until there's a security vuln in an end-of-life piece of core software (the distro, the kernel, lxc, etc) and you need to upgrade the whole thing, and then it's a 4+ week slog of building a new server, testing the new software, fixing bugs, moving the apps, finding out you missed some stuff and moving that stuff, shutting down the old one. Better to occasionally upgrade/reinstall the whole thing with a script and get used to not making one-off changes on servers.
If I were to buy one big server, it would be as a hypervisor. Run Xen or something and that way I can spin up and down VMs as I choose, LVM+XFS for snapshots, logical disk management, RAID, etc. But at that point you're just becoming a personal cloud provider; might as well buy smaller VMs from the cloud with a savings plan, never have to deal with hardware, make complex changes with a single API call. Resizing an instance is one (maybe two?) API call. Or snapshot, create new instance, delete old instance: 3 API calls. Frickin' magic.
"the EC2 Instance Savings Plans offer up to 72% savings compared to On-Demand pricing on your Amazon EC2 Instances" - https://aws.amazon.com/savingsplans/
Huh? Using lxd would be identical to what you suggest (VMs on Xen) from a security upgrade and management perspective. Architecturally and operationally they're basically the equivalent, except that VMs need memory slicing up but lxd containers don't. There are security isolation differences but you're not talking about that here?
I would want the memory slicing + isolation, plus a hypervisor like Xen doesn't need an entire host OS so there's less complexity, vulns, overhead, etc, and I'm not aware if LXD does the kind of isolation that ex. allows for IKE IPSec tunnels? Non-hypervisors don't allow for it iirc. Would rather use Docker for containers because the whole container ecosystem is built around it.
Fine, but then that's your reason. "until there's a security vuln in an end-of-life piece of core software...and then it's a 4+ week slog of building a new server" isn't a difference in the context of comparing Xen VMs and lxd containers. As an aside, lxd does support cgroup memory slicing. It has the advantage that it's not mandatory like it is in VMs, but you can do it if you want it.
> Would rather use Docker for containers because the whole container ecosystem is built around it.
This makes no sense. You're hearing the word "container" and inferring an equivalence that does not exist. The "whole container ecosystem" is something that exists for Docker-style containers, and is entirely irrelevant for lxd containers.
lxd containers are equivalent to full systems, and exist in the "Use one big server" ecosystem. If you're familiar with running a full system into a VM, then you're familiar with the inside of a lxd container. They're the same. In userspace, there's no significant difference.
I use LXC a lot for our relatively small production setup. And yes, I'm treating the servers like pets, not cattle.
What's nice is that I can snapshot a container and move it to another physical machine. Handy for (manual) load balancing and upgrades to the physical infrastructure. It is also easy to run a snapshot of the entire server and then run an upgrade, then if the upgrade fails, you roll back to the old snapshot.
Doesn't the container help with versioning the software inside it, but you're still tied to the host computer's operating system, and so when you upgrade that you have to test every single container to see if anything broke?
Whereas if running a VM you have a lot more OS upgrades to do, but you can do them individually and they have no other impact?
This is the bit I've never understood with containers...
No network RTTs when doing too many MySQL queries on each page - great! But when you want to upgrade one part of that stack... we end up cloning the server, upgrading it, testing everything, and then repeating the upgrade in-place on the production server.
I don't like that. I'd far rather have separate web, DB and Elasticsearch servers where each can be upgraded without fear of impacting the other services.