Hey there, this is Dimitris from the Azure PlayFab Multiplayer Servers (MPS) team, where we develop both the MPS service and Thundernetes, happy to answer any questions. For realtime communication, you can find us on the Microsoft Game Dev on Discord https://aka.ms/msftgamedevdiscord
Is this effectively v3 of Thunderhead? Good to see the pivot towards using k8s, the 20+ minute propping of a 1-2GB container on vanilla Playfab/THv2 really made it a non starter for our company's internal playtesting.
Yup, he is still around :)
I'd call Thundernetes a "spin-off" more than a pivot, tailored to the folks that want full control over their game servers, compared to the more "SaaS" model of MPS (code name "Thunderhead").
Have you tried the MPS service recently? We've done lots of work to improve startup times.
I feel like the secret sauce here and Agones is basically public IPs per node, a pre-warm mechanism (and the software to route games to warmed servers), and an assumption that games are short lived enough that nodes can eventually get scaled down as games fall off.
Seems well and good. I'm not sure if you lose any real DDOS protection by going through a load balancer.
I do have to wonder if its worth running game servers in k8s or just use autoscaling instances without the pod concept. You're really fighting against what k8s does well, for what I assume is no benefit. Thoughts?
This is my field, so even though I’m literally titled “expert”[0]- take what I say with a pinch of salt.
When it comes to gameserver performance what tends to be the biggest issue is predictable performance. So, kubernetes is not inherently antithetical to running game servers as long as you don’t share the cpu core(s) with anything.
In fact there is a project that (while I was not a part of) was spearheaded by ubisoft in collaboration with Google for running game servers on Kubernetes (called Agones)[1].
Ubisoft also used “Thunderhead” (which this project seems to be a variant of) on Azure with rainbow 6 siege. So it’s definitely not considered a show-stopping problem.
When it comes to containers/VMs the biggest hit in terms of performance is in this order: disk/network/memory access/CPU ;; when talking about game servers the most important things go in the reverse of that order, which means that the biggest performance losses are not as important. Network latency might seem like a big deal but you’re limited to frame times in the best case and geographical differentiations will eat much more than the combined weight of frames and Kubernetes networking.
Agones itself bypasses a lot of the Kubernetes networking latency additions, though, not all.
Not necessarily in my experience! Most games don't hit the CPU particularly hard.
Mostly reliant on (hardware offloaded) network latency, memory, and storage. Consistency being the thing of most importance. A spike is felt - interpolation can only go so far
CPU virtualization extensions make the latency cost of VMs hardly noticeable, containers are technically better but it's a split hair.
On an unloaded host and a single VM you can achieve essentially identical performance as bare metal - the CPU extensions, pinning cores, and huge pages are key
Some performance metrics even improve! Disk operations tend to do better from an added layer of memory involvement
Edit: there are options for multi-tenant fairness whichever way you go -- Bare metal, VMs, or containers
Performance isn't much of a worry, more so the 'handles'
Using container technology doesn't necessarily mean adding virtual network devices/proxying/forwarding. In practical terms, you can run in "host" networking mode (no separate network namespace) on Docker or Kubernetes, and most people would still consider those "containers".
In this case they seem to be NAT'ing packets from a host port on the correct Kubernetes node (the one running the container) to a port in the container, which can be done fast enough with iptables (or similar mechanism used by Kubernetes).
If you're running a handful of game servers on each machine, each with a single NATed port, you're fine. At bigger scales it's a problem, but as you mention there are better solutions now.
Not really. Minecraft server notoriously ran in JVM for most of it's existence. Container virtualization is unlikely to add more than a few microseconds of latency.
Nothing about kubernetes implies virtualization. People only use VMs because they're easier to provision. Technically you could provision physical servers with kubernetes and run containers on them with zero overhead.
Or perhaps more realistic you could provision a tier of very low latency VMs.
Not really. The major concern is getting clients to one machine and running a stateful connection so you don't need to pass messages through some kind of slow, multi-hop broadcast system. The CPU overhead of containers or vms is pretty minimal compared to this main concern.
Should "easy" and "kubernetes" ever really be in the same sentance?
Not being snarky, just realistic - I feel like cloud providers sell "kubernetes" as the only way to properly develop apps, because you can be cloud-agnostic, without lock-in, you have full control over your infrastructure.
Without realising, that for most projects, running a kubernetes stack requires a person just for tuning and maintainance. Kubernetes is great - when you hit massive problems of scalability, but most people aren't going to be facing the problems kubernetes solves.
These days it feels like the real value of kubernetes is not the ability to "planet scale" but actually the control-plane API. With tools like kind, it actually is really easy to get going locally, no cloud needed.
I see no reason to not use it, even if your deployment cluster lives on a single vm. The API is worth it.
delivering your service/app via kubernetes is slowly becoming the standard way, just like an universal binary format. so as a developer you should build towards delivery for kubernetes.
but you are certainly right that one should avoid running your own kubernetes infrastructure - at least until you can afford the accompanying expenses
Very cool project, but I wonder about the economics about using public cloud providers as a gaming host where compute and network are cornerstones to their profitability.
It's very true in my experience, and we've looked pretty hard at all the options. Thanks for the offer of the slides, I have first hand experience with this at scale.
It certainly depends a lot on how sophisticated your autoscaling is and how closely you're able to follow demand to limit waste, how well you can manage per-host utilization, whether you are CPU or memory bound, and lots of other factors. But even at truly massive scale the cloud hosting option can be very competitive without nearly as much management overhead.
Interesting! I have the opposite experience and I also operated at quite a decent scale.
I wonder why our experiences are so different.
For context I was running 2,500+ instances at peak with 40vCPU and 256GiB of memory: the most expensive of those of course being in the regions with low density of players like South America and Australia.
We also had a predictive auto-scaler for our cloud components, I would estimate that our waste was 15% at any time, but if we ran bare metal only it would have been cheaper (except for the operational cost and the fact we needed lengthy commitments for hardware)
We run rather a lot more than 2500 instances at peak and a variety of instance types dependent on game mode. Our waste is on the order of 5-7% depending on the time of day -- we can run less waste on the downward slope of the demand curve for instance.
Aha, Fortnite. You guys will get discounts that we will never get, please keep that in mind.
Our waste was the inverse of yours, less waste on the upswing and more waste on the downswing, due to the long lived nature of our instances.
One instance takes roughly 1,000 players, but if you as a player decide to just keep playing, then we won't kick you out of the game for some hours. Some percentage of players will just continue playing as long as possible without matchmaking or map transfers.
This is very true. But never underestimate the cost of running gameservers in cloud. For us (with really good discounts and a fantastic custom made predictive auto-scaler) it was still more than double the cost than using our previously used bare-metal outsourcing provider.
The cost difference would have been enough to hire 2.5x my team as additional resources, but that differs with scale (and so do the discounts)… So YMMV.
You can always run Kubernetes on bare metal. This seems easy in this case because you don't need persistent storage or load balancing to use Thundernetes, and you gain the ability to use the generic Kubernetes tools for self-healing, scaling, rolling deployments, etc.
I guess it makes sense for Microsoft to be able to put Xbox game servers in Azure but also game devs can benefit from the scalability of Kubernetes. If have is low tire not paying for hardware you aren't using and vice versa.
i would perfer ms to fix well known bugs(aks, terraform, az resources, windows) and make agones well integrated (and windows well integrated) than producing their own. why ms could not patch agones?
Pick a cloud platform and write your own system to manage instances. All clouds come with REST APIs for allocating instances and you can manage a warm pool yourself, it's not that hard and for a game this is a core competency. Why bring in something complex like Kubernetes into the mix if you don't strictly need it.
The larger concern here is the egress bandwidth costs from large clouds, which if you don't have Fortnite scale and favorable deals, will cost you ~9c per-GB. If you use bare metal, you usually don't have to pay that, the cost is baked in to the instance.
Cons of bare metal being difficult provisioning, generally having to purchase instances ahead of time whether you need them or not, and having them sit around and not be used when you are off peak time.
Best to write your own service provisioning layer that can support both bare metal and cloud instances, that way you fully understand it, can make changes to it, and it becomes a core competency for your team.
