The only part missing in most Kubernetes write ups is that you probably don't need it. I'm hopeful most engineers recognise this, although my day to day experience paints a darker picture. I know of at least one hot startup deploying Kubernetes just to attract engineering talent.
Yeah +1. I was super skeptical 1,5 months ago when I started my new job on a 22 node GKE cluster. "Surely K8s is overkill", "could've just used Nomad + Consul + Vault + X", "why not just use Fargate", "K8s seems like a distributed systemd", etc pp. Long story short, K8s on GKE just works, saves us tons of time, makes service-oriented architectures/microservices easy as pie and has led to a way better service level than the previous setup (Chef deployed on baremetal).
Yes, it probably is overkill for your 10req/s app, granted. But I highly advise everyone to at least give it a try out of curiosity, because a lot of hard problems at medium scale and above just go poof with K8s.
> because a lot of hard problems at medium scale and above just go poof with K8s.
No, they don't? I don't know why anybody would just assume that something as complex as kubernetes would just run flawlessly once you actually try to run it on thousands of servers. Must be something to do with google PR because people definitely don't seem to assume the same for e.g. hadoop or openstack. Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running?
>when I started my new job on a 22 node GKE cluster
>problems at medium scale
vs
>Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running
K8s obviously is not a silver bullet and of course there's ops work to be done. I can't comment on whether operating K8s clusters in other contexts makes economic sense, but I know it does for our current team.
> makes economic sense [...] for our current team.
But the reason for that is not that it makes "hard problems go poof at scale". The reason is that you're using a hosted service where somebody else (in this case, Google themselves) takes care of the problems for you for a fee and - at small scale - you only have to pay them a fraction of a single operation's engineers salary for it.
So of course it makes economic sense for you to use a hosted service where the sharing economy kicks in, but your recommendation to use kubernetes because it solves hard technical problems at medium scale does not follow from that.
> [Google themselves] takes care of the problems for you for a fee...
GKE's services are, as far as I can tell from their pricing page[0], free. The compute running on top of GKE is charged at the GCE rate, the master node spun up by GKE is free.
Disclaimer: I work at Google Cloud, but nowhere near these offerings.
How does this relate to my point that you're transitively paying somebody else to do ops for you? Maybe google's pricing model rolls this into the normal VM price? Or maybe it's currently offered at a loss to gain traction? Or are you saying they are not paying the SREs anymore and they work for free now?
Per their pricing, GCP doesn't charge for the master node VM that does the orchestration - you just get charged the normal price for VMs as if you'd provisioned them yourself. Thus, given the price of GKE is $0, the only thing I could see you "transitively paying somebody else" is experience - Google engineers become more versed in running managed Kubernetes and you don't. If the fee you perceive is dependency I'd agree - but I'd also opine that many startups/SMBs would be willing to accept that fee rather than onboard engineers to solve it.
As to why they're taking the loss on the master node VM, I don't know. I had previously expected that it was a cost and was quite frankly pleasantly surprised that it wasn't - it seems like the most obvious sell. If I had to guess as to why it's not my best assumption would be that there's far more to be gained in getting companies comfortable with scaling and from angles that go beyond just the strict monetary benefit of them going from 3 compute instances to 300.
If you are running on Google computer anyway, then I don't think you pay anything extra for gke. As far as I can tell, you are only paying for the VMs that you use. So the price for hosted k8s is already baked in whether you use it or not.
Yes, when you use GKE Google is operating K8s for you. Now ask yourself why Google and AWS and Azure offer K8s but they don't offer Swarm, Mesos, Nomad, etc. Maybe K8s solves some problems after all.
No it doesn't. You can not extrapolate from the fact that the hosted version "just works" that kubernetes at scale would also "just work" (kubernetes being the open source product that you run yourself here). Especially if the hosted version is offered by google and everybody knows that they employ truckloads of very good engineers keeping their stuff running.
The original comment asked “what about hosted kubernetes?” and the reply was discussing the benefits of that hosted solution. They didn’t say that running your own cluster would be the same. They didn’t even imply it.
Even if they had, it doesn’t justify your aggression. Maybe take a break from the internet for a bit to clear your head.
Well I am sorry if you felt the wording of my comments was too "aggressive"; personal angle is weird though. Also I think you are trying pretty hard to misunderstand what I am trying to say. Obviously, using google's hosted version will make all your problems go "poof" at any scale, because they are Google's problems now. But that is precisely the point I am trying to make: It's not a feature of using kubernetes - it's a feature of paying Google!
Or Azure. Or AWS. Or probably in the future pretty much anyone. If you want to operate your own cluster you will be able to, for everyone else it will come from their cloud provider as a "cost of getting your business." This is a great thing.
Yes, and that was their point: That the hosted solution did that. They didn’t say it was a feature of kubernetes. The entire thread started with a question about the hosted solutions and they talked about their experiences with that. Can you specifically point out where they say the hosted version’s benefit should translate to the self-hosted version?
Would it be accurate to describe some of the K8S ecosystem as building out equivalent services (monitoring, failover, health checks) to the cloud providers on top of a infra-agnostic layer?
I'm another happy user of this approach. While I generally agree with other posters all over these comments saying it isn't worth it for small or even medium deployments, I'm happy using it with a team of 2-3 for a single client.
Much different use-case than most, though: automated machine learning pipelines via Pachyderm [1]. Would never have touched K8's if it wasn't for Pachd, and between their layer and GKE it's been quite painless. May have given up on Pachyderm if it wasn't for GKE; we're not cut out for maintaining K8's, and our first shot on AWS (pre-AKS) was not pretty.
tl;dr: K8's are great when it's all but completely abstracted away from you
I see a lot of value in Kubernetes. For one thing I don't want to wake up at 1 AM if one of my node goes down and brings down with it important containers. If I scaled the environment with excess capacity then Kubernetes should start up new pods on the remaining nodes.
That is an awesome feature of K8s and similar orchestators. It is however just an HA (high availabilty) problem already solved many times over in many ways. Almost every PaaS system has it. All cloud providers have it. Getting K8s (or any of its competitors) for HA is total overkill.
We use a GKE cluster with dozens of preemptible/spot instances. Great savings for no extra management fee, and with a nice standard kubectl interface for deployments and changes.
VMs are the most reliable thing of any cloud, and often more reliable then their own managed services because of complexity. GKE does live-migration of VMs and Azure/AWS are close to the same thing, so building on that foundation and doing your own "managed" layer via Kubernetes provides a better experience, while also giving you that spot instance discount that you wouldn't get otherwise with PaaS.
Unless you're renting from multiple $10/month services, your service can be only as reliable as your service provider, which is probably less than AWS.
You hit it on the head. Weave Net is participating in the Istio project. I hope they can bring WeaveNet to Istio. That might be able to provide a multi cloud vendor service mesh.
But that’s easily possible – you can use all the French hosters at once. Latency between them is practically nil because their data centers are only a few hundred meters away from each other.
Sure, in case all of Paris goes down, you’re down, too. But in day-to-day usage you can have good reliability on a shoestring budget.
Nobody wants to wake up at 1AM because their singly-homed service just went down. Kubernetes might be a fine tool to achieve that, but I want to point out that there are much simpler failover solutions available. Failover is something you should definitely have, but also something you definitely don't need kubernetes for.
This feature won't even work for anything designed for fault tolerance. I believe it can only work for things that either rely on other properly fault tolerant services or that are completely stateless, idempotent, so they can be retried and eventually consistent (in which case the state still has to be handled by properly fault tolerant systems). Either way kubernetes cannot help here.
You mean a failover solution? It's hard to give a list here because it is such a large space and it completely depends on your product/application. It is more like a category of use cases than one specific use case.
Some ideas for things you could do in a web/website/internet context assuming you have a single point of presence:
One type of "HA" is network-level failover; haproxy (L7), nginx (L7) and pacemaker (usually L3) seem to be very popular options, but I think there are dozens of other alternatives. In terms of network-level failover, things get more interesting once you are running in multiple locations, have more than one physical uplink to the internet or do the routing for your IP block yourself.
For application-level failover and HA, one option for client/server style applications is to move all state into a database which has some form of HA/failover built in (so pretty much every DB). I think this is very common for web applications and also for some intranet applications.
Assuming you have a more complex application running in a datacenter, there is also a lot of very interesting stuff you can do using a lock service like Zookeeper or etcd or really almost any database. Of course, you can also make your app highly available without using an external service; there is a mountain of failover/HA research and algorithms that you can put directly into your app (2pc, paxos, raft, etc). Of course all of these require some cooperation/knowledge from the application. For some apps it might be very hard to make them "HA" without relying on an external system, but for some apps it will be trivial.
Note that when you move away from a web/datacenter context, to something like telecommunication or industrial automation, so something that doesn't run as a process in a datacenter but is implemented {as, on} hardware in the field, failover and high availability will have an entirely different meaning and will be done in a totally different way.
Most single-instance, single-zone failover scenarios can be handled with shell scripts, the AWS API, and cron.
But the parent's comment is missing the point. K8s is not for failover. K8s is literally just a giant monolith of microservices for running microservices. It's not intended to provide failover for non-microservices, it's intended only to run microservices, and as a side-effect of needing to be able to scale them, it inherently provides some failover mechanisms.
FROM golang:1.8-alpine
ADD . /go/src/hello-app
RUN go install hello-app
FROM alpine:latest
COPY --from=0 /go/bin/hello-app .
ENV PORT 8080
CMD ["./hello-app"]
The difference in simplicity is not in the interface that is presented to you as a user. The difference is that your shell script will have a couple hundred lines of code, while the docker and kubectl commands from above will pull in literally hundreds of thousands of lines of additional code (and therefore complexity) into your system.
I'm not saying that is a bad thing by itself, but there definitely is huge amount of added complexity behind the scenes.
That's nice and all as long as it works. If there are any problems with it (network, auth, whatever) have fun with even diagnosing the bottomless complexity behind your innocous lil' kubectl command.
The error messages are quite good. There are extensive logs. There are plenty of probes in the system. Just the other day we `kubectl exec <pod_name> /bin/bash` to test network connectivity for a new service.
To the best of my exposure, Kubernetes is a well engineered system.
The point is that automation software for admin tasks is a zero-sum game: the more a tool does, the less your devops staff will be able to leverage their know-how. The more magic your orchestration does, the less your ops will be able to fix prod problems. And for getting anything done with k8s you'll need a whole staff of very expensive experts.
Look at it this way. Kubernetes is a Tesla Model X, and scripts/aws/cron is an electric scooter.
You can try to go cross-country with either of them. One was engineered to survive extreme environments, protect you from the elements, and move really fast. The other was engineered to leave you exposed, operate in a moderate climate, and go much slower.
If you have problems with the former, it's time to call the technicians. If you have problems with the latter, you might fix it yourself with a pocket screwdriver.
That was my point. I wanted to point out that while some people have only/first heard about failover in the context of kubernetes, it is not something that is specific to kubernetes or even the problem that kubernetes was build to solve.
Of course it is not designed to be a failover solution specifically and using it (exclusively) as such would be ill-advised; I was just trying to be diplomatic while pointing that out.
We're using elastic beanstalk currently (multi docker conatiner). Not that I'm advocating it and I'm really interested in k8s now that aws has eks, but ebs is really simple to use for a simple setup.
It’s probably referring to people using Chef cookbooks blindly without any knowledge of how to configure or maintain the service itself.
I have interacted with people who run production workloads via Helm but struggle to do very basic things with kubectl (like recreating a service).
This is an argument that can be used for any high level abstraction, however. I guess Chef is used since they have very good testing tools so Cookbooks are relatively more stable than other CM definitions. Until things go wrong with the service that is.
That community sourced cookbooks are security nightmares or under maintained. I cannot see how helm charts can stay quality without a curated app store.
Thanks for the clarification. The official Helm charts go through a thorough community review process. We (Bitnami) are committed to making high-quality, up-to-date Helm charts. You can check https://hub.kubeapps.com
You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. To me, this is the analogous case for K8s, you can probably do the same with VMs, but everything is a lot more productive with containers and K8s.
I worked at a place that was deploying K8s as I was leaving and my experience was that K8s was complicated and no one knew exactly how to use it or why we should use it, other than it seemed to be what everyone else was doing. I didn't notice any operational improvements over our prior system (which was largely ad hoc), but the kinks may have been worked out by now.
If you read the Innovators Dillemma, there is a very clear chapter about how new technologies don't really deliver their promises immediately; but as they're adopted and operators/users learn how to work with it, that's when productivity changes are very much visible. It took quite some time for AWS to be mainstream and then to be adopted and operated in a way that delivered high productivity gains, and it seems to be the same for k8s right now.
I really wonder about the causality of that, is it changing things to adapt the new technology makes you clean up / refresh / fix known issues along the way or is it the new technology itself causes less issues down the track
That doesn't really make sense. If you're moving to a new technology, even if you clean up/refresh/ fix known issues along the way, you are likely to accumulate similar such issues in the new technology as well, which would lead to a decrease in productivity. But that has never really been observed.
While I agree that the cleanup can inevitably lead to productivity gains, I believe they are overshadowed by the productivity gains from the actual change that is being made.
That only makes sense when seen as an obstacle to initial adoption from an innovators and sales point of view and could be said just about every innovation. But obviously not every IT innovation makes it.
It depends on how frequently you need to spin up new containers.
As someone on Azure, I think the solution for small-ish guys like me (who still need like 8 VMs in production to run our site due to special requirements) is a greatly simplified container management system. Something that fulfills the basic Kubernetes features, but isn't such a steep learning curve.
I think Microsoft is doing this with Azure Container Instances, but I haven't tried it yet. (No doubt AWS is doing something about it as well.)
Or they're going to take care of all the k8 configuration crap through some neat Visual Studio Tools. In a VS preview I saw a "deploy to Kubernetes" button. I just want something that will give me more web API instances when my server is getting slammed.
AWS ElasticBeanstalk or Google App Engine flex or something from Heroku is what you are talking about. Container management but simplified for the common use case. You can implement the same thing with k8s, but why bother for small stuff. The real value in k8s is large, company-wide installations where everyone shares each others hardware. Even then, I've heard anecdotes that kubernetes doesn't scale to the point Borg or whatever other proprietary system does. Then again, those are niche use cases to begin with.
Unless they've reduced the price, Azure Container Instances is mainly for one-off/bursty tasks. If you need a service running 24/7, it's prohibitively expensive.
I think once their managed kubernetes (AKS) hits GA, the learning curve will be slightly lower.
> You probably don't need the cloud either, but everyone recognizes that it is a productivity gain.
Care to quantify that? Because we don't use the cloud (in fact, in my current role, we don't even make use of virtualisation for production infrastructure) and we have no issues with productivity. Or rather, let's be a little more honest, we have no issues with productivity that would be solved by a move to AWS/Azure/$_some_other_cloud_provider.
Just because you "have no issues with productivity", doesn't necessarily mean that you couldn't be more productive. In my experience it takes a move to the cloud to actually see where those gains lie.
Disclaimer: I have no idea about your actual real-life situation. This is vague and hand wavy on purpose.
Indeed it is not meant to be any kind of business justification - I clearly cannot comment your business and its needs :)
In my experience of moving applications to the cloud, there are always folks who don't believe there is any gain to be made. These folks have always been wrong (so far!)
I suppose I'm only commenting because I hate to see someone snub the potential of cloud. But hey I'm just some guy on HN.
The thing is, in previous positions we've used the cloud plenty, and it's been fine. But for what I'm doing now? No way. Just not the right fit. I therefore become extremely sceptical when people cite $unspecified_but_awesome_gains from switching.
I don't have much practical experience with Kubernetes but I think one argument is that there are other alternatives to it that might give equal or better productivity gains for a lot of people.
For example Azure App Service or equivalent from the other vendors. You don't have to manage a VM.
App service is alright but I'd recommend Heroku over app service any day. However, there is some visible work being done to improve app service like supporting docker compose files to run multiple containers but the basics are still very painful.
Most notably, swapping servers (slots) always caused downtime, accessing logs was flaky and slow, azure interface is just plain slow, continuous deploy was flaky (or it just didn't kick in until 20 min later?) and I had to manually restart servers many times
i am a mainly js engineer and so far have lived entirely within the firebase world as far as my backend goes. in your (and everyone else's) opinion, then: when -do- you need it? set some quantifiable bars for me please?
I could throw the "you'll know it when you see it" but I'll try not to!
1. You have a large amount of underused hardware/VM 's you want to cluster.
2. You have an eloborate microservices system with a dynamic service discovery mechanism.
3. You have the above over many regions / teams.
4. All your services are mostly stateless.
5. Your team of rockstar ops engineers are building their own crappy version of Kubernetes just to deal with the engineering challenges your business faces.
Funnily, my current employer is just that. We predate K8s, so it can be forgiven somewhat. Transitioning, though, is just awful. We have every reason to transition, yet getting the system setup, running, and working with the current infrastructure is a daunting task.
You don't need K8s, until you do, at which point switching can be a nightmare scenario.
If you are able to be cost effective and run services on firebase, then keep doing so. Please!
If you need to do something that doesn’t map well to firebase, then my next stop would be AppEngine.
If what you want to run won’t map well to AppEngine, then start looking at K8S.
For me, this is usually wanting to run some kind of legacy system, or where a higher level of security is needed, or if you want or need to plan to run across multiple cloud vendors or in a mixed workload environment.
If AppEngine has been covered by google’s BAA when I started my current gig, I would have just used that, with k8s to handle the couple of oddball things that we are using that wouldn’t run as well there.
Personally, it’s not about scale so much as certain kinds of architectures or problems. You might have a system that needs to run a smallish number of nodes, but has reasons for needing what kubernetes offers without wanting to reinvent the wheel yet again with the existing cast of characters.
Honestly, I have no idea about Firebase or AppEngine in China. It might entirely be FUD. But, if what you imply is true, then Kubernetes is probably a good solution for your goals.
Run your application on GKE and grow your company. When it comes time to move into China, add a cluster on a cloud provider there.
When you notice that you spend such significant money (i.e. multiple salaries) on hardware capacity planning that employing a team to operate kubernetes might be cheaper instead. Probably not going to happen unless you run on hundreds to thousands of machines.
What's your bar for using webpack (or your js builder of choice)? If you're fluent and at-ease with it, you'll probably use it even for hello-world app. Same (in spirit) for k8.
In your experience with these deployments, which K8s features aren't needed? Is it the orchestration as such that's not necessary, or K8s approach wrt to the workloads deployed?
Sorry for not being clear. There is nothing wrong with K8s features. It solves a specific problem really well. However, most people & companies I interact with don't have that problem. Or that problem is just a very marginal piece of their full set of problems. Running K8s, maintaining it and making your apps work with it is pretty big undertaking. Only go there if the upside is so great it blows any alternative away.
I guess the popular term is "cargo culting". Or maybe Heroku is not hip anymore although it probably is the best advice for many companies.
I strongly disagree. I've built a couple in house PaaS type systems in the past with various degrees of success on more classical tech like straight up EC2 using a variety of Terraform/Ansible/CloudFormation/Scripting but the dev experience has always been painful compared to Kubernetes.
Developers really love working with Kubernetes. It doesn't take a lot to get started and if you're not doing anything too crazy it is a huge productivity boon IMO.
Where things get hairy is persistence, but that's the case regardless of Kubernetes or not.
Ah yes, you're right. I am sure you have a much better understanding of the requirements each of these build outs sitting behind your keyboard over there.
Heroku is still amazing indeed. Our team recently moved all Node.js based systems from AWS to Heroku and it has given us a tremendous productivity gain and reduction in complexity. Before we made that decision we spent about 8 months experimenting with K8s, only to discover it actually made things way more complex for us and reducing productivity without any clear benefits.
In that specific case, Kubernetes is approaching YAGNI territory. But as others have stated, it’s easy to get started with something like Amazon EKS and not have to worry about a thing.
Edit: Put another way, give me APIs instead of config file hell.
Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively.
Wow. The top answer on that one is terrible. I would advise the exact opposite. Istio!? Nice idea, but have you tried using it? That is some serious alpha stage software almost noone needs till very late in their tech maturity.
My team runs a grand total of 1 server, 1 daemon and up to 5 concurrent batch jobs. Sometimes we bring up 1 extra server for experimentation.
It is handy to not break the bank and use GKE autoscaling for the batch jobs. Surely we could setup some batch job manager over some raw VMs, or maybe spend some time to select and learn to use yet-another AWS service.
It is handy to go to the GKE "Workloads" page and see how many servers are currently active, using how many pods. Surely we could pull that off with raw VMs, but it would take me a few minutes extra / a brittle script to pull it off.
Do we _need_ Kubernetes? No. It makes us more productive? Yes. Am I confident that in the next assignment I'll be able to reuse my skills, regardless of what VM provider they use? 90% yes, pending EKS becoming usable.
Echoes of "you don't need C, just use Assembly, it's simpler and faster" from early '90s gaming industry.
I think the recommendation is not to use another orchestration service, but to keep the infrastructure simple and not use any orchestration or service discovery at all.
Adding service discovery and container orchestration will probably not make your product better. Instead it will add more moving parts that can fail to your system and make operations more complex. So IMO a "containerized microservice architecture" is not a "feature" that you should add to your stack just because. It is a feature you should add to your stack once the benefits outweigh the costs, which IMO only happens at huge scale.
Most people know that "Google {does, invented} containers". What not so many developers seem to realize is that a Google Borg "container" is a fundamentally different thing from a Docker container. Borg containers at Google are not really a dependency management mechanism; they solve the problem of scheduling heterogenous workloads developed by tens of thousands of engineers on the same shared compute infrastructure. This however, is a problem that most companies simply do not have, as they are just not running at the required scale. At reasonable scale, buying a bunch of servers will always be cheaper than employing a Kubernetes team.
And if you do decide to add a clever cluster scheduler to your system it will probably not improve reliability, but will actually do the opposite. Even something like borg is not a panacea; you will occasionally have weird interactions from two incompatible workloads that happen to get scheduled on the same hardware, i.e. operational problems you would not have had without the "clever" scheduler. So again, unless you need it, you shouldn't use it.
I think the problem that Docker does solve for small startups is that it gives you a a repeatable and portable environment. It makes it easy to create an image of your software that you are sure will run on the target server without having talk to the sysadmins or ops departments. But you don't need kubernetes to get that. And while I can appreciate this benefit of using docker, I still think shipping a whole linux environment for every application is not the right long-term way to do it. It does not really "solve" the problem of reliable deployments on linux; it is just a (pretty nice) workaround.
Without service discovery, you end up generating configuration all over the place just to wire together your services, letting everybody know about which IP address and port where - super tedious. Add in multi-tenancy, and you're in a bunch of pain. Now try and take that setup and deploy it on-prem for customers that can't use cloud services - you rapidly want something to abstract away the cluster and its wiring at the address allocation level.
I'm not sure I understand this. You have a product that is split into many different components, and when you deploy this product to a customer site, each component runs on different hosts, so you have a bunch of wiring up of service addresses to do for every deployment?
Could something like mDNS be a lightweight solution to that problem?
And also I am genuinely curious how kubernetes would solve that. When you install kubernetes on all of these machines, don't you have to manually do the configuration for that either? So isn't it just choosing to rather configure kubernetes instead of your own application for each deployment? If it is that much simpler to setup a kubernetes cluster than your app, maybe the solution is to put some effort into the configuration management part of your product?
I'm not sure you understand the value k8s proposes based on your comments throughout this entire thread.
Managing many nodes is the reason orchestration software exists. Your suggestion to "put some effort into configuration management" is effectively naive, homegrown orchestration.
Build or buy? That's the same argument - except k8s is free and available from many providers.
Ah, if you look under the covers, Kubernetes service discovery is actually kernel-space virtual load balancers running on each node in your cluster. The DNS record just points to the "load balancer".
If your docker containers all build off of the same OS and then you run them on a host using that OS, won't the containers share the lower filesystem layers with the host?
Services use an internal tool called Apollo to describe the service, deploy it to EC2 VMs, and auto-scale. Apollo inspired AWS CodeDeploy and AWS AutoScaling.
Services are reachable via load balancer sets similar to AWS ELB/ALB/NLB.
You reach a service by the LB-set's DNS name.
If you don't want to use AWS or AWS-specific deployment tools, you could use Puppet Pipelines to do the same on Google Cloud or Azure. Puppet Pipelines was built by people who previously built some of those internal Amazon tools and offers similar functionality but cross-cloud.
And if you want even fewer moving parts, just go PaaS or serverless.
Depends on your definition of "need", it enables me to manage a ton of stuff across multiple dcs and regions easily. It would feel like the dark ages to be without it as this point.
