I think a lot of folks who argue "you should learn how to be a DevOp" are working on a single product that's always in active development. And they tend to see the problem through that lens. (maybe I'm wrong about about the OP, but that's my sense)
If you're a consultant or work on multiple projects, the equation changes significantly. What if you're building a web app for university, for example, with a fixed budget, and once it's feature complete, it goes into "maintenance mode" with very few updates for the next couple years?
Do you want to continually be in charge of DevOps for something that isn't actively being developed for several years, having to ramp back up when issue arises every three months? Is that the best value for the client organization? Or would you rather outsource that? I'd usually choose the latter, and in my experience, it's been a better value for clients.
Does heroku ssh in for you and update your rails installation or other custom software? If not there's no way you should leave any software unattended for that length of time. Most java, python, ruby and php frameworks/apps will all have had vulns in that period for example. Also do they upgrade things like ruby without you redeploying?
What heroku buys you is not having to deal with updating the web server and database, as they'll deal with all of that, but I wasn't aware that they provide a fully managed service like say wordpress.com, and I don't see how they could for custom apps.
You don't need to ssh into the box to take care of Rails vulnerabilities; just redeploy. Heroku is taking care of any vulnerabilities in the rest of the stack: Postgres, Nginx, SSH, etc.
The thing is, that's actually the easy bit - use apt-get upgrade on say Ubuntu lts and you will very rarely see problems with your upgrade of software like Postgres, apache, ssh or nginx - it's so widely used that you're unlikely to run into issues, and upgrading takes a few minutes every few months.
The thing you need to test then deploy are upgrades to your language and framework/app, and those present exactly the same problem on heroku or on say a vps. You can't just redeploy your app without testing on a new config, and deploys to heroku are no easier or harder than deploys to your own server once set up.
I am simultaneously kinda embarassed and also not that I've never actually been able to get Chef to actually work. I've tried two or three times, put two or three hours into it each time, and then said "fuck it" and either did it by hand or "git push heroku."
My time is more valuable than manually managing servers. It's an incredibly basic tradeoff: some people are more price-sensitive, others are more time-sensitive.
In my case, if I end up working on an app that reaches the scale where Heroku isn't appropriate (for cost or 'omg I need metal' reasons) again, I'm sure there will be someone else on my team who lives and breathes ops. I can manage, but I'd rather be doing almost anything else.
(and when I tried to load the article, none of the body text loaded, I had to refresh the page...)
I tracked my time when initially learning chef, and it took me ~100 hours to be comfortable with what I needed (2.5 years ago roughly if I remember well).
The learning curve was fairly steep for me, but then I don't regret it, since I got to use it as a freelancer just right after on multiple gigs, and that I can now extend EngineYard recipes or think about using AWS OpsWork without much sweat.
From a purely single-person building app for yourself kind of perspective though (with one or two servers), it's a deep investment IMO...
People should have a look at http://ansible.cc for an apparently much lightweight alternative.
100...hours? Wow, the documentation for Chef must really be bad because I guess I still don't know what it's meant to do, despite going through the Hello World process. I thought it was just a framework for the batch commands needed to set up a machine. What was the hardest part to feel comfortable with? The syntax? The conventions for maintaining and executing recipes?
Hi Danso, fundamentally Chef, CFEngine and Puppet are all Configuration Management (CM) systems. They are a programmatic way to specify what a "host" or group of "hosts" should look like using a dependency graph. Basically it builds the dependency graph and works from the root to the leaves one level at a time.
As an example let's say you want to install MongoDB on a new Ubuntu machine, you have the following dependencies in approximate order;
* Add MongoDB Apt Repo to apt list.
* Add MongoDB repository key.
* apt-get update.
* apt-get install mongodb.
* Configure /etc/mongodb.conf to your liking.
* Restart mongodb.
Depending on how you implement it CM will allow you to make the process cookie cutter.
Want a new test environment? Spin one up and run your CM against the servers.
Want to try that hot new cloud provider? Ditto.
I think it's worth the investment unless you're just doing apps that have a very short shelf life (e.g. marketing apps that live for no longer than a few months).
The documentation for Chef is not great, but what makes it problematic (at least when I was first learning it) is that it has a tendency to not correspond to the currently released version (sometimes it's outdated, but more annoyingly, sometimes it's pre-dated).
It takes experiences like this to realize one of the advantages of Puppet's architecture, namely that you can bootstrap it by installing one of two system packages (the server or the client). The agent and the master are just processes, whereas in Chef you have to set up MongoDB and a bunch of other network services just to get started, which ends up making your configuration management system Yet Another Precious Service With Lots Of Moving Parts.
- Use chef-solo. (Edit: and use knife-solo to initially bootstrap Chef onto the machine.)
- Your first time, forget about cookbooks and just write a single site-cookbooks/myapp/recipes/default.rb. Use that to list packages, shell code, whatever.
This will get you started, and you can learn/improve from there.
Your second time, use librarian-chef, and as someone else said, freely clone cookbooks so you can adapt them to your needs.
The learning curve of Chef is pretty steep. The article touches on that, saying that documentation is bad, and I would add that the community provided recipes aren't very good either. But, the whole interest is that it's a one time investment. Once you've figured out how to handle your deployments with it, and have a few helpers/examples, you end up saving quite a bit, both in time and money.
> and I would add that the community provided recipes aren't very good either.
This is key. When I first got into using Chef, I assumed things would be no problem because all the services I used already had community recipes.
The community recipes are often wrong, don't work correctly on "clean" first-time installs and have a lot of baked in assumptions about the config and modules to use.
Clone these recipes into your repo, you will be modifying them. Blindly relying on the built in functionality to pull in and update the community repos will lead to pain.
Chef is a powerful tool, but it is also evolving rapidly and requires careful attention and testing.
I think the whole sysadmin/devops thing is an inescapable problem to some degree.
If it's something that nobody on your team can tackle you can easily end up with completely disparate development environments (with devs running different ruby versions, app servers, even databases!) when there's no well documented system for setting up environments.
Heroku makes it dead simple to deploy and add services, but if your team can't manage to do that for local development anyway, things can end up in a really bad state.
To me, the article boils down to an argument that says "You should learn how to be a DevOp. It's not that bad." I don't think that's a trivial thing to pick up. I think it's important to develop some of these skills but I wouldn't describe it as easy.
I think the key argument for Heroku is that it enables you to delay (sometimes indefinitely) learning the skills required to deploy a production application and managing it. In some cases, such as security, it can be a real bear to keep up with the latest developments in addition to managing the application itself.
Ultimately, I view Heroku as similar to an ORM. With an ORM, it will get you most of the way most of the time, but there's some instances where it starts to make sense to dig into the raw SQL below it and get the performance you want.
To extend your analogy, if I may, Heroku is like an ORM which charges per month, and where it costs more to use relational finders. You're paying extra for each convenience and giving up some control, and this is an ongoing cost. For some people this is a great fit and they really don't want to know about this stuff, are building a demo, or working on a smaller site where you don't want to have to babysit the server and are willing to pay a little extra for that.
For others who are on a tight budget, or have an ops person or willing devs who would prefer to control the server setup, heroku is not such a good idea. I think people overestimate how hard it is to get a typical stack up and running and keep it up - for a small server or vps (i.e. most startups) it's probably only a few hours a month work plus a few hours learning setup, if that. The second one you set up can be up in minutes, and it gives you greater control in the long term, plus obviously it's cheaper. If the amount you pay heroku is less than a few hours of your time, then why not use heroku? However if you grow and are spending a lot of money with them, you've probably outgrown them IMHO. I do like their service, and have tried it out, but the convenience comes at a cost, which increases rapidly the busier your site is.
I think you bring up a couple of great points. It can be hard to escape the sort of lock-in effect if you've got a larger app on Heroku. For instance, I'm guessing it would be difficult for RapGenius to move somewhere else at this point. I think you also touch on the key argument for Heroku which, as far as I can tell, is "If the amount you pay is less than few hours of your time, why not?"
Thanks for the amendment to the comparison. In a sense, an ORM can still have ongoing "costs" as it tends to run a little slower and require more processing, so it might require additional hardware eventually as the scale goes up. Maybe I'm belaboring the analogy too much at that point.
I actually thought deployment was a very challenging, nightmarish thing to do. Heroku, and EngineYard are charging me an arm and a leg - of course it must be hard as hell, right?
But I took the afternoon to learn how to actually deploy, and the next day, I cancelled my EngineYard account. It's not trivial, but on a scale of 1-10, it's a 3 or 4.
And even if you can't afford the time/energy to learn it (ie if you're a startup and just need to launch), you can easily do "poor man's" deployment just by sshing, "git pull", "bundle exec unicorn restart" (or something like that). Deploying an app on production is almost the same as deploying on your development machine. Why give Heroku your money when you can do it yourself?
That's true. For me, I view redundancy as one of the values of Heroku. If you're deploying on your own, that may still require provisioning additional servers as needed, manually backing up databases periodically, etc etc. It can be a time cost beyond just deploying it once in some cases. Still, is the cost of Heroku worth it? It's an important question and a personal one for most I think.
I tend to think of deployment as a challenging thing too but the more I discuss it the less I think of it as a mystery. Thank you for sharing.
OP has a good point about deployment. DevOps tools have made huge strides in the last few years and it's feasible to build something that approaches the convenience of Heroku's fully-managed deployment pipeline, but that's not the only value that a PaaS provides.
You really shouldn't underestimate the amount of work that it takes just to keep a product running. For example, dealing with hardware & network failures, maintaining data backups, keeping up with OS-level security, & scaling up. That's a huge part of the value that Heroku provides for us.
There's a sad irony in an article presumably advocating running your own deployment infrastructure that spirals into a miserable re-direct loop when you try to load it.
I'd really be interested in hearing about the stupid things puppet and chef let you do. I'm a puppet user and quite happy with it at this point. Chose it over Chef because I don't want to write Ruby anymore and like the language. Glancing at Salt Stack, I'm certainly interested right off the bat because it's python and will probably take a deeper look.
Still interested in your gripes against chef/puppet, but probably more puppet.
For context I am a ruby developer first, DevOps second. The last cluster I managed was while I was working for Pivotal Labs. Out client had 50 VMs on top of about 30 Metal servers. We used OpenVZ, Scientific Linux (because centos didn't have ipv6 firewalls, WTF?) and chef-server.
We had to add a few patches to get features we wanted such as graceful shutdown and fix up the way the CLI api worked.
Chef was tested, but the way the tests were implemented were done extremely poorly. When testing a class they stubbed their own internal classes. Basically what ended up happening was none of the interaction between the methods were being tested. I was amazed that this class worked at all. This happend quite a bit too. I understand that what chef is trying to do is very difficult but maybe the problem is that they are trying to do too much.
I don't want my ops tool doing everything, I only want it to setup my servers and make sure that they are all the same.
Groupon has a pretty good internal solution that they are trying to open source soon. They released the client a few days ago. Now if only they had some documentation and the server.
as an alternative to relatively complex chef/puppet solutions there is also http://ansible.cc/ - language agnostic and push based over ssh, so no need to install stacks and run resource-wasting clients on the server.
We use Heroku a ton with many of our clients because its quicker, cheaper, and easier to get something setup that is "good enough" for most projects. Informing our clients of the pros and cons is part of the job. Most agree that the pro's out weight the cons initially and we tend to agree. We also use it for many of our own projects and have been happy for a while and will continue to use it.
That said I haven't been happy with the way things are going with Heroku and plan on looking at alternatives.
First it was the Heroku toolbelt that wants to install its own git and foreman, tried the standalone and it just fails before installing. Seriously don't understand this decision, I get that its easier for a new person to get started and that they are moving away from ruby only.
The fact that cedar still doesn't support websockets, we gave up having a seamless varnish cache and nginx in front of our machines in hopes of more options. We got more languages but still limited to basic web technologies.
Support is good but can be super shotty, we've had times where our scheduled tasks fail with cryptic error messages. Simple things like a Rake task that has been running for months just starts failing. We send them the trace and then we hear back "that shouldn't happen" then nothing else. Who knows when it will fail again or why.
I'll probably continue to use Heroku Postgres, its been the best part of Heroku for quite sometime and only continues to get better.
I think its time to look into chef/puppet/vagrant and ec2 or maybe dotcloud. Not to mention I don't want to be stuck with Heroku when Salesforce gets sick of them.
Frankly not every team has the time money or talent for sysadmin. I work for a tiny 4 man company, and we each build apps mostly on our own. We just don't have the bandwidth for sysadmin work. Heroku relieves us of that.
Sure if I was working on a single app full time with 2-3 other devs and a dev who was in love with sys-ops I would go with chef/puppet/vagrant and ec2 no question.
I'm definitely with you that sometimes unexplained exceptions of failed rake tasks occur and that is a little disappointing.
Most of this is ameliorated by time to deploy.
But as it is, despite it's limitations, Heroku has and continues to be fantastic.
I generally agree with the article. We deploy apps to EC2 instances and physical servers. It seems fairly effortless these days, so I don't get the appeal of Heroku, especially considering the cost and price/performance ratio.
What I'd really like is a collection of Chef recipes to replicate a Heroku-like environment, complete with git deployment, buildpack compatibility, logging, etc.
If you're looking for a Heroku-like environment with buildpack compatibility, you should check out OpDemand: http://opdemand.com/
The application stacks are basically a collection of open-source Puppet modules that replicate a Heroku-like environment on EC2, compatible with most Cedar buildpacks (Node, Ruby, Python, Java, Clojure). Real-time logging is piped through the web console. Workflow is to push code to your GitHub repo, press deploy and your code is pulled onto the instances, Puppet policies applied, etc. Command line `opdemand deploy` is available too.
It's also more than just app stacks, you can also deploy/integrate your own custom EC2 "addons" alongside your app stacks and manage them all as a unit (Redis, Mongo, PostgreSQL, custom services, whatever). There's also native support for Heroku apps via the web console which makes it easy to migrate to/from Heroku and EC2.
for me, one of the most important aspects are vendor lock-in. Maybe it's irrational, but I don't like to depend on a single vendor, however good it might be.
I use Linode and AWS, and used Rackspace and probably half a dozen smaller VPS providers in the past. Most of them I ditched, some I still use occasionally, some I might use more in future.
With all of these providers I know I can switch away from in a moment's notice (if the need arises).
If PaaS became standard / compatible across vendors, I would definitely consider Heroku, Engine Yard or whoever else in this space. Until then, I'd rather have this sense of freedom, even if irrational.
I've been doing IT for over 25 years, and I can assure you there's nothing irrational about fearing vendor lock-in.
I'm all in favor of using services like Heroku if you're in a very early start-up phase with less than a handful of people or for weekend side-projects, but once you start building a serious business depending on a PaaS is just as suicidal is letting your business depend on someone else's API.
There's a reason I'm running full nightly mirrors of all of our private and public GitHub repositories...
The same applies to services. They can end tomorrow. Well, maybe not that fast, but how fast can you shift your business to an alternative without paying a terminal penalty?
We can't run without GitHub for more than a week, so we have to be able to get out in less. Same goes for AWS. Besides snapshots we run old-school backups to ensure we can recover on old-fashioned bare-metal if necessary.
Using those services, great. Letting our business depend on them? Never.
A recent good alternative to Chef and Puppet is Salt.
I first had a look at it about a year ago, when they just introduced state management, had lousy docs and rathe small community. It's massively improved in the past year, the docs are plentiful and useful, and the community is growing.
It took me around a day to get started and prepare a masterless (ala chef-solo) config for a typical django setup we use (nginx, postgresql, standard django/pil deps, virtualenv), and a few less-standard things (node, less, apache fop) to try my hand at something a bit more challenging than a trivial setup. Definitely an easier learning curve than Chef, IMHO. There's also a Salt module for Vagrant.
This article does not present a good argument against using Heroku. Yes, you can learn to do for yourself what any given service provides for you. That doesn't mean it is necessarily something that interests you or is worth your time.
>Need a production solr instance? The Heroku way is to use a 3rd party add-on such as websolr. This means creating yet another account to keep track of, adding another bill to your credit card statement, dealing with one more mediocre web interface, and creating an even larger divide between your development and production environments.
Nothing prevents you to use your own solr(or any other server) of your own inside your application deployed in heroku if you want
What a load of crap! We've been setting up our ec2 slices with puppet for a while, and we literally saved tens of IT hours after moving most of our new client projects to heroku. The one constant thing about maintaining configurations is the never ending change. Versions change, requirements change, etc. Not to mantion it's really not easy to properly configure all the services to play together, with monitoring, restarts, and all.
I've been using the new AWS OpsWorks service for my latest project. It's great. You still have to learn chef if you want to write custom recipes, which I did, but overall for just deploying stuff to the cloud it's hard to imagine Heroku coming out on top when you really back out the costs.
Im sorry for the OT but looking as to how a developer/designer working at/on Grouptalent could read this:
I really dislike the typography on the site, the font is way to thin, small and low contrast hence I had a hard time reading it, even when zoomed. It made me only skim this article even though it seems interesting.
yes I'm on Windows and I get to see the Nimbus Sans font. But looking as to how my comment has a rating of -2 I suppose people disagree. Honestly I didn't mean to be a dick, just hoped to provide a little constructive criticism.
If you're a consultant or work on multiple projects, the equation changes significantly. What if you're building a web app for university, for example, with a fixed budget, and once it's feature complete, it goes into "maintenance mode" with very few updates for the next couple years?
Do you want to continually be in charge of DevOps for something that isn't actively being developed for several years, having to ramp back up when issue arises every three months? Is that the best value for the client organization? Or would you rather outsource that? I'd usually choose the latter, and in my experience, it's been a better value for clients.