I've been using Flynn for a while now. Unfortunately there's always something that entirely bricks the cluster eventually. Sometimes a node will reboot unexpectedly (as happens with servers) and the cluster slowly implodes from there, sometimes I'll just get a monitoring alert that all the sites hosted on it have gone down for no apparent reason.
They announced during their 1.0 release that Flynn is now production ready but I have to disagree at this time. I do honestly love the idea of running my own Heroku but it's just not stable enough and doesn't recover well if a node goes offline without manual intervention using the flynn-host fix command (and sometimes that command just chokes up completely).
Next time it happens (when, not if, at this rate) I'll try to set aside some time to gather up the logs to help debug the problem but I'm using Flynn to avoid the sysadmin side of things and just build my product. In my opinion Flynn is tinker-ready, not production-ready. It's infuriating that it has these random problems because like I say I really _really_ love the product and desperately want to be able to use it reliably.
Really sorry to hear you have had issues with your Flynn cluster. Stability is our number one concern and we are pushing stability fixes all the time.
Flynn does self-heal after node reboots, so if that is not working for you then it is a bug we need to fix.
When you do have issues it would be really helpful to us and other users if you could open a GitHub issue or come and chat to us in IRC (#flynn on Freenode) so we can fix any bugs you are encountering. I fully understand you just want to be concentrating on building your product, this is why we are building Flynn!
Aye like I say I'll grab the logs next time, unfortunately it's happened at the worst possible times so far (e.g. preparing to demo the site so can't really be trying to debug a broken cluster!) so I've not had the time to debug things as they've happened yet. If nothing else I can't fault the installer, it's damned easy to whip up a fresh Flynn!
One of those things where I wish I could say it's something I'm doing that's breaking it but in each of the situations I've had I've not been interacting with it at all when it happens. I know my waffling here doesn't help anyone without logs of some sort, this post just caught me a couple of days after a cluster brick.
I'll set it all up again and will grab the debug info and jump on IRC this time if it all goes to pot.
Please do, the node reboot logic has been through a lot of incremental fixes over time and we are always working hard to iron out the wrinkles that are left.
If you don't have time to interactively debug with us on IRC just open a Github issue with what details you can and the output of the `flynn-host collect-debug-info` command and hopefully we will be able to discern what went wrong.
I've scribbled the command down and will do so, thanks!
Just to blow away some of my grump cobwebs I'm not just trying out Flynn, I'm still using Flynn! The hiccups were just going to be temporarily patched by hijacking my ServerPilot controlled machine (WordPress on Flynn.. Haven't managed that one yet!) until it felt a bit more stable in the dev/stage environment. :)
Would be cool to learn how you are approaching Wordpress on Flynn. I have poked around at writing an asset backend that talks to Flynn's blobstore because a family member wanted me to host their Wordpress site but haven't quite gotten around to it.
We merged the MariaDB database appliance a little while back which took care of the other major roadblock to getting Wordpress to "just work" on Flynn.
I'm `jpg@freenode and josephglanville on Github, hope to see you around. :)
That one was more a "I wonder if I can.." than a required thing so I didn't fully give it a go. I was giving Bedrock[1] a try for it which may have complicated matters further trying essentially to learn two new things at once.
I imagine my thought process at the time was "Flynn is like Heroku and Heroku is all about 12 factor apps. Bedrock seems to be a 12 factor WordPress, I'll try that!". It also somewhat matched up with the rest of what I do in Laravel so the standard composer, PHP, .env, MariaDB vibe. I'll give it another shot over the weekend, I don't recall what stopped it working.
Major thanks on adding MariaDB support by the way, I know mysql and the likes get their fair share of flak but it's what I've used for the past decade or so, more comfortable with it than postgres :)
For us to invest time/effort adopting a PAAS solution requires an amazing amount of trust in your ability to execute now and in years down the line. Unlike alternatives, you are a company and not an industry collaboration / foundation, how open can you be to make us believe you are not the next RethinkDb-like shutdown?
e.g. How many in your team, why can they do this, how much funding do you have, what is your burn, when do you become default-alive?
Disclaimer: I founded a DevOps (now Kubernetes) consulting company.
We, up until recently, had our own (closed source) ansible/terraform/packer immutable AWS-based PaaS sort of thing. We abandoned it in July when Kubernetes hit 1.0.
It's not just if Flynn and Convox and other VC-powered startups whomever can even exist another couple of years - but whether they exist and if they can out-compete Docker and CoreOS and Kubernetes and... etc.
My feeling is that a lot of infrastructure companies died in July. Just like Blackberry after the iPhone - gradually, and sometime in the future, suddenly.
What's the future for these PaaS companies? I dunno. Consulting services? Managed PaaS? These are fine, but... have a finite growth curve. Not everyone wants or needs to be a unicorn - for sure - but if you're tying your SaaS or web business to someone's infrastructure code, you kinda want:
a) the thing everyone else is doing (so you can grow your own team eventually - it's easier to hire a kube developer than a eg. flynn dev)
b) something that will be around for a long time and
c) something that has the best blend of features/power/ergonomics (Kubernetes has almost 1,000 contributors. Flynn: 98. Convox: 52. Bigger isn't always better, but...)
For us, that's kube. We're still iterating on our our business model - perhaps we just stay as a kube consulting shop for a good while - but running an Open Source PaaS business is not where we think the puck is heading.
Thanks for the great question. I'm the CEO & co-founder of Flynn.
tldr: we're open to making Flynn a foundation/community project if we could find the right kinds of partners and once the core innovation is complete.
This is something we've thought a lot about over the past few years. Everyone on our team is a hardcore open source believer and contributor and the long term success and sustainability of open source projects and Flynn in particular are something we care deeply about.
When we started working on Flynn (with just a crowdfunding page posted to HN) we had no intentions of commercializing it. Before asking HN to fund the project we reached out the founders of many growing startups (many of which are now unicorns) who we saw as the core users. Our goal was to agree on an architecture and set of APIs, then have each company "own" one component of Flynn, with one engineer at each working on their component full time.
Everyone we talked to said they loved the idea of an open source Heroku that could run anything (not just 12 factor apps) and run anywhere from AWS to their own colos and datacenters, but none had the staff to spare on infrastructure engineering (which is why everyone wanted a PaaS in the first place). Several, like Shopify, contributed to Flynn financially so we could hire a dedicated team to build Flynn for everyone.
So a community/industry collaboration was absolutely our goal from the start, but several things made us question that route. Not only were our first choices for collaborators and partners unable to participate directly, but we spent a lot of time talking to companies in the then-thriving OpenStack community as well as other foundation/industry run infrastructure projects. They shared a number of horror stories of how the most promising new features and capabilities were killed by cross-vendor compatibility concerns and how specs were limited by legacy vendors on committees.
Committees and bureaucracies are great at keeping something alive and maintaining the status quo, but not at innovation.
We also met executives from very large companies in regulated sectors like banking and telecommunications. They were extremely excited about Flynn but had a completely different set of requirements than the younger companies like startups whose needs we knew best, and had experienced ourselves. It was clear that we would have to build a very different project to serve the largest of enterprise customers. Crucially, foundations/industry collaborations on infrastructure attract the largest of vendors who have these mega-companies in their sights as the real customers.
If you want to see what a PaaS built to their (imagined) needs are, look at CloudFoundry built by Pivotal, which is currently owned in part by Dell/EMC, VMWare, GE, Microsoft, Ford or at OpenShift, which was built as a startup but acquired and retooled by RedHat.
While we think Flynn has a lot to offer today and in the future to these large institutional users, they have not been our focus.
Flynn today has the minimum feature set we consider necessary to be useful. Looking towards the future we want it to encompass a lot more. We'll be publishing a comprehensive roadmap/master plan soon, but most of that innovation would be harder or impossible to accomplish if we were locked into a foundation today. Development would be slower, changes would have to work their way through approval to consensus, and many partners would object to new features competing with their own established products.
We've been very careful both to limit external dependencies and to provide end-to-end integration between all our components. As a result we're able to move very quickly when changing APIs, implementation details, even major pieces of architecture, all without impacting customers. This gives us the freedom to stay nimble in a changing landscape, but partners might be more attached to technologies in which they already had a large institutional investment.
Infrastructure companies today are already a world apart from where things were in 2008 when Heroku upended PaaS. The default is now open source, making lock-in (historically the bigger concern) far rarer. There are few companies among our peers battling to become the next Oracle.
Additionally there's a tremendous amount of portability both among unified PaaS's and build-your-own infrastructure because of technologies like buildpacks and containers. As a result most apps that run well on one platform can be moved to another. There are of course features, like Flynn's database appliances, that are not available on most others.
We've made other choices like a small codebase with few external dependencies that make managing Flynn's development and maintenance easier, whether it's our team or yours doing that maintenance.
That being said, we're entirely committed to Flynn's future personally. While we'd love to see it become a huge company, that's not necessary for us personally. We're a team of five, and speaking for at least my cofounder any myself, we're not going anywhere, regardless of the VC market.
We work on Flynn (for several years and many long hours now) because we believe it needs to exist in the world, not because we think it'll make us rich. We're committed to keeping Flynn alive as long as that stays true (basically until it becomes unnecessary or there's something else that does everything we want but better). Since our 1.0 launch we've had huge growth and adoption and are currently fundraising. We're single digit new customers away from default alive with the current team. Whether we become a small, sustainable company to selling to/partnering with a large enterprise vendor, to creating a foundation, everyone on the team is 100% committed to a permanent, sustainable future for the product, with or without venture capital, unicorns, or IPOs.
Thank you for the detailed response - much appreciated. You have tempted me to look into Flynn further.
The headline feature that attracts me is HA postgresql - that has instant relevance. My greatest concern is stability - I'm not seeing many success stories but lots of problems. I want to hear that Flynn is "the one bit of my stack I don't have to worry about". I hope you can publish some case-studies (both successes and post-mortems) including discussion of the attempted architectures.
A fundamental philosophical worry I have is that it is a "wrapper tech". I increasingly stay away from these - the promise is that they make the underlying tech easier/quicker to use but they make assumptions that don't hold for you in the long-term, they serve too many unrelated use cases so become bloated and poorly fitted to your own needs and they reduce the features, flexibility and stability of the tech they wrap. The cost in fixing & working-around the wrapper can quickly exceed the cost/value of building your own glue based on the underlying tech because it also gives you a greater depth of knowledge in the fundamentals and more power.
that's the whole point, you don't need a lot of time to adopt flynn. it's not that "getting BOSH to work was my life for the last 3 weeks" kind of experience.
Not the whole point, for all we know Flynn could be out of business in 6 months and they'll be stuck on a platform with no future and saddled with switching costs. Not saying it's likely, but it's a risk worth considering.
i should have said: that's the whole point for using a buildpack/docker paas in the first place.
a) it's not hard to set up flynn
b) migrating to another buildpack system is not a lot of work, except setting up some environment variables and installing a cli
Experienced Flynn last year, we spent about 2 weeks making it work properly with our solution (We had a complex SaaS solution hosted on Heroku, with tons of addons).
We had some very bad bug that happened at the time (like deploying a new version made the dyno crash and we had to deploy a completly new environment).
Other than pretty bad bug, it was working properly, and did what they sell.
Good point is that the dev team is very responsive and we reported a bunch of bug that got fixed in no time. They also have a smooth updater when a new Flynn release came out, side bad note is at this time, it was kinda bugged (because of our specific setup).
For all those reasons, we choose to stick with Heroku at that time (we also tried Doku https://github.com/dokku/dokku, various AWS services such as BeanStalk, and DEIS http://deis.io/), but for sure it was the best candidate out there and I would love to give it a try again.
We have pushed a ton of stability fixes recently, and we now have a stable release channel which we release to monthly (see https://flynn.io/docs/stability).
I highly suspect many of the bugs you encountered have been fixed, and we make it a priority to investigate and fix reported bugs as quickly as possible, so if you end up giving Flynn a try again let us know if you have issues and we'll fix them.
As for updating, the in-place updater still has some kinks but the backup / restore route is fully tested and is reliable (see https://flynn.io/docs/production#updating).
This is really cool. I am not sure if there is a way to deploy this on public cloud like AWS/Azure. If this were cloud agnostic way of building software, I would be totally into it.
> The CLI includes a local browser-based installer that can boot and configure a Flynn cluster on Amazon Web Services, Digital Ocean, Azure, and your own servers via SSH. It automatically performs all of the steps required to install Flynn.
Thanks! I know this is slightly different from what Flynn is trying to do, but what I was looking out was a way to use standard cloud provided PaaS rather than using cloud as IaaS. This approach may work fine in the short-term but cloud native services provide a lot of extra functionality (monitoring, security, sandboxing, eventing, logging, encryption, etc.,) that would be hard to replicate for something like this. An abstraction to those cloud native service, OTOH, may be useful against lock in. I believe it should be possible given that most of the cloud native services across different clouds provide similar set of features.
Thanks! Flynn is designed to be cloud-independent so you're not locked in to any cloud, or even cloud in general. You can also run Flynn on your own servers.
Flynn also allows portability between clouds. Since we don't use any cloud-specific features (like, say RDS) you can backup a Flynn cluster (including database appliances) from one cloud and deploy onto a new cluster on an entirely different cloud.
I've worked with a group using Flynn to host a score of internal apps. It required one full time developer to keep it running, and saved a bunch of time and effort for the people maintaining those apps - no need for devops, pretty much, as deployment boils down to a ten line bash script. So if that tradeoff works for you, you should consider using Flynn.
In this age of it being next to impossible to hire devops folk, the idea that you can have one of them servicing the devops requirements for a sizable fraction of developed applications is somewhat attractive.
I wouldn't put Flynn in place for external-facing and high-traffic applications at the present time based on the experience I've had with it, however. Not ready for that, I think. You have to have a fair tolerance for short outages.
having worked on cloud foundry platform in the past, paas was extremely exciting at the time (2012), we even came up with something called warden which was way before docker. but since then, i have to ask, is there really that much benefit to having a paas? i understand the arguments about not needing to have devops/sre's, etc, but i think iaas works just fine, sure there's the initial plumbing for things like load balancers, installing an operating system, upgrading/patching that operating system, securing all the network connections, but it's not an everyday task. cloud foundry at the time required a minimum of 20 vm's to get the environment running (cloud controller, bosh, service nodes, dea's, nats, etc), and these weren't your t1 micro aws machines, these were memory and processor hungry instances; to manage the entire environment required a team of sre's and devops. nice to see that flynn only requires 3 nodes minimally, but as the environment scales, you'll need more nodes and i wonder how the overhead scales with flynn. obviously for small environments (let's say 6 vm's, an additional 3 vm's to support flynn is not ideal). but if you get up into the 20+ vm arena, perhaps the overhead starts to pale in comparison?
perhaps it's just me, but the devops part isn't that annoying, yet, but i've got an ultra small environment running nginx, golang binaries, and some services like rdb and k-v store.
I think the main motivation compared to IAAS is to run on your own choice of machine e.g. on-premise or dedicated from OVH/Hetzner, or across different cloud providers while not incurring the full admin overhead of baremetal. The reasons can be portability, control, performance, privacy or the potential for 1-2 orders of magnitude cost-savings.
And yes, scale, high availability, self-healing, auto-scaling & load balancing is where the admin pain comes in.
thanks duncan for the reply, i am still not sold on paas as a long term solution, my reasoning being as such, companies like google that have their own paas (gae), don't even use it to run large portions of their own infrastructure. take for instance, facebook as well, they do not use paas as far as i'm aware for major portions of their sites, and i would say that those 2 companies are representative of extreme scale. so why wouldn't a company of this scale need/want paas? i guess part of it is that they can afford sres and devops, but the counter-argument would be that if they could do without sres/devops, why wouldn't they?
i guess what i don't like about most paas is the fact that you lose some level of control over the environment for the sake of convenience and bliss, but at the end of the day, someone has to fix all those bugs in your paas layer.
I think you could describe the systems that google/fb use internally as paas despite not being externally facing - they have internal teams supporting a platform that then hosts apps and services from 100s-1000s of teams. I know what you mean about control but things like kubernetes were developed for Google's internal paas so they do kind of have control despite the level of abstraction.
For us normal people at more human levels of scale - I agree its wise to be skeptical - serve your users in the simplest manner and don't get sucked into HN hype. Its easy to get an infrastructure inferiority complex :)
* Automatic cluster scaling, so your PaaS deployment is as small as possible. Cloud Foundry doesn't have this, and I wish it did
* Flynn has built-in service discovery, in Cloud Foundry you bring your own (if you want)
* Flynn comes bundled with some data services
* Flynn has no BOSH equivalent, so there's nothing there to bring back VMs when they fail, perform rolling upgrades or manage the IaaS
* Flynn doesn't seem to offer a metrics stream, only logs
* Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack
* I couldn't find anything about role-based access control in Flynn, something Cloud Foundry has (which can be backed by LDAP for enterprises)
* Flynn only appears to route HTTP traffic, so no TCP routing for IoT workloads
* Cloud Foundry is run at massive scale in production for mission-critical workloads by banks, payment processors, IoT data ingesters, manufacturing companies, research companies, governments and everything inbetween. I can't see any evidence of the same being said for Flynn.
* Flynn is owned by one company, Cloud Foundry is owned by a non-profit organisation affiliated with the Linux Foundation and can never be transferred back to a regular company. This prevents the sort of nonsense you've seen in the Docker community recently.
* Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack
You can deploy docker images, at least that was the case one year ago
* Cloud Foundry is run at massive scale in production for mission-critical workloads by banks, payment processors, IoT data ingesters, manufacturing companies, research companies, governments and everything inbetween. I can't see any evidence of the same being said for Flynn.
I think that's the whole idea behind flynn. Instead of being heavy-weight, complicated to deploy (no BOSH) and resource intensive, you can run a cluster with 3 nodes and deploy a ton of services there. It's not targeted at banks, it's targeted at people who don't want to use heroku or pivotal or GCE. And this makes it a lot faster in many ways, at least last time I checked.
That's cool about Docker images. Cloud Foundry has a binary buildpack, and the Java buildpack deals with .jars, which means slightly lower complexity for the end user than needing to roll a Docker image. CF does Docker too now, which is nice for compatibility, but I think people are missing the point by deploying images instead of apps.
Fair points about BOSH, it's got a steep learning curve (cliff). I'd love to see Cloud Foundry have an install experience that's along the lines of "Hey, here's some IP addresses and an SSH key. Go install yourself there."
> Flynn has no BOSH equivalent, so there's nothing there to bring back VMs when they fail, perform rolling upgrades or manage the IaaS
Flynn auto-heals after node restarts (and if it doesn't that is first and foremost a bug we need to fix), but also has a cluster monitor service which brings services back online automatically, and as a last gasp there is `flynn-host fix` for manual intervention.
Flynn has node promotion / demotion so you can totally do rolling upgrades (users have reported they do this with a modified CloudFormation stack for example). There is also a built-in updater to update to a more recent version of Flynn.
> Flynn doesn't seem to offer a metrics stream, only logs
Something I think should be built-in, but there are many off the shelf solutions which can be deployed to Flynn as apps.
> Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack
Flynn has built-in buildpacks but also supports running custom buildpacks (see https://flynn.io/docs/apps#buildpacks). If you just want to deploy binaries, you can either:
> I couldn't find anything about role-based access control in Flynn, something Cloud Foundry has (which can be backed by LDAP for enterprises)
Flynn does not currently have strict access control, but work is very much on the way, see our security improvements project: https://github.com/flynn/flynn/projects/1.
> Flynn only appears to route HTTP traffic, so no TCP routing for IoT workloads
Happy to see Flynn get more adoption, it got featured in the cron.weekly newsletter a few months back, when it was in the earlier stages; https://www.cronweekly.com/issue-39/
what gives you more than docker-compose itself? with docker-compose and docker swarm I can easily add services there, configure the application environment and scale it.
seems interesting but please enumerate things I can do that are not delivered by docker compose and docker swarm.
A main advantage of Flynn over using docker-compose is that Flynn is more aware of how to run applications out-of-the-box, rather than just running containers which the user has to fully configure.
For example, Flynn comes with PostgreSQL, MariaDB and MongoDB appliances which are deployed in a highly available, fault tolerant configuration with sane defaults: https://flynn.io/docs/databases.
Through the use of buildpacks, it is also straight forward to deploy applications written in many popular languages and frameworks without being intimately acquainted with how to containerise those types of applications (e.g deploying a Ruby on Rails app on JRuby: https://flynn.io/docs/languages/ruby).
You also get things like full cluster backups (including all your data), simple git / docker / tarball based deployments, log aggregation, release rollbacks, domain / path based routing, service discovery, and many more!
Both this and other PaaSes remove a lot of the plumbing. Also, the target audience is different. PaaSes offer self-servixe for end users: for instance starting a new database on demand through the user interface. You can achieve much of this with Docker Compose, but your users can't.
The Docker-PaaS movement reminds me of content-management systems in the early 2000s. Everyone (myself included) thought it'd be great to spend loads of engineering effort reinventing the wheel, instead of delivering business value.
The bit that made the decision for me is managing self healing, highly available postgres in the same infrastructure and tooling as the rest of my deployment.
I have yet to see any other PAAS come close to offering HA postgres built in.
I'm working on a similar service, but focused around my own open source stack: https://baasil.io/
The main issue with deployment/orchestration platforms is that they cannot handle ALL different kinds of stacks/apps (each one has unique scalability requirements).
With Baasil.io, our strategy is to get it 100% working and auto-scalable; one stack at a time.
I worked on Mantl for a while last year. To understand the difference you can see Mantl just as an opinionated coalition of tools for running microservices. So, Mantl is not a thing but a collection of things (Mesos, Marathon, Consul, ELK stack, etc) put together using Ansible playbooks.
Flynn is an actual product which, from what I can see, is also a collection of tools but these are written by the Flynn team themselves.
I'm happy to answer any other question you might have about it.
I worked on it for a few weeks while contracting at Cisco. At the time it was a set of tools for deploying a Mesos + Marathon cluster, and K8s was being looked at as a sub-in replacement for Marathon. I think that's now been done + much more.
There wasn't any particular magic in it, more that it was a neat package of terraform+ansible tooling for getting mesos and the surrounding ecosystem going. This was about a year ago, so things may have changed since then. I do like the idea of not inventing your own paas, but rather having a consistent framework for deploying whatever kind of paas you need on whatever cloud is available.
> having a consistent framework for deploying whatever kind of PaaS you need on whatever cloud is available
Indeed that is a great plus, no need to understand all the intricacies of each Cloud platform.
What's your thought comparing it to Heroku or Flynn from the perspective of the (user-)developer? It seems getting little traction on fora like HN or Cloud conferences. Why would that be?
> Is it possible to store objects in something similar to S3 (for example using Minio)?
Flynn has a built-in blobstore for storing files and supports multiple storage backends: PostgreSQL Large Objects (default), Amazon S3, Google Cloud Storage and Azure Storage (to be released into stable next week), see https://flynn.io/docs/production#blobstore-backend.
The documentation about the blobstore says PostgreSQL should be used only for small deployments. What if you deploy Flynn to DigitalOcean, which has no S3-equivalent?
They announced during their 1.0 release that Flynn is now production ready but I have to disagree at this time. I do honestly love the idea of running my own Heroku but it's just not stable enough and doesn't recover well if a node goes offline without manual intervention using the flynn-host fix command (and sometimes that command just chokes up completely).
Next time it happens (when, not if, at this rate) I'll try to set aside some time to gather up the logs to help debug the problem but I'm using Flynn to avoid the sysadmin side of things and just build my product. In my opinion Flynn is tinker-ready, not production-ready. It's infuriating that it has these random problems because like I say I really _really_ love the product and desperately want to be able to use it reliably.