* Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer. If you use AWS RDS or Heroku Postgres, use Multi-AZ deployment. Be mindful of your costs though, because they can skyrocket fast.

* Minimize moving parts (e.g. databases, servers, etc..). If possible, separate your marketing site from your web app. Prefer static sites over dynamic ones.

* Don't deploy at least 2 hours before you go to sleep (or leave your desk). 2 hours is usually enough to spot botched deploys.

* Try to use managed services as much as possible. As a solo founder, you probably have better things to focus on. As I mentioned before, keep an eye on your costs.

* Write unit/integration/system tests. Have a good coverage, but don't beat yourself up for not having 100%.

* Monitor your infrastructure and set up alerts. Whenever my logs match a predefined regex pattern (e.g "fatal" OR "exception" OR "error"), I get notified immediately. To be sure that alerts reach you, route them to multiple channels (e.g. email, SMS, Slack, etc..). Obviously, I'm biased here.

I'm not gonna lie, these things make me anxious, even to this day (it used to be worse). I take my laptop everywhere I go and make sure that my phone is always charged.

[0] https://tryhexadecimal.com

I almost have the regex "fatal|invalid|unknown|error|except|critical|cannot" in muscle memory many years after having last had to type it - must have typed it thousands of times tailing and grepping logs :- )

It takes a bit to work this into the code and infrastructure everywhere but it’s worth it.

    %w[fatal invalid unknown error except critical cannot].any? {|word| 'My critical error message'.include? word}

At fail-over my local machine will also switch from the dev database to prod (connecting to the RDS instance). It'll only has read-only access to relevant data, but it'll keep the site 90% functional.

Is it perfect? Hell no. Pretty hacky, but it's kept my application running twice after a failure in the past two years.

This is exactly how I run my (also monitoring) SaaS and it shows that the OP has learned how to minimize risk and prepare for the worst.

You are married to your mobile & laptop.

I also shows there is no free lunch: you need to invest in redundancy.

I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8 hours every night and I'm not worried. I've been in contact with my customers and provided solutions to keep them alive. If they leave they're going to leave, nothing I can do. I am looking to make sure everything is built uniformly (the server that died was the last of an old build process) and invest in scaling to the cloud in a bit more efficient and orderly manner.

This is not a long term solution though. You need to add people and take a break at some stage.

Very clean and simple, but not trivial or bare. Really good job, jmstfv.

[0]https://tryhexadecimal.com/costs

I've been experimenting with functional (atomic) CSS lately and built a small framework that ended up using in Hexadecimal (if you peek at the source code, you can see weird classes like m-0, p-0, etc..). I liked it, but one should be careful in not overusing it, otherwise, it is an unmaintainable mess, yet again.

Below, I linked my Netlify config [0]. My static files live in the src/ folder, so you should probably change that if you have them in a different folder. If you don't know what Content-Security-Policy (CSP) or Strict-Transport-Security (HSTS) headers do, leave them out. It is easy to shoot yourself in the foot if you don't know what you're doing.

[0] https://privatebin.net/?c794ba90771f6e10#GSnJJ3dnd5r3ZDGgi9R...

Have you considered a trial extending to their first outage notification, rather than 14 days?

I would imagine if they don't see an outage in 14 days they're not seeing the value.

However if their trial expired following their first 'site is down' reminder - that's where your value comes in.

Just my $2c

I'll investigate that. Appreciate the suggestion.

It is highly unlikely that all of them will be knocked down at the same time.

If workers suspect that there is something wrong with your website, they will check it from all locations at once. Website is only marked as down if more than half of them can agree on that. That alone knocks down most of the false positives: https://tryhexadecimal.com/docs/uptime#sequence-and-frequenc...

* Account deletion (DELETE FROM \"accounts\")

* New successful sign up (INSERT INTO "accounts")

* New signup attempt (Started POST "/signup") [0]

* Fatal Exception Error ("fatal" OR "exception" OR "error")

* Warning ("warn")

I do share my costs and tools publicly, in case if you're interested: https://tryhexadecimal.com/costs

[0] I got hit by a botnet attack couple weeks ago, so I keep a tab on their activity (https://news.ycombinator.com/item?id=21327416)

https://devcenter.heroku.com/articles/background-jobs-queuei...

Kids these days.

I had a RAM stick fry in one of the physical machines sitting in a colo 1 hour drive away. Not die, but just start flipping bits here and there, triggering most bizarre alerts you can imagine. On the night of December 24th. Now, that was fun.

--- To add ---

If you are a single founder - expect downtime and expect it to be stressful. Inhale, exhale, fix it, explain, apologize and then make changes to try and prevent it from happening again. Little by little, weak points will get fortified or eliminated and the risk of "incidents" will go down. There's no silver bullet, but with experience things become easier and less scary.

If your workload doesn't fit into such a setup, then you simply need to take investment, optimize your code, or move closer to your colo

Insert punchcard anecdote...

Transistors! We had to replace valves in my day.

Billing issues. What happens if the credit card you use to pay for everything gets hijacked, and you're trapped with a blocked card trying to clean it up but your bank is taking their sweet time and won't give you another card until it's sorted? ALWAYS have a backup credit card.

DNS Registrar. There's a hard SPOF in the DNS, where your registrar essentially holds your domain name hostage. If your DNS gets hijacked, but your registrar is taking a few days to sort out who actually owns it, you're down hard. There's no mitigation for this one, except paying for a registrar with proper security processes. If you do 3FA anywhere, make it here.

AppStore. If your app gets banned, or a critical update blocked, what do you do? Building in a fallback URL (using a different domain name, with a different registrar, can help work around any backend issues. There's not much you can do for the frontend functionality, except using a webapp.

It can be worthwhile looking at risks and possible mitigations beyond just server and database issues, especially when it's just you.

This happened to me this week. Luckily bank got me the new card within 2 business days, but still was a bit stressful and I burned a day getting my payment info updated everywhere.

It doesn't even have to get hijacked, we had our bank reissuing cards because of some bug on the old chip, the card number stayed the same and the expiration dates changed but Google Cloud blocked the paying accounts. Security is the top priority sure, but I don't want some proprietary algorithms decide whether or not our server will be up or not. The fix was to move away from Google, because bouncing on their support took to long.

Random things will go wrong that you can't predict. Boxes will die suddenly and without reason, even after months of working fine without changes, and always at the worst possible moment. Your system needs to be built to withstand that.

I'll take the opposite approach of everyone here and recommend against serverless, kubernetes, and Heroku/PAAS.

You are a solo founder. You should understand your infra from the ground up (note: not understand an API, or a config syntax, but how the underlying systems actually work in great detail). It needs to be simple conceptually for you to do that. If anything goes wrong, you need to be able to identify the cause and fix it quickly.

I've gone through this first-hand and know all the trade-offs. If you'd like, I'm happy to discuss architecture decisions on a call. Email is in my profile.

If something “goes wrong” or you don’t understand how to implement something with managed services, support is just a ticket and a live chat/phone call away. I can speak from personal experience that AWS business support is great even when there isn’t a problem and you just want an “easy button” for someone to tell you what’s wrong with your configuration.

All in all, I think maybe I might have to find some other batch processing system.

But yeah I did have a doozy of an issue with ECS but it was completely my fault. I create a cross account policy for ECR but left out the account that actually contained the registry. Then my containers were in a private subnet without any access to the internet (by design they were behind a load balancer) but they couldn’t get to the ECR endpoint. I just had to either assign a public IP address or use a private link.

Support helped me with both.

My product is single-tenant, which can be tough infrastructure-wise since each app/customer needs a cluster of servers and services (Postgres and RabbitMQ). The Heroku pipelines enables me to have a testing app, staging app, and when I want to push staging to production, I can push the code to all my single-tenant production apps with the push of a button.

In theory I could do this with Bitbucket or Gitlab CI/CD pipelines, but this enabled me to focus on app development instead of devops.

That's just my preference, of course.

It depends on your availability requirements, for me, it's more important and simpler to be able to deploy again(with data) under hour than dealing with HA.

All my SaaS products run on a Windows server, with SQL Server as a database and ASP.NET on IIS running the public sites. You can probably come up with a lot of uncharitable things to say about those technologies, but "flimsy" and "fragile" likely aren't in the list.

As a result, when things go seriously wrong, the application pool will recycle itself and the site will spring back to life a few seconds later. Actual "downtime", of the sort that I learn about before it has fixed itself might happen maybe once ever couple years. At least, I seem to remember it having happened at least once or twice in the last 15 years of running this way.

There's a Staging box in the cage, spun up and ready to go at a moment's notice, in case that ever changes. But thus far it has led a very lonely life.

Even now, the only “Linux deployments” I do are either with Lambda or Fargate (Serverless Docker). I just don’t like managing servers. These days even my EC2 instances are disposable.

Many years ago we used BizSpark, and we wasted far too many days and dollars trying to understand and use the licencing (and using the abysmal web interface that was mandatory). We couldn't end up jumping through the right hoops to use products in production, even though were target audience for the products when we started our business. The licences had very broad exceptions that gave Microsoft the right to pull the plug at any time.

I'm sure things are different now, but I'm also sure they are the same.

A SQL database on Azure runs about $5-$15 a month, depending on how you configure it.

Once you throw in SSL certs and DNS management, you're probably looking at about $100 a month.

https://azure.microsoft.com/en-us/pricing/details/sql-databa...

Basic will do quite a lot; I use one to load test our application and haven't hit limits.

1) pagerduty.com or uptimerobot.com for remote monitoring to make sure you site(s) are up (and get alerts when they're not).

2) Datadog or New Relic if you want deeper monitoring (application performance, database performance, diagnostics/debugging.

3) Rollbar.com (site doesn't seem to respond) for site performance/errors.

4) Roll your own with Prometheus (https://prometheus.io/, or Nagios (https://www.nagios.org/)/IcingA. Or... strangely - I still use MRTG for a few perf monitoring things: https://oss.oetiker.ch/mrtg/

5) If you want to monitor the status of deploys/builds - I love integrating CI/CD systems with Slack - very helpful.

Hope that helps - I've spent a lot of my career monitoring things, and have this mantra that I need to know about services down before customers call to tell me same.

(a lot of these have free tiers)

I am used to rolling my own and use Nagios to make sure my servers and web sites are up and URLs and scripts functioning.

I used to use MRTG and RRDTool in days past when I was responsible for monitoring many servers, switches and routers.

Lambda functions can have cold starts that introduce latency. How do you manage that?

(From my small amount of experience - please prove me wrong.)

That still sucks, so then you can (hopefully) cache some things so that _some_ data begins to stream in. Or you can make it so your very high priority stuff has minimal dependencies - you can get a Lambda cold start in < 1s if your app only uses the standard library.

But still, in my experience, cold starts are a thing. If you have a high-traffic app or use Lambda warming, you decrease the # of people who experience a cold start, but at the end of the day, your p99 is going to be worse than a vanilla VM solution, because _some_ people will get cold starts. For some apps, that's OK - think line of business app where the first few pages can be served from static materials or cached materials, and you trigger the requests in the background.

Second, minimizing the function's code size. Both by making small functions and then optimizing them (there are plugins for the serverless framework to do that)

Third, using a language with a minimized cold start. Some can have cold start latences lower than 1 second [^1]

[1]: https://mikhail.io/serverless/coldstarts/aws/

But then again, I did already know the other fiddly bits of AWS.

I also work for a company that pays for the business support plan, they will help you to an extent with any weirdness in their own platform - but not with third party utilities.

Another advantage with using SAM, is that you can create and configure your lambda/API Gateway from the web console and then export your SAM/CloudFormation template.

All that being said, for APIs, I don’t use either. I would recommend using standard frameworks like C#/WebAPI, JS/Node, Python/Flask/Django instead and using the proxy integration libraries that AWS provides. It lets you develop/debug locally like you are accustom to and it gives you the optionality to move your APIs to Fargate (Serverless Docker) or EC2 instances without any code changes:

C#/Web API

https://aws.amazon.com/blogs/developer/deploy-an-existing-as...

Node:

https://github.com/awslabs/aws-serverless-express

Python/Flask

https://dev.to/apcelent/deploying-flask-on-aws-lambda-4k42

I’ve been told that there are similar frameworks for other languages but those are the three that I use.

If you have any other questions, feel free to email me. My address is in my profile. I’m not a consultant trying to sell anything....

Honestly, the answer is learning how to manage anxiety and stress, particularly doing potentially destructive things under pressure. I think the psychological aspects of this are much more difficult than the technical ones.

If it helps, people are generally very understanding if you explain that you are a solo founder, and take reasonable steps to fix issues in a timely way. Most customers assume every company is a faceless organization; their attitude is much more forgiving when they learn they're dealing with a fellow person.

You cannot be on call 24/7 forever. You will burn out. If you can't hire someone you trust to take over part of this burden, then you have to accept the risk of sometimes not being able to log in for N hours if there is an outage (because you're camping with your spouse, etc.)

For very high-stress situations (database crash, recovery from backup) working from a checklist that you have tested is very valuable.

Good luck to you, and I hope you found useful answers in this thread!

I concur that hiring someone, if you can afford it, even part time, would be a great idea.

Many many businesses can survive on a $25 a month 1x dyno and the free database tier (and free sendgrid, free newrelic, free whatever other addon) and can be pretty sure their site will basically never go down.

It's a fantastic security blanket, but...as your business grows you'll start to pay. The question then becomes "If I do all this myself on AWS (or azure or gcp) how much time am I going to be sacrificing time against building my business dealing with random infrastructure crap?" Or "at what point does it make sense to hire someone to focus on all this infrastructure and how much would that cost vs. just paying for heroku?"

Before thinking about handing over management of the deployment, I would encourage you to think about what the root cause of the outage is and whether something in the app will create that situation again. I invested in setting up DataDog monitoring for all hosts with alerts on key resource metrics that were causing issues (CPU was biggest issue for me).

The other thing that's worked well for me is just keeping things simple. As a solo founder, time spent with customers is more valuable than time spent on infrastructure (assuming all is running well). It's a little dated, but I still think this is a good path to follow as you're building your customer base. A simple stack will let you spend more time learning how your product can help your customers best.

http://highscalability.com/blog/2016/1/11/a-beginners-guide-...

Also if you send custom stats to datadog such as login activity, you can use their Anomaly detection to find suspicious behavior.

Disclaimer: we use datadog at my company and have tried all the other popular options. Hands down datadog is the most feature rich and user friendly.

Just getting logging centralized alone has saved me tons of time, which is in turn more time spent on the product. I've been able to use the log parsing to setup metrics that tell me when an outside integration is acting up and isolate which paths. Take a day to really learn how their logs work and you'll be able to generate metrics / advanced event alerting in no time.

I was hesitant to pay the premium, but the peace of mind has been worth it. You can piece the same thing together with open source tooling. But then you've got another thing to manage.

- metrics system (Prometheus, InfluxDB, Graphite, etc.)

- dashboards (Grafana)

- alerts (both the metrics system and Grafana can handle this)

If you're not a big-data company, you can self-host instances of these products reliably.

Disclosure: My startup, https://HostedMetrics.com, provides turnkey hosted versions of these software packages.

I'd be happy to explain the options that are out there and provide any advice you need. Get in touch: contact info is in my profile.

My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I was trying to insert IDs into a uuid database field, but this new data had non-uuid IDs. Because the original IDs were always generated by me, I was able to guarantee that the data was correct, but this new data was not generated by me. Of course, sufficient defensive programming would have avoided this as this database error shouldn't have prevented other stuff from working, but my point is that mistakes get made (we're humans after all) and things do get overlooked.

The problem is, restarting my service doesn't prevent this external data from getting received again, so it would simply break again as soon as more is sent and the system would be in this endless reboot loop until a human fixes the root cause.

That's a problem that I worry about, no matter how hard I try to make my system auto-healing and resilient (I don't know of any way to fix it other than putting great care into programming defensively), but again, we're human, so something will always slip through eventually...

Some people are suggesting to out-source an on-call person. That seems to me like the only way around this particular case. (The other suggestions can still be used to reduce the amount of times this person gets paged, though)

There's two ways to think about this:

1 - Your product might actually be too complex for a single-person business. You could rotate being on call for situations like this. This means that you'd have to make sure that sales are big enough to support an additional parter or two.

2 - Perhaps you need to simplify your product? Think more critically about error handling? I don't know the details about this part of your service, but if I assume that these bad UUIDs came from HTTP POSTs, why does a series of wonky HTTP posts bring down your entire service? Typically, something like this would trigger some kind of unhandled error that's caught higher up in your web framework and returns some kind of 5xx error.

This paragraph is very C# centric, but it should translate to other languages as well: Typically, I layer my error handling. Each operation is wrapped in a general exception handler that catches EVERYTHING and has some very basic logging. (ASP.Net does this and returns a 5xx error if your code has an unhandled exception.) Furthermore, as I get closer to actual operations that can fail, I catch exceptions that I can anticipate. Finally, I have basic sanity checks for things like making sure a string is really a UUID.

Without knowing much of your service's architecture, it just sounds like you need some high-level error handling. You probably have 100s of other little weird bugs, so high level error handling needs to do the equivalent of returning a 5xx error and logging, so you can fix it when you're able to.

The difference with having a large team is less that all possible failure cases will get protected against (although more eyes and code review does help), but more that someone can always be available to fix it when something unexpected happens.

In my particular case, the majority of the system kept running fine. The part that failed was a streaming system which receives updates in realtime from an external system. The error actually was localised to one particular type of updates, but that type stopped working because I didn't protect defensively enough against errors in that one particular case (I do have my database queries protected against errors, but this one slipped through). This caused other systems to not get these updates, so things that relied on them stopped working. Its not that they crashed, they just never received the updates they were waiting for.

Of course the fix is to trap all exceptions, log/notify, ignore and continue, so that at least one piece of bad update doesn't affect other updates, but again, my main point was that we're human, so can't possibly protect against everything that might cause a non-recoverable (without human intervention) error.

> Finally, I have basic sanity checks for things like making sure a string is really a UUID

Yes, I did add this too after I hit this issue and its a good point: validate EVERYTHING even if you generate it and think you can assume it will be good.

> Don't work alone!

That's the real solution, but sometimes its not possible.

Thanks for your detailed response, though, its appreciated.

Is your product too complicated for a single-person business?

As a solo programmer, I can write and develop extremely complicated systems. These systems can be so complicated that I don't have time to run them, find customers, support customers, ect.

That, ultimately, is why I don't see myself running a single-person business anytime soon. I really enjoy complicated programming, and if I have to also handle ops, support, sales, ect, then what I program needs to be too simple to remain interesting.

I employ a healthy dose of exception trapping and logging, and I get an email whenever it happens.

People aren't perfect, but you can anticipate a lot of failures. It usually involves bad data as in your case. Each time you get bitten, change your code so it fails gracefully.

This is how we handled errors in our video archiving system at Justin.tv, and to my knowledge we never lost a single frame that made it to the broadcast servers. The raw bits were streamed to disk as they came in, and only got removed once the VOD servers had the final version— any errors would retry a few times and then get flagged for manual processing. We did have a few close calls where something broke the whole archiving system, and the broadcast servers came dangerously close to shutting down due to full disks, though.

I handled it with a pipeline that did the following, 1. validation, 2. transform data if needed, 3. load the data. If validation failed, the data would get "quarantined" and I would get an email notification, or slack notification if urgent.

I don't generally write a lot of unit tests, but all my validation and transformation logic would have 100% coverage because things always change and you need to make sure future updates never break the system

I assume also you want a simple way to increase reliability while keeping costs within reasonable limits.

Well, AWS can give you all that if you don’t want to go super fancy. Check Beanstalk to get something simple and reliable. Monitor using CloudWatch. Make sure to leverage redundancy options (multi az, multi region if worth it, etc). These are some general tips but with the information that you provide that’s all I can say.

You can also pay a consultant to get a review of your setup and get some recommendations. It won’t be cheap but it depends how much you value your time and your product.

The question you should be asking is, how can I make my service automatically recover from this problem. It depends why exactly it crashed. If a simple restart fixes the problem, there are different ways you can automate this process, like Kubernetes or just writing scripts.

I’m happy to give more detailed advice if you would like, my email is in my profile.

The question is not whether your system will fail, the question is when.

Have proper monitoring and alerting in place.

But don't over engineer it, sometimes everything seems technically fine, but your support inbox will start getting user complaints.

Resolve the issue, figure out the root cause, make sure this or similar stuff won't happen, apologise to the affected users if necessary, and move on.

You'll learn waaay more failure modes of your application running in the wild, than just thinking about "what could go wrong".

It's a long game of becoming a better developer/devops guy, and not repeating the same mistakes in the future.

On the set up, try your best to solve issues and use tried and true hardware, but things go down sometimes, even big sites like Google, Facebook go down, there is no silver bullet, you can only improve on your past mistakes.

Last, try to find some remote help, on a contract basis, it's not that expensive and it can help alleviate a lot of your stress.

If it's truly critical to have no down time then you probably need to build that resilience in to your architecture.

https://news.ycombinator.com/item?id=6576250

Using serverless, PaaS like Heroku or similar will help.

I use a python monitoring script that tails logs watching for ALERT level log lines and constant order activity combined with a cron watchjob to ensure the process is alive during trading hours. The exception handler in the monitoring script sends alerts if the script itself dies.

If there are any issues I use twilio to text me the exception text/log line. I also use AWS SES to email myself but getting gmail to permanently not block SES is a pain in the ass. By design Twilio + AWS SES are the only external dependencies I have for the monitoring system (too bad SES sucks).

On my phone I have Termius SSH setup so I can log in and check/fix things. I have a bunch of short aliases in my .profile on the trading server to do the most common stuff so that I can type them easily from my phone.

I also do all my work through a compressed SSH tmux including editing and compiling code. So if things get hairy I can pair my phone with my laptop, attach to the tmux right where I left off, and fix things over even a 3G connection.

This compressed SSH trick is a huge quality of life improvement compared to previous finance jobs I've worked where they use Windows + Citrix/RDP just to launch a Putty session into a Linux machine. It's almost like finance IT has never actually had to fix anything while away from work.

It doesn't prevent an app-level outage (corrupted data in the database, bad architecture,...) but at least I don't have to worry about servers going down anymore.

As for the rest, unit & extensive integration tests along with continuous integration and linting. Oh, and a typed language. Moving from Javascript to Typescript was a blessing. But I still miss Swift.

- Atleast have a pool of 2 instances (ideally per service) running under an auto-scaler or a managed K8s (GKE is best) with LB in front. May also want to explore EBS and google cloud run. If you can use them, use them!

- Uptime alerts. pingdom (or newrelic alerts) with pagerduty added.

- Health checks! The trick is to recover the failed container/pod/service before you get that pagerduty call. Ideally, if you have 2 of each service running #2 will handle the requests until the #1 is recreated.

- Sentry + newrelic APM + infra: You should monitor all error stack traces, request throughput, avg response time. For infra, you mainly need to watch memory and CPU usage. Also on each downtime, you should have greater visibility at what caused it. You should set alerts on higher than normal memory usage so you can prevent the crash.

- Logs, your server logs should be stored somewhere (stackdriver on gcloud or cloudwatch on aws).

These might sound overwhelming for a single person but these are one time efforts after which they are mostly automatic.

If you have a good idea of the usage patterns of your service, create metrics backed by the patterns. This can help you find things that CPU/Memory will hide.

2. Pay for a Business Support plan. https://aws.amazon.com/premiumsupport/pricing/

3. Call business support about something "how do I restart my server" - so you know how to file a ticket, get a feel for how quick the response is and how it works.

Do not over think this. EG: terraform templates

My only concern was that my target was very low-cost setups, and I wanted something like Packer but let me provision multiple images onto a single machine. Eg, if I just used Packer, as far as I could tell I would have to have 1 machine per image. It sounds odd, but I didn't want to pay $5*Services, especially when the number of users and load was very small. Being able to deploy in a PaaS fashion to something like Docker on the machine seemed best.

But then I was looking at Terraform + Packer + DockerSomething, and things went back to feeling less simple.

I haven’t used Packer, but could you use CodeBuild/CodeDeploy/CodePipeline? Again if you’re on AWS you might as well take advantage of their support.

You can deploy Docker images using CloudFormation either to Fargate and not have to worry about servers or to EC2 instances (? I haven’t tried).

I've been in the game for a while and every time I run across an idea for a service, there's always a question of whether I'd be OK with sleeping with a pager, remoting to the servers at 4 am on Saturday and generally be slaved to the business. The answer, upon some reflection, is inevitably No. This is the domain of teams.

I'm in the same boat with my solo founder projects (links in profile).

The short answer: I'm married to my phone/laptop.

My test coverage is good. I use managed services when possible so I don't need to play sysadmin. I don't deploy before I leave for something (dinner, shower), and I have some pretty good redundancy across all my services. If one node goes down, I'm safe. If four go down (incredibly unlikely), well, fuck, at least my database was backed up and verified an hour ago.

I invested a large amount of time into admin-y stuff. My admin-y stuff is solid and I can tweak/config/ccrud anything on the fly. I credit being able to relax thanks to my admin-y stuffs. Obviously, if shit really hits the fan with hardware or an OS bug, I need to get to my laptop. But over the last six years, I haven't had to do that yet, and hopefully I won't have to.

I've explored adding staff — mainly for day-to-day operations — but I like the idea of interfacing with my customers and I credit growing things to where I have because I'm in the trenches with them. Things haven't always gone smoothly, and my customers always let me know, but any issues are normally swiftly-resolved.

The scale of one of my products is non-trivial and has a ton of moving parts — some of which I'm in no control of and could change at any time and break _everything_. It sounds terrifying, and it is, but I've made a habit to check things before peak hours. If something's amiss, a quick fix is usually all it takes.

I have a few pieces of advise:

1. Make sure your service can safely fail and be restarted. What I mean is, if somebody is POST'ing data or making database changes, make sure you handle this safely and attempt some recovery. Something not being fully processed is okay as long as you are able to handle it.

2. Self-monitoring. I run all my systems inside a simple bash loop that just restarts them and pop me an email (i.e. "X restarted at Y" and then "X is failing to start" if it continues).

3. External monitoring via a machine at home that rolls the server back to a previous binary (also on the server). It also pulls various logs from the server, as well as the binaries, so they can be analyzed. Okay, it has some reduced functionality, but it's stable and will keep things going until the problem is fixed.

4. Make sure your service fails inconveniently - i.e. returns a `{"status":"bad"}` string or something, or defaults to a "Under maintenance page, please come back soon". Your service going down is one thing, but becoming completely unresponsive is quite another.

One thing I can't prepare for (which happens more than you think) is the server itself crashing, which as you say, means I'm randomly logging into a VPS console and rebooting. I use a bunch of different VPS providers and every one of them has a slightly different console.

You can do it at the OS level: on a windows OS for example: you use EventViewer and assign a task to specific type of log captured by the OS this task can then invoke a small app that sends emails if an error-log occurs or something like that

    Application specific issue:
        you can manually capture exceptions raised within the app and send notifications
            there are many clever ways to do this and not hinder performance, and also not pollute your code-base with exception handling
                you can spawn "fire and forget threads" that send notifications ...
                let me know if need more ideas here

    Integration tests:
        given that you've built a strong suite of integration-tests covering all the functionality on your app
            you can have have your integration tests run every 15min or so and send notifications if tests fail

Managed support will usually only monitor and fix basic infrastructure and respond to support requests from you. They often won't monitor or fix your applications/services; for that you can set up your own application monitoring and tests. NewRelic is a good all-in-one choice, but there are plenty more out there. To call you during an incident, you'd also adopt PagerDuty.

In order to avoid service outage in general, you want to hook up some kind of monitor to something that re-starts your services and infrastructure. This will only fix crashes; it won't fix issues like disks filling up, network outages, application bugs, too many hits to your service, etc.

You should be able to find small businesses who specialize in selling support contracts for all levels of support. By signing a contract and on-boarding a 24/7 support technician, you can get them to do basically whatever you need to be fixed when it goes down. I don't have suggestions for these, maybe someone else does (it used to be common for SMBs in the 2000's).

If I were you, I would use free monitoring services like uptimerobot. There are some other options available. Typically these services provide some basic functionality for free, it would be enough for a small enterprise.

On AWS it is quite easy to create your own external probes for a reasonable price. However, it would require some basic programming skills.

#!/bin/bash

thisHtml=`curl -s "[your site's web address]"`

if [[ $thisHtml != •"<title>[your site's title]</title>"• ]]; then

#echo "Server is down"

ssh -i "[your pem file]" -t ec2-user@[ip address] 'sudo /bin/bash -c "killall -9 node"'

ssh -i "[your pem file]" -t ec2-user@[ip address] 'sudo /bin/bash -c "export PATH="/root/.nvm/versions/node/v8.11.2/bin:/usr/lib/node:/usr/local/bin/node:/usr/bin/node:$PATH" && forever start /var/www/html/[...]/index.js"'

rebootDate=`TZ=":[your time zone]" date '+%Y-%m-%d %H:%M:%S'`

echo "$rebootDate" >> "/home/ec2-user/serverMonitoring/devRestarts.txt"

fi

Yes. There are consulting shops that will do this, as will many of the monitoring tools listed in the thread (though these tools will not fix the problem for you). Broadly speaking, there is a cost associated with this, as well as the cost associated with your downtime. If the cost of your downtime (reputational risk, SLA credits, etc) outweighs the cost of hiring someone to cut your MTTR to 5 minutes (assuming you can playbook out all of the relevant scenarios) + provides some value in stress reduction, then you should do this. If you've been doing this a while, you can math it out. In what experience I've had though, an outside person is unlikely to be able to fix an "unknown unknown", they just won't know your environment as well as you will.

All that said, one hour of service interruption a year is still better than most.

The idea that your server does not perform regular health checks or spin itself back up when it fails just seems weird to me now. I like being spoiled.

Edit: I forgot about HyperiotOS, which is another viable option. MicroK8s has Canonical’s backing so it has quickly become the most well-supported option. It advocates single-node clusters for development but it supports multi-node as well.

To maintain portability you need to pay attention to CPU architecture and the underlying OS. Just make sure that your images are built for the same architecture that you will use in the cloud and that you have the right OS and version. x86_64 is ubiquitous (I think AWS offers an ARM server option, though) so developing on a Raspberry Pi cluster might be a headache to deploy.

For production you will want to use a cloud provider (or not, it’s not the craziest idea to run a home server if your use case permits) and you can do so for development if you can afford it.

The real advantages are service discovery, monitoring options, stateful support and resource allocation. I used Kubernetes Up and Running 2nd Edition, and if you don’t know Docker yet I would work through the Docker Book first because you will want to write your own images. Just make sure that everything lives in the same repository: a single filesystem as the single source of truth.

The advantage of K8s is that it abstracts so much away from you, that you should (in theory) be able to take the same YAML config file from AWS EKS to GCP GKE to Azure AKS...and it runs the same everywhere. Things like loadbalancering and HTTP ingress rules that normally would manual configs on each platform - become part of the K8s config.

This can be as simple as just using Google's GKE service.

Ultimately you can engineer your systems, even if they are quite complex, to be manageable by a single person. It's not one thing though. It's years of experience and gut feel. It's also totally distinct from technology.

Some things that come to mind:

- use queues for background tasks that may need to be retried. If things go down and you have liberal retry policies, things should recover.

- use boring databases. Just stay away from mongo and use something like rds which is proven and reliable.

- be careful in your code about what an error is. Log only things at the error level you need to look at.

- test driven development. Saves a ton of time.

It's all the matter of defining requirements, then solutions and tradeoffs of those solutions and then implementing it with best practices in mind (automation, testing, monitoring, backups, etc.).

Hit me up if you want to discuss it over a pint! :)

That's what we provide at Clever Cloud BTW https://www.clever-cloud.com/

I feel like we over engineer that part. Sure there's plenty of service where you don't want any downtime and it makes sense to over engineer it (like any monitoring service) but for many SaaS, the worst that will happens is a few emails.

Maybe write a simple SLA, something with a 8 hours response over theses kinds of outage. If some client require more, than sell them a better SLA at an higher cost. That should let you invest into better response time for sure.

No cloud machines, no hosted cloud services for production beyond DNS.

* 3 machines in separate data centers (equivalent of AWS AZs) for >= 30 EUR/month each. ECC RAM.

* These machines are /very/ reliable. Uptime of > 300 days are common, reboots happen only for the relevant kernel updates.

* Triple-redundancy Postgres synchronous replication with automatic failover (using Stolon), CephFS as distributed file system. I claim this is the only state you need for most businesses at the beginning. Anything that's not state is easy to make redundant.

* Failure of 1 node can be tolerated, failure of 2 nodes means I go read-only.

* Almost all server code is in Haskell. 0 crash bugs in 4 years.

* DNS based failover using multi-A-response Route53 health checks. If a machine stops serving HTTP, it gets removed from DNS within 10 seconds.

* External monitoring: StatusCake that triggers Slack (vibrates my phone), and after short delay PagerDuty if something is down from the perspective of site visitors.

* Internal monitoring: Consul health checks with consul-alerts that monitor every internal service (each of the 3 Postgres, CephFS, web servers) and ping on Slack if one is down. This is to notice when the system falls into 2-redundancy which is not visible to site visitors.

* I regularly test that both forms of monitoring work and send alerts.

* Everything is configured declaratively with NixOS and deployed with NixOps. Config changes and rollbacks deploy within 5 seconds.

* In case of total disaster at Hetzner, the entire production infrastructure can be deployed to AWS within 15 minutes, using the same NixOps setup but with a different backend. All state is backed up regularly into 2 other countries.

* DB, CephFS and web servers are plain processes supervised by systemd. No Docker or other containers, which allows for easier debugging using strace etc. All systemd services are overridden to restart without systemd's default restart limit, to come back reliably after network failures or out-of-memory situations.

* No proprietary software or hosted services that I cannot debug.

* I set up PagerDuty on Android to override any phone silencing. If it triggers at night, I had to wake up. This motivated me to bring the system to zero alerts very quickly. In the beginning it was tough but I think it paid off given that now I get alerts only every couple months at worst.

* I investigate any downtime or surprising behaviour until a reason is found. "Tire kicking" restarts that magically fix things are not accepted. In the beginning that takes time but after a while you end up with very reliable systems without surprises.

Result: Zero observable downtimes in the last years that were not caused by me deploying wrong configurations.

The total cost of this can be around 100 EUR/month, or 400 EUR/month if you want really beefy servers that have all of fast SDDs, large HDDs, and GPUs.

There are a few ways I'd like to improve this setup in the future, but it's enough for the current needs.

I still take my laptop everywhere to be safe, but didn't have to make use of that for a while.

You enable them to do it for you by creating HA infrastructure. Start by creating an autoscaling group that enforces a certain number of working application endpoints. You probably need an alb too. An app endpoint that fails healthcheck causes the asg to spin up another instance and auto-register with the alb. (You can snapshot your configured and working app endpoint as the base image).

It's just extra redundancy in case something like cloudwatch (which you should use -- with ELBs) also goes down.

It's not perfect, but it's (1) cheap (2) easy (3) quick (the mythical trifecta). It misses some of issues due to high loads (but still technically available) but works perfectly when things actually crash (like queue workers deciding to turn off).

Docker, elastic beanstalk, SNS, and the hidden world of AWS instance performance are all a PITA. Oh yea, certs...

Welcome help as well.

It would be nice to have a stateless tool abstract Terraform a bit, to let you use more providers as a basic PaaS.

I also warns me if the CPU load goes up over 80%.

For the first two years of going live I had this hardwired to my Pebble via real-time mail, but now I know my platform is robust; so I can choose worry about other things.

(though this is essentially a single-line comment, it's earnest, not intended to be sarcastic)

For monitoring, I am using Stackdriver which has easy-to-use health check.

It’s important for critical services, yet if you lose your 2FA device, like a phone, you will be locked out for a while. Like many things, it will happen at a bad time.

Also, send errors through chat platforms like Telegram to be notified of any errors/monitor the servers

These are both reactionary but at least you'll know if things break.

Other options includes using another service that offers 24/7 uptime. Obviously you pay more for that.

This approach is easy to implement and scale

https://en.wikipedia.org/wiki/Bus_factor

Otherwise I'd suggest religiously documenting your outage root causes and contemplating hard what could've avoided that outcome.

Then lastly for monitoring on the cheap:

Sentry.io - alerts.

Opsgenie - on-call management.

Heroku+new relic - heartbeat & performance.

Tldr; Keep your stack small and nimble and try to learn from past outages

Otherwise you allow the problem to persist, pile up with other issues (also fixed by restarts, I assume) and implementing automated restarts in that manner reduces not only your uptime in uncontrollable manner, but also your code/infrastructure quality, increasing your tech debt beyond the point of recovery.

Friends don't let friends fixing things by restarting them ;)

I’m speaking in terms of AWS translate to your chosen infrastructure.

At the bare minimum you should have two redundant servers behind an autoscaling group with a min/max of two with health checks.

When you need to get something up now and you want to keep the crash state, configure the crash instance to be taken out the autoscaling group but not terminate and start up a new instance. You can then troubleshoot.

I assume you have a separated logging mechanism, where all logs are collected, independently from the restarted service. Don't forget to log the state of your system as much as possible for post-mortem analysis