I've been thinking about this quite a bit lately. I've run DevOps for a few organizations and learned quite a bit through that.
Ultimately you can engineer your systems, even if they are quite complex, to be manageable by a single person. It's not one thing though. It's years of experience and gut feel. It's also totally distinct from technology.
Some things that come to mind:
- use queues for background tasks that may need to be retried. If things go down and you have liberal retry policies, things should recover.
- use boring databases. Just stay away from mongo and use something like rds which is proven and reliable.
- be careful in your code about what an error is. Log only things at the error level you need to look at.
Ultimately you can engineer your systems, even if they are quite complex, to be manageable by a single person. It's not one thing though. It's years of experience and gut feel. It's also totally distinct from technology.
Some things that come to mind:
- use queues for background tasks that may need to be retried. If things go down and you have liberal retry policies, things should recover.
- use boring databases. Just stay away from mongo and use something like rds which is proven and reliable.
- be careful in your code about what an error is. Log only things at the error level you need to look at.
- test driven development. Saves a ton of time.