My first gig out of college was maintaining Cobol at a bank for two years. Some of the programs were 20+ years old, and we made changes monthly - it always seemed like a fantastic rate of churn.
If something went wrong during a deployment, heaven help you. One time a pair of transposed digits got through testing and code reviews and into deployment. My colleague spotted it and called the management center to hold jobs... Thirty seconds too late.
That's basically what we did. The problem here was that processing was split into eight or so steps across ~25 streams of work between the backup steps and was far enough along the output files were generated, revving their generation number. Those files fed something like thirty follow-on jobs, so all the file generations had to be backed out before processing could run.
It all seemed horribly complicated and brittle to me at the time. Not sorry I don't work with it any more.
If something went wrong during a deployment, heaven help you. One time a pair of transposed digits got through testing and code reviews and into deployment. My colleague spotted it and called the management center to hold jobs... Thirty seconds too late.
It took four days to back everything out.