The story falls on the category of "it won't happen until it happens". Testing is expensive and even with all the testing in advance we observe emergent behaviors that make us run to the reams of logs to ask the very simple question: "What happened?" My take away: If you have to fail, because you will, fail graciously. Good post.