$600,000, pfft, try £350 million, Niall Fitzgerald at Unilever.

We launched a product with a wonderful new element in it, which had the capacity to clean clothes better than anything that had ever been seen. Unfortunately, if it wasn’t used in exactly the right spec, it cleaned so well that it cleaned away the clothes as well...I was personally responsible for Unilever’s largest marketing disaster.

He quoted his Chairman’s response to this at the time:

We’ve just invested £350m in your education. If you think you’re going to take that somewhere else, forget it.

An old coworker at a clearing firm once accidentally rebooted two very critical servers during a very critical time. From what I'm told, it ended up costing the company about a billion dollars in fees and fines.

He managed to keep his job.. just with no more prod access. :)

Sounds like someone else’s fault, unless he owned and designed the system. No single engineer should be able to cause that kind of damage. Quorum rules, etc.

It was a team of five senior level linux sysadmins who oversaw about 2k servers. Lots of arguments for and against pushing blame here or there, but at the end of the day - he fucked up big time and shoulda known better.

I’m sure that’s true, but my point is that if you have that much money riding on a system you should have to figuratively (if not literally!) put two keys in and turn the lock at the same time to break shit. There should be systematically enforced mandatory reviews, two plus person policy for issuing commands, etc.

You have to expect people to make mistakes. I’m not saying he didn’t fuck up, but if a company is down a billion dollars the story should be of multiple people making multiple mistakes.

You're preaching to the choir. There's a reason the company was bought and completely gutted.

For what it's worth, a billion dollars for this company, while still a lot, isn't world ending. It was probably insured, too.

When people easily make mistakes like that. It’s usually the systems/process/design fault.

Hm. I mean yes, but still - the guy made a legit rookie mistake by not checking the hostname of the host he rebooted before typing "reboot". Kinda 101 stuffs there. :/

One imagines that if he were fired, he could have blown the whistle on the incompetent CIO/CTO who set up such a weak process.

A billion dollars in fines is an unfathomable mistake! It's on the level of a natural disaster, like a hurricane!

Would love to hear more about that :)

That's all I know, really. It was one of the strangest companies I've ever worked at.

Another story from the same place (won't confirm nor deny if it was the same sysadmin) - someone accidentally pushed a puppet config out that changed all hosts' timezones to CDT. A bit later, I received a random and kinda joking text from a buddy from a past job - who couldn't get an answer from my company's tech support - asking why all of their clearing reports had the wrong times.

He was the only person outside of three other sysadmins who ever mentioned anything was wrong. Not even my boss brought it up! It was odd.

What's a clearing?

Clearing (and settlement) is the process of delivering securities and receiving the cash after a securities transaction (ex: stock purchase).

This quote needs a date. It is from, at the very latest, maybe 1940.

Inflation-adjusted, that's like $10 million.

Why would it be 1940 at the very latest? He ran IBM till well into the 50s. It has multiple variants, the longer ones involving a sale and a million dollars.

Woah, what was IBM up to back then?

... business machines?

https://en.wikipedia.org/wiki/Tabulating_machine

Cult of personality: https://arstechnica.com/information-technology/2014/08/tripp...

Genocide. Dehomag and Watson Business Machines.

producing technology and selling it to the third reich

https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

But what about moral hazard! Interestingly, this quote is less and less applicable the more it spreads.

It's about intent. Was the guy's intent to bring the site down? Does he have a track record of making such mistakes? Do they demonstrate a lack of concern or remorse? No?

Then blame is irrelevant. It happened, they learned from it, they now have that experience under their belt.

"What happens if we spend a million training this guy, and he goes somewhere else?"

"What happens if we don't spend a million training him, and he stays?"

Wow, I went the wrong way trying to establish job security.