My manager had got me to look at backups.. but for cheap.
I decided on bacula - I had the clients installed on all the computers in the office, and it worked for some small tests.
My manager decided we would try this with a USB drive attached to one of the servers (somehow this didn't seem like a bad idea).
In the morning, very uncaffinated he sent me to the other site - an unmanned basement office with the servers.
Being uncaffinated I forgot the door password and set off the alarm.
I had to go into the office and phone him with the alarm going to get the code to turn the alarm off.
OK, that was stressful but sorted out at least.
I plugged in the hard drive to the selected server and headed back.
Once I got back it turned out all the websites on that server had gone down - trying to send all the backups to this poor USB harddrive had overwhelmed the IO on that-era Linux server and the poor thing just froze.
Fairly soon after I was let go, and joined my friends at a much more fun company making mobile games.
It's the responsibility of the technical person to uphold engineering ethics, especially in the face of potentially inadequate recovery and security solutions.
I was once let go from a big name university for refusing to weaken and rush changes to a payment processing network (PCI-DSS) when there was "no time" to review them in detail. That's a future FBI press conference sort of thing when it all comes crashing down. Not long after, all SS#s, DOBs, and deets for every employee was stolen from a "rogue" laptop taken by a consultant, likely to be sold on carder and identity theft forums because of an utter failure at data protection processes. That place was a shitshow because they didn't have the professional ethics or leadership backbone to do what was prudent and necessary.
Yeah, for a surprisingly long time, linux had a big write cache problem. You were okay if you used all fast-writing devices. You were okay if you used all slow-writing devices. But if you mixed them, the slow writes could totally fill the write cache and starve everything else.
Not only an USB (1?) could cause it but also a cifs transfer over a 100 Mbps link.
I decided on bacula - I had the clients installed on all the computers in the office, and it worked for some small tests.
My manager decided we would try this with a USB drive attached to one of the servers (somehow this didn't seem like a bad idea).
In the morning, very uncaffinated he sent me to the other site - an unmanned basement office with the servers.
Being uncaffinated I forgot the door password and set off the alarm.
I had to go into the office and phone him with the alarm going to get the code to turn the alarm off.
OK, that was stressful but sorted out at least.
I plugged in the hard drive to the selected server and headed back.
Once I got back it turned out all the websites on that server had gone down - trying to send all the backups to this poor USB harddrive had overwhelmed the IO on that-era Linux server and the poor thing just froze.
Fairly soon after I was let go, and joined my friends at a much more fun company making mobile games.