The real issue here has nothing to do with Linux or Windows. Rolling out updates to production without testing them first is a fail, not matter what OS you're using.
If they'd tested the McAfee definitions on even one of the POS terminals before rolling them out to all the restaurants everything would have been fine. They could have even automated having one restaurant get the definitions one or two days before all the others so only a single location was affected. Or better, set up staggered updates within each restaurant, so they all stay up and at worst lose a single POS terminal.
> Rolling out updates to production without testing them first is a fail
Deploying McAfee in the first place to all those terminals is a painful process by itself, yet many people do it because of the "extra" security it provides. When you can't trust something like McAfee for your production, then you're doing something wrong, and that's not the lack of testing upgrades to McAfee ;)
IMHO, the WTF here is that they've used Windows in the first place for what are in essence dumb terminals that help them manage drive-through orders ... for that you can always deploy a highly customized and minimal Linux OS, with most of the ports blocked and with a minimal number of services running, with reduced capabilities and enhanced with Linux SE if you're a security freak ... making the area of attack for any potential threat a lot smaller.
Then you wouldn't need a piece of shit like McAfee to handle your security ... all anti-viruses lately are pieces of shit IMHO.
> The real issue here has nothing to do with Linux or Windows
I think it does. On that terminal they could've had a small Linux image providing a ssh server, whose purpose would be to boot into it in cases of emergency. And you can control a Linux instance a lot more efficiently remotely. Try doing that with Windows.
Without knowing why they chose a Window POS system, claiming Linux would have been better, or was even an option, is pure speculation. Give them the benefit of the doubt and assume they had reasons that made sense.
It's awesome that Linux saved the day and that the author was resourceful enough to pull it off, but the whole event still could easily have been avoided (and without having to change much at all).
As nice as it is to blame McAfee, only the end user can control the extent to which third parties can affect them.
At Loopt, we stagger both Windows and Ubuntu updates for this very reason. We do the same with our own code updates. It costs very little and has saved us on numerous occasions.
You are right, but it was McAfee's fault, they messed up the testing in the very first hand. You can't test everything yourself, this update was a paid service of a major IT company.
From my time working at McDonalds back in highschool I can tell you they had solved this problem already. The head machine in the back office was running AIX, and all the POS terminals netbooted a linux variant from it. There was also a secondary server which no one touched that did the most critical job of receiving, tracking and reporting orders. Windows on a dumb POS terminal? That just sounds like a straight up bad idea to me.
At one of my first jobs (before college), I worked at a huge multiplex theater.
About six months after I started, they replaced all of the POS terminals that had been there since I began with new shiny ones.
New shiny ones that were dumb terminals to virtual Windows XP instances running out of the back office. Unfortunately, there were approximately 20 POS terminals in the place.
And every single one of them was on the same collision domain. All the Cat5 was ratnested under the counters; some of them were held in to the plethora of cheap 5-port hubs by duct tape. In the networking cupboard behind concessions, the wires were coated in old "butter flavored topping." Occasionally, all the POS terminals in the box office would cut out - usually because someone in the back office accidentally knocked over the five-port hub back there, and one of the connections fell out.
Needless to say, management was super excited to have these shiny new POS terminals. Everyone who had to use them? Well.. we all did our jobs a little bit slower: it's hard to ring up a ticket sale when it takes 3-4 seconds for a touchscreen press to register. The actual hardware of the POS terminals was pretty (needlessly) impressive. A lot of times the people running a place really have no idea what any of "that computer stuff" does - they rely their POS salesman and their crack team of installation specialists to get everything running. And from what I gather, a lot of those companies are pretty much used car salesmen...
(wonder if that's a potential business op? It's a hard market, to be sure, but most store owners I know that use a POS hate the system like they hate fleas, mosquitos, and AIDs)
Interesting thought. Could be a killer iPad application if you could partner with a hardware manufacturer to produce a cash drawer dock. High quality, capacitive touch screen and a integrated cellular modem for trade shows. Cost wise, it's likely to be cheaper than the service contract on normal POS systems.
I think there is still a huge install base of Windows machines in POS environments, especially in small to medium businesses.
In most of those settings (read: no need for remote access) it is per se not worse than deploying Linux unless you have them connected to an open LAN/WLAN and/or the Internet.
It sure made sense for a lot of people during the last 15 years for various reasons, occasionally one can even spot a DOS system still being used (now you know why they still sell floppy disks in 2010).
Big companies like McDonalds clearly had the resources long ago to go for a different approach but only during the last 5 years or so this has become a more viable option for many small-medium shops.
Even though Ubuntu, Google, Wikipedia & Co have "happened" during those Windows POS years we aren't there yet.
I suspect mainstream small-medium business will skip this step anyhow:
For example I myself am working on a distributed POS solution using a certain set of 2010 state of the art technology (the no-brainer variation: "cloud" back end and administration, newest generation of touch devices as "terminals").
It allows for low on site IT infrastructure and maintenance plus a strong emphasize on UI/UX which thanks to Apple is finally becoming ubiquitously expected.
Grml is another Debian based Linux distro for rescue scenarios like this. The small version weighs in a little more at 100MB versus PLoP's 62MB though.
If your deployment environment doesn't offer NFS/SMB/CIFS you can fetch your images via HTTP from your PXE booted kernel too.
Regarding data centers - it can even boot over serial in connection with HP iLO and IBM RSA/AMM out of the box.
I've got a grml usb stick with me at all times (now even supports multiboot 32/64bit!). I just recently recovered a relatives' WinXP after a seemingly failed automatic update and did a dll rollback in a couple of minutes.
It's a great tool plus it shows you how to use the mighty zsh properly!
Tiny Core Linux is another candidate at only 10MB (or its X-less variant, Micro Core, at 6MB). It includes a capable terminal server for PXE booting with support for loading extensions via NFS/TFTP/HTTP:
Hmmm interestingly, the "unix exemption" seems to have been dropped from 1.1 to 1.2 of DSS document.
Part 5.1 used to include this note:
Note: Systems commonly affected by viruses typically do not include UNIX based operating systems or mainframes.
Consumer grade MacAfee that auto updates is still probably a very poor choice for this application as they should have a test bed in the lab that they beta all changes on before they roll out to thousands of machines that likely need a truck roll if something goes wrong.
You can run McAfee on linux now as well (mostly for scanning email and CIFS shared files, I assume?). PCI DSS and DoD IA controls require it.
Enterprise grade McAfee still sucks, it's just a more manageable suck. Despite that, it still blew up with the bad DAT file - those are the same between product lines.
It's still wrong. Anti virus software in general are not very far from virii themselves in what they do to your computer. This whole macafee story just proves the point, but it's not the first time such a thing happens. AVG had that happen to them too a year back.
Its a perfect example of the classic "if it can then it must" brainless rule application.
I once had a car that had a bad bulb socket in the "cyclops" brake light so it wouldn't pass inspection. Did I fix the socket to get past? Hell no. I removed the whole light. If it wasn't there, it didn't have to work. Those were the rules.
It doesn't have to be done. I've worked on POS and smart-weighing systems that were cross-platform (Windows and Linux).
Neither used AV, but they were tightly locked down. My impression was that most customers don't care about the OS. Some prefer Windows, they already have the tech personnel in place. On the other hand, the Linux ports were done because a very big customer said that they had to have Linux... It certainly cost them more than licensing Windows. :)
From a dev point of view, Windows was easier, as it is generally more friendly towards C++ development. The OSes were about equally capable, I didn't see any Linux technical advantages in practice.
Not really sure how accurate that is....most ticket printers, magnetic card readers, weigh scales, barcode scanners, and other POS related devices adhere to the Unified POS standard.
There is a Java implementation of the standard (called, unsurprisingly, JavaPOS) as well as a .NET implementation (called POS for .NET). I've run a Linux based point of sale that used JavaPOS to interface with peripherals without issue.
Lots of vendors are also MS resellers and will turn a profit by selling their client all the Windows and SQL Server licenses they'll need to run the provided solution.
Who sticks with what, where? Windows and Linux both have their uses. I use Windows because in my opinion Visual Studio is the best native code debugger ever created. I'm hardly a fanboy though, and I use Linux a lot (valgrind is awesome).
A previous hotel job I had used a touchscreen Windows-based POS. One day, for whatever reason, I accidentally held down the 'submit transaction' button for half a second too long. The terminal went apeshit and crashed.
Several hours later, after hours of phone calls, hundreds of orders written on paper and totted up by calculator, a confused-looking technician managed to get it working properly again.
Booting off iSCSI is all the rage now. Any server with an Intel 1Gbe network port can be made to boot off iSCSI after flashing with special firmware from Intel.
TRWTF is that they didn't prepare for a catastophic failure such as this one. If you have your POS getting auto-updates then why haven't you considered the possibility that it will go wrong with "WILL NOT BOOT" as a consequence.
It should simply be a matter of phoning the manager and tell him :
"Don't worry, just insert the disk that says "RESCUE DISK" on it, reboot and it will roll back the machine to last week".
Since we know that the machines could PXE boot, they should have just had a server in the back preloaded with a known good image that would clone onto the POS automatically if started via PXE.
Then the fix would have been a matter of simply swapping the dhcp entry for the pos devices on the server with one that had the pxe directive. The pos would have caught that during its next reboot and restored itself.
If they'd tested the McAfee definitions on even one of the POS terminals before rolling them out to all the restaurants everything would have been fine. They could have even automated having one restaurant get the definitions one or two days before all the others so only a single location was affected. Or better, set up staggered updates within each restaurant, so they all stay up and at worst lose a single POS terminal.