Hacker News new | past | comments | ask | show | jobs | submit login
Target Hackers Broke in via HVAC Company (krebsonsecurity.com)
151 points by panarky on Feb 5, 2014 | hide | past | favorite | 85 comments



One of the key parts of this for consumers to keep in mind is that this is a massive PCI violation. Target neglected many of the most basic requirements in terms of network segmentation and data protection. Target is a large tier-one retailer. They had 3rd party audits to "guarantee" PCI compliance.

However, the 3rd party is usually a single 'auditor' who interviews the staff and looks at the network diagrams provided by the IT department. This information may be inaccurate to the point that it may not even exist.

The focus in these audits is almost always ecommerce. I'm sure Target's ecommerce site has been scoped very thoroughly. Almost every retailer is just as exposed. While every client I've worked with has (by the time I left) been PCI compliant on the ecommerce side, the internal networks are often completely flat, even across global locations. SOX is a joke as a result, as there is no separation of concerns.


What lessons beyond "Even major retailers have serious security problems" should a consumer be worried about here?

I don't know much about PCI compliance, but I don't get the idea that it is something I should have to worry about as a user of a credit card.


You should not use any card that does not give you a 100% guarantee that you will not be held responsible for erroneous charges. It really doesn't matter the retailer – they almost all suck. The older the company or the faster its growth, the more you should worry. Push that worry onto Amex or your bank.

PCI is surprisingly easy (and anyone can look it up at https://www.pcisecuritystandards.org/security_standards/docu...). It's basically all about "data at rest," covering what can be stored, how it must be stored, and how it should be stored. If a merchant can't meet the requirements, then they are required to demonstrate "compensating controls" which can literally be "we have a project in place to fix this before the audit a year from now."

"Data at rest" involves (in this case) credit card information stored on "disk" (SSD, etc.) for more than a short period of time. This generally excludes virtual memory, some queueing software, etc. If it touches the disk for ten seconds but is then wiped, you're compliant.

Then there are a list of "musts". You must have firewalls protecting the internal network (and review the rulesets), documentation on any connectivity to cardholder data, dataflow diagrams, only allow essential traffic in the card info environment, wall off wireless access, have an IPS, change all vendor defaults on any devices that could possibly screw you, minimize server responsibility (so don't run your smtp relay on the machine that also encrypts your card info), rotate encryption keys every x days, don't store CID/CVV/track 2/full mag-tape data from the card, scan your systems for changes/compromise/unauthorized access, maintain nonreputable audit trails, etc.

PCI is ratcheting down the requirements so that there is less room for interpretation. Previously, just using Oracle's obfuscation toolkit would be enough. This would protect you (more or less) if someone had access to your block device in raw mode or your data files in what-the-hell-is-your-dba-thinking mode, but an Edward Snowden could log in and SELECT all of your card info.


But why should a credit card user care about any of that (other than the easy disputing of erroneous charges)?


Because the world sucks. Many times, your cc provider will detect suspicious activity and freeze your account, but one of the common patterns of card theft is a validation phase followed by a "hammer it until it cries" phase. It's quite possible to have a few small charges show up on your account that you might miss because your overall bill seems fine. Usually, once the card is proven good, you'll get the lovely bill for a first-class flight to Qatar.

I agree that a cc user should not have to care about any of this, but people responsible for protecting your money are not so responsible.


But there isn't any need to invoke PCI or details about it in order to say "Card info gets leaked so watch your statements".

(And most people are fairly aware of the fact that card numbers are sensitive information; the newish thing here is a large retailer failing so spectacularly)


Remember when people were afraid to enter their credit cards online, so they would call a customer service rep and read it to that person over the phone?

The key piece to keep in mind is that the info on that little piece of plastic is never "safe".

If I were warning my mother, I'd just follow your hypothetical above. Of course, if the outcomes of these data breaches start including other identity theft, such as the "$50,000 twitter name"[1] hijack, more realistic phishing attacks, etc., then things are going to get interesting.

[1] Which is of course silly since the TOS specify that they are not to be sold.


Remember? It's still happening. We get this question constantly.


"(other than the easy disputing of erroneous charges)"

They are not easy nor simple AND you can't use your credit card while they mail you a new one

So yeah, as someone who have had their CC misused, it's a pain in the behind


The lesson is to have more than one credit card, because eventually the one you use regularly will be compromised -- even if you only use it at physical locations. And when it is compromised, you will have to get a replacement, which is an inconvenience (hence the need for a second card). Oh, and keep an eye on your statements. And if you use a debit card, you may be worse off -- when it is compromised, you will have bills bouncing until you can get it straightened out. Even more of an inconvenience.

If you use a debit card, go to your bank and have them turn off the "feature" that lets you overdraft (and get charged a $35.00 fee each time). Set up a separate account (one that doesn't have a debit card), to use for all your bill payments, so at least that doesn't get behind if your main account is cleaned out.


You should really avoid using a debit card at all.

There's numerous disadvantages, the main one is that that you generally have less protections against fraud, and even if you do, you may have to wait for the transactions to post before your bank can reverse them.

There's no upside to using a debit card; get multiple credit cards and pay them off every month. There's also various upsides including more air miles/hotel points, cash back, travel insurance, etc.


>If you use a debit card, go to your bank and have them turn off the "feature" that lets you overdraft (and get charged a $35.00 fee each time).

Better yet, have them issue you a new card that is only an ATM card and cannot be used without furnishing the PIN. That's what I did back in the 90s when "check cards"† were first introduced. I phoned the bank and asked if the Visa logo meant the card be used without a PIN. When they said yes, I told them to close the account. Instead, they said not to worry and sent me an ATM-only card in the next day's mail.

I use my ATM card for banking only. Everything else goes on my AX (or DS or, in extremis, MC or VI).

https://www.usbank.com/checking/us-bank-visa-check-card-debi...


And not just more than one card, but also make sure the cards are from different banks!

It's also recommended to have a Visa and a Mastercard (but that's for an unrelated reason that a small number of merchants take one and not the other - and usually it happens at the worst possible place, like a taxi).


Seems like this was overlooked a little, but one could always use cash.

I understand the reasons why cash can be inconvenient, but I doubt it it less hassle then dealing with a bank/CC company over fraudulent charges, associated overdraft fees, and all the crap.

Now eCommerce transactions over the web is a different story.


Indeed; that's what I do for everything local that less than, say, $750, and it's remarkably worry free. No privacy issues either, no one has any idea exactly what I buy locally.


Pay with cash when that option is available.


I wouldn't be surprised if these auditors were like this or even worse: http://serverfault.com/questions/293217/our-security-auditor...


The way credit cards are designed should be illegal. They are basically a piece of plain text that can be used by anyone to purchase things from someone's account.

It's basically the worst security you could possibly have for one of the highest-risk systems (direct access to money) that exist.

People need to be angry at the credit card industry for designing such an insecure system, not Target. There are plenty of online payment systems that work far, far better than credit cards that cannot, by design, ever have user account information stolen from a retailer.


How is it bad security? As a consumer, there's practically no issue. Data mining for fraud detection is pushed off onto the card provider - not my problem. There's no "direct access to money". It's a bill from a service provider that I can easily dispute. At worst, it's a slight inconvenience if my CC info is compromised.


Well, maybe we should talk about debit cards, which are functionally identical. The Target breach included theft of PINs - a purchase made with your debit card plus PIN is on you, not on a CC company.


Debit cards are terrible for this reason. And as I understand, chip-and-PIN models also start moving the security onto the consumer. Nice for merchants, crappy for consumers.


Get back to us when there's a major practical breach of a chip-and-PIN system and consumers aren't compensated.


"It's okay that the protections we are used to being afforded aren't afforded to us anymore, because this system hasn't been hacked yet."


No. Because when this system is hacked, protections will be afforded to us.

Can you even imagine the media storm in the EU or some other similarly consumer-friendly location if the evil American corporations don't compensate consumers for fraud of no fault of their own like they used to? Millions of people would mail in their cards. This is simply not going to happen.

I'm in Canada and a colleague had her wallet stolen from a locked car, PIN changed over the phone (that's a whole another subject) and purchases made. Faced with a police report confirming the story and possibility of bad press, you think the processor, the issuer, and the merchant won't budge? They budged.


>They budged

See, with the current system, there's no need for "budging". Several times I've had unauthorized charges show up (both Visa and Amex). A quick call, and it's sorted out.

If you've got to invoke the possibility of bad press, that sounds absolutely worse for the consumer.


I was giving an example of a more difficult situation to dispute, with card present and the person's personal information available for PIN reset. Electronic charges with card not present are easy to dispute anytime.

You have as many protections as your issuer decides is good business to give you. You can dispute a transaction but they can rule against your claim.

Try disputing a card present transaction in a Rite-Aid on a card with a $1000 limit and get back to us about consumer protections.


I've disputed a $6000 "card present" transaction at a physical store about 20 miles away from me. AmEx determined the card must have been cloned and sent a new one out immediately. There was no hassle involved: "Nope I haven't been to that town." And done.


You have no idea why I gave the example I did, do you?


Then, if they are going to afford the protections anyway -- they should afford them in writing and contractually.

This way, they could only partially refund and say that we're lucky to get anything at all.


There have probably been small-scale compromised of chip-and-PIN in the UK already due to a design flaw allowing purchases to be made without knowing the PIN. (We don't know for sure since the banks involved erased the logs that would confirm it.) The customers wound up liable for the transactions.


Interesting; I've tried searching but all I'm finding are technological faults, not articles about consumers liable. Can you link or give names to search? Closest I've found is http://www.theguardian.com/money/2012/may/04/banks-pin-card-... and it looks like the card-resend story that would only have been worse with a swipe-and-sign.


And that's why I only ever have used debit cards for banking, and never for purchases.


Exactly. And fraud is low enough that they're happy to do this.


What you say is true.

However, it doesn't really matter. Despite all the hue and cry, credit card fraud losses run in the single-digit basis point range as a fraction of transaction volume and around 1.2% as a fraction of issuer expenses.† Losses due to uncollectible debt (the "charge-off rate") are much higher, at around three or four percent in 2013 (but 10.9% in the second quarter of 2010).††

The pain point isn't financial. It's the bad publicity when breaches occur and the nagging fear is that losses could suddenly get out of hand due to some unanticipated vulnerability.

http://web.archive.org/web/20091229101826/http://www.sas.com...

††http://www.federalreserve.gov/releases/chargeoff/chgallsa.ht...


So this means we're effectively all paying 1.2% extra (indirectly through retailer's credit card fees) to accept these fraud losses. But you have to add to that some amount for overhead of dealing with fraud reports, and the expense of required security practices in order to be PCI compliant, which is entirely a result of the plain-text nature of credit cards. It probably results in 1.5-2% extra to all purchases, entirely hidden to consumers.

And, from your own citation, on average, 10% of Americans are victims of credit card fraud and 7% of debit card fraud - this probably results in about 16% of Americans total if you assume both groups are random samples of the population.

If you've ever dealt with credit card fraud, you'd know that it's an ordeal that can be a huge pain in the ass. It can take many hours of your own time, and time from your card issuer. The cost isn't just directly financial, it's the overhead that comes as a result of the fraud.

I am certain that if given the choice, nearly everyone would take a 1.5% discount on all of their purchases if they had to do two-factor authentication on all of them in exchange.


I am certain that if given the choice, nearly everyone would take a 1.5% discount on all of their purchases if they had to do two-factor authentication on all of them in exchange.

Do you have a trusted friend or family member who is not a technologist? Ask if you can watch them a) sign up for 2FA for their bank and b) complete one end-to-end transaction using 2FA.

You may, in response to this short anthropological study, revise your estimate that nearly everyone would gladly use 2FA for every purchase.

(Additionally, merchants wouldn't be thrilled about any system which makes it difficult for their customers to spend money. If the Stripe API had a field for require_two_factor_authentication I'd set it to "false" or "are you freaking kidding me? no!" simply because I know that will cost me more in lost transactions than I lose to CC fraud.)


My point is that the fraud losses are so far not heavy enough to move the payment processing industry off the dime towards a really secure solution.

A secure system might use your phone as its platform. E.g., the merchant's POS presents the bill to your phone via Bluetooth. If you approve, your phone prepares a payment request, encrypts it with the bank's public key, and sends it back to the POS. The POS then forwards it to the payment processor and receives back a message accepting or declining the charge. Elapsed time: one or two seconds.

The obvious hurdle is winning acceptance for the new protocol and outfitting a zillion POS terminals all over the country and world to run it.


Transaction volume is a poor way to measure the problem. I might use my card 1,000 times before getting hacked and having to replace my card. It's not a .01% loss it 100% as far as the consumer is concerned.


The consumer is almost always made whole in fraud incidents. For the consumer, it's mainly the pain in the ass factor. Which is not insignificant!


> The consumer is almost always made whole in fraud incidents.

By the collective money of other consumers...


Who ultimately bears the loss is a complicated microeconomic question, which depends on price elasticities and the like.

At one extreme, the payment industry and the merchants could end up eating the entire loss in the form of lower profits. At the other extreme, the entire loss might fall on consumers in the form of higher prices for goods and services. In reality, some of the loss will be born by producers (i.e., stock holders) and some by consumers.

The only thing that is clear is that a 7 bp fraud rate means the economy as a whole loses $7 for every $10,000 transacted.


Well the pain point is financial, it just lies with the merchant which gets a chargeback, I believe, when a fraudulent cc is used.


There are plenty of online payment systems that work far, far better than credit cards that cannot, by design, ever have user account information stolen from a retailer.

Why doesn't Target use those payment systems? Wait, I think I know... because their customers don't use those payment systems. Are you saying that customers should "be angry at" themselves?

It has been years since I've used a credit card at a big-box store. They all still take cash.


> People need to be angry at the credit card industry for designing such an insecure system, not Target.

He explicitly said the blame doesn't lie on Target.

Payments systems are a chicken and egg problem, however you would think the entity that feels the most pain from credit card fraud would try harder to replace the broken system.


Exactly! Except the entity that feels the most pain isn't the credit card companies or the banks. It's typically the retailer or the payment portal.


That was actually my point. I am surprised retailers aren't making more of a stink about how insecure payment processing is given they are on the line for fraud 9 times out of 10.


Ever heard of worse is better?

This simple method works fine. The amount of fraud is not enough to make switching worth it. (And don't forget credit cards need to work even without persistent data connection.)


Why should I be angry? My credit card has been comprimised twice in my life and both times my credit card company contacted me immediately (before I even noticed) and refunded all fraudelent transactions immediately. I didn't have to do anything.

Credit card companies should be pissed at themselves for making it this easy.


It's a design that's iterated over many decades, and it hasn't always been easy for them; the payment network is huge.

Plus, if you think security is bad now, remember that before this we were writing checks for most transactions. At least with credit cards I can discover fraud usually the same day it occurs, and they are far more convenient that checks ever were -- as someone who has used both, I'm not all that bothered by the current state of the system.


Yeah but the cool thing about credit cards is : consumer protection laws.


Your checking account is just as insecure. All that's needed to debit your checking account is your bank's routing number (which is public) and your account number. No PIN, nothing else.


What system, exactly?


This doesn't surprise me.

From what I've seen on the provider side HVAC (and access control) companies want their devices right on the Internet. If you push them they'll deal with being behind NAT with a port forward but mention using a VPN and that's too much work for them.

Ask them about security and they hand wave and say it is secure because it is a "appliance" or "controller" as if that magically protects them.

This is changing somewhat as vendors move to a "phone home to the cloud" approach instead of direct access so they can get in on the revenue stream between the end user and the dealer. This removes the direct exposure to the internet but the local/insider threat remains.


I for one welcome the fantastic security enabled by the Internet of Things.


This already started.

TV receivers broadcasting metadata about files on devices connected to the LAN.

Routers allowing anonymous access over ftp to the disks connected to them.

Routers protected on the WAN side with factory set passwords.

Kettles and irons equipped with wifi and looking for open networks to broadcast something to the world.

(Smart)phones with broadband processors with DMA access to the applications processor's memory.

Computers, tablets and game consoles equipped conveniently with cameras and microphones.

We live in the future, do we like this or not.


I have seen TV broadcast equipment with SNMP and web interfaces exposed on public internet addresses. Days ago.


Curious: What are the ethical implications for the thieves? They effectively stole from a giant, multinational corporation, and inconvenienced consumers. At the end of the day, no consumer lost money from this. I wonder how this differentiation affects the consciences of the thieves, who are probably sitting on quite a large pile of money right now...

(I hope this comment doesn't show up in a background check..)


Consumers have not lost money directly, but they certainly will lose it in higher interchange fees, interest rates, store prices, and all the other tiny components that make up the American payment infrastructure.


It’s not immediately clear why Target would have given an HVAC company external network access...


There was a really funny story about a vendor of some equipment at one of Google's new data centers trying to convince Google that it had to have access to the data center network so that it could monitor its doo-dads. As I recall the "solution" was some plenum rated CAT5 that went out of the roof and attached to a DSL line on a pole at the corner of the property.


When you reach a certain size of HVAC system it is common for the dealer to regularly service the system and proactively monitor it. This requires access to the controllers to see a pump has failed, coolant pressure dropped, etc.

They should have ordered a stand-alone Internet connection for such things but probably figured they'd save the money and use the existing network connection.


Really all they would need is a VLAN and a proper set of ACLs.


It's almost as if, at some size, a company's "internal" network should be considered no more secure than the internet itself. Defense at the perimeter only is flawed, especially when the perimeter is the only thing protecting cash registers that update with unsigned firmware. One wouldn't connect such a client directly to the internet, so neither should one connect it directly to a network shared by millions of devices at 1700 stores.


> It's almost as if, at some size, a company's "internal" network should be considered no more secure than the internet itself.

Exactly this.

I realized this a few years ago, at a previous employer. We had _excellent_ security at the firewall boundary.

Inside the network gave me cause for concern. Once could plug into any port, in any building, get an IP address and ping anything on our network.

And how good was site security at a given location? Who knew, really.

When your server's security depends on J. Random Employee not allowing tail-gaters at the back door ... you got problems.


ACLs don't go far enough. You need an air gap. Chances of ACL/VLAN configuration failures are higher than someone purposely connecting a patch cable between networks.


What does this mean? Do you envision Target leasing dark fiber to connect all their stores, even those that aren't within ten miles of a POP? Because that isn't going to happen, for reasons, and without it they can't eliminate VLANs at some level. It's more practical to realize that VLAN configs should be regularly updated and tested.


> Do you envision Target leasing dark fiber to connect all their stores, even those that aren't within ten miles of a POP?

Do you remember the days of T1s and private lines? You can actually get a dry loop (a DSL line with no service, just twisted pair between two points) for $50-100/month depending on the distance between termination points. I'm not suggesting we go back to that level of network segregation though for branch offices.

The intention of my comment was to communicate that when an outside vendor needs internet access for remote access/control/etc, you provide it to them via a separate physical network. In this case, it could have been a seperate switch with a DSL or LTE connection.

PCI standards don't require a separate payments-only network, but would you put remotely accessed gear on your internal network if it was handling payments data? Target liability is estimated at ~$450 million and climbing.

>It's more practical to realize that VLAN configs should be regularly updated and tested.

I agree with this point. The reality is, it rarely happens as often as it should, if at all. If your policy isn't enforced programmatically and with checks/balances, it doesn't exist.


I'm not suggesting we go back to that level of network segregation though for branch offices.

We agree! My point was just that, even if there are physically separate networks within the store (and who knows, that might make sense in some cases...) all of those networks will connect in some fashion to the internet, so the air gap doesn't really exist. You can eliminate some onsite vlan'ing by running more cable, but as long as Target wants to be able to administer cash registers from far away, they'll need some sort of VPN setup, and you're right back where you started.


the real problem here is that they should have had two distinct networks, one for their payment system that is wholly separate from all other network activity.

Its boggling to me that any company would permit another company on the same network when their work is wholly unrelated.


"Its boggling to me that any company would permit another company on the same network when their work is wholly unrelated."

Not really, you've got a store manager who wants instant access to the same refrigeration alarms and control system his vendor sees and live sales figures all from the same PC. I can totally see it happening.

Wholly unrelated would be the Target store manager (or district mgr or whatever) having access to Walmarts backup generator telemetry.


Wow, I'm more surprised that they had $100M "cyber" insurance.


So if I check my actual policy documents, it's described as a "Data Breach Expense And Regulatory Defense Coverage rider to your Errors and Omissions Policy." But hey, computers are involved, so the news media would probably call that my cyberinsurance.


@patio11: "Makes sense - I have one million, as a rider to E&O policy. The insurance companies practically give it away."

https://twitter.com/patio11/status/431219958047965185


Is it common and responsible to name the source of the credential leak like that? I can imagine that maybe other retailers would want to know that information, but I can also imagine the investigation is continuing and thus the opportunity exists for the story to become more complex.


Other retailers should be less concerned about if they have done business with this vendor than they should be able letting any vendor have insecure access to their network.


[deleted]


Still on the Krebs front page...


This kind of reminds me of the original Wall Street movie, where Charlie Sheen broke into companies via their cleaning service, and then stole inside information.


This is a non cached link that appears to still function:

http://krebsonsecurity.com/2014/02/target-hackers-broke-in-v...


Edit: The article is back up now.

---

The article has disappeared. Here's the text of the first part of the post from Google cache.

http://webcache.googleusercontent.com/search?q=cache:mG2INOj...

Feb 14 Target Hackers Broke in Via HVAC Company

Last week, Target told reporters at The Wall Street Journal and Reuters that the initial intrusion into its systems was traced back to network credentials that were stolen from a third party vendor. Sources now tell KrebsOnSecurity that the vendor in question was a refrigeration, heating and air conditioning subcontractor that has worked at a number of locations at Target and other top retailers.

Sources close to the investigation said the attackers first broke into the retailer’s network on Nov. 15, 2013 using network credentials stolen from Fazio Mechanical Services, a Sharpsburg, Penn.-based provider of refrigeration and HVAC systems.

Fazio president Ross Fazio confirmed that the U.S. Secret Service visited his company’s offices in connection with the Target investigation, but said he was not present when the visit occurred. Fazio Vice President Daniel Mitsch declined to answer questions about the visit. According to the company’s homepage, Fazio Mechanical also has done refrigeration and HVAC projects for specific Trader Joe’s, Whole Foods and BJ’s Wholesale Club locations in Pennsylvania, Maryland, Ohio, Virginia and West Virginia.

Target spokeswoman Molly Snyder said the company had no additional information to share, citing a “very active and ongoing investigation.”

It’s not immediately clear why Target would have given an HVAC company external network access, or why that access would not be cordoned off from Target’s payment system network. But according to a cybersecurity expert at a large retailer who asked not to be named because he did not have permission to speak on the record, it is common for large retail operations to have a team that routinely monitors energy consumption and temperatures in stores to save on costs (particularly at night) and to alert store managers if temperatures in the stores fluctuate outside of an acceptable range that could prevent customers from shopping at the store.

“To support this solution, vendors need to be able to remote into the system in order to do maintenance (updates, patches, etc.) or to troubleshoot glitches and connectivity issues with the software,” the source said. “This feeds into the topic of cost savings, with so many solutions in a given organization. And to save on head count, it is sometimes beneficial to allow a vendor to support versus train or hire extra people.”

Continue reading →


It appears to be back up now. It returned right around 3:40PM EST.


It's back now, it seems.


And, evidently, they've broken into Krebsonsecurity.com to delete this.


Go up one level in the URL.


This reminds me of the vulnerable Google HVAC last year.


Finally someone to blame...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: