In my first job as a Software Developer about 5 or 6 years ago I worked at a company which connected to forecourt controllers (which managed ATGs, fuel pumps, etc) on petrol forecourts in the UK, retrieved the data and presented it to a team of analysts for reporting back to the forecourt owners.
To this day, the most difficult software challenge I have had to tackle was to retrieve the data from the serial port of a controller called the DOMS 2000. It had been replaced a number of years earlier by a new model but was still in wide service.
We had very limited documentation and I spent weeks observing the data being transmitted from the serial port while performing actions using a simulator. Eventually I managed to reliably intercept and correctly interpret almost every command.
I believe, to this day, I may well be one of the foremost experts on the DOMS 2000 on the planet.
What would you say was the biggest cause for this to be so difficult? is it "just because" you didn't have the proper documentation? or was the nature of the communication protocol itself hard due to its design? or was it perhaps the fact that both factors would compound terribly?
Have you capitalized on your unique DOMS 2000 knowledge in recent projects? If this is a widely-used controller which is involved in a lot of valuable business, it sounds like you could have a very good consulting gig here.
This is what the NSA and all digital counter-terrorism theatre players should really do with public money: take a good hard look at the network infrastructure underpinning critical business segments, and force them to clean up their act. Be "the common man's pentest agency".
The NSA should be performing signal and human intelligence operations around the world to ensure the US can track global events. While the Internet is a key tool for this, it doesn't make them the "internet cops" of the US.
The FBI should be doing this, not the NSA, considering the FBI is actually the "US Police".
The government has made representations that we're facing a "cyber warfare" threat, and NSA is a paramilitary organization with a big role to play in conducting such a conflict.
The FBI is an investigative agency. They aren't beat cops. They aren't soldiers. They do have a counter-intelligence role.
From a threat POV, consider how dependent on gasoline and diesel the US economy is, and what the impact of disruption would be. If an attacker could disrupt fuel supplies in key areas or highway corridors, they would cause all sorts of chaos and economic damage.
As Americans, we've been very lucky to have two oceans protect us from the shift in warfare from the battlefield to total war, with the exception of the US Civil War. Our global connectivity changed the threat landscape. That's not a police matter.
"NSA/CSS has two interconnected missions: Signals Intelligence (known as SIGINT) and Information Assurance. Through SIGINT, we gather information that America's adversaries wish to keep secret. Through Information Assurance, we protect America's vital national security information and systems from theft or damage by others."
In the NSAs charter, the term "National security information" refers to classified information and "sensitive but unclassified" information, and the scope includes those systems. The scope does not include protecting all systems that might have national security implications, the NSA has no regulatory authority or requirement to do this.
If your corporate operations are vital to national security, congratulations! You've made it big! Also, you should take security seriously.
This should be an agreement between you and the rest of society (read: laws or standards). Government agencies responsible for the security of the nation should probably be looking into these types of things, and if they come across your RS232->Ethernet-dongle-that-explodes-Tampa they should a) notify you, b) offer assistance if required (for a price) and c) fine and/or arrest you if you are a serial offender.
Serial offender? I can't believe I just typed that...
It was established in 1952 by a presidential directive from Harry S. Truman in which he specified its mission as
to provide an effective, unified organization and control of the communications intelligence activities of the United States conducted against foreign governments, to provide for integrated operational policies and procedures pertaining thereto.
It's probably the equipment makers obligation to have this kind of internet-wide scan done on their devices. A central agency may not even know what's out there, and it's not really reasonable or efficient to have it be the gas station owners problem. Well it is kind of their problem, but it's really stupid to expect every station owner to solve an internet security problem themselves.
According to @War, which I just read, NSA has been poking into "vital national interests" like fuel distribution for years. That they haven't discovered (or having discovered, urged people to fix) this issue, is kind of an indictment of their abilities. Of course, having their dirty laundry aired by contractors isn't a great endorsement either. I wouldn't actually mind if they helped the various insecure agencies of the federal government improve. At this point, however, it doesn't seem reasonable for them to bother "the common man".
The NSA is a bit crippled. They are so intensely afraid of someone (anyone) figuring out what they do or how they do it that they can not act on any information they gather. The 9/11 Commission report showed that the NSA had full access to all of Mohammed Attas communications, knew exactly where he was, had translated his messages, etc, yet they did nothing about it. When the CIA requested access to the information they had, the NSA refused. When the FBI requested access to the information, the NSA refused. The degree to which they are obsessively paranoid can not be overstated. It's quite possible that they are very well aware of these weaknesses along with multitudes of others, but they will never say a word about it for fear that someone might figure out what they do.
These issues are known in the control systems security space, and have been for a long time. Convincing the owners/operators of them is very, very hard. They all say the same things:
"But they are vlanned off!"
"Why would anyone know to attack it, we keep that info confidential?"
"They wouldn't target us, we're a small company"
"The probability of this is too small to worry about, the cost/benefit is just not realistic"
"Can this really be used to do anything bad, the attacker wouldn't even know what to do"
I'm sure you're right, but the NSA is great at strong-arming executives into actions they wouldn't otherwise take. The typical dog-and-pony show runs like this: inform the CEO that she has a one-day top-secret clearance, and her attendance is expected at a "secret" meeting of industry executives. The morning portion of the meeting consists of going around the room and pointing out the gaping holes in each executive's company's network. The afternoon portion is "what you can do to help us help you". These recommendations come off as a bit more authoritative than that memo from the network engineer.
See pages 179 and 180 of @War by Shane Harris. Since these briefings are clearly more about NSA nest-feathering than actual security, I think it's not so awesome.
The problem is that the NSA has both the defensive and offensive roles and I think they've failed at the former role so badly because there's a ratchet effect pushing everything towards greater secrecy and a quasi-military role.
We'd be much better off as a country if we split the NSA into half with one side basically being a sort of USDA/EPA/FEMA with the only goal of improving baseline security standards and a healthy budget for things like aggressively auditing widely used code and services.
I have a Customer who uses this equipment (the Vedeer-Root gauges that the article describes) in unattended fueling stations. They moved from dial-up modems in the stations to Internet connections a couple of years ago (to facilitate modern credit card processing) and, at that time, the vendor who handles the tank gauges asked me to forward a port through each of the the stations' firewalls to the gauge. The idea that I was going to terminate a VPN in each station and accept no unsolicited traffic from the Internet was completely foreign to them. With that in mind, I'm not at all surprised by what rapid7 is reporting here. It's likely that those IP addresses where they're finding tank gauges also have other targets (like video surveillance systems, environmental monitoring systems, automated fire suppression systems) receiving Internet traffic on forwarded ports, too.
This sort of thing happens everywhere. Grain dryers, wastewater treatment plants, etc.
You get industry-specific vendors that believe their solution is opaque to anyone that hasn't received their specific training and they run loose. The end-user doesn't care because their expert in [whatever automation] tells them it has a password or proprietary protocol and can't be accessed without a program that's only given to trusted (industry) insiders.
There are some great presentations on YouTube from the Chaos Communication Congress and similar conferences about the abyssmal security of PLC-controlled industry stuff, including prisons. It is exactly as you say. They think their systems are too obscure for anyone to figure out. Of course, they hire the cheapest talent they can find to bang the things together in the first place and constantly look for ways to cut staff and replace developers and security personnel with newer (cheaper) people all the time, so they never end up with anything of any significant complexity. Even many places that think their networks are airgapped turn out not to be. In one prison discussed during one talk I watched, the prison had a commissary or something run by McDonald's... someone had connected to the commissaries Internet connection giving the Internet a direct route into all the insecure systems controlling the prison.
> The end-user doesn't care because their expert in [whatever automation] tells them it has a password or proprietary protocol and can't be accessed without a program that's only given to trusted (industry) insiders.
I doubt a discussion even gets that far. The end-user doesn't care because it works to do what they need, and they never even consider security.
How do you do "responsible disclosure" of a multi-vendor multi-customer vulnerability like this, with vendors not from the IT industry (or at least they don't realize they are in that industry, heh), who almost surely have no easy way to contact them to report vulnerabilities.
Real question. Was this irresponsible disclosure? What should they have done? (Are those doing the disclosing risking criminal prosecution?). I don't know the answers.
There are a few different parts. I keep thinking about amazon, facebook and google's programmable infrastructures. Control needs to be secure, but data acquisition could be public. I can imagine a gas company that supports surge pricing like uber.
If all this stuff is visible as a web service, you can take a regular old programmer and optimize inventory and delivery schedules. I'm sure that happens now, but i'm not sure how sophisticated it is. There's probably some value in a car being able to access the octane rating and price of gas on fillup, which would update your phone with price and performance metrics. I think there's also an aspect of service discovery you're overlooking.
I guess the point is, the words "internet of things" are goofy and overused, but it implies being more pervasive, open and standardized than scada.
Kinda. It's S(CA)DA where each node can have its own TCP connection instead of being RS232, Modbus, etc and then aggregated over TCP through a device server. Basically facilitated by cheap Ethernet hardware.
e.g., I have a BeagleBone monitoring a mousetrap in my basement. Total cost about $60; would be half that with a Raspberry Pi. It's only on my internal network, but it would be trivial to put its webpage online or serve up the data to something like what Pachube used to be.
Access to 3% of US gas station real time data should be enough to give you an edge in open markets. It might be enough to find mispriced stocks before earnings. Not that I'm suggesting anyone should do such a thing.
You wonder how many other internet connected things are exposed this way. I've always wondered if there are internet connected traffic light systems (I know most systems are ancient technology) are vulnerable in the same way.
Sure, the link I had has FTP sites, with absolutely no login required.
What can you do with an FTP site? Host your own phishing site. Modify the owner's own webpages.
Related to this article: if you can narrow the addresses down to ones that also have these gas meters, then you can modify their website to say "Hey; free gas!" and mark their tanks as 'always full'.
The key take-away here is that there is a literal TON of unsecured sites and multiple ways that something bad could be done.
A city with 50,000 compromised heaters drawing 1.5KW lets an attacker play with 75MW on the electric grid. (I just picked the middle value in a list of heaters; probably not a good estimate.)
A moderate county (roughly 2 million people in 800,000 households) has about a 2400MW power supply.
Compromising 6% of households' heaters would allow you to drive spikes on the order of 3% of the average power grid load (which includes things like business uses). With some targeting, you could probably get a larger influence on a localized portion of the grid.
Somehow, I think attackers causing high-ish frequency noise (on the order of ~1 minute waves, or less) with 3% of the average power could cause problems for the electric grid.
I always laughed at the movie Live Free or Die Hard, but now I'm thinking it's more and more likely that hackers can completely bring down society from behind a keyboard.
Fun fact: One of these stations can have almost 100,000 gallons of fuel in it. My cab driver this morning used to be a mechanical engineer working on gas stations in the US. Apparently modern filling stations typically have 2 (gasoline) or 3 (with diesel) 30,000 gallon tanks. The gauges by themselves might not be able to cause a big problem but I would hope anything that involved in the safety of the system is well-built.
"...approximately 5,800 ATGs with TCP port 10001 exposed to the internet and no password set."
Are we talking about simple monitoring information here, or some kind of administrative access?
"In our opinion, remote access to the control port of an ATG could provide an attacker with the ability to reconfigure alarm thresholds, reset the system, and otherwise disrupt the operation of the fuel tank."
Remote access to a control port would indeed conceivably allow these activities, no "in our opinion" required. But you haven't told me whether this is indeed remote access to a control port.
Also, the ISP pie graph just looks like a typical distribution of customers across ISPs. How useful is this supposed to be?
Well, the docs for these things are online and there are hundreds of opcodes. Some are less benign than "inventory status". One hopes that any of the "S" opcodes ("set" as opposed to "inquire" or "I" opcodes) are protected by the " System RS-232 Security Code", but I'm not about to try to find out.
I'm guessing the "Systems RS-232 Security Code" is a 4-digit numeric PIN, or at any rate a fairly low-entropy password normally, and that there is absolutely no rate-limiting or monitoring or other guard against a live brute force attack.
It looks like the documentation's been taken down from the URL included in the OP. I'm sure it's available elsewhere, but I haven't tried to find it.
Decades of experience shows that revealing security problems leads to greater overall security than not doing so.
Coordinated disclosure is to mitigate the effects of a particular case - but vendors routinely abuse it to sit on problems for months or even years (in their own interests), leading to a movement for full disclosure (in their customers' interests).
Also, you seem to be assuming the bad guys don't know already.
(This is why Microsoft's complaints about Google revealing their holes don't convince me: we already know Windows is peppered with holes that are unknown to the public but are literally commodities to criminals and national security organisations. Revealing all of them would increase our security and not decrease it even in the short term.)
How else do you get these people to change? We're not talking about a problem at a single company here. It's a wide spread issue of people apparently being either unaware or just not caring enough about security.
A well-connected industry insider may be able to get things moving. But if industry insider exist who care enough, why hasn't this been fixed a long time ago?
For an outsider, bringing the issue up in public is probably the most effective way to get this fixed. And it's not like they published a script to exploit this.
Obama just doubled-down on the CFAA; our government and society doesn't want to know, and they don't care about 'the most effective way to get this fixed.'
The security community has a long history of releasing flaws publicly. Take this recent libc gettimeofday() exploit, for example. I administer some servers and I am glad that I was alerted to the problem so I could address it. If that information had been hidden, then the problem would not have been fixed and eventually some black hat would exploit it.
Publishing security flaws can cause problems in the short term but it's in everyone's best interests in the long term. It is vital that our infrastructure is protected against attacks and we can't afford to allow companies to sell and operate insecure industrial systems. Making this information public is the best way we have to get these problems fixed.
[The deleted parent mentioned a temperature setting he found in the documentation.]
These are just monitors. That setting sounds like something to adjust the amount being reported in the station tank if it is very cold or very hot, as temperature slightly affects the volume of gasoline.
I can think of two possible attacks:
1) Falsely reporting the station tank is empty, when it is actually close to full, will cause unnecessary deliveries of gasoline for which the station owner will have to pay.
2) Falsely reporting the station tank is full, when it is actually close to empty, could cause a local - possibly regional - gas shortage if enough gas station in particular area are attacked.
I was in on a conference call last week where they said there are currently no crypto protocols which are safe and effective for small footprint iot applications. Of course the NSA is trying to fast track not one but two new ciphers for this application.
I don't totally buy that conclusion. Yes, you won't get any useful security on an Arduino, but there are plenty of small footprint devices that will do full TLS. I'm using TI's CC3200 right now with TLS 1.2 using ECDHE_RSA_WITH_AES_256_CBC_SHA to connect to our APIs and this thing is just a Cortex M3 doing all the crypto. (There's probably AES accelerator, but I doubt there is any hardware doing the PKI stuff.)
I mean, sure, it's a more expensive chip, it's two ARM cores plus a WiFi MAC in a single package, but it's pretty much all the computing and connectivity you would need for a normal iot kind of application in a less than 1 cm^2.
that is why you need a firewall between your IoT devices and the public internet. Then let an intermediary machine be a gateway/gatekeeper to the public internet.
I once wrote software for a Gilbarco gas station pump, basically a "next gen" model of the kind used by ordinary folks all across the country daily. I too was surprised then by how much Internet related tech we were expected to use and interface with. (I was not the overall architect. And I saw the practical advantages of those choices.)
To this day, the most difficult software challenge I have had to tackle was to retrieve the data from the serial port of a controller called the DOMS 2000. It had been replaced a number of years earlier by a new model but was still in wide service.
We had very limited documentation and I spent weeks observing the data being transmitted from the serial port while performing actions using a simulator. Eventually I managed to reliably intercept and correctly interpret almost every command.
I believe, to this day, I may well be one of the foremost experts on the DOMS 2000 on the planet.