Stories about computer security tend to go viral when they bridge the vast divide between geeks and luddites, and this week’s news about a hacker who tried to poison a Florida town’s water supply was understandably front-page material. But for security nerds who’ve been warning about this sort of thing for ages, the most surprising aspect of the incident seems to be that we learned about it at all.
Spend a few minutes searching Twitter, Reddit or any number of other social media sites and you’ll find countless examples of researchers posting proof of being able to access so-called “human-machine interfaces” — basically web pages designed to interact remotely with various complex systems, such as those that monitor and/or control things like power, water, sewage and manufacturing plants.
And yet, there have been precious few known incidents of malicious hackers abusing this access to disrupt these complex systems. That is, until this past Monday, when Florida county sheriff Bob Gualtieri held a remarkably clear-headed and fact-filled news conference about an attempt to poison the water supply of Oldsmar, a town of around 15,000 not far from Tampa.
Gualtieri told the media that someone (they don’t know who yet) remotely accessed a computer for the city’s water treatment system (using Teamviewer) and briefly increased the amount of sodium hydroxide (a.k.a. lye used to control acidity in the water) to 100 times the normal level.
“The city’s water supply was not affected,” The Tampa Bay Times reported. “A supervisor working remotely saw the concentration being changed on his computer screen and immediately reverted it, Gualtieri said. City officials on Monday emphasized that several other safeguards are in place to prevent contaminated water from entering the water supply and said they’ve disabled the remote-access system used in the attack.”
In short, a likely inexperienced intruder somehow learned the credentials needed to remotely access Oldsmar’s water system, did little to hide his activity, and then tried to change settings by such a wide margin that the alterations would be hard to overlook.
“The system wasn’t capable of doing what the attacker wanted,” said Joe Weiss, managing partner at Applied Control Solutions, a consultancy for the control systems industry. “The system isn’t capable of going up by a factor of 100 because there are certain physics problems involved there. Also, the changes he tried to make wouldn’t happen instantaneously. The operators would have had plenty of time to do something about it.”
Weiss was just one of a half-dozen experts steeped in the cybersecurity aspects of industrial control systems that KrebsOnSecurity spoke with this week. While all of those interviewed echoed Weiss’s conclusion, most also said they were concerned about the prospects of a more advanced adversary.
Here are some of the sobering takeaways from those interviews:
- There are approximately 54,000 distinct drinking water systems in the United States.
- The vast majority of those systems serve fewer than 50,000 residents, with many serving just a few hundred or thousand.
- Virtually all of them rely on some type of remote access to monitor and/or administer these facilities.
- Many of these facilities are unattended, underfunded, and do not have someone watching the IT operations 24/7.
- Many facilities have not separated operational technology (the bits that control the switches and levers) from safety systems that might detect and alert on intrusions or potentially dangerous changes.
So, given how easy it is to search the web for and find ways to remotely interact with these HMI systems, why aren’t there more incidents like the one in Oldsmar making the news? One reason may be that these facilities don’t have to disclose such events when they do happen.
NO NEWS IS GOOD NEWS?
The only federal law that applies to the cybersecurity of water treatment facilities in the United States is America’s Water Infrastructure Act of 2018, which requires water systems serving more than 3,300 people “to develop or update risk assessments and emergency response plans.”
There is nothing in the law that requires such facilities to report cybersecurity incidents, such as the one that happened in Oldsmar this past weekend.
“It’s a difficult thing to get organizations to report cybersecurity incidents,” said Michael Arceneaux, managing director of the Water ISAC, an industry group that tries to facilitate information sharing and the adoption of best practices among utilities in the water sector. The Water ISAC’s 450 members serve roughly 200 million Americans, but its membership comprises less than one percent of the overall water utility industry.
“Some utilities are afraid that if their vulnerabilities are shared the hackers will have some inside knowledge on how to hack them,” Arceneaux said. “Utilities are rather hesitant to put that information in a public domain or have it in a database that could become public.”
Weiss said the federal agencies are equally reluctant to discuss such incidents.
“The only reason we knew about this incident in Florida was that the sheriff decided to hold a news conference,” Weiss said. “The FBI, Department of Homeland Security, none of them want to talk about this stuff publicly. Information sharing is broken.”
By way of example, Weiss said that not long ago he was contacted by a federal public defender representing a client who’d been convicted of hacking into a drinking water system. The attorney declined to share his client’s name, or divulge many details about the case. But he wanted to know if Weiss would be willing to serve as an expert witness who could help make the actions of a client sound less scary to a judge at sentencing time.
“He was defending this person who’d hacked into a drinking water system and had gotten all the way to the pumps and control systems,” Weiss recalled. “He said his client had only been in the system for about an hour, and he wanted to know how much damage could his client really could have done in that short a time. He was trying to get a more lenient sentence for the guy.”
Weiss said he’s tried to get more information about the defendant, but suspects the details of the case have been sealed.
Andrew Hildick-Smith is a consultant who served nearly 20 years managing remote access systems for the Massachusetts Water Resources Authority. Hildick-Smith said his experience working with numerous smaller water utilities has driven home the reality that most are severely under-staffed and underfunded.
“A decent portion of small water utilities depend on their community or town’s IT person to help them out with stuff,” he said. “When you’re running a water utility, there are so many things to take care of to keep it all running that there isn’t really enough time to improve what you have. That can spill over into the remote access side, and they may not have a IT person who can look at whether there’s a better way to do things, such as securing remote access and setting up things like two-factor authentication.”
Hildick-Smith said most of the cybersecurity incidents that he’s aware of involving water facilities fall into two categories. The most common are compromises where the systems affected were collateral damage from more opportunistic intrusions.
“There’ve been a bunch of times where water systems have had their control system breached, but it’s most often just sort of by chance, meaning whoever was doing it used the computer for setting up financial transactions, or it was a computer of convenience,” Hildick-Smith siad. “But attacks that involved the step of actually manipulating things is pretty short list.”
The other, increasingly common reason, he said, is of course ransomware attacks on the business side of water utilities.
“Separate from the sort of folks who wander into a SCADA system by mistake on the water side are a bunch of ransomware attacks against the business side of the water systems,” he said. “But even then you generally don’t get to hear the details of the attack.”
Hildick-Smith recalled a recent incident at a fairly large water utility that got hit with the Egregor ransomware strain.
“Things worked out internally for them, and they didn’t need to talk to the outside world or the press about it,” he said. “They made contact with the Water ISAC and the FBI, but it certainly didn’t become a press event, and any lessons they learned haven’t been able to be shared with folks.”
AN INTERNATIONAL CHALLENGE
The situation is no different in Europe and elsewhere, says Marcin Dudek, a control systems security researcher at CERT Polska, the computer emergency response team which handles cyber incident reporting in Poland.
Marcin said if water facilities have not been a major target of profit-minded criminal hackers, it is probably because most of these organizations have very little worth stealing and usually no resources for paying extortionists.
“The access part is quite easy,” he said. “There’s no business case for hacking these types of systems. Quite rarely do they have a proper VPN [virtual private network] for secure remote connection. I think it’s because there is not enough awareness of the problems of cybersecurity, but also because they are not financed enough. This goes not only for the US. It’s very similar here in Poland and different countries as well.”
Many security professionals have sounded off on social media saying public utilities have no business relying on remote access tools like Teamviewer, which by default allows complete control over the host system and is guarded by a simple password.
But Marcin says Teamviewer would actually be an improvement over the types of remote access systems he commonly finds in his own research, which involves HMI systems designed to be used via a publicly-facing website.
“I’ve seen a lot of cases where the HMI was directly available from a web page, where you just log in and are then able to change some parameters,” Marcin said. “This is particularly bad because web pages can have vulnerabilities, and those vulnerabilities can give the attacker full access to the panel.”
According to Marcin, utilities typically have multiple safety systems, and in an ideal environment those are separated from control systems so that a compromise of one will not cascade into the other.
“In reality, it’s not that easy to introduce toxins into the water treatment so that people will get sick, it’s not as easy as some people say,” he said. Still, he worries about more advanced attackers, such as those responsible for multiple incidents last year in which attackers gained access to some of Israel’s water treatment systems and tried to alter water chlorine levels before being detected and stopped.
“Remote access is something we cannot avoid today,” Marcin said. “Most installations are unmanned. If it is a very small water or sewage treatment plant, there will be no people inside and they just login whenever they need to change something.”
SELF EVALUTION TIME
Many smaller water treatment systems may soon be reevaluating their approach to securing remote access. Or at least that’s the hope of the Water Infrastructure Act of 2018, which gives utilities serving fewer than 50,000 residents until the end of June 2021 to complete a cybersecurity risk and resiliency assessment.
“The vast majority of these utilities have yet to really even think about where they stand in terms of cybersecurity,” said Hildick-Smith.
The only problem with this process is there aren’t any consequences for utilities that fail to complete their assessments by that deadline.
Hildick-Smith said while water systems are required to periodically report data about water quality to the U.S. Environmental Protection Agency (EPA), the agency has no real authority to enforce the cybersecurity assessments.
“The EPA has made some kind of vague threats, but they have no enforcement ability here,” he said. “Most water systems are going to wait until close the deadline, and then hire someone to do it for them. Others will probably just self-certify, raise their hands and say, ‘Yeah, we’re good.’”
Update, Feb. 11, 4:15 p.m. ET: Hildick-Smith has asked to qualify his last statement about the EPA’s authority. He says while the EPA is not collecting copies of the risk and resilience assessments and emergency response plans, or enforcing quality controls on the documents, they can fine utilities for not complying with the process and certifying that they have completed the requirements. The EPA explains more here (PDF).