On Jan. 15, a hacker tried to poison a water treatment plant that served parts of the San Francisco Bay Area. It didn't seem hard.
50,000 security disasters waiting to happen: The problem of America's water supplies
"If you could imagine a community center run by two old guys who are plumbers, that's your average water plant," one cybersecurity consultant said.
Illustration of a water tank with digital shark fins swimming in the water
A series of cyberattacks on water facilities highlight the precarious security situation facing U.S. supplies.Doug Chayka / for NBC News
June 17, 2021, 4:33 PM BST / Updated June 17, 2021, 5:20 PM BST
By Kevin Collier
On Jan. 15, a hacker tried to poison a water treatment plant that served parts of the San Francisco Bay Area. It didn't seem hard.
The hacker had the username and password for a former employee's TeamViewer account, a popular program that lets users remotely control their computers, according to a private report compiled by the Northern California Regional Intelligence Center in February and seen by NBC News.
After logging in, the hacker, whose name and motive are unknown and who hasn't been identified by law enforcement, deleted programs that the water plant used to treat drinking water.
Pipeline hack highlights urgency of cybersecurity threat against US infrastructure
MAY 16, 202104:41
The hack wasn't discovered until the following day, and the facility changed its passwords and reinstalled the programs.
"No failures were reported as a result of this incident, and no individuals in the city reported illness from water-related failures," the report, which did not specify which water treatment plant had been breached, noted.
The incident, which has not been previously reported, is one of a growing number of cyberattacks on U.S. water infrastructure that have recently come to light. The Bay Area attack was followed by a similar one in Oldsmar, Florida, a few weeks later. In that one, which made headlines around the world, a hacker also gained access to a TeamViewer account and raised the levels of lye in the drinking water to poisonous levels. An employee quickly caught the computer's mouse moving on its own, and undid the hacker's changes.
The Biden administration and the public are in the middle of a cybersecurity reckoning. Russian and Chinese spies have sneaked into numerous federal government networks, sometimes sitting for months undetected. Criminals have hacked into practically every industry and extorted companies at will, including those that occupy important parts of U.S supply chains.
But of all the country's critical infrastructure, water might be the most vulnerable to hackers: the hardest in which to guarantee everyone follows basic cybersecurity steps, and the easiest in which to cause major, real-world harm to large numbers of people.
Colonial Pipeline CEO speaks out about ransomware hack
JUNE 9, 202100:59
U.S. water infrastructure does have some built-in security — most notably its lack of centralization. A widespread water hack would be difficult to pull off, much like a hack on U.S. elections, because each facility runs independently, not working in tandem with others.
But that also means there's no simple solution to safeguard water facilities. The Bay Area case is still under FBI investigation. How the hacker or hackers got access to those TeamViewer accounts isn't known. But a staple of dark web forums is hackers buying, repackaging and selling login credentials. The usernames and passwords for at least 11 Oldsmar employees have been traded on the dark web, said Kent Backman, a researcher at the cybersecurity company Dragos.
To date, a true catastrophe — where a hacker was able to poison a population's drinking water, causing mass sickness or even death — has not happened. But a number of facilities have been hacked in the past year, though most draw little attention. In Pennsylvania, a state water warning system has reportedly alerted its members to two recent hacks at water plants in the state. In another previously unreported hack, the Camrosa Water District in Southern California was infected with ransomware last summer.
Whether hacks on water plants have recently become more common or just more visible is impossible to tell, because there is no comprehensive federal or industry accounting of water treatment plants' security.
"It's really difficult to apply some kind of uniform cyber hygiene assessment, given the disparate size and capacity and technical capacity of all the water utilities," said Mike Keegan, an analyst at the National Rural Water Association, a trade group for the sector.
"You don't really have a good assessment of what's going on," he said.
Unlike the electric grid, which is largely run by a smaller number of for-profit corporations, most of the more than 50,000 drinking water facilities in the U.S. are nonprofit entities. Some that serve large populations are larger operations with dedicated cybersecurity staff. But rural areas in particular often get their water from small plants, often run by only a handful of employees who aren't dedicated cybersecurity experts, said Bryson Bort, a consultant on industrial cybersecurity systems.
"They're even more fragmented at lower levels than anything we're used to talking about, like the electric grid," he said. "If you could imagine a community center run by two old guys who are plumbers, that's your average water plant."
Recommended
TECH NEWS
Teamsters launch massive nationwide effort to unionize Amazon workers
TECH NEWS
Is a Netflix docuseries creating Formula One mega fans? What industry insiders are saying.
There has never been a nationwide cybersecurity audit of water treatment facilities, and the U.S. government has said it has no plans for one. While individual facilities can ask the federal government for help to protect themselves, few do. In most cases, it's up to individual water plants to protect themselves, and even if they're aware they've been hacked — a big if — they might not be inclined to tell the federal government, much less their customers. That means hacks can take years to come to light, if they do at all.
In March, the acting U.S. attorney in Kansas indicted a former employee of a tiny water treatment plant in Ellsworth County over an incident that had happened two years earlier. A night shift worker who had worked at the Post Rock Rural Water District logged into a remote online control system and tried to shut down the plant's cleaning and disinfecting operations in 2019, the Department of Justice said. The former employee has pleaded not guilty, and his lawyer didn't respond to a request for comment.
Small rural water facilities tend to be reluctant to share their vulnerabilities, said Daryn Martin, a technical assistant at the Kansas Rural Water Association, a trade organization for about 800 Kansas water treatment facilities, including Post Rock.
"Generally, they're not reporting to the federal government. There's some distrust, you know, in small-town, Midwest USA," he said.
But letting employees log on remotely to do basic work offers substantial advantages for rural workers who periodically are alerted to minor issues that need their attention, Martin said.
"Remote access makes it so you don't have to man a facility 24 hours a day," he said. "We have a lot of remote water districts that cover hundreds of miles. To pay a guy to drive 30 miles to turn a pump on and then he might have to turn it off in 3 hours when the tank gets full? He can do all that remotely. That saves money."
The Cybersecurity and Infrastructure Security Agency, the federal government's primary cybersecurity defense agency, is tasked with helping secure the country's infrastructure, including water. But it doesn't regulate the sector and is largely confined to giving advice and assistance to organizations that ask for it.
Only a tiny fraction of the country's water facilities choose to use CISA's services — "several hundred" out of more than the 50,000 across the U.S., Anne Cutler, a spokesperson for the agency, said.
Of those that do, an internal CISA survey conducted earlier this year, the results of which she shared with NBC, found dour results. As many as 1 in 10 water and wastewater plants had recently found a critical cybersecurity vulnerability. Most shocking, more than 80 percent of the major vulnerabilities that the surveyed facilities had were software flaws discovered before 2017, indicating a rampant problem of employees not updating their software.
Some things are marginally improving. Congress recently gave CISA legal authority to force internet providers to turn over the identities of organizations that it or other government agencies see are being targeted by hackers.
The White House plans to launch a voluntary cybersecurity collaboration between the federal government and water facilities, similar to one announced with the energy industry in April, a spokesperson said, though no dates have been announced.
Experts said that no one claims any government initiatives can make American water entirely safe from hackers, however.
"Those two plumbers are in no different a boat than a Fortune 100 company," Bort said.
"If you could imagine a community center run by two old guys who are plumbers, that's your average water plant," one cybersecurity consultant said.
Illustration of a water tank with digital shark fins swimming in the water
A series of cyberattacks on water facilities highlight the precarious security situation facing U.S. supplies.Doug Chayka / for NBC News
June 17, 2021, 4:33 PM BST / Updated June 17, 2021, 5:20 PM BST
By Kevin Collier
On Jan. 15, a hacker tried to poison a water treatment plant that served parts of the San Francisco Bay Area. It didn't seem hard.
The hacker had the username and password for a former employee's TeamViewer account, a popular program that lets users remotely control their computers, according to a private report compiled by the Northern California Regional Intelligence Center in February and seen by NBC News.
After logging in, the hacker, whose name and motive are unknown and who hasn't been identified by law enforcement, deleted programs that the water plant used to treat drinking water.
Pipeline hack highlights urgency of cybersecurity threat against US infrastructure
MAY 16, 202104:41
The hack wasn't discovered until the following day, and the facility changed its passwords and reinstalled the programs.
"No failures were reported as a result of this incident, and no individuals in the city reported illness from water-related failures," the report, which did not specify which water treatment plant had been breached, noted.
The incident, which has not been previously reported, is one of a growing number of cyberattacks on U.S. water infrastructure that have recently come to light. The Bay Area attack was followed by a similar one in Oldsmar, Florida, a few weeks later. In that one, which made headlines around the world, a hacker also gained access to a TeamViewer account and raised the levels of lye in the drinking water to poisonous levels. An employee quickly caught the computer's mouse moving on its own, and undid the hacker's changes.
The Biden administration and the public are in the middle of a cybersecurity reckoning. Russian and Chinese spies have sneaked into numerous federal government networks, sometimes sitting for months undetected. Criminals have hacked into practically every industry and extorted companies at will, including those that occupy important parts of U.S supply chains.
But of all the country's critical infrastructure, water might be the most vulnerable to hackers: the hardest in which to guarantee everyone follows basic cybersecurity steps, and the easiest in which to cause major, real-world harm to large numbers of people.
Colonial Pipeline CEO speaks out about ransomware hack
JUNE 9, 202100:59
U.S. water infrastructure does have some built-in security — most notably its lack of centralization. A widespread water hack would be difficult to pull off, much like a hack on U.S. elections, because each facility runs independently, not working in tandem with others.
But that also means there's no simple solution to safeguard water facilities. The Bay Area case is still under FBI investigation. How the hacker or hackers got access to those TeamViewer accounts isn't known. But a staple of dark web forums is hackers buying, repackaging and selling login credentials. The usernames and passwords for at least 11 Oldsmar employees have been traded on the dark web, said Kent Backman, a researcher at the cybersecurity company Dragos.
To date, a true catastrophe — where a hacker was able to poison a population's drinking water, causing mass sickness or even death — has not happened. But a number of facilities have been hacked in the past year, though most draw little attention. In Pennsylvania, a state water warning system has reportedly alerted its members to two recent hacks at water plants in the state. In another previously unreported hack, the Camrosa Water District in Southern California was infected with ransomware last summer.
Whether hacks on water plants have recently become more common or just more visible is impossible to tell, because there is no comprehensive federal or industry accounting of water treatment plants' security.
"It's really difficult to apply some kind of uniform cyber hygiene assessment, given the disparate size and capacity and technical capacity of all the water utilities," said Mike Keegan, an analyst at the National Rural Water Association, a trade group for the sector.
"You don't really have a good assessment of what's going on," he said.
Unlike the electric grid, which is largely run by a smaller number of for-profit corporations, most of the more than 50,000 drinking water facilities in the U.S. are nonprofit entities. Some that serve large populations are larger operations with dedicated cybersecurity staff. But rural areas in particular often get their water from small plants, often run by only a handful of employees who aren't dedicated cybersecurity experts, said Bryson Bort, a consultant on industrial cybersecurity systems.
"They're even more fragmented at lower levels than anything we're used to talking about, like the electric grid," he said. "If you could imagine a community center run by two old guys who are plumbers, that's your average water plant."
Recommended
TECH NEWS
Teamsters launch massive nationwide effort to unionize Amazon workers
TECH NEWS
Is a Netflix docuseries creating Formula One mega fans? What industry insiders are saying.
There has never been a nationwide cybersecurity audit of water treatment facilities, and the U.S. government has said it has no plans for one. While individual facilities can ask the federal government for help to protect themselves, few do. In most cases, it's up to individual water plants to protect themselves, and even if they're aware they've been hacked — a big if — they might not be inclined to tell the federal government, much less their customers. That means hacks can take years to come to light, if they do at all.
In March, the acting U.S. attorney in Kansas indicted a former employee of a tiny water treatment plant in Ellsworth County over an incident that had happened two years earlier. A night shift worker who had worked at the Post Rock Rural Water District logged into a remote online control system and tried to shut down the plant's cleaning and disinfecting operations in 2019, the Department of Justice said. The former employee has pleaded not guilty, and his lawyer didn't respond to a request for comment.
Small rural water facilities tend to be reluctant to share their vulnerabilities, said Daryn Martin, a technical assistant at the Kansas Rural Water Association, a trade organization for about 800 Kansas water treatment facilities, including Post Rock.
"Generally, they're not reporting to the federal government. There's some distrust, you know, in small-town, Midwest USA," he said.
But letting employees log on remotely to do basic work offers substantial advantages for rural workers who periodically are alerted to minor issues that need their attention, Martin said.
"Remote access makes it so you don't have to man a facility 24 hours a day," he said. "We have a lot of remote water districts that cover hundreds of miles. To pay a guy to drive 30 miles to turn a pump on and then he might have to turn it off in 3 hours when the tank gets full? He can do all that remotely. That saves money."
The Cybersecurity and Infrastructure Security Agency, the federal government's primary cybersecurity defense agency, is tasked with helping secure the country's infrastructure, including water. But it doesn't regulate the sector and is largely confined to giving advice and assistance to organizations that ask for it.
Only a tiny fraction of the country's water facilities choose to use CISA's services — "several hundred" out of more than the 50,000 across the U.S., Anne Cutler, a spokesperson for the agency, said.
Of those that do, an internal CISA survey conducted earlier this year, the results of which she shared with NBC, found dour results. As many as 1 in 10 water and wastewater plants had recently found a critical cybersecurity vulnerability. Most shocking, more than 80 percent of the major vulnerabilities that the surveyed facilities had were software flaws discovered before 2017, indicating a rampant problem of employees not updating their software.
Some things are marginally improving. Congress recently gave CISA legal authority to force internet providers to turn over the identities of organizations that it or other government agencies see are being targeted by hackers.
The White House plans to launch a voluntary cybersecurity collaboration between the federal government and water facilities, similar to one announced with the energy industry in April, a spokesperson said, though no dates have been announced.
Experts said that no one claims any government initiatives can make American water entirely safe from hackers, however.
"Those two plumbers are in no different a boat than a Fortune 100 company," Bort said.
