Three times over the last year, Colonial Pipeline and the Transportation Security Administration discussed scheduling a voluntary, in-depth cybersecurity review — an assessment the federal agency began doing in late 2018 to strengthen the digital defenses of oil and natural gas pipeline companies, according to a company official and an industry official familiar with the matter.

But no such review of Colonial's systems has occurred, according to a Colonial spokesman. And the pipeline company has previously told federal officials it wants to first complete a headquarters move to a new building — probably in November — though the spokesman, Kevin Feeney, said on Friday that it may allow a review sooner.

It's unknown whether the government-run cybersecurity assessment would have helped Colonial avert the ransomware attack that locked up some of Colonial's computer systems this month — and led the company to shut down its entire pipeline, leaving large swaths of the East Coast with fuel shortages.

But a range of current and former officials and cybersecurity experts say the company's ability to avoid a government review underscores how a voluntary, arms-length approach by federal officials over nearly two decades has left key elements of the nation's critical infrastructure at risk.

"I'm very concerned whenever I see a lack of urgency given the potential threats we face," said Rep. Jim Langevin, D-R.I., co-founder of the Congressional Cybersecurity Caucus. "You're leaving so many areas exposed by not having a review — and addressing at least the vulnerabilities that you can identify."

Now, in the attack's wake, the Department of Homeland Security, which houses the TSA, is reversing course, scrapping two decades of a voluntary regime for pipeline cybersecurity and moving for the first time to mandatory rules.

But a review of the TSA's history since it was handed oversight of pipeline security in 2001 shows a government culture of closely partnering with energy giants and industry trade groups in setting guidelines that were voluntary. No penalty resulted for a failure to obey them.

Until the validated architecture design review debuted in December 2018, the TSA had never done a detailed cybersecurity assessment of pipelines. The VADR, as it's called, was what TSA had asked Colonial to do last year. Twenty-three have occurred this year alone — despite the pandemic.

"The cybersecurity landscape is constantly evolving and we must adapt to address new and emerging threats," Homeland Security Secretary Alejandro Mayorkas said in a statement. "The recent ransomware attack on a major petroleum pipeline demonstrates that the cybersecurity of pipeline systems is critical to our homeland security."

Colonial spokesman Feeney said that over the past four years, the company has hired "four leading independent firms" to conduct "comprehensive cybersecurity risk assessments to bolster the company's overarching security and information governance program."

As a result, he said, the company has boosted total information technology spending by about 50 percent since 2017.

The DHS shift in regulatory approach comes as ransomware attacks, in which hackers demand a fee to unlock computers whose data has been encrypted, are spreading from hospitals, universities and municipalities to industrial sectors out of a sense that these critical infrastructure companies will be more willing to pay to regain control of their systems. And it follows the intelligence community's publicly warning in 2019 that nation states had the ability to launch cyber attacks that could disrupt "a natural gas pipeline" for days to weeks.

"The lesson, particularly in recent months, is that we need to step up the incentives for the critical infrastructure to strengthen cybersecurity," said Michael Chertoff, who served as secretary of homeland security in the George W. Bush administration. "Positive incentives are important, but sometimes you need to be mandatory."

The TSA was created in the wake of the September 2001 terrorist attacks to safeguard aviation, but pipelines — seen as a mode of transportation — were moved under its jurisdiction, too. And for the first half of its existence, physical security was the focus.

Personnel at the TSA's pipeline security program ensured that fences around pipeline facilities were high enough to keep out intruders. The threat of cyber-hacking was not a front-burner issue.

But by 2010, things had changed.

Increasingly, energy, oil and gas and pipeline systems were operated by automated systems that checked temperatures, pressures and flows.

And concerns about cyberattacks grew — including at pipelines.

At a House Homeland Security Committee field hearing on pipeline security in Plant City, Fla., that year, the oversight subcommittee chairman turned to the head of TSA's pipeline security program, Jack Fox.

Rep. Christopher Carney, D-Pa., asked Fox whether regulations were needed to ensure pipeline security.

The 9/11 Commission Act of 2007 required the TSA to determine whether they were necessary. But, Fox said, "at this point, we are still working on this partnership that we have with the private industry."

An industry witness offered this view: Regulations, testified Gary Forman, director of corporate security for the natural gas and electric company NiSource Inc., "waste a lot of resources."

Carney, who is no longer in Congress, persisted.

"Our concern," he said, "is that there is no problem until there is a problem, until something happens. Then (people) will say, 'Well, why was that not regulated?'"

Despite having the authority to issue mandatory rules for pipeline companies, the TSA relied on voluntary guidance crafted with industry help that it hoped would motivate companies to improve their cyberdefenses.

In an interview this month, Fox explained the rationale. "If we wanted to make a change, tell them they should do something differently, we could have that change the next day with one phone call," he said. "We felt we were getting better security and faster responses the way we were doing it without going through regulation."

In 2010, the agency issued its first pipeline security guidelines that included a modest cyber component. The guidelines were based on assessing the likelihood of numerous physical risks, such as terrorist attacks, leaks and cyberthreats from a "rogue employee with computer access."

The problem, said Tim Conway, an industrial control systems cybersecurity instructor at the SANS Institute, is that the guidelines were voluntary. They could be ignored. And, he said, they left the risk determination to the company, which could then decide where and whether security controls were needed.

The guidelines bullet-pointed eight security controls. "But that's not a 'do-them,' " Conway said. "That's a 'you should consider these.' "

In that period, the electric industry was in the midst of upgrading its systems to meet government regulations. The pipeline sector, Conway said, wasn't eager to be subject to similar rules.

"The general theme back then was, 'If we can do things to secure our environment, let's do it, but we don't want more regulation,'" said Conway, a former director at an electric utility.

A series of high-profile incidents raised the pipeline sector's awareness of cyberthreats to the industry.

In 2012, a major Chinese cyber-intrusion campaign was underway, targeting natural gas pipeline networks in the United States. It didn't result in any disruptions, but, said one industry official, who was not authorized by their organization to speak on the record, "it was definitely something that made us go, 'Oh, we are a target.' "

Telvent-Canada Ltd., a company that supplied remote monitoring tools to Fortune 100 energy firms, notified clients that fall it had been hacked in what analysts believe was part of the same Chinese government campaign.

"There was a rising level of concern," recalled Conway, who was in industry then. "Discussions arose of, 'What can we count on the government to do to protect us? And how much is on us?' "

The TSA's voluntary approach to guidelines was further weakened by a lack of resources at the agency's pipeline security program. Within the TSA, the pipeline group had a relatively lean budget, making it difficult to recruit people in the highly competitive world of cybersecurity.

At the TSA, "aviation is the big dog at the table," Fox, who was the pipeline program head from 2002 until he retired in 2016, said in an interview. "They get 90 percent of the resources."

Cybersecurity also was not his unit's forte.

Realizing that computer intrusions were a growing threat, Fox in 2008 tried to hire dedicated cyber employees. "I was only looking for a couple," he said. He never got them.

So they did their best.

"We tried to ask questions," he said. "We asked companies to see if they knew about cyber. We estimated or believed they were doing the right thing, but we couldn't be positive. We just didn't have the expertise."

By 2012, pipeline staff peaked at 14. But two years later, TSA officials decided to reorganize, shifting from a model in which each transportation mode — mass transit, freight rail, pipelines, etc. — had its own specialists to one in which analysts covered multiple areas. And the pipeline security program was left with a staff of one: Fox.

That meant analysts on a site visit might not have pipeline expertise. Industry personnel spent more time "educating generalists about pipelines and pipeline security," said Kathleen Judge, director of risk and compliance for global security at National Grid, who testified on behalf of the American Gas Association at a 2016 hearing. After industry raised concerns, the reorganization was abandoned and the TSA returned to using specialists.

Gradually, the pipeline program hired staffers. But it still had only six as recently as 2020.

In 2018, the TSA began taking steps to ensure more pipeline operators were investing in cybersecurity. The agency issued revised guidelines, with a beefed-up cybersecurity section that now included criteria defining a "critical" facility that should have cybersecurity controls.

In recent years, the TSA has included those guidelines in a cybersecurity portion of its on-site corporate security reviews. These assessments of a pipeline owner's policies and procedures are seen more as a "tabletop exercise," said John Cusimano, vice president of industrial cybersecurity at aeSolutions, a consulting firm.

"The assessors they send out don't have a lot of cybersecurity background," he said, and they accept general answers without follow-up. "Getting through one of these TSA cyber assessments is pretty easy."

And James Hoecker, a Washington-based lawyer who represents energy companies and is a former chairman of the Federal Energy Regulatory Commission, says, "Since the standards are voluntary, the pipelines can say, 'Thank you very much we're doing just fine and we don't need to disclose this information.' "

In December 2018, the Government Accountability Office documented a span of more than five years in which the TSA had failed to follow up with companies to ensure that shortcomings uncovered in the corporate security reviews were fixed. That made it difficult, the GAO said, to gauge whether the reviews were effective.

That same month, the TSA and DHS's Cybersecurity and Infrastructure Security Agency teamed up to carry out the first VADRs, the validated architecture design reviews. Separate from the corporate security reviews, these were the first agency assessments devoted exclusively to cybersecurity. Agency personnel visit pipeline operators who agree to be audited. The result are detailed reports on a company's vulnerabilities and recommendations for improvements, the DHS has said.

But again, the recommendations are just that. "You can ignore them if you want," Conway said.

While resources have been a chronic issue, the TSA says it's beginning to turn a corner. Last year, the DHS acknowledged TSA staff lacked the time or expertise to conduct cybersecurity assessments and planned to contract an outside firm to do the audits. Now, the TSA says that since 2020 it has had 34 staffers in its pipeline program, adequate for both cyber and physical security.

And this past week, it issued a directive for the first time requiring pipeline owners to report cyber incidents. It is following up in the coming weeks with mandatory standards that pipeline companies must follow or face financial penalties.

TSA first reached out to Colonial last year to do a VADR, Colonial spokesman Feeney said. At the time, he said, "we were - and still are — in the middle of a physical move to a new building. Additionally, we were in the midst of a covid lockdown."

So the TSA tried again this year.

At the end of March, Sonya Proctor, the TSA's assistant administrator for surface operations, contacted Colonial Chief Information Officer Marie Mouchet, according to Feeney and the person briefed on the matter. The agency and Colonial tentatively discussed scheduling a VADR for mid-May, the person said.

This month after the attack, the two spoke again about lining up a VADR, Feeney said.

"We plan to accommodate their requests once we have fully recovered our systems and completed the investigation into this current incident," he said.

The Wall Street Journal on Wednesday reported that Colonial missed a security review before the cyberattack.

The Colonial incident this month showed that, despite these added measures, the government has struggled to track how companies prepare for and respond to cyberattacks.

On May 7, Colonial shut down its pipeline and called the FBI, Colonial's Feeney said. "We and the FBI then began alerting other relevant federal agencies," he said.

At a closed briefing with lawmakers earlier this month, Colonial executives said they still did not know the method used to gain access to their systems by the hackers, affiliates of a Russia-based group called DarkSide, according to lawmakers and Feeney.

The cyberattack involved ransomware that encrypted computers on the company's business or "information technology" networks. Feeney said in a statement to The Washington Post on Wednesday that "we have no reason to believe as of today that our OT (operational technology) side was compromised, but that is still under active investigation."

Given "the risks to our OT" network, he said, "shutting down our system immediately was the right decision."

Feeney did not explain what the risks to that network were, saying only that suspending operation was an "effort to protect the integrity of our OT systems'' and that "neither the billing system nor problems with the .... scheduling system were the drivers for shutting down the pipeline."

He added that the billing system was still down when the pipeline service was restored.

In another briefing this month, the FBI deputy assistant director for the cyber division, Herb Stapleton, told lawmakers that he had seen unconfirmed news reports that a $5 million ransom had been paid. According to a law enforcement official, he said he had nothing further to share on that point.

Less than an hour after the briefing ended, the Wall Street Journal posted a story quoting Colonial CEO Joseph Blount saying that he had approved a $4.4 million ransom payment on the same day the attack took place.

The following day, Colonial Pipeline government affairs director Drew Lohoff told lawmakers the firm had notified the FBI in San Francisco after the ransom was paid, according to two Democratic aides.

The FBI and Colonial declined to comment.

"I felt (the payment) really set a bad precedent," said Rep. Yvette Clarke, D-N.Y., adding she was troubled by "the fact that they had not reached out to the FBI before making the decision."

She said she pressed Lohoff on whom they contacted before paying the ransom and he essentially said, "No, they did not contact or consult with any federal agencies before (paying)."

Had the government been given a heads up "they were in the midst of negotiating something," she said, "the FBI might have been able to do some more tracking on the culprits."

The Washington Post's Magda Jean-Louis, Julie Tate and Aaron Schaffer contributed to this report.

related stories

• DHS to issue cybersecurity regulations for pipelines after Colonial hack