Hard lessons of the SolarWinds hack - The Verge
InIn December, details came out on one of the most massive breaches of US cybersecurity in recent history. A group of hackers, likely from the Russian government, had gotten into a network management company called SolarWinds and infiltrated its customers’ networks. This access was then used to breach everything from Microsoft to US government agencies, including the US Treasury and departments of Homeland Security, State, Defense, and Commerce.
On today’s episode of Decoder, I’m joined by Joseph Menn, a reporter at Reuters who focuses on cybersecurity investigations and the author of the new book Cult of the Dead Cow: How the Original Hacking Supergroup Might Just Save the World. We discuss what this breach means for US security and the companies in SolarWinds’ supply chain that might have been affected.
The SolarWinds hack hasn’t really gotten the attention it deserves because it happened during the chaos after the presidential election — but it’s a big deal. And it raises a lot of questions about how to respond to such a massive attack and the responsibility of the private sector when it comes to national security. There aren’t a lot of easy answers here, but it’s clear that change is coming with the Biden administration.
Okay, Joseph Menn. Here we go.
Below is a lightly edited excerpt from our conversation.
Joseph Menn, you’re the technology projects reporter at Reuters. You focus on investigative cybersecurity stories. Welcome to Decoder.
Thanks for having me.
I really wanted to have you on, because I feel like in the transition between the Trump administration and the Biden administration, the story of the SolarWinds hack and its fallout kind of got lost. I know it’s a huge story, but so many other things [are happening] in our country that, to me, it feels like it’s not getting the attention that it deserves. Is that kind of your sense of it, too?
Well, I agree with that, and I think it’s by design. The roots of this attack go back more than a year, but the activity escalated, and it came at a perfect time, because the US was so distracted. In particular, the US government, and even the security people within the US government, were busy worrying about securing the elections. So you couldn’t have picked a better time to launch a massive spying attack on the government.
So from the very beginning, tell people what SolarWinds is and how it was attacked.
Sure. So SolarWinds is a company that most folks haven’t heard of unless they work in big companies. They make mainly network management software. So you download it, you install it, it sits on your network, and it lets you know how things are working on your network. It helps things run smoothly. It’s one of the many, many boring enterprise and infrastructure software makers, but it happens to be used by a large percentage of the biggest companies in the United States and the biggest government agencies in the United States.
The hackers compromised SolarWinds, and then who was affected?
So we don’t actually yet know how the hackers got into SolarWinds. There are a number of theories. SolarWinds itself uses software from other providers. So this is what’s known as a supply chain attack, which is among the scariest kinds of attacks. SolarWinds was compromised in order to attack the customers of SolarWinds. But that also may have been how the hackers got into SolarWinds in the first place. It could have been an employee gone bad. It could have been a direct hacking technique that the company hasn’t discovered yet, or at least hasn’t disclosed yet.
But one way or another, they got in there, and got into the code-building environment and, in a very sophisticated way, were able to insert a backdoor into SolarWinds’ Orion network management software code. They did it in such a way that it only happens when the code is being compiled at the last minute. So it was almost impossible to find, but once they were in there, anybody who downloaded at least two relatively recent updates of the Orion software last year downloaded this backdoor. Then it looked around to see where it was, it connected with the original attackers, and then the attackers could decide whether to deploy additional code and really exploit it further.
So the universe of customers that downloaded the tainted SolarWinds code is around 18,000 customers. But so far, it only looks like around 50 of the most important customers got that secondary infection where the attackers were really interested.
That list of customers, that’s the State Department, Treasury, Homeland Security, right? Major parts of the government were involved in this attack.
Right, and major technology companies. So we’re talking Microsoft, Cisco, and some security companies, which is another reason to be alarmed by all of this.
Let me just take one step back and make sure I understand. So … you’re the CIO of a big organization. You’re responsible for the network. You need to set up a bunch of servers and switches and network management tools. You’re probably running some of your own applications in some environment somewhere. You need to manage all that. You buy Orion, which is software from SolarWinds, and that helps you manage your whole network.
That’s right.
Then SolarWinds itself was compromised. We’re saying we kind of don’t know how. The hackers compromised it so badly they were able to inject code into Orion updates that were almost undetectable. Those updates were shipped to 18,000 customers.
Right.
One of the things that really has caught my attention from the beginning is that the State Department didn’t catch it. Treasury didn’t catch it. Microsoft didn’t catch it. A much smaller cybersecurity company caught it. How did that happen?
Yeah, so this is not good. Not just the companies and the agencies that you mentioned, but NSA, the National Security Agency, the home of most of the brains in the government of cybersecurity expertise, they are a customer of SolarWinds. They had this stuff, and they didn’t catch it. And one of the NSA’s many jobs is reviewing code of software suppliers to particularly DoD, one would assume, and they missed it. That’s a really big red flag for the system we have.
So what happened is that FireEye found it, and FireEye, which subsumed Mandiant a few years ago, is among the very best-known, most sophisticated cybersecurity companies on the planet. They’re quite famous in the industry, and justly so. You can imagine that they have all kinds of security, because you might think that security companies aren’t an obvious target for serious hackers, because they’re more likely to be detected. But actually, they’re a major target, because they have awesome access inside all sorts of things, and you can find out how they find out about you as a hacker. So it’s a major prize and a really effective means if you can compromise security software.
But FireEye does have good defenses, and one of the things they have is two-factor authentication for their employees. There was a notification that one of the employees had activated a new device to verify himself coming into the network. So they caught that, and they asked the employee, “Hey, do you have a new phone?” The employee said no, and then FireEye began digging. They’ve done a number of things that were really, really good on this. So one is that they didn’t ignore this as a potential false positive. They actually went nuts, and they couldn’t figure out how the bad guys had gotten in as far as they had. So they went digging for how they could have gotten in, and eliminated basically all the more straightforward ways in. Then they started tearing down the code of the software that was on the servers that were compromised, and that is a nightmare. I mean, that is not something you want to do.
But they did it, and they actually found what had gone wrong. It was in the SolarWinds code, and then they disclosed it. They warned everybody. They found out what the hackers had taken, which included some of their tools for Red Team, pertaining to hacking and testing their clients’ systems. Then they disclosed it, before knowing which of SolarWinds’ customers were affected. They said, “Hey, this needs to get out now,” and that’s how it got out.
What’s been the SolarWinds response?
So SolarWinds, they were notified by FireEye. They confirmed that that code was tainted. They figured out which other editions of their code had the backdoor, and then they hired a bunch of firms. They hired CrowdStrike, which is another prominent security firm, kind of a rival of FireEye. They hired one of the big consulting accounting firms to do a forensics dive, and then they hired this new firm run by Chris Krebs, the former head of the Cybersecurity and Infrastructure Security Agency within [the Department of Homeland Security], and Alex Stamos. They’ve hired them to tell them what to do better in the future, sort of like how to prioritize the culture overhaul and best security practices, because SolarWinds’ name is really on the line here. If you sell something and it’s used to attack all your customers, that’s a potentially existential crisis. So they have a new CEO, by coincidence, and he said that this is jobs one, two, and three for him.
As well it should be an existential crisis, I think.
Yeah.
It feels remarkable to me that this many major American companies have all converged on one software provider for network management. As I think about security broadly, honestly, as I think about the technology industry, it feels like so many things have converged onto one or two providers that they’re rich targets for attackers, right? If you can break SolarWinds, you get the Treasury Department, but you also get Microsoft and Cisco and whoever else, as opposed to if there was a broader spectrum of service providers at this level, the effects of the attacks would be limited. Is that part of the puzzle here, that SolarWinds is just kind of big and dominant and maybe got a little lazy?
So everything you said is true, except for the idea that one or two players allow you broad access. The real problem is that there are 100 companies like this. Yeah, SolarWinds has this major, major position in network management, but there are all these other companies that are also completely dominant. And this has been a known point of weakness for the security posture of the country for a long time. Way back when, at ATstake — actually [Alex] Stamos’s old company — the CTO there wrote a paper about Microsoft being a threat to the world because everybody used it. So if you hack Microsoft, then you can hack everybody. And it’s still true, really, of Microsoft, and SolarWinds, and Oracle, and 100 other companies you’ve never heard of. It’s really scary. And that’s why supply chain attacks are so alarming.
One of the questions I have about supply chain attacks here in this context is the attackers got in, they modified an update, the updates got sent out. We now live in a world of automatic software updates, or, I don’t want to say [we’re] careless, but we’re conditioned to software updates.
That’s right. And we should be.
Right. But the trade-off here is you should update your computer a lot for security reasons because people might try to get in your computer, but then that makes the software update itself a rich, rich target. How has that dynamic changed in the industry?
So automatic software updates are a terrific thing because the vast majority of hacks are not some super-secret zero day nation-state evil genius. It’s garden-variety, known flaws that people haven’t gotten around to patching. And it’s certainly understandable for individuals and smaller companies, but I think people still don’t get that at big companies, they don’t just automatically install updates because they can conflict with other software configurations and crash those. So there has to be a delay as you test it, before you put it out there. But nonetheless, automatic updates are a terrific thing, and nobody should stop automatic updates because of this attack. It’s just that, yeah, that is a grand prize, and there are other bits of that too. So there’s digital certificates, code signing is a really good thing that didn’t used to be a thing, but now there’s an authentication process. So you know that the code comes as authorized by the vendor. Unfortunately, in this SolarWinds attack, they also stole the digital signing. The hackers did that at SolarWinds itself so it looked like it was approved by them. And they’ve done that in other major attacks. It’s still a good thing, but unfortunately it’s kind of turtles all the way down.
It feels like at some point trade-offs have to be made, and the trade-off is to concentrate attack surfaces in one place and ideally have that place be well fortified. But even that can’t be perfect.
What is happening at SolarWinds now? They’ve hired the three companies. They’ve got a new CEO. He says it’s jobs one, two, and three, but there’s also however many clients. There’s the 50 high-value targets, there’s the government. At the simple level of “I’m in charge of the network at the Treasury Department, and I got to fix it,” what are the next steps?
Well, going back a bit, If you’re SolarWinds, you’ve had a pretty lousy holiday break. This stuff was coming out over the break, and we haven’t even talked about the additional vectors. So SolarWinds wasn’t the only way in. Some of the attacker behavior was found at sites where they didn’t download SolarWinds, the tainted updates, or they didn’t have SolarWinds at all. So there’s still unknown ways in. The US government has said that there’s at least password spraying, password guessing, and automated attacks of that sort. It looks like there are others, and it turns out that Microsoft is also a big vector here. Their cloud architecture is complicated and they have a system of resellers that sell you Office 365 if you’re a big company or an agency. And then those resellers frequently maintain access. It looks like some of them were compromised. So that’s how the attackers tried to get into CrowdStrike, the security firm. It also looks like that may have been how they got into Malwarebytes, which is another security firm that doesn’t use SolarWinds. So if you’re in charge of an agency and you’re trying to defend it, one of the things you’re looking for are these other techniques that they came in [through]. Was there another way in?
The bad guys coming in, their first job is to make sure it’s a clean entry and get rid of all of their logs to make it confusing about how they got in. But job two is putting in additional backdoors. Those could be planted in all kinds of places, which is why this cleanup is going to take months or actually years to be sure they’re actually out if you’ve got a really big network. There’s some folks who say you’ve got to burn down the entire network and rebuild it from scratch.
And that might be true for the highest value targets, like bits of the Defense Department, and I should say that there’s no evidence yet that they’ve got into classified networks, which are segregated, but if I was running these agencies I’d want to be sure of that. The other problem is that certainly one of the goals of the attackers was looking at source code, presumably to find additional vulnerabilities. So they did view Microsoft source code. They probably viewed the source code of other big companies. And then they can find new ways of attacks through other means that have nothing to do with SolarWinds. So you got to worry about that too.
The Microsoft part of the story to me is very confusing. So they got into Azure, and then Microsoft basically said, “This isn’t a big deal,” even though all of the reporting around it says, “This is a big deal.” And then Microsoft seems to have come around. Can you just walk me through that? Because as I’ve been looking at it, it seems like everyone wants this to be minimized and not quite so public. And so all of the statements are a little opaque, but tell me what exactly happened with Microsoft here.
We know Microsoft did download the tainted code, but that’s true of thousands of companies. So we reported that, and Microsoft said, “Yeah, but there’s no evidence that they got into any production systems. It looks like it was contained.” I and others reported that, “Well, it looks like Microsoft code was used in various other attacks.” And then a bit later, Microsoft says, “Well okay, actually it looks like they did get into our source code. They could view our source code. They couldn’t modify our source code.” So it’s still unclear. The problem is when you get to this level of crisis, there’s a lot of lawyers and PR people involved — and God love lawyers and PR people, they should all make good livings — but sometimes they’re the enemy of clarity, and what security folks here needed is clarity.
And also these investigations are going to be going on for quite a while. So there are actually a number of companies saying that there is no evidence of X, but … absence of evidence isn’t evidence of absence. We know they were in places where they shouldn’t be. They had control of some Microsoft employee, at least one Microsoft employee account. Maybe more, and the permissions that went with that. And we know Microsoft was used in lots of attacks. We don’t know really how we got from A to B, or if it was just through the Microsoft resellers yet. This will come out at some point.
But it is murky. And there are a number of big victims that haven’t disclosed yet, that haven’t said that they were compromised. And there’ll be at least a trickle, if not a stampede, of companies that say that they were impacted. There’s a weakness in disclosure law where you have to disclose if you’re in one of, I think something like 40 states, and there’s personal information of people that was compromised. You have to disclose that. And if you’re a publicly traded company and it’s material to your revenue, then you have to disclose it. But everything else, by and large, you don’t. That’s actually most breaches.
So one of the worst examples of that is when the F-35 plans got compromised, and you would see these major defense companies, which I knew had been hacked, not disclose it in their SEC reports. And the reason is that it’s not material, because the government isn’t going to buy an F-35 from somebody else. So it actually doesn’t hurt their bottom line. There are big systemic incentive issues here, and that includes disclosure, and that’s why it’s patchwork. And in my opinion, there should be a federal law that sort of harmonizes this sort of thing.
I very pointedly have not asked you who was responsible for this, because I think that deserves some focus. So to the best of our knowledge, who was responsible for these attacks?
The US intelligence community, and the top-flight, private people who often, unfortunately, know as much as the government people, all say: signs point to Russia. There’s some discrepancy over which agency within Russia. The tradecraft is really, really high.
When you say the tradecraft is high, what specifically do you mean?
The amazing disappearing code. They left almost no artifacts from their code. So there’d be all this careful work, de-obfuscating what happened, and then they get there and then there’s nothing. They made all their code disappear. That thing, the fact that they got the code signing certificate, the fact that the backdoor was inserted only when a certain product was being compiled at the last minute — that’s all really, really high-end stuff that pretty much by itself rules out anything but a nation-state. People underestimate— this is a complicated area because if you’re the victim of a hack, particularly if you’re a publicly traded company with investors, you definitely want to say this was a nation-state of evil geniuses that you couldn’t possibly have defended against. And sometimes it turns out to be a 16-year-old, and that’s embarrassing. That actually happened with Twitter. I’m not saying that Twitter said it was a nation-state, but it looked like a very sophisticated hack, and then it literally turned out to be teenagers.
This is not that. This really is a nation-state. I would be very surprised if it’s not Russia, which does deny it. There is some overlapping code. It is also true that countries have gotten good at imitating each other’s stuff, but I can tell you it wasn’t the US. And it was somebody that is not after money, because while there was a broad net, the 50 or so known secondary targets they really went after are classic espionage targets. The Department of Defense, the State Department, the Treasury Department, the Commerce Department. That pretty much rules out an economic motive. So we’re down to a handful of suspects, and there’s no reason to not believe the US assessment that it’s Russia.
It’s complicated that the hack happened in an election year. The Trump administration, and President Trump himself, was strangely cozy with Russia. Now, there’s a new administration. It seems like Biden’s going to take a more aggressive posture toward Russia. In the middle of all this, there’s the election security noise, Trump fires Christopher Krebs, who is in charge of cybersecurity. How does that all play into this? Is it “We didn’t want to say it was Russia too loudly, and now we’re saying it a little more loudly”? Is it “The Trump administration did not have a good cybersecurity infrastructure”? The shape of that seems really fuzzy to me.
I agree. I don’t think that’s something that we know yet. It may have been a bit of insurance on the part of the Russians that, under Trump, the United States government did not aggressively punish — particularly, the executive branch did not aggressively punish — Russia for a lot of really bad behavior that other governments would have done more about. There were sanctions, but it was driven by congressional action. So you don’t have to be a big conspiracy theorist to say that the Russians [would] think, “Well, we’ve gotten away with invading Ukraine. We can do this big hack, and even if we get caught, this is the least likely White House in memory to sound the alarm and rattle a saber at us.” So from here, I certainly doubt that there was any foreknowledge of this in the administration. But from the attacker perspective, the US is distracted, and has a president that’s less likely than predecessors, even if he finds us, to yell and scream and threaten sanctions if we are found out.
We’ve got a new president now. Has the posture toward particularly this act, but also Russia cyber operations, shifted at all yet?
[Sighs]
That was a very telling sigh.
The administration’s been on the job for a few days, and in the heated political atmosphere, there’s a wide spectrum of noise before things are sorted out. During the transition, Biden said that this is something specifically that is going to be responded to in a big way. Other folks have said this is an act of war. The people who’ve been at this for a long time and don’t have a particular axe to grind are not saying it’s an act of war. There’s no evidence of destruction. There’s no human life lost. This is classic espionage. It’s just that we got owned really badly. It’s similar, in my mind, to the hack of the Office of Personnel Management widely attributed to China some years ago where they got classified personnel files on the majority of folks in the US government, and outside of the US government, with a secret clearance or above. That was really, really bad, but it wasn’t warfare. That’s an espionage win. And we’re trying to do exactly that sort of thing to China and Russia and other governments.
Introducing Decoder, a new podcast about big ideas and other problems. Join The Verge’s Nilay Patel as he explores the future of business, technology, and the world they create. Subscribe now!
But I think it’s clear that there will be some kind of response. There are going to be hearings on this, but ... like many things cyber, it has a lot of aspects. Like, “Are we sure that country X was behind this? Can we prove that to the world’s satisfaction? And then do we respond economically or diplomatically and other ways?” Hopefully not militarily, though I suppose that’s a possibility. And then, “How do we stop this from happening again?” And that’s really hard and complicated.
It involves defense versus offense. It involves asking how you secure the supply chain. What do you do about employees in other countries and contractors in other countries? It’s similar to the trade war with China. Our computers and their software … go back and forth dozens of times before they wind up on your desk. And it is pretty impossible to secure completely. So what do you do about that? Do you try and undo all these global relationships because you’re sometimes rivals? If you do, you’re going to hurt the economy in a pretty major way. So there are big thorny issues, and it’d be nice if the new administration and Congress take that seriously and come up with a plan. It hasn’t really happened before.
A breach of this scale, involving the biggest companies in America and the American government itself, is usually the thing that catalyzes change, that leads to a disclosure law or a reframing of the American posture toward offensive cyberattacks. But because of the transition [from Trump to the Biden administration] and the sort of instant quiet from the attacked parties, it doesn’t feel like this is that moment. Is there a group of people who are going to tackle this in the Biden administration? Does he have an appointee set who has the expertise to raise the profile of this again and build the political capital to actually make the change?
Well, that’s really interesting. And this is playing out in real time. Biden has appointed most of the top cyber people. As of this morning, there were one or two key holes. But among other things, for the first time, there’s a deputy national security adviser for cyber, Anne Neuberger. Anne is very well-regarded, was at the NSA for many, many years, and, among other things, was fulfilling part of the NSA’s mission to cooperate with industries on defense. There are a number of people who have really strong military and government experience. As of this weekend, the suspected new cyber czar inside the White House is Jen Easterly, who was one of the people that helped create Cyber Command as a separate unit of the Pentagon, the people responsible for running cyberattacks in other countries.
So you have really intelligent, really experienced people. Do they have the kind of broad, blue-sky, strategic thinking that might help turn around this really gnarly problem? I don’t know. We’ll see. The fact that both houses of Congress are from the same party probably helps, as does the fact that a lot of this isn’t that partisan in a terribly polarized environment. Nobody’s a big fan of getting hacked to pieces by the other countries. So I am more optimistic now than I have been in 20 years of covering this, but that doesn’t mean I’d actually bet on a complete turnaround.
So you’re starting from a low bar, is what you’re saying.
Yes.
You mentioned that this looks like an espionage operation. What’s interesting about that is you come from Microsoft’s view of espionage, but kind of the way Americans would see it is, “Well, that’s Microsoft’s problem. We don’t need the government or the Pentagon...” or “That doesn’t merit a military response,” which is kind of what you’re describing. But at the same time, this is a major national security problem. How does that play together?
So that’s a really good question. And you put your finger on one of, I think, the biggest issues here, but let me separate out the Pentagon offense response part of it. The US government responded when the North Koreans attacked Sony for dumb reasons. I can’t believe we’re in an era where national governments attack each other over dumb movies, but here we are.
You’re talking about when North Korea hacked Sony over the movie The Interview about a pair of journalists — played by Seth Rogen and James Franco — who go to North Korea to interview Kim Jong Un and eventually assassinate him. Which North Korea obviously didn’t appreciate.
It wasn’t even a good movie. It’s not like it was Chinatown, or...
We gave the movie a bad review and Seth Rogen was very mad at us. It was like the weirdest outcome of that entire debacle, that Seth Rogen was mad at The Verge.
For a variety of reasons after intense discussions at the highest levels of government, the US had a response mostly under the table, but there was some cyber action taken against North Korea. So sometimes that is appropriate. Just because another government attacks a piece of civilian infrastructure doesn’t mean there should not be any response. I mean if a private power utility gets taken down in the United States by, say, Iran. We certainly reserve the right to attack Iran militarily over that. Again, this isn’t that. This is espionage.
Very, very rarely do you see military response to legit espionage targets being attacked. But there’s the separate issue, which you hit on, which is, is this a Microsoft problem? In my opinion, it is not fair to expect private companies, no matter how large, to fend off entire nation-states. The job of the US government should be to defend private enterprise from other countries.
It’s really, really hard when you try to get into the weeds on that, because sometimes a nation-state will use the same techniques as a 16- or 17-year-old. So there should be some reasonable standard of defense that is expected of companies. But again, at the really, really high end, if the Russians got into NSA or the Chinese got into the classified personnel files, it doesn’t matter how big a company you are. You’re going to get owned if they really want you to. So that is one of the big strategic issues that I would hope that the White House and Congress address. Where do you draw the line? What kind of help can be provided? And what’s overreach?
I think about the big nation-state actors that you would think about. The Russians, the Chinese and North Koreans, the Iranians. There’s a tighter nexus between government and industry in those countries. I think that’s the most polite way of saying it. Where the government does some espionage, and they’re going to hand over that information potentially to one of the large corporations. There’s a level of corruption, or maybe state design, there that incentivizes both sides stacked as one. I don’t think we have that here … our incentives [are] to go off and have our government hack a bunch of companies in other countries. Their incentives are very different. So how do you align all of that stuff here beyond just simple deterrence: If you hack us, we’ll hack you back?
Again, one of those big, hairy aspects to this. You’re right that there’s tighter integration between private companies and the state in many other nations, and it not only helps them on offense, it helps them on defense. If the government can simply order everybody in the power industry to apply a patch, then they do it. The United States government does not actually have that power, which is kind of scary. It is also true that the companies that are based in the United States, many of them get a majority of their money overseas. So even if they think of themselves as — or for regulatory purposes, they are — American companies, that doesn’t mean they’re going to march in lockstep with the American government, which makes it really interesting.
But remember, you talked about the US government hacking companies in other countries, well, sometimes, as Snowden showed us, they’re hacking American companies in other places. They did that to Google and others in order to spy. So it’s really messy. If there is this sort of retreat, the sort of trade war 10x, where we need a completely American supply chain, then I guess you would have tighter integration, or at least a framework for tighter integration, between the government and the private sector.
That’s not obviously a good thing. That would erode American business overseas. And the economy is part of what goes into national security. A stronger economy’s in a better position to defend itself. So it gets quite tricky. And I think one of the big problems in discussing any of this in a grown-up way is the overclassification that we have in this country. I mean, the government tells even members of Congress very little about capabilities and what’s actually happening out there. And so there is this kind of shadow warfare happening, this low level stuff, lots and lots of cyberoperations happening. And even members of Congress don’t know what’s going on. So it is hard to have a grown-up discussion about these sorts of trade-offs without greater openness.
One of the questions I’ve been wrestling with; we do it too. As you’ve been saying—
In spades. Yes.
Yeah. The United States hacks a lot of things all the time inside and outside of the country. Is there a way to just pull that back internationally? Is there a treaty or a set of regulations or rules of engagement that exist or have been proposed, to say, “Hey, this is getting out of hand and all of our economies are [under] threat.”
So in theory, yes. I mean, you can certainly have international agreements. For many years under the Obama administration, it was all about norms, setting norms. And there was an attempt to do that kind of globally, and then, “Well, maybe we’ll do it in a given region.” And it hasn’t really led to very much. And when there’ve been big tests, like it is widely understood that Russia shut out the lights in Ukraine a couple of Christmases ago. The NotPetya attacks were terrible. The WannaCry attacks, which shut down hospitals, you would think that that would have given rise to some sort of international agreements, if anything would, and it hasn’t yet. The president of Microsoft, Brad Smith, has been kind of a spokesman on this for the industry; [he] talks about a Geneva Convention sort of a thing that would, among other things, exempt civilian infrastructure from these sorts of attacks.
But the problem is verification. I mean, we can barely verify nuclear treaty compliance. There are so many proxies, both literal and figurative. The line between nation-state and crime is deliberately fuzzed up in a number of places. It’s really, really hard to ensure that something wouldn’t happen. And again ... there has never been a deal that really stopped intelligence exploitation. So if there is going to be some kind of global treaty or understanding or something, it would probably be about physical destruction and not about intelligence gathering, which, again, this is.
Several months ago, I interviewed Andy Greenberg, who wrote Sandworm, which is a book about NotPetya and shutting off the lights in Ukraine. His argument to me was that was the moment when the Obama administration could have stopped it, but instead they set a new norm that this is how we’re doing it. We reserve the right to attack back. The international outcry around turning off the lights in Ukraine was not big enough. And the message was sent, this is in some way acceptable behavior. Do you think that’s grown? Was that the moment, was that an inflection point, or has it just been a steady drumbeat of other things?
I think that was a missed opportunity. … And I think a lot of people think more should have been done there, but again, that was Russia attacking Ukraine. It wasn’t attacking us. I think the response would have been very different if Russia had attacked us in that way. Unfortunately, Ukraine, like some of the Middle East, is kind of a proving ground for a lot of these capabilities. We haven’t seen a lot of what nation-states could do to key parts of our infrastructure because people ... haven’t so far felt like it’s worth it to bloody our nose, but they can; we probably can too. So I think if Ukraine had been in a position to respond, it would have. You could certainly make an argument that NATO should have done something. We’ll see if, with the passage of time, with people who care more about international alliances, if that stuff gets firmed up.
I think a lot of NATO saw the lack of response to that as a mistake. So we’ll see how strongly the Biden administration feels about NATO and this issue. There are a lot of things in play, but I think cyber is back on the table now. The Biden administration is going to make it a priority and I think it’s going to be a part of basically all major conflicts going forward, so there’s no reason why it would not also be subject to treaties, agreements, responses, norms, that sort of thing.
Just before we started recording, you were saying there’s a little bit of a tie between the book you just wrote, Cult of the Dead Cow, and some of the characters involved in the SolarWinds response — explain that to me.
Well, the book tells the history of the “good guy impulse” in security, and it tells the story of these old-school hackers from before there was a web, from back in the 80s, to the present day, and their sort of moral evolution, and a lot of them turned pro. They were teenagers, pranksters and whatever, and they turned pro in various ways and they invented hacktivism. But one of the things they did is that in order to make a difference, some of them went to work inside the government, including DARPA, and some of them went into the private sector and founded really important security companies because that was, they thought, the best way to actually help with security. And one of the companies they founded was called ATstake, which was an early security boutique where hackers went inside companies like Microsoft, and big banks, and whatever, and said, “This is what you’re doing wrong.” It sort of brought the hacker mindset inside these companies.
A lot of really amazing people came out of ATstake and went to work inside every major tech company in the United States, and one of them is Alex Stamos. So he joined ATstake because he admired these guys who had testified before Congress about how fragile the internet was. And Stamos went on from there to, among other things, work at Yahoo as a chief security guy, and then at Facebook where he dealt with the Russian disinformation in 2016. And now he teaches at Stanford on big disinformation and related issues, and he is in business with Chris Krebs, [who is] ex-US government, ex-Microsoft, and they’re running around trying to help SolarWinds and other companies learn the lessons of this latest hacking spree.
So it’s kind of this continuum, and it’s like there’s these coaching trees where great basketball coach X trained all these people, who then went on and coached all these other teams, and they all go back to the same root of the tree. The Cult of the Dead Cow story is kind of about that.
Alex has been in and around The Verge and our shows for a long time. One of the things he has said about the 2016 election was, we were all focused on the wrong thing at Facebook. They didn’t know what it would look like, and then everyone in 2020 was worried about Russian disinformation and the threat ended up looking very different and coming from a different place. I think Chris Krebs has said something similar: we were all focused on the election, we missed this thing. How do you build a broad enough array of sensors and detection mechanisms to say, okay, there’s a major election going, but there are these gigantic fat targets in the United States that need constant surveillance and protection?
Well, first of all, it’s in the nature to be fighting the last thing, to be fighting the last war. I’m not sure that we still need to be taking off our shoes at the airport because one guy tried that thing that one time. Your really high-end terrorists or other adversaries are not going to do the same thing that they’ve already taught us how to defend against, and this is a classic case of that. As to how you defend against this kind of intelligence-driven attack, I think you need to deal with some really fundamental questions that include supply chain, vendor relationship, and to my mind, it definitely includes the defense-offense balance. I did some stories a few years ago that said that 90 percent of what the US government spends on cyber-y things, at least back then, was about offense or intelligence gathering, not about defense.
And the NSA is the agency, at the time, [that] was charged with defense. Now DHS has taken up some of that, but their budget is minuscule compared to the NSA’s cyber budget. And the NSA had a whole division, the Information Assurance Division, that was responsible for at least securing the US government, and after the Snowden debacle, the Obama White House had a commission of five folks to look at this and figure out a whole bunch of things that we should do different to avoid another Snowden situation. And one of their recommendations was [to] spin out the defensive division of the NSA from NSA proper, because nobody trusts the NSA, and because the offensive mission so dominates that you can’t be sure that they’re not going to subvert defense — which in fact they did, and that would emerge from the Snowden leaks. But they didn’t spin it out. Instead they did a disappearing act where NSA scattered the defensive mission lower down within the agency so there was no longer a No. 2 at NSA whose mission was defense. They got rid of that position. So they went the opposite way.
I think defense has to be prioritized a lot more than it has been, because if everybody’s good at offense, then you’re going to have a lot more offense around the world. The way to actually win is to get really, really good at defense and I think we have not been focused that way at all.
That “if everyone’s good at offense, you just end up with more offense,” that sounds to me like pretty classic foreign policy, military deterrence language, right? I mean, this is the Cold War, right? You’re speaking in the language of the Cold War in some way. What breaks it? Is it just that we get so good at defense that no one tries anymore?
That’d be nice, wouldn’t it? People will keep trying, but you can raise the cost. It obviously has to be a multi-faceted thing. I mean, if you’re talking about the Cold War, then the big thing is deterrence and the certainty that country X will retaliate against country Y. It’s true we have not seen that, or not very much of that, so that needs to be done too. But what I’m talking about is more Reagan’s Star Wars vision of being able to shoot down missiles as they come at us. Now that’s what would actually change things.
So yeah, the idea is that they would keep trying to get into Langley, or Maryland, and would fail. That would be super awesome, but we haven’t actually even tried. There’s really cool stuff happening in the private sector. There’s been a lot of advances in defense, but there hasn’t been a government-wide embrace of that initiative, where you put lots and lots of money into the National Academy of Sciences or others to really be fed up. There’s Skunk Works inside DARPA, [but] there hasn’t been a giant thing and there needs to be a giant thing.
Do you think there’s a cultural shift with the new generation of lawmakers? I mean, we have some younger lawmakers now. We have lots of younger people who’ve come up in things like Cult of the Dead Cow in parts of the government. People are good at computers now in a way that maybe they weren’t so good at computers 10 years ago.
Yes, this is one of the good things. I mean, in the olden days when I started covering this, the only good thing you could say about cybersecurity was, “Well, awareness is rising.” And now it’s true. There are people in Congress that actually understand. There are actual engineers in Congress. This is a new thing. You still have a hearing where they drag in [Facebook CEO Mark] Zuckerberg and members of Congress ask embarrassing questions, but it is a big change from where it was, and there are tech-savvy staffers at all levels. They’re digital natives and they understand these trade-offs. I think that there’s a better chance than we’ve ever had of people having a real discussion about this.
But again, I am concerned more about the establishment, the four-star generals, the people running intelligence agencies, people in the White House who still think of warfare and intelligence in the old terms and don’t get into questions of the private sector versus the public sector stuff because there aren’t really straightforward answers. Right now, our government has been so dysfunctional that you couldn’t get the two houses to agree on pizza toppings, so how are you going to tackle something like this? I mean, the Chamber of Commerce, the private lobbying group, was outraged that folks in the Department of Energy and [Homeland Security] wanted to put out voluntary guidelines for best practices to protect nuclear plants or power plants from hackers, because they thought that was a slippery slope that would lead to more regulation. We can’t have that crap anymore. We need people actually willing to give and take and deal with complicated issues, or we’re going to keep getting owned like this.
What’s the next thing you’re looking for, specific to the SolarWinds story? What’s the next turn of the screw, do you think?
Well, a turn of the screw would be [if] giant companies, X, Y, and Z say that they were also breached by the same folks, they also had access to their source code. Maybe there were other updates that got their customers, that sort of thing. That’s pretty likely. There may be, in these hearings, or there will likely be an intelligence report that is made public about why they think it’s Russia. And there will probably be some big review over practices and some other commission. On the other hand, maybe it’s going to wind up like the Snowden commission where some of the stuff gets adopted and some of it doesn’t. So I don’t know about that. I think a federal disclosure law is plausible because companies don’t want to have to deal with this patchwork of states. It would also be nice if there’s a federal privacy law. So maybe those initiatives go together and both of those things happen.
On today’s episode of Decoder, I’m joined by Joseph Menn, a reporter at Reuters who focuses on cybersecurity investigations and the author of the new book Cult of the Dead Cow: How the Original Hacking Supergroup Might Just Save the World. We discuss what this breach means for US security and the companies in SolarWinds’ supply chain that might have been affected.
The SolarWinds hack hasn’t really gotten the attention it deserves because it happened during the chaos after the presidential election — but it’s a big deal. And it raises a lot of questions about how to respond to such a massive attack and the responsibility of the private sector when it comes to national security. There aren’t a lot of easy answers here, but it’s clear that change is coming with the Biden administration.
Okay, Joseph Menn. Here we go.
Below is a lightly edited excerpt from our conversation.
Joseph Menn, you’re the technology projects reporter at Reuters. You focus on investigative cybersecurity stories. Welcome to Decoder.
Thanks for having me.
I really wanted to have you on, because I feel like in the transition between the Trump administration and the Biden administration, the story of the SolarWinds hack and its fallout kind of got lost. I know it’s a huge story, but so many other things [are happening] in our country that, to me, it feels like it’s not getting the attention that it deserves. Is that kind of your sense of it, too?
Well, I agree with that, and I think it’s by design. The roots of this attack go back more than a year, but the activity escalated, and it came at a perfect time, because the US was so distracted. In particular, the US government, and even the security people within the US government, were busy worrying about securing the elections. So you couldn’t have picked a better time to launch a massive spying attack on the government.
So from the very beginning, tell people what SolarWinds is and how it was attacked.
Sure. So SolarWinds is a company that most folks haven’t heard of unless they work in big companies. They make mainly network management software. So you download it, you install it, it sits on your network, and it lets you know how things are working on your network. It helps things run smoothly. It’s one of the many, many boring enterprise and infrastructure software makers, but it happens to be used by a large percentage of the biggest companies in the United States and the biggest government agencies in the United States.
The hackers compromised SolarWinds, and then who was affected?
So we don’t actually yet know how the hackers got into SolarWinds. There are a number of theories. SolarWinds itself uses software from other providers. So this is what’s known as a supply chain attack, which is among the scariest kinds of attacks. SolarWinds was compromised in order to attack the customers of SolarWinds. But that also may have been how the hackers got into SolarWinds in the first place. It could have been an employee gone bad. It could have been a direct hacking technique that the company hasn’t discovered yet, or at least hasn’t disclosed yet.
But one way or another, they got in there, and got into the code-building environment and, in a very sophisticated way, were able to insert a backdoor into SolarWinds’ Orion network management software code. They did it in such a way that it only happens when the code is being compiled at the last minute. So it was almost impossible to find, but once they were in there, anybody who downloaded at least two relatively recent updates of the Orion software last year downloaded this backdoor. Then it looked around to see where it was, it connected with the original attackers, and then the attackers could decide whether to deploy additional code and really exploit it further.
So the universe of customers that downloaded the tainted SolarWinds code is around 18,000 customers. But so far, it only looks like around 50 of the most important customers got that secondary infection where the attackers were really interested.
That list of customers, that’s the State Department, Treasury, Homeland Security, right? Major parts of the government were involved in this attack.
Right, and major technology companies. So we’re talking Microsoft, Cisco, and some security companies, which is another reason to be alarmed by all of this.
Let me just take one step back and make sure I understand. So … you’re the CIO of a big organization. You’re responsible for the network. You need to set up a bunch of servers and switches and network management tools. You’re probably running some of your own applications in some environment somewhere. You need to manage all that. You buy Orion, which is software from SolarWinds, and that helps you manage your whole network.
That’s right.
Then SolarWinds itself was compromised. We’re saying we kind of don’t know how. The hackers compromised it so badly they were able to inject code into Orion updates that were almost undetectable. Those updates were shipped to 18,000 customers.
Right.
One of the things that really has caught my attention from the beginning is that the State Department didn’t catch it. Treasury didn’t catch it. Microsoft didn’t catch it. A much smaller cybersecurity company caught it. How did that happen?
Yeah, so this is not good. Not just the companies and the agencies that you mentioned, but NSA, the National Security Agency, the home of most of the brains in the government of cybersecurity expertise, they are a customer of SolarWinds. They had this stuff, and they didn’t catch it. And one of the NSA’s many jobs is reviewing code of software suppliers to particularly DoD, one would assume, and they missed it. That’s a really big red flag for the system we have.
So what happened is that FireEye found it, and FireEye, which subsumed Mandiant a few years ago, is among the very best-known, most sophisticated cybersecurity companies on the planet. They’re quite famous in the industry, and justly so. You can imagine that they have all kinds of security, because you might think that security companies aren’t an obvious target for serious hackers, because they’re more likely to be detected. But actually, they’re a major target, because they have awesome access inside all sorts of things, and you can find out how they find out about you as a hacker. So it’s a major prize and a really effective means if you can compromise security software.
But FireEye does have good defenses, and one of the things they have is two-factor authentication for their employees. There was a notification that one of the employees had activated a new device to verify himself coming into the network. So they caught that, and they asked the employee, “Hey, do you have a new phone?” The employee said no, and then FireEye began digging. They’ve done a number of things that were really, really good on this. So one is that they didn’t ignore this as a potential false positive. They actually went nuts, and they couldn’t figure out how the bad guys had gotten in as far as they had. So they went digging for how they could have gotten in, and eliminated basically all the more straightforward ways in. Then they started tearing down the code of the software that was on the servers that were compromised, and that is a nightmare. I mean, that is not something you want to do.
But they did it, and they actually found what had gone wrong. It was in the SolarWinds code, and then they disclosed it. They warned everybody. They found out what the hackers had taken, which included some of their tools for Red Team, pertaining to hacking and testing their clients’ systems. Then they disclosed it, before knowing which of SolarWinds’ customers were affected. They said, “Hey, this needs to get out now,” and that’s how it got out.
What’s been the SolarWinds response?
So SolarWinds, they were notified by FireEye. They confirmed that that code was tainted. They figured out which other editions of their code had the backdoor, and then they hired a bunch of firms. They hired CrowdStrike, which is another prominent security firm, kind of a rival of FireEye. They hired one of the big consulting accounting firms to do a forensics dive, and then they hired this new firm run by Chris Krebs, the former head of the Cybersecurity and Infrastructure Security Agency within [the Department of Homeland Security], and Alex Stamos. They’ve hired them to tell them what to do better in the future, sort of like how to prioritize the culture overhaul and best security practices, because SolarWinds’ name is really on the line here. If you sell something and it’s used to attack all your customers, that’s a potentially existential crisis. So they have a new CEO, by coincidence, and he said that this is jobs one, two, and three for him.
As well it should be an existential crisis, I think.
Yeah.
It feels remarkable to me that this many major American companies have all converged on one software provider for network management. As I think about security broadly, honestly, as I think about the technology industry, it feels like so many things have converged onto one or two providers that they’re rich targets for attackers, right? If you can break SolarWinds, you get the Treasury Department, but you also get Microsoft and Cisco and whoever else, as opposed to if there was a broader spectrum of service providers at this level, the effects of the attacks would be limited. Is that part of the puzzle here, that SolarWinds is just kind of big and dominant and maybe got a little lazy?
So everything you said is true, except for the idea that one or two players allow you broad access. The real problem is that there are 100 companies like this. Yeah, SolarWinds has this major, major position in network management, but there are all these other companies that are also completely dominant. And this has been a known point of weakness for the security posture of the country for a long time. Way back when, at ATstake — actually [Alex] Stamos’s old company — the CTO there wrote a paper about Microsoft being a threat to the world because everybody used it. So if you hack Microsoft, then you can hack everybody. And it’s still true, really, of Microsoft, and SolarWinds, and Oracle, and 100 other companies you’ve never heard of. It’s really scary. And that’s why supply chain attacks are so alarming.
One of the questions I have about supply chain attacks here in this context is the attackers got in, they modified an update, the updates got sent out. We now live in a world of automatic software updates, or, I don’t want to say [we’re] careless, but we’re conditioned to software updates.
That’s right. And we should be.
Right. But the trade-off here is you should update your computer a lot for security reasons because people might try to get in your computer, but then that makes the software update itself a rich, rich target. How has that dynamic changed in the industry?
So automatic software updates are a terrific thing because the vast majority of hacks are not some super-secret zero day nation-state evil genius. It’s garden-variety, known flaws that people haven’t gotten around to patching. And it’s certainly understandable for individuals and smaller companies, but I think people still don’t get that at big companies, they don’t just automatically install updates because they can conflict with other software configurations and crash those. So there has to be a delay as you test it, before you put it out there. But nonetheless, automatic updates are a terrific thing, and nobody should stop automatic updates because of this attack. It’s just that, yeah, that is a grand prize, and there are other bits of that too. So there’s digital certificates, code signing is a really good thing that didn’t used to be a thing, but now there’s an authentication process. So you know that the code comes as authorized by the vendor. Unfortunately, in this SolarWinds attack, they also stole the digital signing. The hackers did that at SolarWinds itself so it looked like it was approved by them. And they’ve done that in other major attacks. It’s still a good thing, but unfortunately it’s kind of turtles all the way down.
It feels like at some point trade-offs have to be made, and the trade-off is to concentrate attack surfaces in one place and ideally have that place be well fortified. But even that can’t be perfect.
What is happening at SolarWinds now? They’ve hired the three companies. They’ve got a new CEO. He says it’s jobs one, two, and three, but there’s also however many clients. There’s the 50 high-value targets, there’s the government. At the simple level of “I’m in charge of the network at the Treasury Department, and I got to fix it,” what are the next steps?
Well, going back a bit, If you’re SolarWinds, you’ve had a pretty lousy holiday break. This stuff was coming out over the break, and we haven’t even talked about the additional vectors. So SolarWinds wasn’t the only way in. Some of the attacker behavior was found at sites where they didn’t download SolarWinds, the tainted updates, or they didn’t have SolarWinds at all. So there’s still unknown ways in. The US government has said that there’s at least password spraying, password guessing, and automated attacks of that sort. It looks like there are others, and it turns out that Microsoft is also a big vector here. Their cloud architecture is complicated and they have a system of resellers that sell you Office 365 if you’re a big company or an agency. And then those resellers frequently maintain access. It looks like some of them were compromised. So that’s how the attackers tried to get into CrowdStrike, the security firm. It also looks like that may have been how they got into Malwarebytes, which is another security firm that doesn’t use SolarWinds. So if you’re in charge of an agency and you’re trying to defend it, one of the things you’re looking for are these other techniques that they came in [through]. Was there another way in?
The bad guys coming in, their first job is to make sure it’s a clean entry and get rid of all of their logs to make it confusing about how they got in. But job two is putting in additional backdoors. Those could be planted in all kinds of places, which is why this cleanup is going to take months or actually years to be sure they’re actually out if you’ve got a really big network. There’s some folks who say you’ve got to burn down the entire network and rebuild it from scratch.
And that might be true for the highest value targets, like bits of the Defense Department, and I should say that there’s no evidence yet that they’ve got into classified networks, which are segregated, but if I was running these agencies I’d want to be sure of that. The other problem is that certainly one of the goals of the attackers was looking at source code, presumably to find additional vulnerabilities. So they did view Microsoft source code. They probably viewed the source code of other big companies. And then they can find new ways of attacks through other means that have nothing to do with SolarWinds. So you got to worry about that too.
The Microsoft part of the story to me is very confusing. So they got into Azure, and then Microsoft basically said, “This isn’t a big deal,” even though all of the reporting around it says, “This is a big deal.” And then Microsoft seems to have come around. Can you just walk me through that? Because as I’ve been looking at it, it seems like everyone wants this to be minimized and not quite so public. And so all of the statements are a little opaque, but tell me what exactly happened with Microsoft here.
We know Microsoft did download the tainted code, but that’s true of thousands of companies. So we reported that, and Microsoft said, “Yeah, but there’s no evidence that they got into any production systems. It looks like it was contained.” I and others reported that, “Well, it looks like Microsoft code was used in various other attacks.” And then a bit later, Microsoft says, “Well okay, actually it looks like they did get into our source code. They could view our source code. They couldn’t modify our source code.” So it’s still unclear. The problem is when you get to this level of crisis, there’s a lot of lawyers and PR people involved — and God love lawyers and PR people, they should all make good livings — but sometimes they’re the enemy of clarity, and what security folks here needed is clarity.
And also these investigations are going to be going on for quite a while. So there are actually a number of companies saying that there is no evidence of X, but … absence of evidence isn’t evidence of absence. We know they were in places where they shouldn’t be. They had control of some Microsoft employee, at least one Microsoft employee account. Maybe more, and the permissions that went with that. And we know Microsoft was used in lots of attacks. We don’t know really how we got from A to B, or if it was just through the Microsoft resellers yet. This will come out at some point.
But it is murky. And there are a number of big victims that haven’t disclosed yet, that haven’t said that they were compromised. And there’ll be at least a trickle, if not a stampede, of companies that say that they were impacted. There’s a weakness in disclosure law where you have to disclose if you’re in one of, I think something like 40 states, and there’s personal information of people that was compromised. You have to disclose that. And if you’re a publicly traded company and it’s material to your revenue, then you have to disclose it. But everything else, by and large, you don’t. That’s actually most breaches.
So one of the worst examples of that is when the F-35 plans got compromised, and you would see these major defense companies, which I knew had been hacked, not disclose it in their SEC reports. And the reason is that it’s not material, because the government isn’t going to buy an F-35 from somebody else. So it actually doesn’t hurt their bottom line. There are big systemic incentive issues here, and that includes disclosure, and that’s why it’s patchwork. And in my opinion, there should be a federal law that sort of harmonizes this sort of thing.
I very pointedly have not asked you who was responsible for this, because I think that deserves some focus. So to the best of our knowledge, who was responsible for these attacks?
The US intelligence community, and the top-flight, private people who often, unfortunately, know as much as the government people, all say: signs point to Russia. There’s some discrepancy over which agency within Russia. The tradecraft is really, really high.
When you say the tradecraft is high, what specifically do you mean?
The amazing disappearing code. They left almost no artifacts from their code. So there’d be all this careful work, de-obfuscating what happened, and then they get there and then there’s nothing. They made all their code disappear. That thing, the fact that they got the code signing certificate, the fact that the backdoor was inserted only when a certain product was being compiled at the last minute — that’s all really, really high-end stuff that pretty much by itself rules out anything but a nation-state. People underestimate— this is a complicated area because if you’re the victim of a hack, particularly if you’re a publicly traded company with investors, you definitely want to say this was a nation-state of evil geniuses that you couldn’t possibly have defended against. And sometimes it turns out to be a 16-year-old, and that’s embarrassing. That actually happened with Twitter. I’m not saying that Twitter said it was a nation-state, but it looked like a very sophisticated hack, and then it literally turned out to be teenagers.
This is not that. This really is a nation-state. I would be very surprised if it’s not Russia, which does deny it. There is some overlapping code. It is also true that countries have gotten good at imitating each other’s stuff, but I can tell you it wasn’t the US. And it was somebody that is not after money, because while there was a broad net, the 50 or so known secondary targets they really went after are classic espionage targets. The Department of Defense, the State Department, the Treasury Department, the Commerce Department. That pretty much rules out an economic motive. So we’re down to a handful of suspects, and there’s no reason to not believe the US assessment that it’s Russia.
It’s complicated that the hack happened in an election year. The Trump administration, and President Trump himself, was strangely cozy with Russia. Now, there’s a new administration. It seems like Biden’s going to take a more aggressive posture toward Russia. In the middle of all this, there’s the election security noise, Trump fires Christopher Krebs, who is in charge of cybersecurity. How does that all play into this? Is it “We didn’t want to say it was Russia too loudly, and now we’re saying it a little more loudly”? Is it “The Trump administration did not have a good cybersecurity infrastructure”? The shape of that seems really fuzzy to me.
I agree. I don’t think that’s something that we know yet. It may have been a bit of insurance on the part of the Russians that, under Trump, the United States government did not aggressively punish — particularly, the executive branch did not aggressively punish — Russia for a lot of really bad behavior that other governments would have done more about. There were sanctions, but it was driven by congressional action. So you don’t have to be a big conspiracy theorist to say that the Russians [would] think, “Well, we’ve gotten away with invading Ukraine. We can do this big hack, and even if we get caught, this is the least likely White House in memory to sound the alarm and rattle a saber at us.” So from here, I certainly doubt that there was any foreknowledge of this in the administration. But from the attacker perspective, the US is distracted, and has a president that’s less likely than predecessors, even if he finds us, to yell and scream and threaten sanctions if we are found out.
We’ve got a new president now. Has the posture toward particularly this act, but also Russia cyber operations, shifted at all yet?
[Sighs]
That was a very telling sigh.
The administration’s been on the job for a few days, and in the heated political atmosphere, there’s a wide spectrum of noise before things are sorted out. During the transition, Biden said that this is something specifically that is going to be responded to in a big way. Other folks have said this is an act of war. The people who’ve been at this for a long time and don’t have a particular axe to grind are not saying it’s an act of war. There’s no evidence of destruction. There’s no human life lost. This is classic espionage. It’s just that we got owned really badly. It’s similar, in my mind, to the hack of the Office of Personnel Management widely attributed to China some years ago where they got classified personnel files on the majority of folks in the US government, and outside of the US government, with a secret clearance or above. That was really, really bad, but it wasn’t warfare. That’s an espionage win. And we’re trying to do exactly that sort of thing to China and Russia and other governments.
Introducing Decoder, a new podcast about big ideas and other problems. Join The Verge’s Nilay Patel as he explores the future of business, technology, and the world they create. Subscribe now!
But I think it’s clear that there will be some kind of response. There are going to be hearings on this, but ... like many things cyber, it has a lot of aspects. Like, “Are we sure that country X was behind this? Can we prove that to the world’s satisfaction? And then do we respond economically or diplomatically and other ways?” Hopefully not militarily, though I suppose that’s a possibility. And then, “How do we stop this from happening again?” And that’s really hard and complicated.
It involves defense versus offense. It involves asking how you secure the supply chain. What do you do about employees in other countries and contractors in other countries? It’s similar to the trade war with China. Our computers and their software … go back and forth dozens of times before they wind up on your desk. And it is pretty impossible to secure completely. So what do you do about that? Do you try and undo all these global relationships because you’re sometimes rivals? If you do, you’re going to hurt the economy in a pretty major way. So there are big thorny issues, and it’d be nice if the new administration and Congress take that seriously and come up with a plan. It hasn’t really happened before.
A breach of this scale, involving the biggest companies in America and the American government itself, is usually the thing that catalyzes change, that leads to a disclosure law or a reframing of the American posture toward offensive cyberattacks. But because of the transition [from Trump to the Biden administration] and the sort of instant quiet from the attacked parties, it doesn’t feel like this is that moment. Is there a group of people who are going to tackle this in the Biden administration? Does he have an appointee set who has the expertise to raise the profile of this again and build the political capital to actually make the change?
Well, that’s really interesting. And this is playing out in real time. Biden has appointed most of the top cyber people. As of this morning, there were one or two key holes. But among other things, for the first time, there’s a deputy national security adviser for cyber, Anne Neuberger. Anne is very well-regarded, was at the NSA for many, many years, and, among other things, was fulfilling part of the NSA’s mission to cooperate with industries on defense. There are a number of people who have really strong military and government experience. As of this weekend, the suspected new cyber czar inside the White House is Jen Easterly, who was one of the people that helped create Cyber Command as a separate unit of the Pentagon, the people responsible for running cyberattacks in other countries.
So you have really intelligent, really experienced people. Do they have the kind of broad, blue-sky, strategic thinking that might help turn around this really gnarly problem? I don’t know. We’ll see. The fact that both houses of Congress are from the same party probably helps, as does the fact that a lot of this isn’t that partisan in a terribly polarized environment. Nobody’s a big fan of getting hacked to pieces by the other countries. So I am more optimistic now than I have been in 20 years of covering this, but that doesn’t mean I’d actually bet on a complete turnaround.
So you’re starting from a low bar, is what you’re saying.
Yes.
You mentioned that this looks like an espionage operation. What’s interesting about that is you come from Microsoft’s view of espionage, but kind of the way Americans would see it is, “Well, that’s Microsoft’s problem. We don’t need the government or the Pentagon...” or “That doesn’t merit a military response,” which is kind of what you’re describing. But at the same time, this is a major national security problem. How does that play together?
So that’s a really good question. And you put your finger on one of, I think, the biggest issues here, but let me separate out the Pentagon offense response part of it. The US government responded when the North Koreans attacked Sony for dumb reasons. I can’t believe we’re in an era where national governments attack each other over dumb movies, but here we are.
You’re talking about when North Korea hacked Sony over the movie The Interview about a pair of journalists — played by Seth Rogen and James Franco — who go to North Korea to interview Kim Jong Un and eventually assassinate him. Which North Korea obviously didn’t appreciate.
It wasn’t even a good movie. It’s not like it was Chinatown, or...
We gave the movie a bad review and Seth Rogen was very mad at us. It was like the weirdest outcome of that entire debacle, that Seth Rogen was mad at The Verge.
For a variety of reasons after intense discussions at the highest levels of government, the US had a response mostly under the table, but there was some cyber action taken against North Korea. So sometimes that is appropriate. Just because another government attacks a piece of civilian infrastructure doesn’t mean there should not be any response. I mean if a private power utility gets taken down in the United States by, say, Iran. We certainly reserve the right to attack Iran militarily over that. Again, this isn’t that. This is espionage.
Very, very rarely do you see military response to legit espionage targets being attacked. But there’s the separate issue, which you hit on, which is, is this a Microsoft problem? In my opinion, it is not fair to expect private companies, no matter how large, to fend off entire nation-states. The job of the US government should be to defend private enterprise from other countries.
It’s really, really hard when you try to get into the weeds on that, because sometimes a nation-state will use the same techniques as a 16- or 17-year-old. So there should be some reasonable standard of defense that is expected of companies. But again, at the really, really high end, if the Russians got into NSA or the Chinese got into the classified personnel files, it doesn’t matter how big a company you are. You’re going to get owned if they really want you to. So that is one of the big strategic issues that I would hope that the White House and Congress address. Where do you draw the line? What kind of help can be provided? And what’s overreach?
I think about the big nation-state actors that you would think about. The Russians, the Chinese and North Koreans, the Iranians. There’s a tighter nexus between government and industry in those countries. I think that’s the most polite way of saying it. Where the government does some espionage, and they’re going to hand over that information potentially to one of the large corporations. There’s a level of corruption, or maybe state design, there that incentivizes both sides stacked as one. I don’t think we have that here … our incentives [are] to go off and have our government hack a bunch of companies in other countries. Their incentives are very different. So how do you align all of that stuff here beyond just simple deterrence: If you hack us, we’ll hack you back?
Again, one of those big, hairy aspects to this. You’re right that there’s tighter integration between private companies and the state in many other nations, and it not only helps them on offense, it helps them on defense. If the government can simply order everybody in the power industry to apply a patch, then they do it. The United States government does not actually have that power, which is kind of scary. It is also true that the companies that are based in the United States, many of them get a majority of their money overseas. So even if they think of themselves as — or for regulatory purposes, they are — American companies, that doesn’t mean they’re going to march in lockstep with the American government, which makes it really interesting.
But remember, you talked about the US government hacking companies in other countries, well, sometimes, as Snowden showed us, they’re hacking American companies in other places. They did that to Google and others in order to spy. So it’s really messy. If there is this sort of retreat, the sort of trade war 10x, where we need a completely American supply chain, then I guess you would have tighter integration, or at least a framework for tighter integration, between the government and the private sector.
That’s not obviously a good thing. That would erode American business overseas. And the economy is part of what goes into national security. A stronger economy’s in a better position to defend itself. So it gets quite tricky. And I think one of the big problems in discussing any of this in a grown-up way is the overclassification that we have in this country. I mean, the government tells even members of Congress very little about capabilities and what’s actually happening out there. And so there is this kind of shadow warfare happening, this low level stuff, lots and lots of cyberoperations happening. And even members of Congress don’t know what’s going on. So it is hard to have a grown-up discussion about these sorts of trade-offs without greater openness.
One of the questions I’ve been wrestling with; we do it too. As you’ve been saying—
In spades. Yes.
Yeah. The United States hacks a lot of things all the time inside and outside of the country. Is there a way to just pull that back internationally? Is there a treaty or a set of regulations or rules of engagement that exist or have been proposed, to say, “Hey, this is getting out of hand and all of our economies are [under] threat.”
So in theory, yes. I mean, you can certainly have international agreements. For many years under the Obama administration, it was all about norms, setting norms. And there was an attempt to do that kind of globally, and then, “Well, maybe we’ll do it in a given region.” And it hasn’t really led to very much. And when there’ve been big tests, like it is widely understood that Russia shut out the lights in Ukraine a couple of Christmases ago. The NotPetya attacks were terrible. The WannaCry attacks, which shut down hospitals, you would think that that would have given rise to some sort of international agreements, if anything would, and it hasn’t yet. The president of Microsoft, Brad Smith, has been kind of a spokesman on this for the industry; [he] talks about a Geneva Convention sort of a thing that would, among other things, exempt civilian infrastructure from these sorts of attacks.
But the problem is verification. I mean, we can barely verify nuclear treaty compliance. There are so many proxies, both literal and figurative. The line between nation-state and crime is deliberately fuzzed up in a number of places. It’s really, really hard to ensure that something wouldn’t happen. And again ... there has never been a deal that really stopped intelligence exploitation. So if there is going to be some kind of global treaty or understanding or something, it would probably be about physical destruction and not about intelligence gathering, which, again, this is.
Several months ago, I interviewed Andy Greenberg, who wrote Sandworm, which is a book about NotPetya and shutting off the lights in Ukraine. His argument to me was that was the moment when the Obama administration could have stopped it, but instead they set a new norm that this is how we’re doing it. We reserve the right to attack back. The international outcry around turning off the lights in Ukraine was not big enough. And the message was sent, this is in some way acceptable behavior. Do you think that’s grown? Was that the moment, was that an inflection point, or has it just been a steady drumbeat of other things?
I think that was a missed opportunity. … And I think a lot of people think more should have been done there, but again, that was Russia attacking Ukraine. It wasn’t attacking us. I think the response would have been very different if Russia had attacked us in that way. Unfortunately, Ukraine, like some of the Middle East, is kind of a proving ground for a lot of these capabilities. We haven’t seen a lot of what nation-states could do to key parts of our infrastructure because people ... haven’t so far felt like it’s worth it to bloody our nose, but they can; we probably can too. So I think if Ukraine had been in a position to respond, it would have. You could certainly make an argument that NATO should have done something. We’ll see if, with the passage of time, with people who care more about international alliances, if that stuff gets firmed up.
I think a lot of NATO saw the lack of response to that as a mistake. So we’ll see how strongly the Biden administration feels about NATO and this issue. There are a lot of things in play, but I think cyber is back on the table now. The Biden administration is going to make it a priority and I think it’s going to be a part of basically all major conflicts going forward, so there’s no reason why it would not also be subject to treaties, agreements, responses, norms, that sort of thing.
Just before we started recording, you were saying there’s a little bit of a tie between the book you just wrote, Cult of the Dead Cow, and some of the characters involved in the SolarWinds response — explain that to me.
Well, the book tells the history of the “good guy impulse” in security, and it tells the story of these old-school hackers from before there was a web, from back in the 80s, to the present day, and their sort of moral evolution, and a lot of them turned pro. They were teenagers, pranksters and whatever, and they turned pro in various ways and they invented hacktivism. But one of the things they did is that in order to make a difference, some of them went to work inside the government, including DARPA, and some of them went into the private sector and founded really important security companies because that was, they thought, the best way to actually help with security. And one of the companies they founded was called ATstake, which was an early security boutique where hackers went inside companies like Microsoft, and big banks, and whatever, and said, “This is what you’re doing wrong.” It sort of brought the hacker mindset inside these companies.
A lot of really amazing people came out of ATstake and went to work inside every major tech company in the United States, and one of them is Alex Stamos. So he joined ATstake because he admired these guys who had testified before Congress about how fragile the internet was. And Stamos went on from there to, among other things, work at Yahoo as a chief security guy, and then at Facebook where he dealt with the Russian disinformation in 2016. And now he teaches at Stanford on big disinformation and related issues, and he is in business with Chris Krebs, [who is] ex-US government, ex-Microsoft, and they’re running around trying to help SolarWinds and other companies learn the lessons of this latest hacking spree.
So it’s kind of this continuum, and it’s like there’s these coaching trees where great basketball coach X trained all these people, who then went on and coached all these other teams, and they all go back to the same root of the tree. The Cult of the Dead Cow story is kind of about that.
Alex has been in and around The Verge and our shows for a long time. One of the things he has said about the 2016 election was, we were all focused on the wrong thing at Facebook. They didn’t know what it would look like, and then everyone in 2020 was worried about Russian disinformation and the threat ended up looking very different and coming from a different place. I think Chris Krebs has said something similar: we were all focused on the election, we missed this thing. How do you build a broad enough array of sensors and detection mechanisms to say, okay, there’s a major election going, but there are these gigantic fat targets in the United States that need constant surveillance and protection?
Well, first of all, it’s in the nature to be fighting the last thing, to be fighting the last war. I’m not sure that we still need to be taking off our shoes at the airport because one guy tried that thing that one time. Your really high-end terrorists or other adversaries are not going to do the same thing that they’ve already taught us how to defend against, and this is a classic case of that. As to how you defend against this kind of intelligence-driven attack, I think you need to deal with some really fundamental questions that include supply chain, vendor relationship, and to my mind, it definitely includes the defense-offense balance. I did some stories a few years ago that said that 90 percent of what the US government spends on cyber-y things, at least back then, was about offense or intelligence gathering, not about defense.
And the NSA is the agency, at the time, [that] was charged with defense. Now DHS has taken up some of that, but their budget is minuscule compared to the NSA’s cyber budget. And the NSA had a whole division, the Information Assurance Division, that was responsible for at least securing the US government, and after the Snowden debacle, the Obama White House had a commission of five folks to look at this and figure out a whole bunch of things that we should do different to avoid another Snowden situation. And one of their recommendations was [to] spin out the defensive division of the NSA from NSA proper, because nobody trusts the NSA, and because the offensive mission so dominates that you can’t be sure that they’re not going to subvert defense — which in fact they did, and that would emerge from the Snowden leaks. But they didn’t spin it out. Instead they did a disappearing act where NSA scattered the defensive mission lower down within the agency so there was no longer a No. 2 at NSA whose mission was defense. They got rid of that position. So they went the opposite way.
I think defense has to be prioritized a lot more than it has been, because if everybody’s good at offense, then you’re going to have a lot more offense around the world. The way to actually win is to get really, really good at defense and I think we have not been focused that way at all.
That “if everyone’s good at offense, you just end up with more offense,” that sounds to me like pretty classic foreign policy, military deterrence language, right? I mean, this is the Cold War, right? You’re speaking in the language of the Cold War in some way. What breaks it? Is it just that we get so good at defense that no one tries anymore?
That’d be nice, wouldn’t it? People will keep trying, but you can raise the cost. It obviously has to be a multi-faceted thing. I mean, if you’re talking about the Cold War, then the big thing is deterrence and the certainty that country X will retaliate against country Y. It’s true we have not seen that, or not very much of that, so that needs to be done too. But what I’m talking about is more Reagan’s Star Wars vision of being able to shoot down missiles as they come at us. Now that’s what would actually change things.
So yeah, the idea is that they would keep trying to get into Langley, or Maryland, and would fail. That would be super awesome, but we haven’t actually even tried. There’s really cool stuff happening in the private sector. There’s been a lot of advances in defense, but there hasn’t been a government-wide embrace of that initiative, where you put lots and lots of money into the National Academy of Sciences or others to really be fed up. There’s Skunk Works inside DARPA, [but] there hasn’t been a giant thing and there needs to be a giant thing.
Do you think there’s a cultural shift with the new generation of lawmakers? I mean, we have some younger lawmakers now. We have lots of younger people who’ve come up in things like Cult of the Dead Cow in parts of the government. People are good at computers now in a way that maybe they weren’t so good at computers 10 years ago.
Yes, this is one of the good things. I mean, in the olden days when I started covering this, the only good thing you could say about cybersecurity was, “Well, awareness is rising.” And now it’s true. There are people in Congress that actually understand. There are actual engineers in Congress. This is a new thing. You still have a hearing where they drag in [Facebook CEO Mark] Zuckerberg and members of Congress ask embarrassing questions, but it is a big change from where it was, and there are tech-savvy staffers at all levels. They’re digital natives and they understand these trade-offs. I think that there’s a better chance than we’ve ever had of people having a real discussion about this.
But again, I am concerned more about the establishment, the four-star generals, the people running intelligence agencies, people in the White House who still think of warfare and intelligence in the old terms and don’t get into questions of the private sector versus the public sector stuff because there aren’t really straightforward answers. Right now, our government has been so dysfunctional that you couldn’t get the two houses to agree on pizza toppings, so how are you going to tackle something like this? I mean, the Chamber of Commerce, the private lobbying group, was outraged that folks in the Department of Energy and [Homeland Security] wanted to put out voluntary guidelines for best practices to protect nuclear plants or power plants from hackers, because they thought that was a slippery slope that would lead to more regulation. We can’t have that crap anymore. We need people actually willing to give and take and deal with complicated issues, or we’re going to keep getting owned like this.
What’s the next thing you’re looking for, specific to the SolarWinds story? What’s the next turn of the screw, do you think?
Well, a turn of the screw would be [if] giant companies, X, Y, and Z say that they were also breached by the same folks, they also had access to their source code. Maybe there were other updates that got their customers, that sort of thing. That’s pretty likely. There may be, in these hearings, or there will likely be an intelligence report that is made public about why they think it’s Russia. And there will probably be some big review over practices and some other commission. On the other hand, maybe it’s going to wind up like the Snowden commission where some of the stuff gets adopted and some of it doesn’t. So I don’t know about that. I think a federal disclosure law is plausible because companies don’t want to have to deal with this patchwork of states. It would also be nice if there’s a federal privacy law. So maybe those initiatives go together and both of those things happen.
