AI
Nobody in the Extinction Debate Has Done a Drill
A generation lived for decades beside a technology that really could end everything, and built hotlines and treaties instead of panicking or pretending. The AI risk debate talks as if this is the first time.
I was standing in a hallway with a coworker a few weeks ago, talking about the AI labs telling everybody to slow down, and I heard myself say, “When we were all in school, it’s like once every six months we were learning how to deal with nuclear catastrophes.”
That’s how I remember it, anyway. If you’re old enough, you probably remember the drills too. I grew up during the Cold War, and that’s not a credential, exactly, but it’s a piece of context I keep noticing is missing from the argument about whether AI is going to end the world. “None of these people grew up in that,” is how I put it in the hallway. “We already lived through this Cold War thing.”
For a long stretch of the twentieth century, the end of the world wasn’t a thought experiment. It was policy. Two countries kept thousands of nuclear warheads ready to launch at each other, and everybody knew it. Hiroshima and Nagasaki had already happened, and the test shots had been filmed, so nobody had to imagine what the weapons did. Schools ran drills. Buildings had fallout shelter signs. In 1983 ABC aired a whole movie about the day after, and something like a hundred million Americans watched it. Kids filed into hallways and put their hands over their heads for a threat that was real, built by people, and aimed at the places they lived.
So when people running AI labs say their technology might kill us all, I don’t hear something new. I hear something my generation already lived with. What bothers me is that the conversation acts like this is the first time.
Living under that taught people a few things I don’t think you can get from a model. The first is that civilization-scale risk can be survivable. It isn’t guaranteed, but it can be survived. We came close more than once and got through it, partly by luck and partly through a lot of dull, careful work by people on both sides who took their jobs very seriously.
Look at what got built. After the Cuban Missile Crisis, Washington and Moscow agreed in 1963 to set up a direct hotline so the two leaders could talk before a misunderstanding became a war. The same year brought the Limited Test Ban Treaty, and the Non-Proliferation Treaty was signed in 1968. Weapons got coded locks so they couldn’t be armed without authorization, and custody rules that kept any one person from handling them alone. Doctrine got written down, argued over, and rewritten. None of it was perfect, and a lot of it came after a scare instead of before one. Nobody wrote the hotline in advance. Somebody wrote it after thirteen days in October.
And there were close calls. In 1983 a Soviet officer named Stanislav Petrov watched his early-warning system report incoming American missiles and decided it was a malfunction. He was right. Nobody calls Petrov a doomer. He’s the reason the controls mattered: a person inside a system, with enough judgment and enough standing to say no.
The second thing people learned is that a lot of the visible safety work was theater. Everybody knew a school desk wasn’t going to stop a nuclear blast. I said as much in that hallway: it was all nonsense, but still, we had to. The drill was mostly a ritual that gave people something to do with their fear. That’s worth remembering when you watch today’s safety announcements and try to sort out which parts are protection and which parts are ritual.
The third is that the people actually managing the risk spent most of their time on boring work. Checklists, procedures, inspections, the same routine over and over so that nothing interesting ever happened. Boring is what competent risk management looks like from the inside.
“None of these people grew up in that” is a little too broad, though, and I know it. My sense is that most of the people building these systems came of age after the Berlin Wall came down, and a lot of the online extinction debate is shaped by people who’ve only ever modeled a threat like this. They’ve built careful arguments and probability estimates, and some of those arguments are very good. Some of the loudest worriers are old enough to remember the Cold War themselves, and they deserve a real hearing. But for a lot of the people setting the tone, existential risk has only ever been a spreadsheet.
That’s not a reason to dismiss anyone. Being young doesn’t make you wrong, and living through something doesn’t make you right. Plenty of people who lived through the Cold War misjudged it in both directions. It is a reason to ask what everyone’s priors are made of. A sense of the odds built from decades of living next to an engineered threat is going to land somewhere different from one built on expected-value math. I don’t think either one is automatically better. They’re different instruments, and you’d want both readings.
I’ll admit my own bias. I think people are hard to wipe out. We’re a lot like cockroaches that way. And my honest read of history is that it usually takes an event or two before the serious controls go in, and then they do go in. That’s not comforting, but it’s how the last existential technology got managed. The thing that worked had a mechanism and a control you could point to. We could see exactly how the harm would happen, so we could build specific things to stop it. The AI version of this conversation has a lot of probability talk and very little “here’s how it would happen, and here’s what we’d put in the way.” The question I kept coming back to in that hallway was the plain one: how is it going to kill us?
The slowdown pledges don’t answer it either. In the same conversation I called them a total prisoner’s dilemma. Every lab does better if everyone slows down, and worse if it’s the only one that does, so a pledge mostly tells you what a lab would like the others to do.
There’s a fair objection, and I’d like to have it out in the open. Nuclear weapons needed states, rare materials, and enormous industrial effort, which made them possible to count and control. AI may not work that way. If that’s where the analogy breaks, it’s worth saying exactly how.
What I’d ask of the debate: Talk to people who’ve lived under one of these. Ask them what actually reduced the risk, what was theater, and what it was like to keep living normal lives under a threat nobody could make go away. Then ask yourself whether your sense of the odds comes from something you lived or something you calculated. And while we’re at it, who in AI is building the hotline?