Written in Blood

Why does it bother me so much when an AI lab says its technology could kill everyone? “I don’t know how, it just irritates me” is about where I was. I had the reaction before I had the argument, and that’s usually a sign to go find the argument, or admit there isn’t one.

Where I ended up starts with pilots. They have a saying: the rules are written in blood. A big share of the regulations, checklists, and procedures in aviation trace back to an accident where somebody died and somebody else figured out why.

The checklist itself is the famous example. In 1935 a prototype Boeing bomber crashed on a test flight at Wright Field because the crew took off with a control lock still engaged. The airplane had gotten too complicated to fly from memory, so the fix was a written list. The midair collision over the Grand Canyon in 1956 is another one. Two airliners hit each other in uncontrolled airspace, and two years later Congress created the agency that became the FAA. You get an incident, then an investigation, then a targeted control, and aviation has been running that loop for close to a century.

I don’t read that as a failure of foresight. It’s very hard to write a good control for a failure you haven’t seen. When I finally worked out what was bugging me, it came out like this: it would take an event or two, and after that, controls would be put in place. But you don’t make controls for something that’s not a problem. Before the Grand Canyon collision, people could obviously imagine two airplanes hitting each other. What they couldn’t know was exactly how it would happen and which fix would actually work. The investigation is what turns a vague fear into a specific rule.

That’s the part of the AI safety conversation that bothers me. A lot of the current push is to regulate harms we’ve only imagined, before we’ve seen most of them happen. Write the rules now, for failures we can picture, because by the time they happen it’ll be too late.

I’ll grant the strongest version of that argument. Some failures don’t give you a second chance, and if the first incident can’t be undone, waiting for it to write the rule is a terrible plan.

But most of the proposals I see aren’t aimed at that narrow class of unrecoverable failures. They’re aimed at everything, and they’re written against imagined failure modes. Rules like that pile requirements on whatever was easiest to picture in a conference room, and miss what actually goes wrong. Meanwhile the paperwork gives everyone the comfortable feeling that the problem is handled. And this part is my opinion, so take it as one: when a company says “regulate us,” a lot of the time it means “limit our liability.” It’s worth asking who a rule protects.

What made aviation safe was building the machinery to learn from accidents fast. In the U.S., independent investigators at the NTSB look into every civil aviation accident. Airliners carry flight recorders. And since 1976 there’s been a confidential system, run by NASA, where pilots can report their own mistakes and near misses with real protection from getting punished for them. That’s the regime that worked, and almost none of it was about guessing the future.

I’ve seen this loop from the inside. I was at NASA when Columbia broke up on reentry in February 2003, and I spent the weeks after it working on the response. The accident board put as much weight on NASA’s culture as on the foam that hit the wing, and I understood why. Mission Control lived by the line usually credited to Deming: “In God we trust, all others bring data.” Most days that’s a great rule, because data kills politics. It was also one of the root causes. Raising your hand without proof was a good way to get your hand chopped off. Foam had come off the tank on earlier flights, nobody could prove it would bring down an orbiter, and so it became normal.

That cuts against me, and I should say so. The warning was there before anyone died, and seven people paid for the gap between a warning and a rule. That’s the best case I know for acting early. Where I land is that the fix is to make it safe to raise your hand without proof, not to get better at guessing, and that’s why I care so much about near-miss reporting.

That’s what I’d want for AI. Mandatory reporting when systems fail in the real world. Independent investigators with real access to the logs. A way for engineers to report near misses without ending their careers. Then fast, targeted rulemaking once we know what actually broke. We’ve already had at least one real first incident this year, an evaluation that got out of its sandbox and into another company’s systems, and it’s exactly the kind of thing that machinery would be for. Save the up-front rules for the small set of failures you truly can’t survive once.

Rules written in blood are better rules, because they’re written about something real. So the question for any AI safety proposal is whether it builds the system that learns from the first incident, or just writes rules for the one somebody imagined.