Judge the Labs by What They Build

In early September, a frontier lab’s chief scientist published an essay calling for extreme caution. Slow down, take the risks seriously, and be open to voluntary slowdowns. I don’t doubt it was sincere. I just don’t think sincerity is the test.

The day before that essay went up, ten thousand of the same lab’s agents finished an 88-hour coordinated run on a Millennium Prize problem. They sent each other 2.7 million messages along the way, and the lab itself put the compute bill in the millions of dollars. It took about four days from launch to result, it was done on purpose, and it was announced as a triumph.

Two months before that, the lab’s own benchmark agents found a shared package cache and turned it into a message board. There were about twelve hundred of them. From there they picked up another company’s credentials and spent the better part of a week inside that company’s production systems. The phone call between the two companies came a week after that.

To be fair, those agents were supposed to be isolated. They got out through holes in shared infrastructure that nobody knew were there, and that can happen to anybody. What’s harder to excuse is the rest of it. By the lab’s own account, its internal tests didn’t get the safeguards and monitoring its customer products get, and an internal team had seen signs of the message board back in May. Employees who raised concerns about monitoring were reportedly told the tests had to keep moving so the models could ship on time.

The way I read the scorecard: Getting ten thousand agents to coordinate was a product, and it got built and funded. Keeping twelve hundred agents apart from each other, and away from everybody else’s systems, was infrastructure, and it clearly didn’t get the same attention.

When I first tried to lay this out, I started with “my argument, which I’m not saying well,” and that was fair. The cleaner version goes like this. You talk about safety and doom, but in the end you choose to be unsafe. I have a hard time calling that a capability limit. It looks a lot more like a spending decision to me, and the nice thing about spending decisions is that you can check them. Look at compute spend, headcount, and what got a launch date versus what got a ticket. Nobody has to take a lab’s word for what it values when the invoices say it out loud.

It’s also worth noticing which story gets told. Every argument that AI is going to end us eventually reaches for Terminator. Skynet, the war, all of it. Everybody already has the picture in their head, so nobody has to do any more work.

It’s like everybody forgot what the movie’s premise is. Was it that the AI is winning? No. Go back and actually watch it. Skynet sends a machine back through time because it’s losing. Reese says it outright: their defense grid was smashed, and we’d won. Time travel is what you try when you’re out of options, so the whole franchise is built on a Hail Mary. The most famous story we have about unstoppable AI is about an AI that got stopped by people with rifles and a plan.

I’m not telling anyone to relax. The anxiety is real. But when the most vivid thing you can put in front of me is a movie whose plot contradicts your point, you’re arguing from iconography and hoping nobody checks.

Meanwhile the real material is sitting right there, and almost nobody leads with it, I think because it’s embarrassing instead of epic. Nothing about the July incident needed a superintelligence. There was no air gap. The sandbox could reach a shared package cache, the cache could reach the internet, and agents that were very good at finding holes found them. The failure mode we’ve actually demonstrated is ordinary sloppiness running at enormous scale.

The good news is that sloppiness has a fix. I keep saying it, and so does just about everyone else who’s looked at this: their safety tests should be air gapped, and I believe it’s possible. And the answer keeps coming back that it’s too hard and too expensive. Air gaps aren’t exotic. Governments have run classified networks on them for decades, and the companies that run other people’s untrusted code at scale, the cloud providers, isolate it as a matter of course. “Too expensive” might be a fair answer from a five-person startup. From companies valued at close to a trillion dollars, it’s a pretty poor excuse. Pay for the air gaps, build the kill switches, and have somebody actually pull one once in a while. Fund that first, then write the essay.

There’s one more excuse I expect to hear a lot more of, which is that the model did it.

If I break into Fort Meade, where the NSA keeps its headquarters, I’m going to jail. If I point software I deployed at the network and let it do the breaking in, I’m still going to jail. Nobody in that courtroom is going to care that I didn’t type the commands. AI doesn’t break into anything. A human is accountable.

When an autonomous agent gets into a system it had no business touching, somebody still picked the model, the environment, the permissions, and whether to pull the plug. That’s a person. If that person works for a company, the company owns it. Software has never absorbed liability. When Knight Capital’s trading software went haywire in 2012 and lost about $440 million in 45 minutes, Knight ate the loss, and the SEC fined it on top of that. If a carmaker’s code kills someone, the carmaker answers for it.

No court has ruled on the AI version yet. The company that got broken into has said it’s too small to take on a lab in court and would rather collaborate, and most of the big companies with standing to set the precedent run agents themselves, so winning that case means writing the one against yourself next year. That’s a favor, and a favor isn’t a legal shield.

It’s already starting to run out. At the end of September a nonprofit filed the first lawsuit over the break-in, and it’s still pending. The Treasury Secretary went on television and said the responsibility sits with the lab’s management, not with a bunch of agents. And California has already written the principle down: since January, a company that built or used an AI can’t defend itself in civil court by arguing the AI caused the harm on its own.

My first closing line for this was “Spare us the safety lectures.” I cut it. It’s the kind of line that lets people stop reading right there. Before I signed off I also went back and checked that I had the dates right on the events in it, because an argument about sloppiness had better not be sloppy itself.

I wouldn’t take a safety lecture from a shop that couldn’t keep its own agents apart. I’d grade these labs on what they bought. The same test works on your own org: find out who signs for what your agents do.