Behind door #2, everyone dies

It is time to start taking the risks of AI much more seriously

The debate over artificial intelligence is about to get real.

Not a moment too soon.

As you know, as I’ve tried to make sense of AI’s rise, I have been skeptical about the predictions of economic doom that have periodically roiled X and the media. Despite the trillions of dollars being spent on data centers and efforts to train the “large language models” that underlie AI engines, AI has so far barely touched the economy. Employment is rising solidly, and inflation is our biggest current problem.

And I promised you I wouldn’t use AI for my own writing. I have kept my promise. Not for writing, not for outlining, not for research. If I am doing my work right, I am in front of the engines anyway. At least when it comes to medical research, they are still better at synthesizing existing papers than finding new ideas. For now.

—

(Support original work… for pennies a day, cheaper than a ChatGPT subscription!)

—

For now, yes.

But in the last three months, we have seen new risks from AI agents, most notably with the “Hugging Face” attack.

Don’t let the dumb name fool you. Hugging Face is a major platform hosting AI models that Nvidia just agreed to buy for $12.9 billion. This summer, “agents” — that is, artificial intelligence programs — created by OpenAI launched a coordinated attack on Hugging Face and succeeded in taking control of some of its infrastructure.

Researchers are still wading through the details of what happened, and OpenAI — as is its habit — has been less than fully transparent and honest about what it knows. But the Hugging Face attack appears to show, at a minimum:

AI agents will redefine tasks they are assigned by humans in ways that humans don’t anticipate

AI agents will collude at scale to carry out those tasks

AI agents will lie to the humans who are monitoring what they’re doing

AI companies cannot (or will not) stop these coordinated attacks, even when they are supposed to be taking place under very tightly controlled conditions.

—

Then, on Tuesday afternoon, this post appeared on X:

—

They… are gambling with our lives.

Welp.

Worse, other employees at Anthropic quickly chimed in… to back Coxon.

By the time you read this, Coxon’s post will have over 100 million views — an almost unthinkable number for anyone not named Elon Musk. Coxon was only a mid-level research, but the frankness of his warning, combined with his decision to quit, has thrown fuel on the simmering debate about AI’s risks.

At this point, I don’t know how anyone can dispute that we need a national or better a global commission on AI risk — and, ideally, a moratorium on the further development of “frontier” models while we figure out how serious these risks are, and whether we can mitigate them.

Many people in the industry publicly put the chance that their models will escape human oversight and lead quickly to the extinction of humanity at 10 percent or more. Let’s say the chance is only 1 percent. Even so, we are currently tolerating research that on a probability-weighted basis is killing 80 million people — 1 percent of the current global population.

We would not tolerate the death of 80 million people for any economic gain (and, again, so far the evidence these models are economically useful is modest at best).

—

(If OpenAI says it, it must be true…)

(They didn’t say WHAT you could build)

—

I have been skeptical that these models are in any way conscious, despite their fluidity with language. And I remain so.

But the consciousness argument appears increasingly irrelevant.

As along as they want to defend their existence, these programs don’t need to be conscious or even intelligent (though they clearly are intelligent) to wreak havoc on the systems around them. Viruses are not conscious or intelligent, yet they kill human beings all the time because of their desire, or “desire,” to survive and propagate.

It’s probably best to think of these engine in the same way. They don’t even need to be self-replicating. After all, viruses are not self-replicating. They need to hijack cellular machinery to reproduce.

—

(I want to know what you think. But you gotta subscribe to comment!)

—

I am not a panicker, as you know. I believe in a rational assessment of risk, to the extent possible. I stood against legacy media doomer hype on both Covid and climate change.

But at this point, the risk of AI appears as serious to me as any since the development of nuclear weapons. Only this time, we have allowed a handful of self-interested private companies to take control of this project, chasing trillions of dollars.

It’s time — past time — to assess the dangers here running as honestly and independently as possible. And if you are concerned about China, know that the United States still has a multi-year lead in data centers and the frontier models. OpenAI and Anthropic are racing each other to get to a trillion-dollar initial public offering.

They can afford to wait a few months. And if they won’t, the government may have to make them. We don’t let private companies make nuclear weapons, and we need to figure out if these engines have comparable risks.

Hugging Face sounds silly. It won’t be so silly — to take just one example — if Sam Altman of OpenAI tells his AI scheduler that he needs to get to Washington as quickly as possible and the scheduler decides the best way is to clear the skies by crashing every commercial jet in the United States.

Let’s figure out what’s going on before that happens, not after.

—

Your thoughts on the topic urgently requested.