AI warnings are mounting

I don’t know what to do either. I do know we would be foolish to dismiss them.

Since last week, when an Anthropic computer scientist resigned and posted a warning to X about the rising risks of artificial intelligence, the debate over AI has exploded.

On Saturday, Dario Amodei, head of Anthropic, which along with OpenAI and Google is leading the creation of these engines, posted a long article called “We Must Pace The Frontier.”

Amodei called for “government support” so the companies can work together to slow their development and put adequate “safety standards” in place.

I don’t think these questions have any easy answers, particularly because China has shown no inclination to slow down the development of its own AI models. And I am wary of granting antitrust exemptions to any industry. At a minimum such exemptions should be time-limited and come with strict regulatory oversight.

—

(The Last Frontier?)

—

But I am struck by how dismissive many on the right (including some UT readers) are of these AI concerns, or of any efforts to regulate AI companies.

Yes, AI has far had only marginal economic or societal impact, except of course for the money spent on building out its infrastructure, now the main driver of American economic growth, for better or worse.

But underestimating these models is a mistake. They did not exist a decade ago; they barely functioned five years ago. Today they solve previously unanswerable math problems, draft legal briefs, diagnose illnesses, design proteins, and write books — and software code.

The engines can’t manage all these tasks better than the most skilled humans can in each domain, but they have narrowed the gap with millions of years of evolution almost to zero — in a few years. Unless their ability to improve suddenly ends practically overnight, they will pass us within months, possibly a year or two.

Further, whether AI is a dangerous weapon (either under its own independent control or human direction) is not really related to its economic impact. A nine-year-old can pilot a drone, but autonomous drones are already killing people in the Ukraine war. AI is becoming excellent at handling skills like that at scale - like a million-drones-at-once-scale.

—

(Chasing the truth, wherever it leads. Independent of Big Pharma, Big Tech, and Big everything. With your help.)

—

Yet many people on the right, particularly on X, seem to think the companies and the people inside them simply want some kind of regulatory protection, or a bailout (they don’t need a bailout, Anthropic is the most valuable private company in history), or at best have fallen victim to their own marketing hype.

The right compares these warnings to the early Covid panic. It sneers at Coxon, calling him a junior engineer who’d barely worked at these companies, and says the interest in his comments must have been coordinated, a “psyop.”

The psyop argument is nonsense.

We saw what coordinated media, government, and public health action looked like early in Covid. It had a specific aim, to force people to stay home and shame those who did not, even if they were at low risk from the illness. It made use of specific phrases. It was not confined to elite media outlets like The New York Times, but spread everywhere within days.

Nothing similar is happening here. No government or World Health Organization-style infrastructure is supporting the people raising these alarms. They mostly come from inside the companies. (Compare that situation to Covid, when virologists tried to downplay any possibility they themselves had caused the crisis.)

I will say more soon about why I think some of the specific arguments coming across X don’t hold up.

The broader point, though, is that the companies are only drawing attention to themselves and risking the anger of President Trump by raising awareness of these risks. They have a good thing going, and no reason to step on a rake. The Occam’s Razor should be that their fears are real. Maybe overstated, but real.

—

So maybe… just maybe… they are genuinely nervous?

The most recent example came in an X post yesterday from Daniel Selsam, a veteran and highly cited researcher at OpenAI. (Notably, Selsam does not even have his X account. Another AI “doomer,” Daniel Kokotajlo, posted his note.)

Jacob Coxon, the researcher who sounded the alarm last week, is 27 and a relatively junior scientist. Selsam falls in a different category. He’s 42, and as he explains, he has spent most of his adult life working on AI. He clearly enjoys it both as intellectual challenge and for its economic and scientific potential.

The original note is very long, but I think his comments are worth reproducing (I have made edits where possible to keep the length manageable). Even more than the words, I hope you will take note of Selsam’s tone. He is not hysterical, more sorrowful.

He sounds like nothing so much as a father who fears he's raised a psychopath.

—

Dan Selsam's Personal Statement on AI Risk:

…Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.

The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled…[Emphasis added]

I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future.

Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.

I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us…

That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase…

—

(I have nothing witty to say about this picture, AI probably would.)

—

It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics.

Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years…

But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:

[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.

[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

—

(Capturing the essence, even when I’d rather not.)

—

These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended.

Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.

If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine…

One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. [Emphasis added] Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for…

Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day…

In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.

I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me.

I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns.

Daniel Selsam
September 14, 2026