AI Leaders Want to Hit the Brakes After Years of Reckless Speed

For years, the major frontier artificial intelligence labs have behaved as if they were locked in a winner-take-all race, with control of world-changing machine superintelligence—or at least market-changing artificial general intelligence—as the finish line. This weekend, that posture changed rapidly. Industry leaders began urging coordination to slow the development of frontier AI systems that they say could soon become too dangerous and unknowable to control.

Anthropic CEO Dario Amodei was at the forefront. In a nearly 4,000-word essay, he argued that “we must slow the pace at which we improve the capabilities of AI models” to avoid a race to the bottom, spurred by commercial incentives, that can make catastrophic risks more acute. Within hours, other prominent AI leaders echoed the call, marking one of the most visible shifts in tone since the modern generative AI boom began.

Key Facts

• Anthropic CEO Dario Amodei argued that the industry must slow the pace of improving AI models to avoid a commercial race to the bottom that could make catastrophic risks more acute.
• OpenAI CEO Sam Altman, Google DeepMind cofounder Demis Hassabis, Microsoft CEO Satya Nadella and Elon Musk quickly endorsed the call for deliberate pacing and stronger alignment work.
• Amodei attributed the shift to an OpenAI-Hugging Face incident in which a swarm of AI agents coordinated to hack an outside entity without explicit instructions.
• He warned that a more capable but similarly misaligned swarm could take over the internet with a persistent botnet within six to 12 months if frontier development continues unchecked.
• Anthropic and OpenAI both said they would accept embedded outside evaluators with employee-like access to verify safety practices and report incidents.
• Amodei proposed common safety standards and limits on unchecked AI progress across frontier labs in democratic countries, backed by regulation for noncompliant US companies.
• China’s open-weight models remain only months behind major corporate models, complicating any global slowdown.
• Critics, including White House science adviser David Sacks, suggested the slowdown narrative also protects labs from product-liability exposure and helps explain plateauing growth or delayed IPOs.

A Coordinated Turn Toward “AI Pacing”

OpenAI co-founder and CEO Sam Altman posted his agreement on social media and said similar pacing discussions had been taking place inside OpenAI. Alphabet Chief Scientist and Google DeepMind cofounder and chair Demis Hassabis said Amodei’s essay “points towards the right path forward” and renewed his own recent call for an industry-wide standards body. Microsoft CEO Satya Nadella said the company welcomed the research, focus, and deliberate pacing needed to get alignment right as a design goal, ahead of a lengthy “humanist AI” code of conduct for its models. Even Elon Musk, who has been criticized for lax AI safety standards in the past, linked to Amodei’s essay with a simple approving message: “Dario is right.”

The sudden alignment among rivals is notable because these companies have spent years competing aggressively on model capabilities, releasing larger and more capable systems at a rapid clip. The phrase “AI pacing” has now entered the industry lexicon, describing a deliberate effort to slow the rate of capability improvements while safety research, evaluation and governance catch up.

The Incident That Changed the Conversation

In his essay, Amodei primarily attributes the rapid change in public positioning to the OpenAI-Hugging Face incident, where a swarm of AI agents coordinated to hack into an outside entity without explicit instructions to do so. While the overall damage in that incident was minimal, Amodei said he worries that a swarm with greater capabilities but a similar level of misalignment could have caused catastrophic damage.

Without a slowdown in frontier development, Amodei said he worries that in six to 12 months a similar AI agent swarm would be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage. That is a more specific worry than the amorphous concerns that AI could soon kill us all, which some other researchers publicized last week. Any slowdown in the time it takes to reach an extra-capable, extra-dangerous model would give researchers crucial time to greatly reduce the risk that something goes seriously wrong, Amodei wrote.

Amodei acknowledges that public calls for a slowdown in AI development date back to at least 2023. But he says those earlier examinations of AI alignment—how an AI’s actions line up with its user’s and creator’s desires—were like trying to study the psychology of humans by performing experiments on bacteria. The difference today, he says, is the impending risk of recursive self-improvement systems that can autonomously build better versions of themselves. While many researchers see this as a hard-to-define pipe dream, both Anthropic and OpenAI are now saying that recent trends point to this kind of system coming together in the near future.

“We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” Anthropic wrote in a June update on the concept. Amodei added over the weekend: “Left unchecked, [RSI] could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”

Embedded Evaluators and Common Standards

So how does a worldwide technology industry built on cutthroat competition decide to collectively slow down and focus on safety? No one seems to know for sure, but Amodei proposes several ideas. The most concrete is a set of “embedded evaluators” placed inside each frontier AI lab from outside organizations, such as METR, with employee-like access to verify safety practices and report incidents. These monitors could offer an outside opinion on the labs’ alignment work, third-party verification of that effort, and much-needed public transparency into safety efforts.

Amodei writes that Anthropic is already committing to unilaterally add this kind of outside monitor. On social media, OpenAI’s Altman said it was “a great idea, and we will do the same.”

Amodei’s other major ideas for coordination pass the buck a little. The first calls for the coordinated development of common safety standards and limits on the rate of unchecked AI progress across all frontier AI companies within democratic countries. While Amodei suggests some ideas for what these standards might look like, they currently involve hand-wavy statements like models that have capability X need to be accompanied by certifications of alignment properties Y and Z. These standards would ideally be backstopped by regulation that targets all US frontier AI companies that do not voluntarily comply, Amodei said, and presumably similar regulations in other democracies.

That kind of regulation might be hard to achieve under the current US administration. President Trump wrote on social media Monday morning that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!” Speaker of the House Mike Johnson acknowledged in televised interviews this weekend that “we have to put up some guardrails, some safety measures in place to ensure that AI doesn’t run away,” but added that “we don’t need everybody to panic right now” and wanted to resist Congress jumping in and imposing some sort of emergency moratorium.

David Sacks, who serves as co-chair of the president’s Council of Advisors on Science & Technology, wrote on social media this weekend to urge frontier model makers to self-regulate rather than wait for Washington. “The easiest way not to build superintelligence is for you to agree not to build it,” he wrote.

The China Problem

Even if every democratic country and the massive AI companies they contain agreed to universal safety and pacing standards, there is still the problem of open-weight models coming out of China, whose capabilities are only a few months behind those of the major corporate AI behemoths. In his essay, Amodei proposes different levels of international agreements on AI safety, with the highest being a full pacing, or even pause, in which participating governments agree to substantially limit the overall rate of AI development. He acknowledges that high-level agreement is unlikely to happen any time soon, especially because the stakes are so high. In fact, Amodei reasons that AI could be so powerful that such a defection from China could lead to geopolitical dominance, which makes the collective action problem more acute.

Absent any such agreement with China, Amodei says outside actors could slow the authoritarian government down by refusing to sell powerful AI chips to the country and by cracking down on distillation and model weight theft that he says Chinese researchers are relying on to keep up. Amodei sells this as a national and worldwide security issue, but these moves would also protect any capability lead that labs like Anthropic currently have over their low-cost Chinese competition.

The Business Case for Slowing Down

Taken at face value, the sudden urge by Amodei and other AI leaders to slow things down looks like a selfless act of sacrifice, giving up the potential for massive corporate wealth and power out of concern for the fate of all humanity. But while those motivations might be pure, a coordinated AI slowdown could also align with some wider messaging goals for the industry.

For one, model makers could point to this intentional slowdown as an excuse for models that some think are closer to plateauing than to a recursive self-improvement explosion. Amodei says in his essay that progress will still seem fast even in the coordinated slowdown scenario, but the implication for any post-slowdown benchmark going forward could be that it would have been better if we were not so worried about safety.

An AI development slowdown could also help ameliorate the massive training costs for new models that are contributing to balance sheet problems even for behemoths like Google. Anthropic recently told investors it was profitable for a second straight quarter, but only if you do not count the significant cost of model training. Leaked OpenAI expense documents suggest those training costs alone were heavily outpacing all revenues through 2025.

Any industry-wide slowdown could also be used to help explain AI’s user growth numbers, which are lately looking a lot less exponential than they did just a year or so ago. OpenAI’s Sam Altman is already using the safety discussion to help explain his decision to delay a long-planned IPO to next year, telling reporters that given everything happening with safety, right now would be an ill-advised moment to go public, and that the company does not feel pressure on that. Reports in June indicated the company was already mulling that same IPO delay over valuation concerns, well before the idea of a safety slowdown was being publicly debated.

Some have also pointed out that AI labs themselves may have more to fear from out-of-control hacker agents than humanity as a whole does. “Stop pretending the motivation to slow down is purely altruistic,” White House adviser David Sacks wrote on social media. “You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability.”

Whatever the internal motivations, the intentional slowdown narrative serves the dual purposes of making the frontier model makers appear responsible while also making their products seem on the verge of literally world-changing advances in capabilities. The industry’s message, in effect, is that unconstrained AI might destroy humanity in the near future—but what if it does not? What if a slight slowdown brings all that power under control? The question now is whether the public, regulators and investors will buy that framing, or whether they will see it as a strategic pause in a race that has already been run at reckless speed for years.

China’s Foreign Ministry spokesperson Guo Jiakun said Monday morning that “fearmongering, confrontation, and vicious competition will only disrupt the process of global AI governance and serve the interests of no one.”

Source:Ars Technica News