Military & AI
When AI Builds Itself: Anthropic's Case for a Verifiable Pause, Read as an Arms-Control Problem
The Anthropic Institute's essay argues AI is already speeding up AI development and that the world should have a verifiable option to pause. It never mentions militaries, but its proposal is an arms-control problem: detection, verification and the incentive to defect.
Akmal Alif · 10 October 2026 MYT
On 4 June 2026 the Anthropic Institute published When AI builds itself, by Marina Favaro and Jack Clark. It argues that AI is already speeding up the development of AI, sets out where that could lead, and proposes that the world should have the option to slow down or pause frontier development. On 18 September 2026 the essay was updated with new data (Favaro & Clark, 2026).
The essay does not mention militaries, weapons or national security. It is in our Military & AI section for another reason. Its central policy proposal is an arms-control problem, and the essay itself uses that phrase. It turns on detection, verification and the incentive to defect, under geopolitical pressure.
One caution: Anthropic builds Claude, and most of the evidence below is the company's own internal data about its own models. The authors flag several of their figures as likely overstatements, and we keep those caveats.
What "recursive self-improvement" means
The essay defines the end point as an AI system able to fully autonomously design and develop its own successor. The authors say plainly that Anthropic is not there yet and that it is not inevitable. Their concern is timing: it could come sooner than most institutions are prepared for.
They sketch the path so far:
2021 to 2023. People wrote code and documents by hand.
2023 to 2025. Chatbots produced short snippets that people copied into editors.
2025 to 2026. Coding agents wrote and edited whole files.
Today. Agents run code themselves and delegate hours of work to other agents.
Some future year. Agents could build and train models, so that Claude improves Claude.
The evidence, in brief
From public benchmarks, the authors cite the following (Favaro & Clark, 2026):
Task length. The length of software tasks models can reliably finish alone has been doubling roughly every four months, up from an earlier seven. Claude Opus 3 (March 2024) handled tasks that take a person about four minutes. Claude Opus 4.6, two years later, handled 12-hour tasks.
Benchmarks. SWE-bench went from low single digits to saturation in two years. CORE-Bench, which tests whether a model can reproduce published research, went from about 20% to saturation in fifteen months.
From inside Anthropic:
Code. As of May 2026, more than 80% of code merged into Anthropic's codebase was written by Claude. Before Claude Code's research preview in February 2025, the figure was in the low single digits.
Output. Lines of code merged per engineer per day were about eight times higher in the second quarter of 2026 than in 2024. The authors call this almost certainly an overstatement of the real productivity gain.
Open-ended work. Claude's success rate on the most open-ended engineering tasks reached 76% in May 2026, up 50 percentage points in six months. The September update shows success across all task types converging around 88 to 92%, with open-ended problems rising from about 26% to 91%.
Experiments. On a fixed code-speedup task, Claude went from about 3x (Claude Opus 4, May 2025) to about 52x (Claude Mythos Preview, April 2026). A skilled human reaches about 4x in four to eight hours.
Research. On an open AI-safety problem, two researchers recovered about 23% of the performance gap in a week. Claude-powered agents recovered 97% over 800 cumulative hours, at roughly $18,000 in compute. Humans still chose the problem and the scoring, and the result did not transfer cleanly to production-scale models.
Judgement. Shown 129 real research moments where a human's next step had room for improvement, the best model beat the human's choice 51% of the time in November 2025 and 64% in April 2026. On a control set where the human's move was already strong, models were judged better only about 20% of the time.
The authors' reading: humans now mostly supply direction and judgement, and even that gap is narrowing.
Three possible futures
The essay sketches three scenarios:
The trend stalls. The exponentials turn into S-curves, or chips, power or an outside shock become the limit. The authors think this is unlikely. It would also give societies the most time to adapt.
Compounding efficiency. AI development becomes largely automated while humans still set direction. The authors think the evidence points here. They warn that a 100-person organisation doing the work of a far larger one could improve government services, but could also power authoritarian surveillance of whole populations and influence operations tailored to each person.
Full recursive self-improvement. AI systems build their successors, and human effort shifts to oversight and verification. The authors say they are least certain how alignment turns out in this world.
They also lean on Amdahl's law: speeding up one step moves the bottleneck elsewhere. At Anthropic, human code review has already become that bottleneck. In the wider world, the pace of change will still be set by trials, elections, institutions and relationships.
The proposal
The authors make a conditional offer:
An option to pause. It would be good for the world to have the option to slow or temporarily pause frontier development, so that institutions and alignment research can keep up.
Anthropic would join. They say Anthropic would expect to slow down or pause if other developers at or near the frontier did so in a verifiable way.
Unilateral pauses are not enough. One lab pausing alone would only change who leads, without creating the wider deliberation that is missing.
Verification systems. The Anthropic Institute will research, with others, the systems that would let developers confirm that others have actually stopped or slowed.
Rules for a pause. A credible pause has to say what triggers it, what lifts it and who adjudicates.
Public deliberation. Anthropic will convene policymakers, researchers, civil society and other AI companies, and publish what comes out.
Why this is an arms-control problem
This is where the essay most resembles a defence-policy paper. In the authors' framing:
It needs many parties in many countries. A slowdown only works if several well-resourced labs at the frontier stop under the same conditions.
Defection is tempting. Whoever keeps going while others pause could inherit the lead, so the incentive to defect quietly is enormous.
A pause can backfire. If a slowdown lets the least cautious actors catch up, everyone could end up less safe. Without a global coordination mechanism, companies and governments will make safety decisions under competitive and geopolitical pressure.
Detection is hard. The authors note that training runs are far easier to conceal than missile silos, and that their inputs are general-purpose.
Verification regimes take time. The world has built them before, for example for the Intermediate-Range Nuclear Forces Treaty. Those regimes took decades to build infrastructure and trust, and the authors argue there is not that long.
Our reading: the hardest part is not agreeing that a pause could be wise. It is building the inspection machinery before anyone needs it. With missiles, inspectors could count launchers. With AI, the equivalent would have to be compute, data centres and training runs, and the essay is candid that nobody yet knows how to inspect those with confidence.
Where it touches security today
Two passages connect directly to security:
Cyber defence. The essay cites Project Glasswing. In its first weeks, Claude Mythos Preview found more than ten thousand high- and critical-severity vulnerabilities across important software. The authors say the bottleneck in cyber defence has already shifted from finding vulnerabilities to patching them fast enough. That holds even if model capabilities stopped improving today.
Surveillance and manipulation. The same efficiency that could transform government services could also enable population-scale surveillance and personalised influence operations.
How it was received
Coverage focused on the pause. Fortune's report doubted that rival labs, racing each other and preparing for public listings, would agree to stop together (Nolan, 2026). That is the essay's own dilemma restated: a pause is only as credible as the slowest party's willingness to be checked.
The rest of this section looks at AI already inside defence and intelligence work. The Palantir Gotham series covers how the software is sold and debated. This essay asks what happens when the systems that write that software start to speed up their own development.
References
Favaro, M., & Clark, J. (2026, June 4). When AI builds itself (Updated September 18, 2026). The Anthropic Institute, Anthropic. https://www.anthropic.com/institute/recursive-self-improvement
Nolan, B. (2026, June 5). Anthropic warns AI could soon build itself without human involvement—and urges a global pause on development. Fortune. https://fortune.com/2026/06/05/anthropic-ai-pause-development-recursive-self-improvement