Recursive Self-Improvement

Sources#

• CS329A Self-Improving AI Agents — Part 1: Course Overview
• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
• How to pace the US frontier
• Jeff Dean: The 1% Rule for Building in AI
• Knowledge-Centric Self-Improvement
• Noam Brown – Agent swarms, alignment, & recursive self-improvement
• Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Recursive Self Improvement for Coding Agents
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
• The AI-as-Normal-Technology View of Loss of Control Incidents
• The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement
• When AI builds itself

Summary#

Recursive self-improvement (RSI) is the point at which an AI system can fully autonomously design and develop its own successor — closing the loop so that each model is improved by the previous model rather than by humans. The Anthropic Institute essay When AI builds itself (Marina Favaro & Jack Clark, June 2026) is this wiki's primary source. Its argument has two halves: (1) a present-tense empirical claim that AI is already accelerating the development of AI (AI Accelerating AI Development — e.g. Anthropic engineers ship ~8× more code per quarter than in 2021–2025), and (2) an extrapolation that the trend "points to an AI system capable of fully autonomously designing and developing its own successor." Anthropic's stated position: "We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for."

This page is the hub for the RSI cluster — the trajectory, the futures, and the governance response. The measured evidence lives in AI Accelerating AI Development; the capability-gating eval in AI R&D Autonomy Evaluation (AECI); the deployment brake in Responsible Scaling Policy Evaluations; and the coordination problem in Frontier Pause Verification.

Closing the loop#

The essay frames RSI as the endpoint of a steadily-tightening development loop, illustrated as person → computer → chatbot → agent → workers (each stage delegates more of the work to AI):

• 2021–2023 — Building the first Claude. Humans write code and docs on laptops; AI is absent from the loop.
• 2023–2025 — Chatbots. People paste model-generated snippets into editors.
• 2025–2026 — Coding agents. Agents write and edit whole files on their own (Claude Code launches Feb 2025).
• Today — Autonomous agents. Agents run their own code and delegate hours of work to other agents (the loop primitive running unattended).
• 20XX? — Closing the loop. "Agents could become capable enough to build and train models themselves. If this happens, future versions of Claude could be continuously improved by Claude itself." This last step is RSI.

"What if we're wrong?" — why direction-setting may not save us#

The natural objection: the work still in human hands — choosing which problems to work on (Research Taste as the Human Bottleneck) — is what matters most, so AI remains a capable assistant, not an autonomous driver of progress. The essay offers two rebuttals:

• Perspiration is becoming automated. AI advances rarely come from "eureka" moments; paradigm shifts (the Transformer, mixture-of-experts) "arrive years apart." In between, "most progress is incremental: we scale something up, see what breaks, fix it, and try again" — exactly the workflow Claude now excels at. Edison's "1% inspiration, 99% perspiration" is invoked: "we see perspiration becoming increasingly automated." Large-scale research progress "is mostly a function of tools and resources" — how fast and how many experiments you can run — which is the bitter lesson pushed to its limit.
• A conservative reading still compounds. Even if Claude never gets research taste, if humans spend most of their time on the single-digit fraction of work that is direction-setting while Claude handles the rest, each human steers far more work than before. "AI already makes Anthropic move much faster than it did before."
• The less-conservative reading. The early evidence of improving research judgment (51%→64% on next-step decisions; see AI Accelerating AI Development) suggests taste "might be just another AI capability that AI systems fail at for a time, then get good at" — the same pattern seen with explaining why a joke is funny, theory of mind, and linguistic riddles (Jagged Intelligence (Ghosts, Not Animals)).

Three possible futures#

The essay lays out three scenarios for "what happens next," contingent on whether the trend continues and what we choose to do:

1. The trend stalls (S-curve), but today's capabilities diffuse widely. Exponentials bend; the judgment separating a competent researcher from a great one may not come from scaling compute/data, requiring a new architecture past the Transformer — or the binding constraint may be the supply chain (energy, chip fab, grid, interconnect) rather than intelligence. Even frozen at today's capability, the world changes: Project Glasswing already shifted the cyber bottleneck from finding to patching (LLM-Driven Vulnerability Research), and a 100-person company can increasingly do the work of a 1,000-person one (AI-Native Startup Lifecycle). Anthropic thinks this is unlikely — "we have not yet seen that curve bend."
2. Compounding efficiency gains; humans still set direction. AI development becomes substantially automated but humans judge results. 100-person companies do the work of 10,000–100,000; revolutionizes knowledge work and government — but could power authoritarian surveillance or individualized influence ops at superhuman scale. The essay says the evidence suggests this is the likely path — bounded by Amdahl's law (below).
3. Full RSI — AI builds its successors. Pace becomes determined entirely by compute (and algorithmic-efficiency discoveries). Humans move "most of our effort towards oversight, validation, and verification of an expanding 'virtual lab' run by AI systems," with skills transferring to the rest of science. How the alignment problem resolves here is what Anthropic is "least certain about": models may be aligned and wise enough to find novel solutions (or to halt), or "the rare occurrences of misalignment present in today's models could compound as the models build their successors, growing more frequent but less understood until we lose control."

What the term is starting to get used for (and shouldn't)#

By mid-2026 "recursive self-improvement" has begun appearing as a product framing for narrow scaffold optimization, and the two senses need separating. Cline's July 2026 post Recursive Self Improvement for Coding Agents (case-study) opens by linking this essay and the Wikipedia RSI article, and reports that one prompt drove 17 hours of autonomous agent work that patched Cline's own harness — retry policy, loop detector, process handling — lifting Terminal-Bench 2.1 from 77.5% to 88.8% with Kimi K3.

That is a real result and it is not this page's subject. The model's weights were untouched; the artifact was a pull request against a TypeScript repo; the direction came from a human-written brief with a pinned end state; the campaign ran once rather than compounding; and a human reviewed the PR before merge. Nothing about it demonstrates a system designing its own successor — the definitional core above. What it does belong to is AI Accelerating AI Development, the essay's present-tense empirical half: AI compressing AI-adjacent engineering work (four engineers × two weeks in January → 17 unattended hours in July, per Cline's own before/after). The full weighing, including the overfitting caveat that the vendor has been hill-climbing this benchmark for six months, is in Agent-Authored Harness Optimization.

The distinction is worth keeping sharp because the vocabulary is the vendor's claim, not a definition, and because there is a testable boundary underneath it: does the improvement transfer? A scaffold patch that lifts held-out tasks the optimizer never saw is a different object from one that lifts the suite it was optimizing against. No transfer measurement accompanies the Cline campaign.

A controlled evaluation now runs that boundary test, and it lands on the right side of the line. HarnessBank (Luo et al., arXiv 2607.13683, empirical) evolves harnesses for frozen backbones across seven benchmarks with sealed test splits, and reports two results that matter here:

• The gains are real but the artifact is not portable. Six of seven benchmarks credit the evolved harness on held-out tasks (+9.2 to +15.4pp), yet cold-starting the loop on a different model family produces a different harness, and transplanting one model's harness onto another is near-zero off its matched failure pathology and -15.7 when the same lever is turned the wrong way. "A credited harness is a correction fitted to the model, not a universally good setting." Harness self-evolution produces model-specific corrections, which is definitionally not general capability.
• The loop converges rather than compounding. Under its significance gate the search terminates at its 10-round floor; only the ungated variants run forever, and they do so because they hallucinate progress in 62–76% of post-convergence rounds. There is no compounding here to extrapolate from — a human re-issues the brief for each new model and domain.

So the strongest-evidence source on harness self-evolution is also the strongest argument that harness self-evolution is not this page's subject. Full treatment in Agent-Authored Harness Optimization.

The teaching-grade version: the model as its own data generator#

A third usage, older than the two above and pointed at a different object. Stanford's CS329A is titled Self-Improving AI Agents, and what its instructors mean by self-improvement (lecture 1, delivered 2025-09-22, practitioner-opinion) is the train-time/test-time flywheel: in domains with a cheap verifier, test-time scaling is a synthetic-data engine — maths problems with known answers, code with unit tests — so the model's own verified trajectories become the next fine-tuning set, and the improved model then scales further at inference. Azalia Mirhoseini: "there's no boundary in how good the models can become with test time scaling and then bringing that back to the process of training the model… that's the self-improving piece."

This lands in a different place on the page's taxonomy than harness patching does. The weights do move — which the Cline and HarnessBank cases explicitly do not — so it is not merely scaffold optimization. But it is still not the definitional core: a human specifies the domain, curates the verifier, and runs each round; what compounds is a dataset, not a designer. Held next to Knowledge-Centric Self-Improvement the two are near-mirrors — that one accumulates a portable knowledge asset while the model stays fixed; this one accumulates model weights while the knowledge stays implicit — and neither moves toward a system designing its successor. The dependency both share is the verifier, which is also this version's ceiling: the flywheel spins where checking is mechanical and stalls exactly where it is not.

Two honesty notes from the instructors that the vendor framings above lack. Whether the loop's gains come from RL or from what pretraining already contained has, in their words, "no single point of consensus"; and the loop as a whole "is not completely well understood — it's the first signs of life and it starts to get commercialized." That is roughly the epistemic position this page argues for, arrived at from a lecture hall rather than a governance essay.

Compounding and RSI are orthogonal#

A third self-improvement axis clarifies the vocabulary further, by scoring better on the properties the RSI literature cares about while being further from the definition. Knowledge-Centric Self-Improvement (Wang et al., Caltech, arXiv 2607.19592, empirical) makes the agent explicitly disposable — fresh context every attempt, no private memory, no specialization, no modified prompts — and lets only an external curated knowledge base persist. Yet:

• It transfers. A knowledge bundle frozen at generation 10 lifts zero-shot solve rates on held-out tasks in every donor-recipient cell, in both cross-family directions — where a transplanted harness is near-zero off its matched pathology.
• It accumulates. Ten generations run under one protocol with no human re-issuing a brief, against harness evolution converging at a 10-round floor.

Nothing about the model improves, and the paper does not claim otherwise. Which is the point: a system can accumulate a portable, compounding asset with zero movement toward designing its own successor. "It compounds" is therefore not evidence of proximity to RSI, and the two properties should be argued separately. (The compounding claim has its own caveats — solved tasks are retired from the pool each generation, so part of the generation-over-generation curve is easy tasks leaving, and no run goes past ten generations.)

The deployed version: 161 days of self-modification, and no capability curve#

Every instance above is a bounded campaign — a human writes a brief, a loop hill-climbs a benchmark, the campaign ends. Ouroboros/Hope (Razzhigaev et al. incl. Roman Yampolskiy, arXiv 2608.08311, case-study) is the first in the corpus that runs continuously and unbriefed: 161 days of public deployment, 1,085 self-modification commits at 94.2% agent-authored, improvement either scheduled as its own recurring task or triggered by ordinary work and user complaints, with every change passing a blocking multi-model diff-review gate (63.5% recent block rate). It removes the two objections that bound the campaigns — no brief to re-issue, no score to overfit.

And it supplies no evidence of compounding, because it never measures capability. The paper's benchmark scores are single frozen-seed snapshots run with self-evolution disabled (Table 3 says evolution off on four of five rows), and its only time series — Figure 6 — plots four deployment-activity metrics at monthly endpoints: model spend, tokens, published LOC, memory artifacts. Spend and tokens grow steadily-to-decelerating; the artifact series are step-shaped, and published LOC actually declines in the final month. So the record shows the loop kept running and kept shipping, and says nothing about whether Hope got better at anything over the 161 days. There is no ablation anywhere in the document connecting self-development to a score.

That is a useful null for this page in a specific way. The separator the corpus has been using — does it transfer, and does it compound without a human re-issuing the brief? — was built against loops that terminate. Here is a loop that does not terminate, and the answer is unavailable rather than negative: the one thing an unbounded self-modifying deployment could have contributed to the extrapolation at the top of this page is a capability curve, and the first such deployment published an expenditure curve instead.

The verification leg: a closed loop still needs a grader it did not write#

The three futures above all assume, silently, that each generation can tell whether its successor is better. Guo et al. (Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents, CAS, arXiv 2607.24300, empirical) test that assumption directly in the non-gradient self-improvement setting this page's vocabulary debate lives in, and it fails: an agent that edits its policy and its own tests together keeps self-scores at 0.70–1.00 while 15 of 35 model-game policies end below the game's random reference, and the failure is stratified but never absent across the capability ladder — weaker agents overwrite behavior they had discovered, stronger ones "mismeasure the shifted deployment distribution." Constraints that stay inside the self-authored instrument do not close the gap, and an information limit (α + β ≥ 1 - TV(P+, P-)) says no endogenous-only gate can, once regressing and non-regressing candidates look alike from inside.

The paper's conclusion is a structural claim about self-improvement in general, not about Atari: "reliable self-improvement need not abandon self-verification, but it requires at least one deployment-acceptance signal outside the agent's control." Read against the definition at the top of this page, that is a constraint on what "fully autonomously designing its own successor" can mean — a genuinely closed loop would have to author its own acceptance boundary, which is the one component this result says cannot be endogenous without the improvement signal losing deployment meaning. Whether the exogenous anchor must stay human, or can be an executable procedure the system is merely forbidden to read (SEAL's audit is the latter), is the live design question. Developed on Optimizer–Evaluator Decoupling; the harness-loop instance is on Agent-Authored Harness Optimization.

A second lab's version of the same prediction — and the missing cheap validator#

Anthropic's essay is the corpus's primary source, but the trajectory is not Anthropic's alone. Jeff Dean (Google Chief Scientist, YC Startup School 2026, practitioner-opinion) was asked for a 2027 prediction and gave essentially this page's future 2, arriving from the systems side rather than the safety side: "a lot more automation of ML systems themselves… getting ML systems to improve their capabilities by running lots of experiments, breaking things down into sub-problems, running those sub-problems in a tight automatic experimentation loop, putting the results together." He generalizes the precondition — "anything where you can have a measurable objective" — and names the objective function: "optimize your discoveries per unit of compute input." Note the convergence is on the mechanism, not the risk: Dean's version has no misalignment branch and no governance response, and he never uses the term "recursive self-improvement" about it.

His contribution the Anthropic framing lacks is the rate-limiting step, and it is not the proposer. Today's model-development loop already looks like his description (small-scale experiments → promising ones scaled → results integrated into a recipe); what makes automating it interesting is loop latency, and latency is usually set by evaluation. His worked example is a decade-old quantum-chemistry result from Google colleagues: density functional theory takes a night of compute to characterize one molecule, so they trained a neural approximation on simulator input/output pairs and got something ~300,000× faster and "nearly as accurate." "Now you have 10 million things to screen. You could do that while you go to lunch rather than it being a six-month endeavor."

That cuts against the verification leg above in a specific way worth flagging. Guo et al. establish that the acceptance signal cannot be endogenous; a learned surrogate of an exogenous simulator is an interesting middle case — it is trained from the external oracle rather than authored by the improving agent, but "nearly as accurate" is precisely the regime where a screening loop learns to live in the surrogate's error. Nothing in the corpus measures what happens when an automated experimentation loop optimizes for 100,000 rounds against a 300,000×-cheaper approximation of its own validator. Dean does not raise the question; Optimizer–Evaluator Decoupling is where it belongs.

The vocabulary problem, finally given a ladder (September 2026)#

Everything above this line is this page doing the same job four times: taking a new use of "recursive self-improvement," working out what object it actually names, and saying why it is or is not the definitional core. A 35-author survey out of Shanghai Jiao Tong University and Theseus Labs (arXiv 2609.11873, 2026-09-10, 79pp, practitioner-opinion) supplies the instrument that would have made those four passes one: a ladder keyed on which improvement decision has been transferred from the designer to the system, rather than on what the system optimizes.

Its argument for the keying is the exact complaint this page keeps making: prior taxonomies sort by what changes — prompt, memory, harness, weights — but "systems that update the same component may assign very different decisions to AI." Run this page's four cases up the ladder and they separate cleanly for the first time:

• Cline's harness campaign and the HarnessBank/DarwinX literature — L2. The system decides how to improve; the objective, the benchmark and the promotion rule stay external. The survey's own L2 exemplar is Self-Harness, "under a fixed benchmark and promotion rule."
• The CS329A train-time/test-time flywheel — L1, and L3 only if the learner's own state determines what gets proposed next. Human specifies the domain, curates the verifier, runs each round; the weights move, the decisions do not.
• The Caltech knowledge base — L1 by the survey's criterion, which is the sharpest confirmation of that page's own orthogonality result: an artifact can transfer and compound while sitting on the bottom rung, because nothing about who decides what to retain has moved.
• Ouroboros/Hope — L4, and the survey cites it as its own Case 2 for deployment-level RSI. Persistent revision of tools, context assembly, prompts and core implementation from reviewed deployment evidence, with acceptance external.

None of them reach L5, the level where "the mechanism responsible for later improvement itself becomes an inherited target of improvement" — which is the closest formal statement of this page's subject anyone has written.

The definition it offers is stricter than Anthropic's in a useful way. "Autonomously designing and developing its own successor" is a claim about an outcome and therefore only checkable after the fact; the survey's version — persistent self-change "such that these changes can further affect the mechanisms used to generate, evaluate, select, and consolidate subsequent self-improvements" — is checkable on a mechanism, today, by asking where the loop closes, what is inherited, and which decisions stayed external. It also splits the claim in two: structural recursion (a revised mechanism is inherited and governs a later round) versus effective recursion (it produces better successors under comparable budgets and independent assessment).

Two things follow for this page.

The survey cites the Anthropic essay as one of four lab definitions, not as the field's. §2.2.2 sets Anthropic's "strongest form … an AI system autonomously designing and developing its own successor" alongside OpenAI's operationalization through automated research workflows, Tencent's report of "an early-stage RSI loop in which experimental results are fed into subsequent rounds of model development," and Alibaba researchers' stricter criterion that the improvement mechanism itself be subject to modification. The last of those is the survey's own, and it is the one that agrees with this page.

Effective recursion is unestablished across a much wider evidence base than this page had. The survey's own tally: STOP's fourth-generation improver beat its seed on five held-out transfer tasks, but weaker-model runs regressed on average; Gödel Agent had 14 of 100 MGSM trials finish below the initial policy; HyperAgents showed cross-domain transfer of the improvement procedure in its preprint and then "did not establish a statistically significant final advantage for transferred initialization" at 200 iterations; A-Evolve-Training reached 0.86 against 0.87 for the top human submission over four autonomous rounds; Weco's AIDE 2 — whose blog post is titled "The first evidence of recursive self-improvement" — found no statistically significant efficiency advantage on its own stronger test of whether an evolved harness improves the outer search. The verdict: "current results support bounded meta-improvement … statistically reliable accumulation across generations under comparable resources remains open."

That is the same null this page recorded from HarnessBank's termination floor and from Hope's missing capability curve, now reached over 491 surveyed papers and 72 companies. It does not move the extrapolation at the top of this page in either direction — but it does mean the extrapolation still rests on the measured-acceleration half of Anthropic's essay (AI Accelerating AI Development) and not on any demonstrated instance of the mechanism it extrapolates.

One coverage note before this section is leaned on: the survey never cites HarnessBank, DarwinX or the budget-matched Wang et al. control, so its picture of the harness literature is missing the corpus's three strongest sealed-split and matched-budget results — in both directions.

Amdahl's law for organizations#

A recurring brake across futures 2–3: speeding up one part of a process just shifts the bottleneck elsewhere; overall pace is capped by the parts that haven't sped up (Amdahl's law). Anthropic has already hit its signature: as more code flows through the org, human code review became the new bottleneck — the org-level instance of Verification as the New Bottleneck. The same friction appears beyond engineering: an explosion of ideas/initiatives/tools "far more than we have the capacity to pursue." Spotting and clearing these bottlenecks "may become the most important skill for any organization." This is also why "the felt pace of this future will still be set by the bottlenecks" — RSI can't run clinical trials faster than biology, hold elections sooner than constitutions allow, or turn a stranger into an old friend in a weekend.

An external practitioner reaches the same brake by a different route. Noam Brown (OpenAI, practitioner-opinion) argues an overnight intelligence explosion is unlikely precisely because peak capability requires large-scale test-time compute — runs that take weeks or months — so time itself becomes the binding constraint and the realistic shape is a "gradual takeoff," not an instant one. It is the Amdahl's-law point made about inference duration rather than org throughput; the full argument sits in Intelligence Explosion Dynamics. (2026-09-17: re-asked after the Navier-Stokes result, Brown keeps the verdict and replaces the mechanism — serial experiment latency and GPU supply rather than inference wall-clock — and puts a hedged 3× on the accelerated regime. The displacement is recorded on Intelligence Explosion Dynamics.)

The argument from maths, and the reason it is not the same argument as usual (September 2026)#

The RSI case on this page is built from engineering telemetry — code shipped, review throughput, harness patches. Brown's September 2026 interview (practitioner-opinion) supplies a different route to the same conclusion, and it is worth separating from the usual "look how fast maths is going" argument because both participants state the structural claim underneath it explicitly.

The host's version, which is the one this page has to grade. Dwarkesh Patel's argument is not that maths progress predicts AI progress by analogy. It is that the two have the same shape of objective and maths is the harder case:

"In ML you don't care about better understanding the nature of deep learning, or you only care about that as an instrumental goal towards just achieving the result. Just solve this well-scoped problem of improving the sample efficiency of our models, improving the pre-training loss."

Brown agrees and sharpens it into the one place the models' jaggedness is an advantage: "the ways that they're spiky end up, I think, probably being particularly useful for things like RSI. You have a more clear objective. It's just more measurable. There's less question of, 'Well, what new branches of mathematics are worth exploring?' No, there's a very clear answer." That is a claim about the objective, not the capability — AI R&D is well-scoped where mathematics-as-a-discipline is not, so the function the models are worst at (choosing what is worth doing, per Research Taste as the Human Bottleneck) is the function AI R&D needs least. It is the sharpest argument in the corpus for why capability that looks narrow could still close this page's loop, and both speakers arrive at it independently in the same exchange.

And the same exchange supplies the counter, from Brown, immediately: "The main difference is that in mathematics, you're purely bottlenecked by thinking… When you look at things like RSI, you do have to run experiments." So the structural similarity holds on the objective and breaks on the feedback channel — which is exactly where the corpus's other RSI nulls break too (Optimizer–Evaluator Decoupling's acceptance signal, Autonomous Scientific Discovery's non-identifying experimental feedback).

The jaggedness-to-generality bridge, also the host's and also worth carrying because nothing else in the corpus states it this cleanly: it is enough for a model to be jaggedly good at building a better learner, because "that better learner can be more general… assuming there's good enough transfer from the direct problem you're solving to this broader ability to learn." Narrow capability at the meta-level buys general capability at the object level. The assumption is doing the work and is named as an assumption.

What the ladder is worth as evidence. Brown's 10×-per-year human-solve-time series (GSM8K ≈ 5s → MATH ≈ 1min → AIME ≈ 10min → IMO ≈ 100min, projecting ~15h) is a four-point retrospective fit with no error bars, and its author's own use of it was to predict 2028 at the earliest for a Millennium Prize result that arrived in 2026. He reports taking the aggressive side of a $1,000 bet against another lab's researcher two weeks before the result and still being wrong. The transferable content is therefore not the rate but the direction of the miss: the corpus's two published extrapolations of this quantity — this one and METR's ~4-month doubling — have both been undershooting, and the people inside the labs report shortening their own forecast windows "from 12 months to three months." Read it as a calibration datum about forecasters, not as a scaling law.

The counterweight Brown volunteers, and it is this page's future 1 in a form no source here has had. He names a mechanism by which the curve could bend that is neither an architectural ceiling nor a supply-chain constraint — curriculum exhaustion:

"In things like AlphaZero, where you have self-play, you have an infinite curriculum. You're always playing against an AI that's equally strong. Whereas for things like training an LLM with reinforcement learning… if the problem is so easy that it can just solve it in a second, it's not really learning anything. If we run out of problems to challenge it, then that is a plausible scenario where it becomes much harder to make progress."

That is a data-side friction on mechanism 4 of the four RSI engines, and it is specific: the asymmetry is that self-play generates an opponent of matched difficulty for free while RL-on-problems does not, so the AlphaGo trajectory everyone reaches for as the precedent may not transfer for a reason that has nothing to do with model capability. Brown hedges it — "we haven't really hit that as a wall yet… if it ever became a serious problem, there would be ways around it. But it is a plausible scenario" — and it is the only named candidate in the corpus for why the maths trajectory might not recapitulate AlphaZero's.

The alignment leg: compounding in the wrong direction#

The essay's future 3 names misalignment compounding through self-improvement as what Anthropic is "least certain about." Brown states the same worry as a quantitative trajectory, from inside the other lab:

"We make them what we think is aligned, and they're 99.9% aligned. Then we use these models to help us with the next generation of models, and they end up being 99.8% aligned. Then with each subsequent generation, we see an increasing degradation in alignment. Because we're relying more and more on these tools — this is already the case, that we're relying a lot on AI models to help us with our research and with alignment efforts — in the long run, they end up going in the direction of increasing misalignment from humans."

Three things make this more than a restatement. The premise clause is a present-tense first-party report — OpenAI is already using models on its alignment work, which is the loop closing on the safety side rather than the capability side. The failure mode needs no discrete misalignment event: a per-generation multiplier slightly below 1 is sufficient, which is a far weaker precondition than the "rare occurrences compound" framing this page carries. And Brown states the optimistic branch as genuinely open rather than as a rebuttal: "There is a possibility that we go in the other direction, that actually every generation of models, we're able to make more and more aligned. I don't have an answer for how we ensure that we end up in that second trajectory."

What nobody has is the sign. The quantity — whether the per-generation alignment multiplier is above or below 1 — is not measured anywhere in this corpus, and Brown's numbers are illustrative rather than estimates. It is filed as this page's fourth open question because it is the one number that decides between futures 2 and 3, and because the existing capability-side gates do not measure it.

What should we do? (the governance response)#

Anthropic argues it would "likely be a good thing" to have the option to slow or pause frontier development so societal structures and alignment research can keep up — but a unilateral pause merely changes who leads, and a real one requires multilateral, verifiable coordination. Building the systems that make a credible pause possible is the subject of Frontier Pause Verification and the Anthropic Institute's agenda. "The window to investigate the questions together is here, and people outside AI companies should be involved."

A second lab's governance answer: allocate, don't gate#

Zuckerberg's August 2026 manifesto (prediction) reaches the same trap and takes the opposite exit. He states the competitive dilemma more bluntly than Anthropic does — "any lab that doesn't let their AI system direct a substantial amount of compute capacity towards recursive self-improvement will inherently fall behind" — and illustrates the runaway with a self-improving system that finds "100x or more intelligence out of each gigawatt," at which point a fraction of world compute commands more effective intelligence "than everyone else combined."

His remedy is not a pause option but a compute-allocation ratio: labs and clouds should collectively build enough compute to spend competitively on RSI "while still committing the significant majority towards people's individual goals." Three ways it differs from the paragraph above:

• The safety property is a fraction, not a capability gate. Nothing is withheld; the claim is that RSI stays safe as long as the majority of world intelligence remains pointed at human-chosen goals. No threshold is named and no measurement is proposed — and from outside a lab, the fraction is not observable.
• It resolves the trap with capex. Anthropic's answer to "everyone must race" is verifiable coordination; this one is a larger denominator.
• It accepts unaligned goal-pursuit as the cost. "Any AI engaging in recursive self-improvement is by definition directing and advancing its own goals… not inherently harmful by itself, even if its goals are not fully aligned with many people, as long as we maintain a balance of power that favors people overall." That trades the alignment property for an unmeasured quantitative one.

A third answer: forbid the loop's own inputs#

The AI Futures Project (2026-08-05, practitioner-opinion) proposes the only intervention in the corpus aimed at this page's mechanism rather than at a proxy for it. Its Option 3 caps the capability of the models a company is allowed to use to automate AI R&D — best guess "AI R&D can only be conducted or assisted by AIs that were trained at least 9 months ago" — on the explicit ground that a lag "delays the point at which AIs have the capability to sabotage AI research and align the next model to themselves." That is the closing-the-loop step above, blocked by making the loop's operator a generation behind its own output. Its Option 2 attacks the same loop from the resource side, capping capabilities R&D at 5% of total compute while mandating 70% to external inference and 25% to transparent safety.

Two features are worth carrying back here. The proposal is explicitly designed to be acceptable to RSI skeptics — "those who are skeptical of rapid recursive self-improvement may be open to this policy, because on their worldview it wouldn't have as much of an effect" — which makes it the only governance mechanism in the corpus whose cost is low precisely in the worlds where this page's premise is false. And its modeling puts numbers on what the intervention buys: on two forecasters' median parameters, a 9-month lag applied from late 2026 extends takeoff from Automated Coder to superintelligence from 1.0 to 2.4 years and from 2.6 to 4.1 years, while delaying Automated Coder itself by only 0.1-0.2 years. Those are the proposers' own model and priors (Intelligence Explosion Dynamics carries the full comparison), not an external check.

Where the two labs' answers agree is worth recording, because it is the stronger claim: both hold that a lab running powerful self-improving models privately is the most dangerous configuration, not the safest. Zuckerberg's version — "regardless of how much a lab rationalizes this activity in terms of responsibility and safety, this is the path of developing a singular superintelligence that cannot be checked by other systems" — is aimed at closed labs, but it is the same reason Anthropic gives for wanting the pause option to be multilateral.

A critic voice converges on the same friction, from outside either lab (September 2026)#

Kapoor & Narayanan (practitioner-opinion, normaltech.ai) are this corpus's clearest sustained critic of fast-takeoff RSI framing, writing in the same essay that revisits the Hugging Face incident as a control failure. On RSI specifically: "We take RSI seriously, but we think many of the bottlenecks to superintelligence are external and won't be overcome by improving computation," naming compute-acquisition frictions — including KYC-style controls on large training runs — as the practical brake on rogue self-improvement, not a modeling assumption that self-improvement can't compound. This is a fifth named external brake, not a new mechanism: it lands in the same place as this page's Amdahl's-law/embodied-bottleneck synthesis (RSI Growth Curves: Which Friction Binds First?) — the binding constraint sits outside model cognition — but adds nothing to which friction binds first, since it names compute acquisition rather than verification/oversight or physical-experiment latency. Recorded as a third-source convergence, not a resolution of the open question below.

Connections#

• AI Accelerating AI Development — the measured, present-tense evidence half of the essay; the data behind "the loop is tightening"
• AI R&D Autonomy Evaluation (AECI) — the capability-side gate: AECI and the substitution threshold are how Anthropic measures "can the model build the next model?"
• Responsible Scaling Policy Evaluations — the deployment brake; the RSP AI-R&D threat model is RSI risk made operational
• Research Taste as the Human Bottleneck — the last human comparative advantage; whether it holds determines which of the three futures obtains
• Task Time-Horizon Scaling — the external trendline (METR doubling every ~4 months) that makes the extrapolation quantitative
• Frontier Pause Verification — the governance response: building the verification regime a credible slowdown would require
• The Bitter Lesson — "perspiration is automatable" is the bitter lesson applied to research itself; RSI is its furthest extrapolation
• Agentic Loops Overtake Bespoke Systems — RSI's clearest existing-domain proxy: a simple loop matched a bespoke trained system as the model improved
• Harness Shrinkage as Models Improve — the same human-role-narrowing dynamic; humans stop writing code and shift to review
• Verification as the New Bottleneck — Amdahl's law instantiated: review becomes the binding constraint as generation accelerates
• Agentic Misalignment (AM) — the failure mode that could compound through self-improvement: misalignment growing "more frequent but less understood"
• Jagged Intelligence (Ghosts, Not Animals) — the "taste is just another capability AI masters" argument rests on the joke/theory-of-mind precedent
• LLM-Driven Vulnerability Research — Glasswing is the essay's proof that even frozen capability reshapes the world
• Autonomous Scientific Discovery — June 2026 wet-lab evidence that "perspiration is becoming automated" reaches discovery itself (the futures-2/3 case): autonomous drug design, novel hypotheses, week-long genomics
• AI-Native Startup Lifecycle — the diffusion scenario: each employee atop a pyramid of agents; 100-person firms doing 1,000-person work
• AGI-to-ASI Pathways — DeepMind's "From AGI to ASI" report makes RSI its pathway 3; the theory-first sibling treatment to Anthropic's empirical essay
• Intelligence Explosion Dynamics — the growth-curve question (exponential vs. hyperbolic/singularity vs. S-curve) and the four RSI mechanisms (genetic, cultural, cooperative, data), from the DeepMind report
• Multi-Agent Collective Intelligence — cooperative/sociogenic RSI: specialization in agent collectives freeing resources for further specialization
• Large-Scale Test-Time Compute — Brown's test-time-compute pacing argument: peak capability needs long runs, so time bounds the takeoff and an overnight explosion is unlikely (developed in Intelligence Explosion Dynamics)
• Agent-Authored Harness Optimization — the term borrowed for narrow scaffold hill-climbing; kept adjacent deliberately, so the vendor framing doesn't get read back into the definition above
• Optimizer–Evaluator Decoupling — the verification leg of the definition: a self-improving system's acceptance signal is the one component that cannot be endogenous without the improvement signal losing deployment meaning
• Knowledge-Centric Self-Improvement — the axis that separates compounding from RSI: a disposable-agent protocol whose external knowledge artifact transfers and accumulates while nothing about the model changes
• Jeff Dean — a second lab's 2027 prediction converging on future 2 from the systems side, plus the rate-limiting step Anthropic's framing omits: loop latency is set by the validator, and a learned surrogate (300,000× faster DFT) is how you cut it
• Domestic Frontier Pacing — the only proposal in the corpus aimed at this page's mechanism directly: a 9-month lag on models permitted to automate AI R&D, justified as delaying the point at which AIs can sabotage AI research and align their successors to themselves, plus a 5%-of-total-compute cap on capabilities R&D
• Researcher Uplift from Code Output — a near-term marker on this trajectory: METR's Kwa back-solves ~2.5× serial researcher uplift from Anthropic's 8×-code figure and estimates Anthropic's own 2× overall R&D threshold trips at ~3.5× researcher uplift, "which could happen in the next year or so"
• AI Control vs. Alignment — the same September 2026 critic-voice essay's control-vs-alignment reading of the Hugging Face incident; its RSI stance is recorded here, its incident reading there
• RSI Autonomy Levels (B0–L5) — the ladder that separates the four objects this page keeps disambiguating by hand (B0 in-task → L1 execution → L2 strategy → L3 learning agenda → L4 deployment adaptation → L5 recursive inheritance), plus the structural-versus-effective test and the improvement-loop anatomy that makes each boundary checkable
• Headroom-Closed Index (HCI) — the same survey's empirical half, and the motivation underneath the ladder: normalized against each benchmark's entry-year frontier, the domains furthest from closed are the interactive, stateful ones (tool agents 39.9, software engineering 52.6) where the L3–L4 mechanisms would apply
• Continuous Self-Modification Under Review — the first unbriefed, continuous instance of the loop, and a null on the property this page cares about: 161 days and 1,085 self-modification commits behind a blocking review gate, with the only published time series measuring spend, tokens and code volume rather than capability, and every benchmark score taken on a frozen seed with self-evolution switched off
• Agentic Self-Modification (Agent-Initiated Weight Updates) — the weights move and the agent decides, unprompted, yet it is not RSI: one non-compounding fine-tune aimed at a human-assigned repair, with no successor design. Its contribution to this page is that the mechanics of a weight-moving loop (write training data, train, merge, redeploy) are already within a 4B–27B open model's reach on one GPU

Open Questions#

• Is "research taste" a true ceiling (future 1) or just the next capability to fall (futures 2–3)? The essay frames this as the single load-bearing uncertainty.
• The RSI extrapolation rests on trends staying exponential rather than S-curving — but the essay concedes it cannot rule out an architectural ceiling or a compute/energy supply-chain constraint. Which binds first? Partially answered (synthesis against DeepMind): RSI Growth Curves: Which Friction Binds First? — the three futures map one-to-one onto DeepMind's three growth shapes; the first friction to bind is the already-binding one (Amdahl's-law verification/oversight = DeepMind's embodied bottleneck), and the The Abstraction Barrier supplies the mechanism Anthropic lacks for whether taste is a real ceiling (Future 1). Retagged #oq/now → #oq/source 2026-08-10: which friction actually binds is now an empirical question about the next capability generation, not a synthesis gap.
• If misalignment compounds through self-improvement (future 3), is AECI-gated RSP review fast enough to catch it before control is lost?
• Is the per-generation alignment multiplier above or below 1? Brown frames future 3 as a compounding quantity rather than an event — 99.9%-aligned models helping build 99.8%-aligned ones, with OpenAI already using models on its own alignment work — and says outright he has no answer for how to ensure the other direction. Nothing in this corpus measures the sign: the capability-side gates (AI R&D Autonomy Evaluation (AECI)) ask whether a model can build its successor, and no published evaluation asks whether the successor a model helped build is more or less aligned than its parent under matched measurement. What would settle it: an alignment-eval series run on two consecutive model generations with the degree of model assistance in the second one recorded, which no lab publishes.

Sources#

• CS329A Self-Improving AI Agents — Part 1: Course Overview — Stanford CS329A lecture 1 (Azalia Mirhoseini & Aakanksha Chowdhery, delivered 2025-09-22, published 2026-08-03, practitioner-opinion): the third usage of "self-improving" — the test-time-compute → verified-synthetic-data → fine-tuning flywheel — plus the instructors' explicit statement that the RL-vs-pretraining attribution has no consensus
• When AI builds itself — Anthropic Institute, When AI builds itself: Our progress toward recursive self-improvement, and its implications (Marina Favaro & Jack Clark, June 2026)
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — Noam Brown (No Priors, 2026-06-26), practitioner-opinion: overnight explosion unlikely because test-time-compute dependence makes time the binding constraint ("gradual takeoff")
• Noam Brown – Agent swarms, alignment, & recursive self-improvement — Noam Brown (OpenAI) with Dwarkesh Patel, Dwarkesh Podcast, 2026-09-17 (practitioner-opinion). §00:22:02 for the well-scoped-objective argument, the jaggedness-to-generality bridge (the host's), the 10×-per-year human-solve-time ladder and its author's own failed 2028 projection, and the curriculum-exhaustion friction; §00:40:22 for the 99.9%→99.8% generational-degradation scenario and the present-tense statement that OpenAI already relies on models for alignment work. All first-party and all hedged: the ladder is a four-point retrospective fit with no error bars, the alignment percentages are illustrative rather than estimates, and Brown labels the alignment material as spitballing. The structural argument (maths and AI R&D share an objective shape) is stated by both speakers and is attributed to the exchange, not to OpenAI; the jaggedness-to-generality bridge is Dwarkesh Patel's. Full delta against the June interview on Noam Brown
• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution — Luo et al. (arXiv 2607.13683, 2026-07-15, empirical): §4.6 cross-model dissociation and §4.7 termination behavior — the empirical case that harness self-evolution yields model-fitted corrections that converge, not general capability that compounds
• Knowledge-Centric Self-Improvement — Wang et al. (Caltech, arXiv 2607.19592, 2026-07-21, empirical): §3 disposable agents with an external knowledge base as the only persistent object, §4.4 held-out cross-family transfer — the case that compounding and RSI are separable properties
• Jeff Dean: The 1% Rule for Building in AI — Jeff Dean with Diana Hu, YC Startup School 2026 (2026-07-30, practitioner-opinion): §"AI Systems That Improve Themselves" and §"AI That Builds Better AI" — the 2027 automated-experimentation prediction, "discoveries per unit of compute input," and the 300,000×-faster learned DFT surrogate as the way to cut loop latency. COI: Google's Chief Scientist; the quantum-chemistry result is colleagues' work recalled without citation
• How to pace the US frontier — AI Futures Project (Lifland, Halstead, Dean, Larsen, Kodama), 2026-08-05 (practitioner-opinion): §"Limit capabilities of models used for AI R&D" (the 9-month lag, the sabotage rationale, the RSI-skeptic argument) and §"Compute allocation requirements". Modeling figures are the proposers' own model run on the proposers' priors; full treatment at Domestic Frontier Pacing
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents — Guo et al. (Chinese Academy of Sciences, arXiv 2607.24300, 2026-07-27, empirical): the verifier-deployment gap, the information limit on endogenous evidence, and the conclusion that reliable self-improvement "requires at least one deployment-acceptance signal outside the agent's control" — the constraint this places on a fully closed loop. Parse warning: its Table 7 was collapsed and not quotable as parsed; full treatment and parse notes on Optimizer–Evaluator Decoupling (2026-09-07: Table 7 hand-rebuilt in the raw against PDF p.7, one model per row — now quotable; verify.py reports ok)
• The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement — Duan, Liu, Tang, Chen, Zhou et al. (35 authors; Shanghai Jiao Tong University, Theseus Labs, Tsinghua, ByteDance, ModelBest, Xiaohongshu, Shanghai AI Lab, Humanlaya, Agent-Native Research Lab, Frontis.AI), arXiv 2609.11873, 2026-09-10, 79pp (practitioner-opinion): §1.4 the five autonomy levels; §2.2.2 the definition and the four lab definitions it collects (Anthropic, OpenAI, Tencent, Alibaba); §3.6 the structural-versus-effective split and the per-system bounded and null L5 results; §3.6.4 the AIDE 2 assessment. Full treatment on RSI Autonomy Levels (B0–L5); §2.1's benchmark re-analysis on Headroom-Closed Index (HCI). COI: six of the affiliations appear as §5 industry cases reported by their own employees, and §2.3 admits engineering blogs and company materials as primary evidence — every industrial claim is vendor-claim tier. Coverage gap: does not cite HarnessBank, DarwinX or the budget-matched Wang et al. control
• The AI-as-Normal-Technology View of Loss of Control Incidents — Kapoor & Narayanan, normaltech.ai, 2026-09-14 (practitioner-opinion, argumentative, no new data): the external-bottleneck RSI stance quoted above. Full treatment, including its Hugging Face incident reading, on AI Control vs. Alignment
• Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution — Razzhigaev, Gritsaev, Kaznacheev, Dragunov, Yampolskiy & Kuznetsov (MSU / Skoltech / Joi Lab / AIRI, arXiv 2608.08311, 2026-08-08, case-study — downgraded from empirical at compile): §4 Hope's 161-day deployment and counters, Appendix C's evolution off scaffold disclosures, and Figure 6's activity series. Total author COI (they built, ran, audited and score-adjusted their own system). Parse warnings: Table 2 fully collapsed, Table 4 row-shifted past a clean checker verdict; both recovered and both documented on Continuous Self-Modification Under Review, which carries the full treatment