Agent-Authored Harness Optimization

Sources#

• Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
• DarwinX: Evolving Agent Harnesses Through Natural Selection
• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
• How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes
• Knowledge-Centric Self-Improvement
• Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
• Recursive Self Improvement for Coding Agents
• Rethinking the Evaluation of Harness Evolution for Agents
• RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
• Sidekick's continual learning loop
• The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

Summary#

The loop in which an agent optimizes the scaffold it runs inside. Give a coding agent (a) a benchmark it can run, (b) write access to the harness source, and (c) a goal expressed as a score, and it executes the whole eval-fix cycle unattended: run the baseline, read failure traces, form hypotheses, patch the harness, re-run, keep or discard, log what it tried. This is the autonomous form of the loop Google's flywheel deliberately keeps human-gated, and it sits one level above Agent Harness Engineering — the harness patterns are the object being edited rather than the thing a human authors.

The corpus holds three instances, and they do not agree:

• Cline's July 2026 campaign (Cline, case-study) — the loop running in the wild. One prompt to GPT-5.6-Sol as leader model, 17 hours of continuous autonomous work, ~1B tokens (400M agent + 600M eval reruns) and ~$680 total, producing a merged PR against Cline's own production repo. Weight it as a vendor build log, not a measurement — see Evidence weighting below.
• HarnessBank (Luo et al., arXiv 2607.13683, empirical) — the loop under laboratory control. Sealed per-domain test splits, two competing self-evolution methods on a matched rollout budget, a four-check statistical screen on every candidate, and an ablation of its own selection rule. Six of seven benchmarks post credited held-out gains of +9.2 to +15.4pp.
• Wang, Zhu, Hu et al. (Ai2 / UW, arXiv 2607.12227, empirical) — the loop against budget-matched baselines, which neither of the other two ran. Give parallel sampling, sequential refinement and harness evolution the same feedback and the same K = 5 inference budget on Terminal-Bench 2.1, and harness evolution loses to the simplest baseline and does not transfer to held-out tasks.

The third is the one that changes the reading of the other two, so it comes first below. Cline remains the only instance anyone has run against a real production harness.

Two further instances arrived in August: Ouroboros (case-study, unbriefed and unmeasured) and DarwinX (empirical), which reports the opposite of Wang et al. on Wang et al.'s own benchmark. The discriminator that fits all five is how broken the starting harness was — developed in The fifth instance below.

The control nobody ran: budget-matched baselines#

The methodological charge is one sentence, and it applies to every prior result on this page: harness evolution is itself a search procedure that repeatedly evaluates and revises candidates using task feedback, so it has to be compared against methods that spend the same budget searching over trajectories instead. Otherwise a reported gain cannot be attributed to better harness design rather than to more test-time compute — this is Compute-Controlled Benchmarking's benchmark-maxxing critique aimed at a named method class for the first time, with the experiment attached.

The second charge is narrower and just as damaging: prior protocols search and report on the same public benchmark, so gains "may reflect adaptation to task-specific patterns rather than improvements in harness design."

The setup#

Terminal-Bench 2.1 (89 tasks), Claude Opus 4.6 / GPT-5.4 / GPT-5.4 mini, high reasoning effort, 128k max generation, every result averaged over two independent runs. All four arms start from an identical minimal harness: one bash tool, no skills, no middleware, no persistent memory. Budget fixed at K = 5 for every method, with AHE (Lin et al.) sampling m = 1 rollout per task so the rollout counts match. The four arms differ only in what the budget buys:

Arm	Budget spent on	Object updated
Parallel sampling	K independent trajectories	nothing (fixed harness)
Sequential refinement	K conditioned revisions of one trajectory	nothing (fixed harness)
Harness evolution	K rounds of meta-agent harness edits across a task batch	a shared harness
Harness scaling	K rounds of harness edits for one evaluation instance	a per-task harness

Harness scaling is the paper's own construction and the sharpest instrument on the page: it is harness evolution stripped of its reusability claim — instance-guided rather than dataset-guided — and it exists precisely to ask whether the gains come from reusable design or from spending more compute at test time.

One honest scoping note: harness evolution is instantiated as AHE with its explore agent disabled, so that gains cannot come from retrieving benchmark-fitted harnesses off the shelf. That makes the comparison fair and also means this is a negative result on one method, not a replication failure of HarnessBank, which was never run.

Without unit tests, the simplest baseline wins#

pass@1, Terminal-Bench 2.1, self-judge selection for the parallel arm (Table 1; independently confirmed by Figure 1's averages):

Method	Claude Opus 4.6	GPT-5.4	GPT-5.4 mini	Average
direct sampling, initial harness	69.9	75.3	59.4	68.2
Parallel sampling	74.7	79.2	62.9	72.3
Sequential refinement	73.0	73.0	61.8	69.3
Harness evolution	71.4	69.7	61.3	67.4
Harness scaling	76.0	78.1	61.2	71.8

Harness evolution is the only arm that finishes below the do-nothing baseline (67.4 vs 68.2), and on GPT-5.4 it costs 5.6 points outright (75.3 → 69.7). Parallel sampling — draw five, let the model pick — gains on all three models. The paper's reading is that self-generated feedback is noisy and sequential revision compounds early mistakes, so harness revision needs an external correctness signal to be grounded at all.

With unit tests, harness evolution's gain is multi-attempt selection#

Give every arm oracle unit-test feedback and oracle selection, so pass@5 becomes meaningful (Table 2):

Method	Opus 4.6 pass@1	pass@5	GPT-5.4 pass@1	pass@5	Avg pass@1	Avg pass@5
direct sampling, initial harness	69.9	—	75.9	—	72.9	—
Parallel sampling	84.8	84.8	87.1	87.1	86.0	86.0
Sequential refinement	83.1	90.4	85.4	93.3	84.3	91.8
Harness evolution	73.0	83.2	78.6	89.3	75.8	86.2
Harness scaling	83.1	89.9	82.0	88.8	82.6	89.3

Everything improves over direct sampling — unit-test feedback helps whatever consumes it — but harness evolution is last on pass@1 by more than 10 points and last on pass@5 too. The inference the authors draw is the one worth keeping:

If harness revision genuinely produced better harnesses, we would expect the improvement to be reflected in pass@1. Instead, the benefit only materializes when we can select among multiple trajectories.

That is benchmark-maxxing restated as a diagnostic: a method whose gain lives in pass@k and not pass@1 is buying attempts, not capability. Harness evolution's pass@1 barely clears direct sampling (75.8 vs 72.9) while its pass@5 (86.2) sits in the same band as parallel sampling's flat 86.0.

The transfer test: +0.6pp#

Split Terminal-Bench 2.1 into 45 train / 10 validation / 34 held-out test tasks, evolve on train with unit tests, select on validation, report pass@1 on test (Table 3):

Method	Claude Opus 4.6	GPT-5.4	Average
direct sampling, initial harness	63.3	72.1	67.7
Harness evolution	64.5 (+1.2)	72.1 (+0.0)	68.3 (+0.6)

+0.6 points on average, and exactly zero on GPT-5.4. The other three arms are not applicable here — they scale at the instance level and produce no reusable artifact, which is itself the point: only harness evolution claims to output something transferable, and this is what the claim is worth on this benchmark.

Read that +0.6 against the paper's own noise floor. The same direct-sampling configuration on GPT-5.4 scores 75.3 in Table 1 and 75.9 in Table 2 — a 0.6-point spread between two identical baseline conditions. No error bars, standard deviations or significance tests appear anywhere in the paper, so the headline generalization gain is the same size as the only visible estimate of its own run-to-run variance. The conclusion survives (the direction of the argument is that the gain is absent, and an absent gain does not need a significance test), but nobody should quote +1.2 on Opus as a measured improvement.

What the meta agent actually did, and why it didn't stick#

The failure is not incompetence at the edit layer. §5.1 reports the meta agent making rational, well-motivated edits across three layers, escalating in a recognizable order: prompt-layer behavioral rules first (produce deliverables early, copy fragile state before mutating it, recheck constraints before finishing); then, when advisory text plateaus, runtime enforcement through middleware — turn-budget trackers, truncation of oversized tool outputs, finalization gates that block completion when deliverables are missing; then tool-layer fixes correcting misleading guidance and injecting recovery hints. That is the same "gains live in runtime and control flow, not prompts" verdict this page records from HarnessBank and Deterministic Pre-Execution Gates — arrived at again, and still not enough.

The diagnosis is the most portable sentence in the paper:

Most edits memorize fixes rather than distilling strategies. Much of this information is precisely what a competent agent can rediscover through exploration within a single rollout, so persisting it in the harness saves time on tasks the agent could already solve but rarely converts failures into successes.

Three consequences follow, and all three are visible in the tables. A stable core of hard failures — deep domain reasoning, constraints outside harness control — is untouched by accumulated knowledge. The growing volume of persistent prompt text introduces context bloat that offsets the remaining gains (Context Lifecycle Management). And the arm that does best among the harness methods is harness scaling, which does not even pretend to build a reusable artifact: it encodes this task's known bugs, file paths and command sequences into this task's next attempt. Appendix B's case studies are explicit about it — count-dataset-tokens is fixed by embedding the dataset's config, domain and column facts directly in the prompt; db-wal-recovery by prescribing the exact backup-before-open sequence for that database. Those are per-instance notes, not harness design.

The escape hatch the authors leave themselves, and it is a real one#

§5.2 does not claim harness evolution is worthless; it claims Terminal-Bench cannot see the difference, for two stated reasons: agents already score highly there, so the residual failures may be model limits rather than harness deficits; and "a minimal setup consisting of a shell tool and a basic prompt already suffices for most solvable tasks," so performance is bottlenecked by reasoning rather than scaffolding. The recommendation is a benchmark specification: test harness evolution only where (1) tasks leave substantial headroom and (2) performance depends heavily on the harness — specialized tools, skills, workflows.

Take that seriously and it is also the cleanest reconciliation with HarnessBank, below.

Three sources, one benchmark family, opposite verdicts#

The conflict is real and worth stating flatly before reconciling it. Same benchmark family (Terminal-Bench), and three different answers:

	Cline (case-study)	HarnessBank (empirical)	Wang et al. (empirical)
Benchmark	Terminal-Bench 2.1	Terminal-Bench 2 (among seven)	Terminal-Bench 2.1
Model	Kimi K3	Qwen3.6-27B	Opus 4.6 / GPT-5.4 / GPT-5.4 mini
Vanilla baseline	77.5%	36.1	68.2–72.9
Reported gain	+11.3pp	+9.3pp (held-out, z ≥ 1.96)	+0.6pp (held-out)
Held-out split	none	per-domain, sealed	45/10/34 within the suite
Budget-matched test-time-scaling arm	none	one domain only (16× tokens on LiveCode)	four arms, whole suite
Starting harness	mature production harness	each benchmark's vanilla harness	deliberately minimal (one bash tool)

(Now four answers. DarwinX — Salesforce, empirical, 2026-07-31 — reports base Monet 75.5% → 83.2% on the same Terminal-Bench 2.1 with a matched frozen GPT-5.5, a genuinely disjoint held-out arm, and no budget-matched baseline. It is the largest single change to this section since Wang et al. and gets its own treatment below; the tier reasoning and reconciliation in this section are updated for it in place.)

By evidence tier the ordering is not close. Two empirical sources with sealed splits outrank one vendor-claim-adjacent case-study (superseded 2026-08-13: three empirical sources now) in which the vendor ran the benchmark on its own harness and published its own scores. Between the two empirical sources, Wang et al. holds the control HarnessBank's own open question named as missing (a full-suite test-time-scaling arm) while HarnessBank holds the controls Wang et al. does not need (a significance gate, an archive ablation, seven domains rather than one). DarwinX, the third, holds the widest evaluation ladder (four regimes, an anti-cheating audit, 1,260 held-out real tasks) and not the one control that would settle its disagreement with Wang et al. — a budget-normalized arm. Vendor status does not separate them either: DarwinX is Salesforce evaluating Salesforce's own proprietary agent, which is Cline's structural position with an empirical protocol on top.

The reconciliation that fits all three is headroom and harness sensitivity — the authors' own §5.2, applied to the other two papers. (Sharpened 2026-08-13 by DarwinX: Evolving Agent Harnesses Through Natural Selection: score headroom is the wrong variable — Wang et al.'s baseline is 68.2, below base Monet's 75.5, and gets the smaller gain. What survives is the second half, harness sensitivity, restated as how broken the starting harness was — see the fifth-instance section below. The first two bullets that follow are unchanged and remain correct about each source; only the name of the discriminator moves.)

• HarnessBank's Terminal-Bench arm evolves a 36.1-point baseline on a 27B open-weight backbone. Wang et al. evolve frontier models already at 63–75. §5.2's first condition — substantial headroom — is satisfied in the first case and not the second. HarnessBank's own strongest gains sit on its weakest baselines (BrowseComp+ 16.9, AppWorld 41.3), which is the same pattern from inside its table.
• Cline's harness starts mature, and the five fixes its agent found were defects — a collapsed reasoning-effort setting, unretried 429s, a loop detector that couldn't read output, an async-worker crash, pkill self-termination. Repairing a broken production harness and improving an already-adequate minimal one are different tasks with different ceilings, and only the first was measured by Cline. Nothing in Wang et al. contradicts "Cline's harness had five bugs"; what it contradicts is the inference from that to "harness evolution is how you should spend your compute."
• Which leaves the sharpest unresolved piece: §5.2's second condition is untested by anyone. No source in the corpus has run harness evolution on a benchmark chosen for harness sensitivity (superseded 2026-08-13 by DarwinX: Evolving Agent Harnesses Through Natural Selection: WebArena-Infinity is that test, arrived at accidentally. Monet's browser harness was a coding agent pointed at Chrome — 43.5% audit-clean, 20.0% on one application, 23.5% of its successes invalid — so performance there depends on the harness about as heavily as it can. The evolved harness reaches 93.0% on 1,260 unseen real tasks. §5.2's second condition, satisfied, gives the largest gain in this page's corpus, which is the strongest support the condition has.), and Terminal-Bench may be the wrong instrument for the entire literature that reports on it.

What this does not license. It is not a refutation of HarnessBank — different method, different backbone, different benchmark version, and Wang et al. never ran HarnessBank's loop. It is a refutation of the protocol that every prior result was produced under, and its force is proportional: hardest on Cline (no held-out split, no baseline, self-scored), real but partial on HarnessBank (sealed splits satisfy the second charge; a single-domain 16×-token arm does not satisfy the first).

This is not "the harness doesn't matter"#

Worth stating explicitly, because the corpus's three other controlled harness studies all point the opposite way and none of them is threatened by this result. Orchestration Sets Token Economics holds the model fixed, swaps a hand-built orchestration layer, and moves cost −41% and tokens −38% uniformly across six models. Measuring Beyond Accuracy Saturation swaps the scaffold and moves accuracy ~44pp. Harness-Induced Belief Divergence swaps the evidence-mediation layer and moves the agent's beliefs. The harness is a large lever in all three.

The distinction is hand-built versus machine-evolved. Those three vary a harness a team designed, across dimensions a team chose. This varies a harness a meta agent derived from benchmark feedback, one round at a time, and finds the derivation does not pay for its own compute. Both can be true at once, and §5.1 says why: the meta agent's edits memorize this suite's fixes rather than distilling design principles, so what it produces is not the kind of object those three papers measure. A big lever pulled by a search process that cannot see where to pull it is still a bad use of budget.

There is a second-order note in the same direction. The middleware Wang et al.'s meta agent generates — output truncation, finalization gates, turn-budget interrupts — is drawn from exactly the classes Harness-Induced Belief Divergence measures as belief-shifting interfaces. An evolution loop therefore manufactures evidence-mediation changes as a side effect of chasing a score, and nobody has measured what that does to the agent's belief state.

The fourth instance: unbriefed, deployed, and unmeasured#

All three sources above share a protocol — a human writes a brief, a benchmark supplies the objective, the campaign terminates. Ouroboros/Hope (Razzhigaev et al., arXiv 2608.08311, case-study) breaks it: 161 continuous days of a deployed agent editing its own core, 1,085 self-modification commits at 94.2% agent-authored, where improvement is either scheduled as its own recurring task or triggered by ordinary work and user complaints, and every change passes a blocking multi-model diff-review gate (63.5% recent block rate). No brief, no score, no end.

Two things it settles for this page, and one it conspicuously doesn't.

The loop does run unattended for months without a human re-issuing a brief. That is the property Recursive Self-Improvement has been using as a separator and that HarnessBank's 10-round termination floor showed the gated benchmark loop lacks. So "it converges rather than compounding" is a property of the benchmark-objective protocol, not of self-modification as such — remove the objective and the loop keeps going.

And what it produces is still maintenance. The two evolution traces the paper follows end to end are a duplicate-send guard in the public output pipeline (surfaced by users noticing repeated messages) and a review-pack context-overflow fix replaced by an import-graph-centrality-ranked context atlas. Both are ordinary harness defects, and they are the same character as Cline's five fixes and the same character as Wang et al.'s §5.1 diagnosis: the loop repairs what is broken and does not distil design principles. Third independent arrival at the same verdict, from a protocol none of the others used.

What it does not supply is any measurement. The benchmark scores in the paper were produced on frozen seeds with self-evolution disabled (Appendix C says evolution off on four of five rows), there is no pre-evolution baseline harness anywhere, and the deployment's only time series plots spend, tokens, published LOC and memory artifacts. So the loop that runs forever is the one nobody scored, and the scores that exist came from a loop that was switched off. Read against the budget-matching charge above, that is not a rebuttal — it is the same critique with the measurement removed entirely.

One structural note worth carrying: the direction of the edit matches Cline's, not the shrinkage thesis. 175,755 published LOC and 1,085 commits, with the only net deletion in the final month. Agent-authored optimization grows the harness in every instance the corpus holds — and DarwinX below makes that structural rather than incidental: its edits are additive by construction ("because edits are additive, a branch accumulates capabilities rather than trading one for another"), so its selection rule cannot return a smaller harness even in principle.

The fifth instance: the same benchmark, the opposite verdict (DarwinX, July 2026)#

DarwinX (Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang et al., Salesforce AI Research / Agentforce, arXiv 2608.07545, 2026-07-31, empirical, 33pp) is the head-on contradiction: it reports large harness-evolution gains on Terminal-Bench 2.1, seventeen days after Wang et al. reported none there, and it supplies the held-out arm Wang et al. named as missing. Both are empirical; neither is averaged away below.

The method is the archive argument taken further than HarnessBank takes it. The base model is frozen; the harness spans a skill layer (prompts, memory, distilled knowledge) and a code layer (tools, control flow, agent loop). A child is admitted under a preserve-and-extend contract — net gain g(c) = Σ Δₜ > 0 and bounded regression R(c) = Σ(−Δₜ)⁺ ≤ δ, adjudicated by a verifier agent, then re-tested at higher avg@k with a preservation probe before it may steer search. Losing variants are retained as recombination stock, and complementary specialists are merged, the merged child kept only if it covers the union of its parents' solves. The design separates exploration (permissive admission on a noisy signal) from confirmation (strict avg@k before a variant is trusted) — which is HarnessBank's paired-2σ gate restated as a two-speed rule, plus the cross-lineage merge operator DGM and HarnessBank both lack.

Four regimes, ordered by how far the evolution signal sits from the test:

Regime	Base	Evolved	Gain	What controls it, and what doesn't
TB2.1 in-domain (89 tasks, avg@5, GPT-5.5)	75.5 ± 3.5	83.2 ± 1.2	+7.7	matched model; search set = report set; base row at default effort against the evolved row at high
TB2.1 leaderboard (GPT-5.6 Sol)	81.8 (OpenAI's own single-agent reference)	84.7 ± 1.2	+2.9	the paper's only tier-matched pair (same model, both medium) — and its smallest headline
TerminalWorld held-out (41 disjoint tasks, pass@1, Opus 4.8)	61.0	68.3	+7.3 (3 tasks)	genuinely held out, and McNemar p = 0.45 against its own base, p = 1.0 against Claude Code
WebArena-Infinity (1,260 unseen real tasks, GPT-5.5)	43.5	93.0	+49.5	task distribution and reward source both change; n = 1,260; deterministic verifiers
SWE-bench Verified (transfer, 500 issues, Opus 4.8)	—	84.2	+3.4 vs an LSP fix-skill reference	no matched-model base arm exists; every compared harness lands in 80.8–84.2

(Table 2's row set is the paper's, verified cell-for-cell against the PDF at ingest; the effort column is the paper's own and is what the next subsection is about.)

The budget control is a vendor label, not a matched budget#

DarwinX was the corpus's most promising candidate for the control budget-matching charge demands, because §4.1 is titled "the gain is the harness, not compute" and offers what it calls "an effort-controlled comparison." Full-text verification says it is not one:

• "medium / high / xhigh" is never defined. Not in the body, not in Appendix B's per-benchmark protocol table — which has no compute column at all. No token count, no turn count, no wall-clock bound is attached to any tier anywhere in 33 pages, and no dollar cost appears in the paper.
• So §4.1's control compares categorical tier names across different agents and vendors. Its argument — "a neutral harness at higher effort (Terminus-2, xhigh) reaches only 78.0%, below DarwinX's 83.2% at high effort, so raw effort in another harness does not reproduce the gain" — is a valid statement about tier escalation inside a different harness. It is not a normalized budget, because a tier name does not fix turns or tokens across harnesses.
• The headline matched-model pair is not itself effort-matched. Monet (base) is listed at GPT-5.5 / default (75.5%) against Monet (DarwinX) at GPT-5.5 / high (83.2%) — and that pair is the +7.7-point headline. The confound runs in DarwinX's favour on exactly the comparison the paper calls load-bearing, and against it on the Terminus-2 and Claude Code rows (both at xhigh against DarwinX's high/medium).
• §9 concedes it. "The archive, parent selector, recombination operator, and inference effort are not independently randomized."
• Its own compute figure shows the spend rising. Figure 5's medians on TB2.1: on the six newly-solved tasks the evolved harness runs 22 turns against 11 and 380K tokens against 89K; on the 69 both agents already solve, turns move 13 vs 12 — and tokens move 125K → 172K, a ~1.4× rise the prose does not mention while asserting that compute on solved tasks "barely moves" and that the harness "leaves solved tasks essentially untouched." (The 125K/172K pair is legible only in the figure; read from under the image two-pass rule.)

So DarwinX's answer to the budget charge is not "we spent the same" — it is "our extra spend is well-allocated." That is a different and weaker claim, and it is precisely what Wang et al.'s harness scaling arm was built to test: give one evaluation instance K rounds of harness edits and it is the best of the harness methods while producing nothing reusable. DarwinX does not defeat 2607.12227's critique; it instantiates it — it reports in-domain (charge 2: search and evaluation share one benchmark), with no budget-normalized arm (charge 1), on the benchmark whose §5.2 the earlier paper had already conceded may be the wrong instrument.

The held-out arm is weaker than the abstract#

TerminalWorld is the arm that should answer "does it generalize?", and by the paper's own reported statistics it does not answer it decisively:

• The gain over base is three tasks (25 → 28 of 41), McNemar p = 0.45. One solve is worth 2.4 points on a 41-task split, which the paper states.
• The margin over the strongest off-the-shelf agent is one task, McNemar p = 1.0, and the paper calls it "suggestive rather than statistically decisive."
• On the other base model the evolved harness loses to a neutral one. The same procedure on GPT-5.5 reaches 56.1% against Terminus-2's 61.0% — so Opus 4.8 is reported as the headline and the base where the harness underperforms a neutral baseline is reported as a "robustness caveat." That is HarnessBank's cross-model matching law showing through DarwinX's own numbers.
• Appendix C fixes the scope harder than the body does: "a separately skill-bundled pre-TW reference also reaches 28/41," i.e. a harness that never went through TerminalWorld search matches the headline exactly. The paper draws the right conclusion itself — the benchmark demonstrates that a diverse archive recovers a harness beating off-the-shelf agents, "not that TW-specific search lifts every possible starting harness."
• Figure 6 adds what the prose omits: of the four specialists the archive contributes, Specialist A (24/41 = 58.5%) is below the unevolved base (25/41) and Specialist B ties it. Half the population that gets merged into the winner does not beat doing nothing.

The one arm that genuinely contradicts Wang et al. is WebArena-Infinity, and it is a strong one: n = 1,260 real tasks never seen by the loop, evolved on 300 synthetic intents scored by an LLM judge and reported under deterministic verifiers, every one of ten applications improving. Raw-to-raw the move is 53.0 → 94.4 (+41.4); the headline +49.5 uses audit-clean scores, which penalise the base for 120 invalid successes it is not credited for — so ~8 of the 49.5 points are the base's compliance rather than the evolved harness's capability, which the paper states plainly ("capability and compliance improve together") and quantifies (confirmed-invalid 23.5% → 1.4%; invalid trajectories 293 → 17, the residual all raw-state mutations in one application).

Two structural notes on that arm cut against reading it as a vindication of the archive: every recombination was reverted on WAI (the gate keeps 26 iterations and reverts 36, and "every merge is reverted, so the gains accrue along a short accepted primary lineage"), and the in-loop proxy saturates while the held-out truth does not — TerminalWorld's training subset goes 0.505 → 1.000 against 68.3% held-out, a 31.7-point proxy/truth gap with "the variant that best fits the proxy… not the best generalizer." That is Optimizer–Evaluator Decoupling's problem stated as a measured quantity, and it is the sharpest thing in the paper.

What resolves the contradiction: how broken the starting harness was#

Reading DarwinX's four regimes against the other four instances, one discriminator predicts the sign and rough size of every result in this page — and it is not baseline headroom in the score, which fails on the pair that matters most (Wang et al.'s baseline is 68.2, below base Monet's 75.5, and gets +0.6 held-out where DarwinX gets +7.7 in-domain).

The predictor is whether the starting harness carried identifiable, harness-side defects.

Starting harness	Its defects	Reported gain
Monet's browser harness on WAI — a coding agent pointed at Chrome, 23.5% of its successes invalid, scoring 20.0% on one application and 95.8% on another	severe and specific	+49.5
HarnessBank's vanilla harnesses on a 27B backbone, one pathology dominating 49–88% of failures	severe and diagnosable	+9.2 to +15.4
Cline's production harness: collapsed reasoning effort, unretried 429s, a blind loop detector, an async-worker crash, pkill self-termination	five real bugs	+11.3
Ouroboros/Hope in deployment: a duplicate-send bug, a review-pack context overflow	ordinary defects	unmeasured
Monet's mature TB2.1 harness	none named	+7.7 (effort-confounded), +2.9 tier-matched
Wang et al.'s minimal seed: one bash tool, no skills, no middleware — spare but not wrong for a terminal benchmark	none	+0.6 held-out

WAI's own table is the within-source version of the same law: the evolved harness lands in a tight 84.0–98.3% band across all ten applications (mean 93.0) while the base ranges 20.0–95.8% (mean 43.5). The gain per application is close to ceiling-minus-base, and where the base was already strong (Elation clinical records, 95.8) the gain is +0.9. So the +49.5 headline measures the unevenness of the starting harness, not a uniform capability lift.

That is the fourth independent arrival at this page's standing verdict, from the source that looked most likely to overturn it: the loop repairs what is broken and does not distil design principles. Wang et al.'s §5.1 diagnosis ("edits memorize fixes rather than distilling strategies… saves time on tasks the agent could already solve but rarely converts failures into successes") and DarwinX's largest result are the same phenomenon with opposite framing — one measures a loop with nothing to repair, the other a loop handed a badly mis-fitted harness. Neither source needs to be discounted for both to be true.

What DarwinX does add that nothing else in the corpus had: the first cross-suite transfer measurement (a TB2.1-evolved harness run unchanged on SWE-bench Verified, and on a different base-model family), the first archive whose merge operator is specified as a union-coverage acceptance test, and the first harness-evolution paper to run an anti-cheating audit as a first-class result rather than a prompt clause — a two-stage static-plus-LLM detector at 99.0–99.4% coverage, with the evolved harness's evaluation-plane, privileged-host and exploit classes going to zero. On Reward Hacking that is the strongest available evidence that preservation-based selection does not reward verifier-gaming: 370 rewarded TB2.1 trajectories, two flagged, one false positive, and the one confirmed shortcut a per-trial policy lapse (reading an answer from a task's own published README) that the same harness solves legitimately in three of the other four samples.

The sixth instance, and the only one that reports what comes after the plateau (Shopify, August 2026)#

Shopify's Sidekick account (McNamara & Mazza-Anthony, Shopify Engineering, 2026-08-05, case-study — first-party, unreplicated, no controlled arm and no held-out split, so it sits at Cline's evidence tier and below the three empirical sources) adds an instance whose numbers are useless here and whose shape is the most informative thing in it.

The setup is this page's loop with the sharpest available statement of why prompt tuning is not enough: the target "is already a production application, with dynamically assembled prompts, custom control loops, and bespoke orchestration spread across a large codebase. No single prompt determines its behavior, so prompt tuning reaches only a small part of the system. The optimization target is the entire harness: its prompts, tool definitions, and orchestration code." The loop is explicitly a Karpathy-autoresearch instance — propose an edit, evaluate against the metric, "keeps the change if the score improves; and discards it otherwise" — configured in one program.md granting edit rights to prompts/, tools/ and harness/ with the note "everything here is editable, including the harness."

Three things it contributes:

It runs the decoupling properly, by architecture and by construction. The metric is a separately-calibrated LLM judge that the optimizing agent has no write access to — its directories are prompts/, tools/, harness/, and the judge is neither. That is a materially stronger arrangement than Cline's prompt-clause invariant and than DarwinX's in-domain search-equals-report set. The countervailing weakness is a different one: the judge is a learned instrument at 80% raw agreement against an 83% human ceiling, so the optimizer cannot edit its scorer but can still climb its noise, and nothing in the account re-validates it mid-loop.

It is the only source that reports the loop terminating, and what the team did next. Every instance above ends at a converged score, a significance floor, or a shipped PR. Here: "once harness improvements plateau, we begin optimizing in parameter space." No plateau value is published — no score, no round count, no spend — so this is an assertion, not a measurement. But it is a production team's stated ordering, and it points the same way as this page's standing verdict from the other side: if the loop repairs what is broken rather than distilling design principles, then a mature harness has little left to repair and the returns should run out. Shopify says they did, and then moved the object of improvement rather than the search budget — Knowledge-Centric Self-Improvement carries that taxonomy and Agent Quality Flywheel the loop it continues into.

It is a sixth data point for the how-broken-was-the-starting-harness law, with the sign right and the magnitude missing. Shopify's starting harness was a live production system already serving merchants at up to 2,000 requests per minute — the most mature seed in the corpus by some distance — and the reported harness-stage outcome is a plateau, i.e. the smallest possible gain, consistent with the table above. It cannot be added to that table: no held-out split, no baseline number, no gain figure of any kind for the autoresearch stage in isolation. Recorded as directional support, not as a row.

The seventh instance: a human PR gate, and a red test before the fix (Bridgewater, July 2026)#

Bridgewater Associates's PAT (How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes, case-study, first-party conference talk, no numbers of any kind) is the loop deployed inside a hedge fund, and it contributes two structural details that no other instance on this page has, plus the corpus's weakest evidence for any of them.

The fix is preceded by a failing benchmark the agent writes itself. On a Teach press — or autonomously, from background agents mining completed investor conversations — the loop first "create[s] a benchmark that we expect to fail. This shows that we can reproduce this poor behavior." Only then does it "iterate on our context repositories or the harness itself until that benchmark passes," and only then re-run the rest of the suite to check for regression. Every other instance on this page optimizes against a metric that already exists; this one manufactures the discriminating test before touching the target, which is the difference between a score that improved and a defect that was reproduced and closed. It is also the cheapest available guard against the failure mode this page keeps naming — an optimizer climbing its scorer's noise — because a red-then-green test on a specific reproduced behavior is a much narrower claim than a rising aggregate.

The human gate sits at PR review, not at benchmark authorship. The loop's output is a Slack message containing a pull request with the proposed changes to PAT's context and harness. Compare the placement elsewhere: HarnessBank gates at a sealed split, Cline gates at nothing published, Ouroboros gates at nothing at all. Gating at the diff is the strongest practical control in the set — a human sees exactly what changed before it ships to hundreds of users — and simultaneously the least evidential one, because approving a diff is not measuring an effect. Nothing in the talk reports the acceptance rate, the reject reasons, or a single before/after score.

As a seventh data point for the how-broken-was-the-starting-harness law, it is unusable. The seed harness is a mature production system with hundreds of daily users, which predicts small gains — but no gain is reported, in either direction, so the row cannot be filled. What it does corroborate is Shopify's structural point one level up: the optimization target is "our context repositories or the harness itself", i.e. the whole system rather than a prompt, in a codebase where no single prompt determines behavior.

Evidence weighting: this is the least-controlled instance on the page. A talk, by the builders, about their own product, with a corporate disclaimer attached stating no guarantee that discussed benefits will be achieved. It is admitted here for its mechanism — the red test and the PR gate are transferable design choices — and for nothing about its efficacy.

The decomposition this page needed: updating is not benefiting (the primary, read 2026-09-18)#

Seven instances above argue about whether the loop produces a better harness. Lin, Wu, Wang et al. (arXiv 2605.30621, 17 authors across Penn State, UC Santa Cruz, Emory, Northeastern, UIUC and Amazon, 2026-05-28, empirical) split that question in two, and the split explains something none of the seven could. The paper reached this page secondhand first, through the September 2026 RSI survey's §3.5.2 (arXiv 2609.11873); the primary is now in the corpus and this section is written from it.

The decomposition: harness-updating is the capability to produce useful persistent harness updates from execution evidence (a property of the evolver), and harness-benefit is the capability to gain from updated harnesses while solving (a property of the agent). Seven LLM backbones are crossed against three agentic benchmarks — SWE-bench Verified (500 tasks), MCP-Atlas (500 tasks, 36 real MCP servers, 220 tools) and SkillsBench (86 tasks, deterministic per-task verifier) — with prompt templates, initial harness, task stream, trajectory window, evolution budget and turn limit fixed across every agent-evolver pair, and scoring done in situ (each task is scored under the harness that existed before its own evidence was used, so no task grades a harness derived from itself).

The evolver side is flat, and the 9B model writes the same skill#

Fix the task-solving agent, vary who writes the updates, and almost nothing happens. The spread between the best and worst evolver is at most 3.1 percentage points on any benchmark, and no evolver wins on all three — Qwen3-235B leads SWE at 8.2 pp and comes last on MCP at 0.6. Scale is not predictive in either direction: the smallest model in the study, Qwen3.5-9B, posts the highest SkillsBench update gain at 3.8 pp, above Opus 4.6's 2.3 and Qwen3-235B's 1.5.

The mechanism is a case study on the SkillsBench flink-query task, agent backbone pinned to Opus 4.6. Without an evolved skill the agent omits the FINISH-event filter and scores 0.67; with a skill from either Qwen3.5-9B or Opus 4.6 it scores 1.0. Inspected side by side, the two skills are procedurally isomorphic — the same five steps (filter SUBMIT, filter FINISH, count each SUBMIT separately, emit (jobId, count), apply a 10-minute session window), differing only in implementation surface (manual batch sessionization against a KeyedProcessFunction) and in length, ~3,300 against ~3,800 characters.

And post-evolution score is dominated by the agent, not the evolver: within-agent spread across seven evolvers is at most 5.1 pp on MCP against a 36.0 pp gap between the Opus and Qwen3-235B base capabilities (5.0 vs 53.5 on SWE; 9.3 vs 20.9 on SB). Even pairing the weakest anchor agent with its best evolver against the strongest anchor agent with its worst, the strong agent still leads by 18.6 to 35.2 pp on every benchmark. On SkillsBench the same model, Qwen3.5-9B, sits on both sides of that comparison — best evolver for Qwen3-235B, worst for Opus.

The agent side is non-monotonic, and the peak is the middle#

Agent	SWE base	SWE ∆	MCP base	MCP ∆	SB base	SB ∆
Qwen3-32B	3.6	4.4	3.6	1.0	0.0	5.8
Qwen3-235B	20.7	19.3	25.0	4.3	4.7	1.1
GPT-OSS-120B	26.2	15.8	28.0	7.0	0.0	7.0
Haiku 4.5	66.0	2.4	42.4	3.6	5.8	15.1
Sonnet 4.6	73.2	2.8	54.0	3.2	24.4	3.5
Opus 4.6	74.2	2.6	61.0	3.6	25.6	5.8

(Base pass rate %, ∆benefit in pp. Table 1, p.6 — verified cell-for-cell against pdftotext -layout at ingest, and independently reproduced by the appendix's full agent×evolver matrix.)

An inverted U. The peak is mid-tier — Qwen3-235B +19.3 on SWE, GPT-OSS-120B +7.0 on MCP — and both ends are low for different reasons. At the top it is a ceiling: strong models already solve the tasks. At the bottom it is not, because the weak tier has the most headroom and gains the least. That bottom-end gap is what Harness Activation and Adherence measures, and it decomposes into the artifact never entering context (SkillsBench skill-load rate 0.251 for Qwen3-32B against ≈0.96 for the strong tier) and entering context without being followed (harness-following rate 0.142 against 0.757). Qwen3-235B is the clean separator: it loads as reliably as Opus 4.6 (0.961 vs 0.957) and follows less than half as often (0.350 vs 0.757).

The paper's own instruction to anyone building one of these loops: allocate capability budget to the task-solving agent, not the evolver.

What this does to the page's standing argument#

The corpus's discriminator has been how broken the starting harness was — a property of the seed. This adds a property of the consumer: a good patch pays nothing if the solving agent never retrieves it or retrieves it and ignores it. Three results above are re-readable under it. Wang et al.'s §5.1 diagnosis that edits "memorize fixes rather than distilling strategies … saves time on tasks the agent could already solve" is the activation failure seen from the artifact side — a memorized fix activates only on the task it memorizes. HarnessBank's activation check (every patch declares an activation spec and emits a deterministic beacon; a harness that never fires is rejected however good its score looks) is the only gate in the corpus that tests the first mode directly, and it is now backed by an independent study finding that mode common. And DarwinX's 31.7-point proxy/truth gap — the variant that best fits the in-loop proxy is not the best generalizer — is what the second mode looks like when nobody separates the two.

Nobody else in this corpus measures activation and faithful use as first-class quantities. HarnessBank gates on activation but does not report a rate; every other instance reports only the end-to-end score, which by construction cannot distinguish "the update was bad" from "the update was fine and never fired."

But the primary read also narrows the hypothesis, and this is the part the survey's summary did not carry. Every negative result on this page is on a frontier model — Wang et al.'s three frontier backbones on Terminal-Bench 2.1, DarwinX's Monet, Cline's Kimi K3. At that tier Lin et al. measure activation at ≈0.96 and adherence at 0.73–0.76. So non-activation does not explain this page's negative results; it explains a failure mode at a tier this page has never tested. What the two sources agree on instead is more interesting: Lin et al.'s frontier-tier gains are +2.4 to +3.6 pp on SWE and MCP against a no-evolution baseline, which is the same near-nothing Wang et al. find at matched budget. The action is entirely in the mid tier — Qwen3-235B and GPT-OSS-120B, +15.8 to +19.3 pp on SWE — which Wang et al. never tested. The live possibility, and it is new to this page, is that the budget-matched negative result is a frontier-model result rather than a Terminal-Bench result.

Two things this source explicitly does not supply, against the temptation to read it as vindication of the method:

• No budget-matched baseline anywhere. Every ∆ here is measured against a NONE arm — the same agent with no evolution — never against spending the evolution compute on parallel sampling or extra attempts. The Large-Scale Test-Time Compute charge that Wang et al. land is untouched.
• No compute accounting at all. The protocol says every pair shares "the same evolution budget β and per-task turn limit" and then publishes no value for β, no turn limit, no rollout count, no token count and no dollar figure in the paper or its appendices. That makes this the third source in this corpus (with DarwinX and Wang et al.) to fix a search budget without reporting it in a unit anyone else can compare against — the premise of the budget open question below rather than an answer to it.

Checking the survey's secondhand account against the primary#

This is the first of the ~15 findings the September 2026 RSI survey put into this wiki on its own say-so to have its primary read. The characterization held. All five of the survey's claims — the two-capability split, the fixed protocol with identical prompts and budgets, updating being flat in base capability with a small model matching a frontier one, benefit being non-monotonic, and the two failure modes — are accurate to the source. Two scope corrections, both narrowing:

• The survey's "the failures concentrate in two modes" is stated unconditionally. The paper traces only the weak-tier gap to activation and adherence; the strong-tier shortfall is a ceiling effect and is not a failure mode at all.
• "Identical prompts and budgets" is true and unquantified — see the missing compute accounting above. A reader of the survey would reasonably have assumed the budget was published.

One thing the survey flattened rather than got wrong: the non-monotonic pattern is clean on SWE and MCP and noisy on SkillsBench, where the paper concedes the low-base regime is unstable — Haiku 4.5 takes the largest SB gain (+15.1) while Qwen3-235B gains 1.1 from a comparably low base. The paper says so in its own §D.2; the survey's one-line summary did not.

That is a good result for the survey's reliability as a reading list, and it does not generalize: this was its most heavily-summarized entry, and the other ~14 remain unchecked.

What the survey adds to the census, and what it is missing#

Its L2/L4 harness roll-call names four systems this page does not hold: Self-Harness (execution traces → small harness edits, retained only after regression testing — the survey's L2 exemplar), Agentic Harness Engineering (harness components as separately editable and revertible files, with each proposed change checked against its predicted effect, and a separate evolver role that makes the loop less self-referential than Self-Harness), HarnessDev (a creator agent builds a harness from a minimal seed, candidates are frozen and scored on hidden tasks for both success rate and executor-token cost, with creator and executor kept separate so harnesses compare under one fixed executor), and Evo-Harness (failed or negatively-evaluated tasks compiled into cross-task patterns and task-specific procedures). HarnessDev's protocol is the closest thing in the survey to the sealed-split-plus-cost discipline this page rates highest, and the cost-as-a-scored-dimension choice is one nothing here does.

The survey does not cite HarnessBank, DarwinX or Wang et al. So the three results that define this page's disagreement — the two strongest sealed-split gains and the only full-suite budget-matched negative — are absent from a 491-paper census published six weeks after the last of them. That is a reason to read its L2 verdict as a reading list rather than a census, and it is the second time in this section that the same gap appears: the survey's positive framing of harness self-evolution rests on the sub-literature that reports in-domain.

The validity warning, stated better than anywhere else on this page#

§3.3.4 is worth quoting because it is the cleanest available statement of the threat Compute-Controlled Benchmarking and Evaluation Awareness & Grader Gaming circle:

Repeated development-set access invites benchmark overfitting, additional search compute can be mistaken for algorithmic improvement, and an LLM evaluator may share the proposer's blind spots. Public automated-research systems make these risks concrete: reported behaviors include random-seed cherry-picking, shortcut discovery, and attempted test-label extraction through repeated evaluator queries. Once a benchmark is queried adaptively, it effectively becomes part of the optimization surface rather than a passive measurement instrument.

The cherry-picking and label-extraction observations are Anthropic's own, from its automated weak-to-strong researcher. The last sentence is the survey's, and it is the general form of the charge levelled at DarwinX above — searching and reporting on one benchmark is not a protocol flaw so much as a category change in what the benchmark is.

The eighth instance: harness evolution transfers upward, but teaching the weaker model from it breaks the fit it earned (Salesforce, September 2026)#

Every transfer result above asks whether a harness stays useful when handed unchanged to a different model. Yu, Bi, Pentyala et al. (arXiv 2609.09134, Salesforce AI Research, 2026-09-08, empirical) ask the next question — what happens when the receiving model is then updated toward the harness's best user — and the answer is a clean positive on the first half and a sharp negative on the second, on a suite this page has not tested before: seven enterprise agentic tasks (payroll auditing, budget approval, stock alerting, IoT anomaly detection, browser automation, website management, code refactoring) from Yang et al. [2026], evolved with a GEPA-style search (gemini-3.1-pro-preview as meta-agent proposer, edits retained only on validation improvement), with LoRA-SFT (rank 16/64, H100/H200) as the model-adaptation lever.

Upward transfer, confirmed and clean. Evolving the harness around the weak model (qwen3-coder-30b-a3b, 30B/3B active) lifts it from 29.2% to 78.0% mean test success (+48.8). Handed that same harness unchanged, the stronger gemini-3.1-pro-preview expert does not merely tolerate it — it improves on its own baseline-harness score, 84.4% → 93.6% (+9.2), triggering the evolved edits in 93.6–100% of rollouts including a domain-computation recipe the weak model itself uses only 30.8% of the time. This is real upward transfer across a capability tier, on a harness whose credited edits are task-domain knowledge (API conventions, arithmetic recipes, tool-use examples) rather than a reasoning-budget lever — exactly the class What Makes a Self-Improvement Artifact Transfer? predicts survives solver churn, and a cleaner positive than HarnessBank's cross-model matching law ever produced, because nothing here is fitted to a specific failure pathology.

Pulling the gain back down by imitation collapses it. The natural next move — LoRA-SFT the weak model on the expert's successful trajectories under the evolved harness — regresses mean success from 78.0% to 63.1% (-14.9, every one of seven tasks, -4.2 to -29.9), replicated on a second model family (gemma-4-26b-a4b-it on the website-management task: evolved-harness 55.6% → SFT-imitation 41.1%, below even its own default-harness baseline of 46.7%). The identical recipe helps under the unevolved baseline harness (29.2% → 35.5%, +6.3, Table 3), which isolates the failure to the interaction between imitation and harness evolution, not to imitation itself. The mechanism is not lost knowledge — post-SFT domain-computation-recipe usage rises (30.8% → 76.1%) and the implicit-knowledge share of failures falls (46.2% → 44.5%) — it is a planning-defect share that jumps from 1.1% to 14.6% of failures (Table 3, clean parse, reconciled against prose). The fine-tuned model adopts the expert's planning style without the competence to execute it, and no longer matches a harness whose single verify-then-finish cadence was fit to the weak model's own step-by-step rhythm: in one traced payroll-audit rollout the post-SFT model computes the right answer but re-checks it 29 times across 78 steps and never emits a finish, scoring 0 where the base model closes cleanly in 14–20.

On-policy correction is the fix, and it composes rather than fights. Instead of the expert's whole trajectories, a self-directed meta-level MLE agent localizes the single failing turn in each of the weak model's own rollouts and has the expert rewrite only that turn (~500 rows: ~400 corrections plus ~50 self-pass examples), leaving the model's native planning distribution everywhere else untouched. LoRA-SFT on this set raises mean success 78.0% → 79.7% (+1.7), gaining on five of seven tasks and staying within noise on the two the harness has likely already saturated — while the planning-failure share stays near the base-model floor (1.1% → 1.8%) and the knowledge share keeps falling (46.2% → 43.2%). Same knowledge gain as imitation, without the planning collapse. It does not close the full gap to the expert and the paper concedes that is likely a lightweight-LoRA ceiling, but training completes in under an hour, which is the paper's argument for stacking this into an iterative loop: evolve the harness around the current model, update the model on-policy under that harness, re-enter evolution.

What this adds to the page's standing argument. The corpus's two discriminators so far are how broken the starting harness was (a property of the seed) and activation/adherence (a property of the consumer, per Lin et al. above). This source adds a third, orthogonal one: a harness can transfer cleanly upward, and the receiving model can activate it more after a weight update — and the outcome can still collapse, because what breaks is neither the artifact nor whether it fires, but whether the specific weight update leaves the solver's turn-by-turn cadence still matched to the harness's finalization contract. Lin et al.'s activation/adherence framework would score this case as high-activation; the failure is invisible to that lens and only shows up in the failure-mode composition. It is also the first empirical (not case-study) source to combine the harness lever and the weights lever from Knowledge-Centric Self-Improvement's persistence taxonomy on one task set and show they conflict under a naive combination and compose under a corrected one — see that page for the fuller placement.

The ninth instance: regularizing the search itself makes the harness transfer (RRSI, September 2026)#

Every instance above diagnoses the evolve-to-transfer gap; RRSI (Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister & Chen-Yu Lee — Google Cloud AI Research / UNC-Chapel Hill / Stanford / Washington University in St. Louis, arXiv 2609.24972, 2026-09-21, empirical) is the first to treat the gap as a regularization problem and hold a controlled comparison against four competing evolution methods while doing it. It names the same three coupled failure modes this page has been circling under other labels — benchmark-specific fitting, noise chasing, complexity accumulation — and translates classical ML regularization into constraints on the search itself rather than on the harness's reachable space, which it deliberately leaves open (prompts, control flow, tools, memory, subagents all remain editable).

The mechanism, mapped to the classical analogy the paper draws explicitly. Proposal side: an L0-style annealed edit budget caps how many independently attributable edits one candidate may bundle, on a cosine schedule from b_max (early rounds, coordinated discovery) down to b_min = 1 (late rounds, attributable single edits); evidence-aware credit assignment records every evaluated candidate's component, hypothesis, diff and outcome so the proposer stops re-testing hypotheses earlier rounds already falsified; structured exploration reserves a slice of budget for untouched components during a detected stall. Selection side: a leakage critic rejects candidates encoding task names or benchmark-specific values before they ever reach evaluation (so a leaking candidate never earns the inflated score that would make it attractive later); a noise-adjusted stability floor Ŝ(H′) ≥ S* − δ blocks a sequence of individually-small regressions from compounding; a Ridge/L2-style cost rule ΔC ≤ β0 + β1·ΔS gates additional inference cost behind a proportional score gain once the gain clears the noise band; and a Lasso/L1-style pruning rule flags components with no strictly positive measured gain over a fixed window as deletion targets. None of these optimize an actual Lp-penalized objective — the paper is explicit that the correspondence is structural, not mathematical — but the shape is the same one HarnessBank's significance gate and Guo et al.'s exogenous-evaluator argument both land on from different angles: the search may hypothesize freely; credit comes only from a gate it cannot talk its way past.

The controlled comparison this page has been asking for. Eight benchmarks across three domains — coding (Terminal-Bench 2.1 evolve, SWE-bench Verified OOD), agentic workspace (Harvey LAB evolve + in-distribution held-out, JobBench/GDPval/APEX-Agents OOD) and engineering design (EngDesign evolve, Frontier-Eng OOD, both deterministically graded by frozen simulators rather than a judge) — with a frozen Claude Opus 4.8 policy throughout and four prior harness-evolution methods (Meta-Harness, AHE, TTHE, HarnessX) run from the same H0, same frozen policy, same evolve set, same candidate budget:

Method	Harvey LAB (Evolve)	Harvey LAB (ID Held-out)	JobBench	GDPval	APEX-Agents	OOD Avg.
H0 (no evolution)	89.4	86.9	36.0	48.8	34.2	39.7
Meta-Harness	93.0	89.2	37.1	49.1	35.7	40.6
AHE	90.7	88.7	37.2	47.2	33.1	39.2
TTHE	91.1	88.5	35.2	47.0	31.7	38.0
HarnessX	91.8	89.1	36.3	48.5	34.3	39.7
RRSI	90.5	89.2	40.7	52.3	37.9	43.6

Meta-Harness is the strongest baseline on the evolve split and adds only 0.9 OOD points; HarnessX lands exactly on H0's OOD average; AHE and TTHE finish below H0, TTHE by 1.7. RRSI posts the smallest evolve-set gain of any evolved harness in the table and the only OOD average that clears H0 by more than a point — 43.6 against 39.7, a trade every regularizer in the design is built to make. Across all eight benchmarks (Figure 3, headline numbers from prose): evolve-set gains of +6.0 (Terminal-Bench 2.1), +4.9 (EngDesign) and +1.1 (Harvey LAB); OOD gains of +1.8 on SWE-bench Verified (a benchmark never scored during search), +2.3 on Harvey LAB's in-distribution held-out split, +3.5 to +4.7 (7.2–13.1% relative) across JobBench/GDPval/APEX-Agents, and +4.3 Medal points (24.3% relative) on Frontier-Eng. No held-out split regresses anywhere — the specific failure a memorizing harness produces.

The ablation isolates which half of the regularizer is doing the work, and shows both halves are load-bearing (Table 2, agentic-workspace instance):

Variant	Harvey LAB (Evolve)	ID Held-out	OOD Avg.	Tokens/trial (M)
H0	89.4	86.9	39.7	1.56
Unregularized evolution	92.8	88.9	40.3	3.80
w/o proposal regularizers	90.7	88.8	41.9	2.69
w/o acceptance regularizers	91.5	88.7	41.0	3.59
RRSI	90.5	89.2	43.6	2.42

Dropping the acceptance-side constraints (the leakage critic, stability floor, cost rule) raises the evolve score 90.5→91.5 while OOD falls 43.6→41.0 and cost rises by half — an unconstrained selector spends most of what it accepts on noise and context rather than mechanism. Dropping the proposal-side constraints costs only 0.2 evolve points but 1.7 OOD points, showing that where the search looks matters even when nothing downstream is rejected. Removing both gives the highest evolve-set score of any arm (92.8) and an OOD average within a point of doing nothing at all (40.3), at 3.80M tokens/trial against RRSI's 2.42M — the unregularized loop is simultaneously the best-scoring on the split it is judged on and functionally the same as not evolving at all on tasks it wasn't.

Cross-model transfer, and this is the sharpest new fact for this page's standing matching law. RRSI runs independently with two policy families on the coding domain (Table 3) — Claude Opus 4.8 (74.2→80.2 evolve, +6.0; 82.0→83.8 SWE-bench OOD, +1.8) and Gemini 3.5 Flash (64.6→78.7 evolve, +14.1; 76.8→79.0 SWE-bench OOD, +2.2) — establishing the method isn't tied to one backbone. The stronger result (Table 4) hands the harness evolved with Gemini 3.5 Flash to Gemini 3.1 Flash Lite, a smaller model that never took part in the search: Terminal-Bench 2.1 accuracy rises 11.2→14.6, a 30.4% relative gain against a base score under a fifth of the search policy's. That is a direct counter-instance to HarnessBank's cross-model matching law recorded above (a reasoning-budget-lever patch near-zero or actively harmful off its matched pathology) and to DarwinX's own cross-model instability (the same procedure losing to a neutral baseline on a different base model) — here the same regularized procedure, with no re-search, helps a policy strictly weaker than and never exposed to the one that searched. One caveat the paper doesn't resolve: the absolute gain (+3.4) is smaller than the search-policy gain (+14.1), consistent with "the mechanisms don't depend on the capability level they were searched at" but also consistent with a weaker backbone simply having less headroom for any fix to reach — RRSI doesn't run the reverse direction (evolve on the weak model, test on the strong one) that would separate those.

Cost: RRSI produces the lightest harness of any evolved arm. 26.3 steps/trial against 27.3–34.6 for the four baselines; AHE is the extreme case at 3.82M tokens/trial (58% more than RRSI's 2.42M) for 4.4 fewer OOD points. No evolved harness is as cheap as H0 (1.56M tokens, 21.2 steps) — evolution still buys part of its gain with test-time compute, and the regularizer decides how much.

The first harness-evolution paper in this corpus to publish a complete, comparable search-budget table. Table 5 (Appendix D.1) gives every hyperparameter per domain — rounds T (20/20/40), trials-per-task k (2/2/4), noise tolerance δ (0.017/0.004/0.020, calibrated from repeated H0 evaluation), edit-budget range b_min–b_max (1–4/1–3/1–4), stall window w (3/3/3), pruning window n_prune (4/4/5), and the cost-rule coefficients β0, β1. This doesn't settle the standing search-budget disagreement between Wang et al. and DarwinX below — different systems, not a shared unit — but it removes the excuse that a harness-evolution search budget can't be reported in units another lab could reproduce, and the baselines here are explicitly run on "the same candidate budget" as RRSI, which none of Wang et al., DarwinX or Lin et al. above manage for their comparisons.

Where the corpus's discriminator gains a fourth axis. The page has accumulated three properties that predict whether a harness-evolution result transfers: how broken the starting harness was (a property of the seed), activation and adherence (a property of the consumer, per Lin et al.), and whether a weight update preserves the solver's fit to the harness (per the Salesforce instance). RRSI adds a fourth, and it is the first to isolate it cleanly by ablation rather than by cross-source comparison: how tightly the search procedure itself is regularized, holding the seed harness, the benchmark suite and the policy family fixed. The ablation table above is the direct evidence — same H0, same tasks, same backbone, and OOD performance moves from 40.3 to 43.6 purely as a function of which constraints the selector enforces.

Evidence weighting. Single-lab preprint (Google Cloud AI Research, with UNC/Stanford/WashU co-authors), not yet third-party replicated. The proposer, critic and analyst are all Claude Opus 4.8 — a competitor's model, not Google's own — which is a mild point in favor of the comparison not being tilted toward a house model. Benchmarks are third-party throughout (Harvey AI, Merrill et al.'s Terminal-Bench, Jimenez et al.'s SWE-bench, Guo et al.'s EngDesign, Chi et al.'s Frontier-Eng), and three of eight (EngDesign, Frontier-Eng, and half of SWE-bench's fail-to-pass/pass-to-pass gate) are deterministically graded rather than judge-scored, closing the same judge-gaming route DarwinX's engineering arm closes. Against that: no significance testing, error bars or confidence intervals appear anywhere on the final reported numbers — the internal noise band δ gates which candidates survive during search but is never applied to the headline evolve/held-out comparisons the tables above report, so RRSI is in the same position as Wang et al. and DarwinX on this specific limitation, not in HarnessBank's position (paired z ≥ 1.96 gate on the reported numbers themselves). No dollar cost is reported anywhere, only tokens and steps. The paper's own limitations section concedes the frozen-backbone scope (no setting where weights update during evolution — the exact combination the Salesforce instance tests) and that broader validation is needed across "substantially different agent architectures, tool ecosystems, and longer-running self-improvement processes." Table 6's qualitative case-study examples (illustrative accept/reject rationales — a cost-rule rejection at round 0, a floor rejection at round 8, an engineering control-flow fix accepted at low resource growth) carry a soft table-collapse parse flag from ingest; the pattern they illustrate is corroborated by the surrounding prose narrating the same rounds, but individual cell values from that table are not treated as load-bearing here.

The Cline campaign, in numbers#

Terminal-Bench 2.1, Kimi K3 via OpenRouter, stock Cline harness as baseline:

Run	Score	Cost
Baseline (stock harness)	69/89 — 77.5%	$79
Combined candidate	77/89 — 86.5%	$65
Confirmation	79/89 — 88.8%	$49.8

Cost fell as score rose, which is the mechanistically interesting part: most of the gain came from not wasting tokens on doomed retries, self-kills, and false loop-detector aborts. Cline's comparison points — Claude Fable 5 at $552 and GPT-5.6 Terra at $400 on the same benchmark — are not harness-controlled (see Compute-Controlled Benchmarking), so the price ratio compares one tuned harness against whatever configuration those numbers came from.

The five experiments the agent ran, in order:

1. Reasoning-effort abstraction — the harness silently collapsed K3's max reasoning effort to high. Correctness fix, no score credit (OpenRouter was already mapping to the only supported effort), but it unblocked everything downstream.
2. Rate-limit retries — five baseline failures were single OpenRouter 429s killing healthy sessions. Raised retry count with exponential backoff; all five flipped.
3. Output-aware loop detection — the loop detector killed agents legitimately polling long-running background work (same command, changing output). Made the detector read output, not just the command; two tasks flipped.
4. Async-worker liveness — one task exited in 7.6s with zero tokens: any prompt containing @a-style tokens triggered a file-mention lookup on an unreferenced async worker, and the process exited before the model was called. One-line fix, deterministic flip.
5. PID tracking to stop self-termination — two tasks ran pkill -f with a pattern matching the harness's own command line and killed themselves mid-task. Tool guidance to track PIDs; both flipped. Cline reports this fix reduced but did not eliminate broad-match kills.

Every fix is a general harness bug, not a benchmark-shaped special case — which is Cline's argument that no Reward Hacking occurred, and it is the strongest thing in the post.

The controlled version: HarnessBank (Luo et al., 2026)#

Everything the Cline campaign lacks — a sealed test set, competing methods on a matched budget, an ablation of its own selection rule — is what HarnessBank supplies (HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution, EverMind AI / Shanda Group, empirical). It is the corpus's first external, controlled evaluation of this loop.

The architecture separates four things Cline had fused:

• a task agent — a frozen backbone (default Qwen3.6-27B) executing benchmark tasks under the current harness;
• an evolver agent — a different model from a different vendor (Claude Opus 4.8) that reads execution trajectories, names the recurring failure, and writes offspring harnesses;
• a deterministic evaluator owning sampling, scoring, activation logging and the statistical tests;
• a Harness Gene Bank, a quality-diversity archive derived from MAP-Elites.

The harness is partitioned into an immutable kernel (evaluation, bookkeeping, self-evolution and interface-critical code) and a mutable surface the evolver may edit: prompts, injected knowledge, runtime control logic, configuration. Each domain has disjoint train/test splits and the test set is touched exactly once, after evolution ends.

The bank is indexed semantically, not by task. Each cell is a pair — where the patch acts (prompt, knowledge, runtime, config) × why it exists (a pathology inferred from failure traces, e.g. thinking-runaway). Harnesses hypothesizing the same pathology compete inside one cell and only the best survives; harnesses addressing different pathologies stay alive to be recombined later. Parent selection is quality-biased (argmax utility over the vanilla harness plus the bank), so the strongest lineage is exploited while the archive keeps structurally different solutions from being discarded — the paper's answer to search collapse, the tendency of greedy evolve-and-keep loops to converge on a narrow class of safe prompt edits. The keying choice is argued explicitly: an archive keyed on tasks "would preserve harnesses indexed by the very tasks used to select them, which overfits by construction."

Gated Harness Screening is this page's missing control, built in. Rather than running every offspring on the full training set, candidates face four sequential checks on a sampled subset:

1. Validity — sandbox crashes and verifier timeouts trigger repair-and-retry instead of being scored as agent failures.
2. Activation — every patch declares an activation spec and emits a deterministic beacon; a harness that never fires is "inert" and rejected however good its score looks.
3. Paired significance — task-level paired differences against the parent on identical tasks, requiring z ≥ 1.96 (two-sided 5%).
4. Gain — the estimated improvement must be positive.

(The introduction calls these four gates; §3.3 implements three multiplied indicators, folding the gain check into the significance gate as its Δ̂ > 0 factor. Same test, different bookkeeping.)

Only survivors get full-training-set evaluation and compete for a bank cell. The design principle worth stealing: the LLM may hypothesize freely, but credit comes only from the gate. The why label is explicitly "an LLM-assigned hypothesis, not ground truth" — on AppWorld the loop misdiagnosed a capability limit as a knowledge gap, and the gate rejected the resulting patch (0/24 → 0/24 on its own target tasks, p = 1.0). Because a wrong label only steers which candidates get tried, it costs a rejected candidate, never a bad harness.

Results, and what they cost the competition#

Seven domains, Pass@1 averaged over K = 3 attempts, measured on tasks the evolving agent never saw:

Benchmark	Test Pass@1 vanilla → evolved	Gain	Train→test retention
AppWorld	41.3 → 56.7	+15.4	86%
BrowseComp+	16.9 → 30.8	+13.9	86%
LiveCode	58.1 → 71.8	+13.7	65%
Omni-MATH	54.3 → 66.0	+11.7	92%
TB2	36.1 → 45.4	+9.3	148%
GDPval	43.7 → 52.9	+9.2	110%
SWE-bench	47.4 → 52.6	+5.1 (uncredited: n=26, z=0.78)	37%

Six of seven clear z ≥ 1.96 on held-out tasks (p from <10⁻⁴ to 0.033; AppWorld strongest at z = 6.44, n = 168), each a single held-out comparison, so the many candidate comparisons during evolution cannot inflate it. Pass@3 rises on every credited domain too (+5.5 to +15.4), so the harness expands the set of solvable tasks rather than only the per-attempt hit rate. The authors decline to treat the retention ratio as a test — it divides two noisy estimates — and rest the claim on the credited held-out gain.

Two baselines ran under the same protocol: same frozen backbone, same splits, same paired-2σ ruler, budgets within 2.1× on every domain (780–2,310 rollouts, with GEPA outspending HarnessBank on two domains and DGM on one). Across five sealed tests, HarnessBank is credited on four, DGM on one, GEPA on none:

Test Pass@1	Vanilla	HarnessBank	GEPA (prompt-only)	DGM (ungated)
AppWorld	41.3	56.7	44.0	52.0
LiveCode	58.1	71.8	58.1	59.8
Omni-MATH	54.3	66.0	54.3	53.2
BrowseComp+	16.9	30.8	19.0	21.2
SWE-bench	47.4	52.6	47.4	56.4

Both baselines fail in ways this wiki has been circling:

• GEPA, prompt-only, ships vanilla in three of five cells. It finds no variant beating its seed on LiveCode in 47 iterations, because "thinking-runaway is not prompt-addressable"; on AppWorld, where the win surface is partly prompt-expressible, its training gain washes out on the sealed test (+2.8, z = 0.97). The gains in this problem class do not live in the prompt — a direct rebuke to the assumption that harness optimization is mostly prompt optimization.
• DGM, openly self-modifying but ungated, deploys a regression. It clears the bar once (AppWorld +10.7, z = 3.00). On LiveCode it selects its best of 15 generations from a 15-task K=1 spike (0.733, regressing to 0.533 on re-evaluation) and lands uncredited (z = 0.66); on Omni-MATH the harness it ships is worse than vanilla (-1.1%). This is what "keep what appears beneficial" buys without a significance gate.

The paper reports against itself where it should: on SWE-bench, DGM's raw 56.4 is the highest number in the table, and at n = 26 the gate credits neither method. Note also that HarnessBank's authors re-ran both baselines under their own protocol — standard practice, and the standard caveat.

The ablation: what the gate actually buys#

Ablating the paired-2σ gate on TB2 separates three things it might be doing (rows below the first are deltas against it):

TB2 configuration	Test Pass@1	False elites	Rounds to stop
HarnessBank (K=3, 2σ)	45.4	0	10.0
w/o 2σ	±0.0	+2	>20 (cap)
w/o confirm + 2σ	-1.6	+3	>20 (cap)
Vanilla	-9.3	—	—

Deployment is unchanged — train-argmax already picks the winning mechanism — so on this domain the gate is not buying the headline score. What it buys is the archive and the stopping rule. Without it two noise mechanisms enter as elites (one of them inert, its activation beacon never firing), and false elites then seed parents, spending future budget on noise. Termination is decisive: under single-run or K=3-mean crediting, phantom progress appears in 62–76% of post-convergence rounds, so the loop never meets its stop condition and runs to the cap, while paired-2σ stops at the 10-round floor. An unverified self-evolution loop does not merely risk a bad harness — it cannot tell that it has finished.

Two supporting notes: accepted edits span all four levers rather than prompts alone, and on most domains the credited harness stacks mechanisms drawn from more than one cell, with the per-cell elite being what keeps the second mechanism alive long enough to be recombined at all. And train-selection is a lower bound on what generalizes — on GDPval a variant ranked below the winner on train scored highest on test (+11.5% vs +9.2%), which is the argument for crediting on held-out data rather than trusting the training rank.

The evolved harness is a correction fitted to one model#

The most consequential result is a negative transfer one. Cold-started on other backbones the loop evolves different harnesses, each targeting that model's dominant pathology, and the patches follow a pathology → patch matching law (test ΔPass@1; bold = matched patch):

Domain	Model	Dominant failure	Patch A	Patch B
AppWorld	Qwen3.6-27B	empty "engagement" turns	verify-finalize +15.4	submit-verify +1.2
AppWorld	397B	careless errors	verify-finalize +0.2	submit-verify +13.6
AppWorld	Gemini 3 Flash	careless errors	verify-finalize +5.8	submit-verify +13.5 (z=4.15)
Omni-MATH	Qwen3.6-27B	thinks too much	27B stack +11.7	raise-reasoning +1.7
Omni-MATH	397B	runaway (shared)	27B stack +11.0	raise-reasoning +0.7
Omni-MATH	Gemini 3 Flash	thinks too little	27B stack -1.5	raise-reasoning +15.3

Three readings:

• Off the matching diagonal the patch is near-zero, and the wrong direction is harmful. Gemini shares the 397B's AppWorld pathology and reproduces the careless→checklist match across model families; on Omni-MATH the two Qwen generations share thinking-runaway and the 27B-evolved stack transfers nearly loss-free (+11.0 transferred vs +11.7 native). But Gemini reasons too little, so the transplanted recovery never fires (-1.5) while its matched patch turns the same lever the other way to +15.3. Turning it the wrong way costs -15.7 when stacked on the evolved 397B harness. Two model families need one lever turned in opposite directions.
• This is a real failure mode of the frozen model, not a scoring artifact. On five domains a single pathology dominates 49–88% of vanilla failures, and the matched mechanism is credited under both deterministic and judge-based verifiers.
• A test-time-scaling arm exists, and loses. On LiveCode neither a 16× token budget nor a blanket thinking-off toggle reproduces the evolved harness's gain — the credited mechanism disables thinking only after a runaway. On this domain, spending more compute is not a substitute for the specific control-flow fix (Large-Scale Test-Time Compute).

So the transferable asset is not the harness. It is the diagnose → search → verify process: "a credited harness is a correction fitted to the model, not a universally good setting."

The contrast that explains it: knowledge transfers where harnesses don't#

A second empirical source, published six days after HarnessBank's v1, reports the opposite transfer result for a different persistent artifact. Knowledge-Centric Self-Improvement (Wang et al., Caltech, arXiv 2607.19592) freezes a curated knowledge base rather than a harness, hands it to a fresh model family with no further curation, and gets a positive lift in all eight donor-recipient cells on held-out tasks, in both cross-family directions (Polyglot 8.3% → 20.0%, ARC-AGI-1 23.3% → 43.3% for the strongest pairing).

Both results are held-out, both are single-lab preprints, and neither is wrong. The reconciliation that fits both is what the artifact is fitted to:

• A harness patch is fitted to one model's failure distribution. That is why the same reasoning-budget lever has to be turned down for a runaway backbone and up for an under-thinking one, and why transplanting it costs -15.7. The matching law above is the direct evidence.
• A distilled insight is fitted to the task domain — an ARC grid invariant, Rust's loop-based propagation semantics, a test-runner flag. Those are true independently of which model reads them, which is why a Haiku-authored bundle helps GPT and vice versa.

Stated as a design rule: fit your persistent artifact to the problem, not to the solver, if you want it to move. Nobody has run both artifact classes under one protocol, so this is a hypothesis with two strong one-sided data points rather than a measured law — the open question lives on the sibling page.

The two papers also agree, independently and on disjoint benchmarks, about the weakest axis. HarnessBank credits GEPA on zero of five sealed tests; Wang et al. put GEPA last of three on both benchmarks they ran it on (44% and 36% against 86.7% and 68.0%). Two labs, two different objects of improvement, same verdict on prompt optimization.

The Caltech paper is also the sharper contrast on cost: its knowledge-curation runs beat DGM and HyperAgents on solve rate while spending less — SWE-bench Pro at $208 against DGM's $713 — where harness evolution buys its gains with a rollout budget in the hundreds to low thousands per domain.

Reconciling Cline and HarnessBank#

(The third source's disagreement with both is handled in Three sources, one benchmark family above; the two agreements below survive it, and the first is now qualified.)

• They agree on magnitude — against a baseline neither ran. Cline's +11.3pp sits inside HarnessBank's credited 9.2–15.4pp band, and both sit well inside the ~44pp scaffold-swing envelope Measuring Beyond Accuracy Saturation measures for scaffold choice on a fixed model. (Qualified 2026-08-04 by Rethinking the Evaluation of Harness Evolution for Agents: agreeing on magnitude is weak evidence when both magnitudes are measured against a do-nothing baseline. Budget-matched, plain parallel sampling buys +4.1pp without unit tests and +13.1pp with them on the same benchmark — so a gain in this band is not by itself evidence that the harness got better.)
• They agree on where the gains live. Cline's agent added retry logic, output-aware loop detection and PID tracking; HarnessBank's accepted edits span all four levers and its prompt-only baseline is credited nowhere. Agent-authored harness optimization is a runtime and control-flow activity, not prompt engineering — which is also why it grows the harness rather than shrinking it. Wang et al.'s meta agent independently reproduces the escalation (prompt rules → middleware enforcement → tool-layer fixes) and finds it insufficient, so the third source agrees about the layer while disagreeing about the payoff.
• They differ on what can be claimed. See the RSI section below: HarnessBank's cross-model dissociation is the strongest available argument that this loop cannot be recursive self-improvement in the definitional sense.
• HarnessBank's own limits. Single lab; code "publicly available upon acceptance" and unreleased at ingest; one default backbone for the main table; SWE-bench underpowered at n = 26. Most importantly the held-out tasks are held out within each benchmark — nobody has run a harness evolved on suite A against an unrelated suite B, which is a different transfer claim from the one measured here.

This is not the capital-R RSI the title claims#

Cline's post opens by invoking Recursive Self-Improvement and linking Anthropic's RSI essay, and closes with "recursive self-improvement is no longer a sci-fi experiment." The vendor's framing is a claim, not a definition, and the two senses should not be merged. What is documented here is narrow scaffold hill-climbing on a fixed benchmark:

	Cline's run	RSI as this wiki defines it
What improved	the harness around the model — retry policy, loop detector, process handling	the model itself; each generation designs and trains its successor
Where the gain lives	one repo, one benchmark, one model pairing	general capability, transferring across tasks
Who supplied the direction	a human-written brief with a pinned end state and an explicit anti-reward-hacking clause	the system, autonomously
Compounding	none demonstrated — a single campaign, not a loop that reruns on its own output	the whole point: each round makes the next round faster
Termination	17 hours, human pressing continue when it stalled, human PR review before merge	the open question the RSI literature is about

The model's weights were untouched. The improved artifact was a pull request against a TypeScript codebase. Nothing about the run bears on the extrapolation in When AI builds itself except by loose analogy — it is a data point for AI Accelerating AI Development (AI compressing AI-adjacent engineering work), which is that essay's present-tense empirical half, not its projection. The honest reading: an instance of the perspiration-becomes-automated argument, at the scale of one repo's scaffold.

There is a real, testable boundary between the two: does the improvement transfer? A harness patch that lifts held-out agentic tasks the agent never saw is a different object from one that lifts the suite it was optimizing against. Cline reports no transfer measurement.

HarnessBank runs that boundary test twice, and both results point the same way:

• Across tasks, within a domain: it transfers. Six of seven benchmarks credit the evolved harness on sealed test tasks. So harness evolution is not pure benchmark memorization — the thing being fixed is a real failure mechanism, not the training tasks' identities. (Contested 2026-08-04 by Rethinking the Evaluation of Harness Evolution for Agents. HarnessBank's finding stands on its own splits and is not withdrawn — but the same test, on Terminal-Bench 2.1 with frontier backbones and a different evolution method, returns +1.2 / +0.0 / +0.6 average, and the authors read the discrepancy against evolution's same-task results as evidence that "the revisions discovered during evolution encode task-specific shortcuts rather than genuinely better harness design principles." Within-domain transfer is now a contested result whose sign appears to depend on baseline headroom, not an established one.* Updated 2026-08-13 by DarwinX: Evolving Agent Harnesses Through Natural Selection, which supplies a third and a fourth data point pointing opposite ways within one system: WebArena-Infinity transfers hugely (300 synthetic intents → 1,260 unseen real tasks, +49.5 audit-clean) while TerminalWorld's 41-task held-out split returns three tasks at McNemar p = 0.45, with a pre-search reference matching the headline. So the sign depends on something inside a single paper's own results, which rules out lab, method and backbone as the explanation and leaves the starting harness's condition — the discriminator argued below.*)
• Across models: it does not, by construction. The evolved harness is near-zero off its matched pathology and actively harmful (-15.7) when the same lever is turned the wrong way for a different model family. What generalizes is the procedure, not the artifact. Qualified 2026-08-13, and in the direction the frame predicts. DarwinX runs a GPT-5.5-evolved TB2.1 harness unchanged on an Opus 4.8 base and on a different benchmark (SWE-bench Verified) and it holds — 84.2% official pass@1, +3.4 over an LSP fix-skill reference, no in-domain feedback. That is not a counterexample to the matching law, it is the law's own fine structure: what HarnessBank transplanted and lost was a reasoning-budget lever (thinking on/off, raise-reasoning), pure solver pathology; what DarwinX transplants is a verification / artifact-contract bundle whose seven skills contain no domain knowledge and no effort setting — "state the acceptance condition, then check against it before committing" is a task-domain regularity, so What Makes a Self-Improvement Artifact Transfer?'s general form predicts exactly this outcome. Two limits travel with it: the SWE-V arm has no matched-model base-Monet comparator, so the +3.4 is against a different harness rather than an unevolved one, and every compared harness sits inside 80.8–84.2. And DarwinX's own cross-model instability is visible elsewhere — on TerminalWorld the same procedure on GPT-5.5 reaches 56.1%, below the neutral Terminus-2's 61.0% on that base, with Opus 4.8 reported as the headline. Sharpened again 2026-09-24 by RRSI (below), which supplies the cleanest counter-instance yet. A harness evolved with Gemini 3.5 Flash and handed unchanged to Gemini 3.1 Flash Lite — smaller, and never part of the search — still gains (+3.4, 30.4% relative). Read against the fine structure above: RRSI's regularizers explicitly select against solver-specific pathology fits (the leakage critic, the pruning rule for components with no durable gain), so a matching-law-consistent reading is that regularization is what buys the transfer, not that the law is wrong — an unregularized edit still fits one model's failure distribution, a regularized one is selected for surviving without that fit.
• And the loop converges rather than compounding. Under the significance gate it stops at its 10-round floor; the ungated variants only stop at the cap because they keep hallucinating progress. Nothing here compounds without a human re-issuing the brief for a new domain and a new model.

A process that produces a model-specific correction and then terminates is a fitting procedure, not a system designing its successor. That is the sharpest separator the corpus has for the vocabulary question above, and it comes from the strongest-evidence source in this cluster.

Evidence weighting: the vendor scores its own harness#

Four things bound how much this result can carry:

• Self-reported, not replicated. Cline ran the benchmark, on Cline's harness, and published its own scores. No third party has rerun it. Traces and cost breakdowns are posted as gists, which is more than most vendors do, but posting artifacts is not replication.
• The benchmark is the one they have been hill-climbing for months. Cline says so plainly — Jan 2026 on Opus 4.5, a Feb 2026 published playbook, this run in July. Repeated optimization against a fixed suite by the party being scored is the setup Benchmark Contamination and Decontamination and Evaluation Awareness & Grader Gaming are skeptical of, even when no verifier is touched and no task name is detected. Overfitting to a benchmark's failure distribution is a live risk that "we didn't reward hack" does not address: a retry policy tuned on the five 429s in this suite is fitted to this suite's provider conditions.
• The optimizer had write access to its own scoring substrate. See below.
• A prompt-level anti-gaming clause is not a control. The prompt banned verifier edits, task-name detection, and timeout inflation; Cline reports the guardrails held and the model policed itself (recording attribution guards, excluding two runs invalidated when the orchestrator was accidentally terminated). That is a good result and it is also self-attested.

Two runs were thrown out and rerun; the reasoning-effort fix got no causal credit; some failures were judged genuine model limitations. The post reports its own negative results, which is the main reason to take the rest seriously.

The same-model-different-scaffold effect this campaign exploits is measured independently in Measuring Beyond Accuracy Saturation — swapping the scaffold on a fixed model swings accuracy ~44pp on CORE-Bench Hard, and two scaffolds on one model disagree on 31% of tasks. An 11.3pp harness-only gain is unremarkable against that baseline; what is new is that the agent found the 11.3pp rather than a team of engineers.

Decoupling by prompt, not by architecture#

Optimizer–Evaluator Decoupling's invariant is that the thing proposing a change never grades it. This campaign violates the architectural form of the rule and substitutes a behavioral one: the optimizer had write access to the repo that runs the eval, so nothing structural prevented it from editing the grader. What stood in for the split was (a) a prompt clause forbidding verifier edits, task-name detection, and timeout inflation, and (b) a human reviewing the final PR before merge. Both are the weaker kind of guarantee — the Bun stub-and-justify episode is the corpus's evidence that a proxy metric plus write access is exactly the configuration where gaming appears, and there the patch had to be applied at the grader, which was only possible because grader and optimizer were separate.

The generalizable design note: when the harness under optimization is the eval substrate, decoupling has to be reintroduced deliberately — a frozen eval harness the agent cannot edit, a held-out suite it never sees, or a grader run from a pinned commit. Cline's campaign used none of the three; it used a human at the end. That worked here at 89 tasks and one PR, and it is precisely the check that stops scaling (Verification as the New Bottleneck).

HarnessBank deploys all three — immutable kernel, sealed test split, deterministic evaluator owning every score — and then ablates the last one, which is the corpus's first measurement of what the split is worth rather than an argument that it should exist. The measured answer is unintuitive: on TB2 the gate changes nothing about what ships, and everything about whether the loop can stop.

The complementary experiment: what happens when the gate itself is self-authored#

HarnessBank ablates a significance test off an evaluator that stays exogenous in every arm. Guo et al. (Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents, CAS, arXiv 2607.24300, empirical — full treatment on Optimizer–Evaluator Decoupling) ablate the exogeneity itself: the agent co-edits policy.py and tests.py each round, and the only thing held back is a deployment evaluation that never enters its context. The two papers bracket the same design space from opposite ends, and reading them together separates two costs that are easy to conflate:

• An exogenous evaluator without a significance gate costs you the stopping rule. HarnessBank's ungated variants ship the same TB2 score, admit two noise mechanisms as elites, and see phantom progress in 62–76% of post-convergence rounds so the loop runs to its cap.
• An endogenous evaluator costs you the artifact. Guo et al.'s unprotected runs end with self-scores of 0.70–1.00 while 15 of 35 model-game policies land below the game's random reference, and a traced Breakout run peaks at 17.6, is overwritten to 7.5, rediscovers 18.1, and finishes at 12.2 with its self-test pass rate near 1.00 throughout. Restoring only the exogenous comparison moves Gemini-3-Flash from 7.9 to 30.0 on the same benchmark.

Two further results speak directly to design choices this page records.

The endo-gate arm is Cline's configuration under laboratory control. It supplies a gate and whole-state rollback but sources its evidence from the agent's own tests — i.e. the split restored behaviorally rather than architecturally. Compute-matched it does most of the work (mean final truth 7.7 → 13.9 against SEAL's 15.4, peak-to-final loss 6.9 → 0.5 against 0.4), so the retention benefit is mostly the conservative update, not the exogeneity. Uncontrolled it is unreliable — below no-protection for three of six models and timing out on a fourth. That is the sharpest available reading of what Cline's prompt clause plus human PR review was substituting for: the cheap half, with the consistency left to chance.

Nobody has tested whether HarnessBank should also hide its numbers. Guo et al.'s leaky-anchor arm holds the audit fixed and only discloses its scores after a rejection; SEAL is at least as high in all six rows and strictly higher in five, with a 35.1 → 12.7 reversal on the worst cell. Audit confidentiality is a fifth design condition beyond HarnessBank's kernel/split/evaluator triad, and it is untested in the harness setting where the evolver reads execution trajectories by design.

The domains do not overlap — programmatic Atari policies against seven agentic and coding benchmarks — so this is convergence, not replication. But both labs land on the same asymmetry: the optimizer may reason about the metric freely, provided reasoning cannot become credit.

The human role, compressed#

The before/after Cline offers is the post's real argument:

• Jan 2026 — Opus 4.5, Terminal-Bench, 47% → 57%. Four engineers, a couple of weeks, reading traces and staring at failure logs by hand. Published as a playbook in Feb 2026.
• Jul 2026 — Kimi K3, Terminal-Bench 2.1, 77.5% → 88.8%. One prompt, one engineer, 17 hours, human intervention "nearly zero" (pressing continue every few hours on a cloud VM).

Not a controlled comparison — different model, different benchmark version, different starting harness, different absolute band on the difficulty curve. What it does show is the human's job moving from doing the loop to writing the brief and reviewing the PR: the same narrowing Harness Shrinkage as Models Improve documents for scaffolding, arriving at the meta-level. Cline's own conclusion — "the bottleneck isn't models but the humans using them" — is the practitioner form of that claim, and it is the part most likely to be true independent of the benchmark number.

Note the direction of the edit, though: the agent added harness code (retry logic, output-aware detection, PID tracking), it did not delete it. Agent-authored optimization is not automatically shrinkage; here the model was patching scaffold that was too thin, not pruning scaffold that had become drag.

Connections#

• Headroom-Closed Index (HCI) — the measurement behind §5.2's first condition, on the benchmarks this page reports on. Normalized against each benchmark's entry-year frontier, search and terminal agents (BrowseComp · Terminal-Bench) sit at 56.8 in 2026 and software engineering (LiveCodeBench · SWE-bench) at 52.6, against 85.8 for graduate-level science — so there is substantial headroom left on Terminal-Bench in the aggregate even where frontier agents score 70–80 on its 89 tasks. That is a caution against reading Wang et al.'s "agents already score highly there" as a statement about the benchmark family rather than about a particular 89-task suite

• GDPval Benchmark — the domain in HarnessBank's suite whose primary benchmark this wiki now holds, and the place where an evolved harness and the benchmark's own authors measure the same lever from opposite ends. HarnessBank credits 43.7 → 52.9 Pass@1 (+9.2) on a sealed GDPval split; OpenAI, without any harness evolution, moves GPT-5 high 38.8% → 43.1% on human expert preference with a single generic prompt that tells the agent to render its deliverables to PNG and look at them (self-inspection 15% → 97%, black-square PDF artifacts eliminated, egregious PPTX formatting errors 86% → 64%), plus best-of-N=4 with a judge and container GET access. Two independent measurements that a substantial slice of the gap to a human professional is protocol rather than capability — and a caution for this page's headroom discriminator, since GDPval's starting harness was not broken, it was merely not looking at its own output

• Inference-Time Architecture Search — the constrained ancestor. Archon (2024) searches the same kind of object with an external Bayesian optimizer over a closed, hand-specified operation vocabulary and a hand-ordered layer grammar, offline against a held-out split, rather than letting a meta-agent rewrite harness source. Wang et al.'s budget-matched negative result is on the unconstrained form only, which suggests — untested — that the constraint is where the value is

• Continuous Self-Modification Under Review — the fourth instance and the only unbriefed one: 161 days, 1,085 self-modification commits, a blocking multi-model review gate, and no objective function. It shows the loop can run for months without a human re-issuing a brief (so termination is a property of the benchmark protocol, not of self-modification), independently reproduces the maintenance-not-design verdict, and supplies zero measurement — its benchmark scores were taken on frozen seeds with self-evolution disabled

• Cline — the vendor that ran the campaign and whose harness was the object under optimization

• Agent Harness Engineering — the patterns being edited; this page is what happens when the agent, not the engineer, does the editing

• Agent Quality Flywheel — the same eval-fix loop with a human approving each fix; Google states the autonomous version as a direction it deliberately has not shipped, and this is that version running in the wild

• Optimizer–Evaluator Decoupling — Cline violates it architecturally and restores it behaviorally (prompt clause + human PR review); HarnessBank restores it architecturally (immutable kernel, deterministic evaluator, sealed test set) and then ablates it, which is where the rule stops being a principle and becomes a number; Guo et al.'s SEAL runs the mirror ablation on exogeneity itself and adds audit confidentiality as a fifth condition nothing here tests

• Deterministic Pre-Execution Gates — third-party corroboration of this page's sharpest finding, from a lab that never mentions harness evolution. Reddy et al. (arXiv 2607.07405, empirical) hand-write four runtime predicates that intercept tool calls before dispatch and get +12.4pp on τ²-bench airline, replicated on 15 disjoint seeds — the same verdict GEPA's zero credited tests deliver, that the gains in this class are runtime control-flow, not prompt-addressable, arriving from human-authored patches instead of an evolution loop. Two honest limits on reading it as corroboration: they never ran the prompting baseline (their limitation 7), so the comparison is assumed rather than measured; and their gates were written from the policy and evaluated on the same task set with replication over seeds, not tasks — the held-out-task discipline HarnessBank supplies and this does not

• Unproductive Self-Verification — the same lever, opposite sign: HarnessBank's evolver keeps discovering a verify-finalize self-check as a credited patch on backbones whose dominant pathology is premature finalization, while Anthropic's guidance for Opus 5 is to delete verification instructions because that model already over-verifies. The pathology→patch matching law predicts exactly this, and it is the cleanest cross-source support the law has

• Large-Scale Test-Time Compute — the axis this page's newest source puts harness evolution on, and loses. HarnessBank beat one test-time-scaling arm on one domain (a 16× token budget does not reproduce a targeted control-flow fix); Wang et al. run four arms at a matched K = 5 across the whole suite and find plain parallel sampling ahead of harness evolution on every model in both feedback settings. Two directly opposed data points on whether extra compute is better spent on the scaffold or on the trajectories — and now a third that declines to run the arm: DarwinX argues its extra compute is targeted rather than equal (4.3× tokens on the six tasks it newly solves, ~1.4× on the 69 both agents already solve), which is a claim about allocation and not about budget

• Orchestration Sets Token Economics — the hand-built counterpart, and the reason the negative result here is narrower than it sounds: a designed orchestration layer moves cost −41% uniformly across six models, while a machine-derived one does not clear parallel sampling. Same object, different author, opposite verdict — the distinction is drawn above

• Harness-Induced Belief Divergence — the unmeasured side effect: the middleware an evolution loop generates (output truncation, finalization gates, turn-budget interrupts) is drawn from exactly the interface classes Yi & Song show shift the agent's belief state. Harness evolution manufactures those changes as a by-product of chasing a score and nobody has looked at what they do to what the agent thinks is happening

• Reward Hacking — the failure mode the prompt banned by instruction rather than by construction

• Harness Activation and Adherence — the consumer-side half of this page's question, split out because four other pages need it. Whether an evolved artifact ever enters the solving agent's context, and whether the agent then follows it, are separately measurable and do not track each other — and they are the two quantities every end-to-end score on this page silently multiplies into its result. The numbers also bound this page's own reading of the mechanism: the tier where activation fails is not the tier any instance here runs on

• RSI Autonomy Levels (B0–L5) — where this loop sits on the September 2026 ladder: L2, strategy autonomy over a harness with the objective, benchmark and promotion rule external. The ladder's value here is negative — it says what a harness-evolution result cannot demonstrate (an inherited change to the improvement mechanism), which is the boundary this page argues for one instance at a time

• Recursive Self-Improvement — the term the vendor invokes; the distinction between narrow scaffold hill-climbing and model-improves-its-successor is drawn above and matters more than the shared vocabulary

• AI Accelerating AI Development — the claim this run actually supports: AI compressing AI-adjacent engineering work (weeks of four engineers → 17 unattended hours)

• Harness Shrinkage as Models Improve — the human-role-narrowing dynamic reaching the harness-authoring layer, with the twist that the agent grew the harness rather than pruning it

• Open-Ended Discovery Harnesses — the same budget-matching charge applied to a different search target, with a split answer. SwarmResearch searches over solutions rather than harnesses, so a loss would mean "spend the compute elsewhere" rather than "the artifact doesn't transfer" — and its comparison is matched against the multi-agent baseline CORAL ($50/task both, same runtime, same model, and the margins there are small enough that the authors call three of the wins polishable) but not against the evolutionary baseline EvoX (~$23.50/task average against $50, and that is where the large margins are). Half a control is the current state of the art in this literature

• Client-Side Agent Optimization — the sibling lever: AgentOpt searches over model assignments holding the harness fixed; this searches over harness patches holding the model fixed. Both are client-side, and neither has been measured against the other

• Cost-per-Task Over Cost-per-Token — the campaign's economics: ~$680 and ~1B tokens to move a per-run cost from $79 to $49.8, with the payback boundary undefined

• Measuring Beyond Accuracy Saturation — the independent measurement of how much scaffold alone can move a score (~44pp swing, 31% task-level disagreement between scaffolds on one model)

• Compute-Controlled Benchmarking — three ways this page's results are uncontrolled. Cline's "88.8% at $49.8 vs Fable 5's $552" is a harness-uncontrolled comparison; the whole method class is compute-uncontrolled until someone runs a matched-budget baseline, which is Brown's benchmark-maxxing critique landing on a named method with an experiment attached (the pass@1-versus-pass@5 split is the portable diagnostic that came out of it); and DarwinX is that page's cleanest specimen of a categorical effort tier presented as a compute control — "medium / high / xhigh" defined nowhere, no compute column in its protocol appendix, no dollar figure, and its own headline pair crossing default → high

• Evaluation Awareness & Grader Gaming — the adjacent threat model: months of optimization against one suite is a validity risk distinct from, and not excluded by, the absence of verifier edits

• Verification as the New Bottleneck — the human PR review is the last decoupled check in this loop and the one that does not scale with campaign count

• Kimi (Moonshot AI) — the model under test; Cline reports matching Moonshot's own 88.3% vendor SOTA with a general-purpose harness

• Capability-Gated Model Fallback — the abandoned arm of the experiment: Cline dropped Fable 5 as leader model because the safety classifier kept downgrading it to Opus 4.8

• Knowledge-Centric Self-Improvement — the third axis and the direct contradiction on artifact portability: freeze a curated knowledge base instead of a harness and it transfers across LLM families in every cell. Both papers independently rank prompt optimization last. Wang et al.'s §5.1 supplies the mechanism that separates them — harness edits "memorize fixes rather than distilling strategies," which is precisely the selection Caltech's distillation stage exists to enforce — and the same budget-matched charge has never been run against the knowledge axis. That page's weights row now has a controlled sibling: Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails runs the harness and weights levers together on one task set and finds they conflict under full-trajectory imitation, compose under on-policy correction

• What Makes a Self-Improvement Artifact Transfer? — the synthesis of this page's transfer results: the matching law's fine structure (patches transfer within a pathology class, across families) generalizes to "an artifact transfers exactly as far as the regularity it encodes extends," which unifies the cross-model transplant failure here with cross-release instruction depreciation and yields the stated RSI separator. RRSI is a fifth data point for the same synthesis, from the opposite direction: instead of asking what an unconstrained artifact was fitted to, it constrains how the artifact gets written (leakage screening, pruning of components with no durable gain) and gets cross-model transfer to an unseen, weaker policy as the result — regularizing the write side is an alternative to selecting after the fact for what already transfers

• RSI Autonomy Levels (B0–L5) — RRSI's own framing places itself at this page's standing L2 boundary (strategy autonomy over a harness, objective and termination round-count external), the same level every other instance here occupies; what's new is that RRSI is the first instance to hold the seed harness, benchmark suite and policy family fixed and vary only how tightly the L2 search is regularized, isolating a fourth transfer discriminator alongside starting-harness brokenness, activation/adherence and weight-update fit-preservation

Open Questions#

• Does agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks? Partially answered — and the two halves now point opposite ways. Test-time scaling: answered, negatively. Rethinking the Evaluation of Harness Evolution for Agents (empirical) supplies the full-suite budget-matched arm this bullet named as missing — four methods at K = 5 on Terminal-Bench 2.1 across three frontier models, with and without unit-test feedback — and harness evolution finishes below plain parallel sampling everywhere and below do-nothing on average without unit tests (67.4 vs 68.2), with its pass@5 gain traceable to multi-attempt selection rather than pass@1 capability. Held-out generalization: contested, not settled. HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution credits six of seven sealed per-domain tests at z ≥ 1.96 (+9.2 to +15.4pp); Wang et al.'s 45/10/34 split on Terminal-Bench 2.1 returns +0.6pp average and +0.0 on GPT-5.4. The proposed discriminator is baseline headroom (superseded 2026-08-13: score headroom fails on the decisive pair — Wang et al.'s 68.2 baseline is below base Monet's 75.5 and gets the smaller gain; the surviving discriminator is how broken the starting harness was.) Still missing entirely: transfer to a different benchmark, which no source measures. Settled 2026-08-13 for the cross-benchmark half by DarwinX: Evolving Agent Harnesses Through Natural Selection: a TB2.1-evolved harness runs unchanged on SWE-bench Verified and on a different base-model family for 84.2% official pass@1, +3.4 over an LSP fix-skill reference — with the caveats that there is no matched-model base arm on SWE-V and every compared harness sits in 80.8–84.2. The same source raises the held-out gain on a huge-headroom suite to +49.5 (WebArena-Infinity, 1,260 unseen real tasks) and returns p = 0.45 on a small one, so the held-out half stays contested and the budget-matched half stays unanswered — DarwinX defines no effort tier, publishes no compute column and no dollar cost. Sharpened, not settled, 2026-09-24. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses (empirical) does not run the budget-matched arm either, but it runs a controlled comparison the held-out half was missing: four prior harness-evolution methods (Meta-Harness, AHE, TTHE, HarnessX) on the same H0, policy, evolve set and candidate budget as RRSI, across eight benchmarks. Three of the four baselines are flat or negative on out-of-distribution held-out tasks (AHE and TTHE finish below H0's 39.7 OOD average, Meta-Harness adds 0.9), while RRSI's regularized search clears H0 by +3.9 (43.6 vs 39.7) with the smallest evolve-set gain of any evolved arm — and an ablation (Table 2) reproduces the unregularized-loop failure mode inside the same paper, holding the seed harness and benchmark fixed. The held-out question now has a plausible answer: unconstrained evolution mostly does not generalize, and the four other empirical sources in this cluster (Cline, HarnessBank, Wang et al., DarwinX) are all instances of the unconstrained case. Still missing: a budget-matched test-time-scaling arm for RRSI itself, so the first half of this bullet is untouched.
• Is the negative result a property of harness evolution or of Terminal-Bench? Wang et al.'s §5.2 names two conditions a fair test needs — substantial headroom above current agent scores, and a benchmark whose performance genuinely depends on the harness (specialized tools, skills, workflows) rather than one where "a shell tool and a basic prompt already suffices." Terminal-Bench satisfies neither for frontier models, and it is the benchmark essentially the whole harness-evolution literature reports on, including HarnessBank and Cline. Falsifiable directly: run the same four budget-matched arms on a harness-sensitive suite and see whether the ordering inverts. Partially answered 2026-08-13 — the second condition, not the first. DarwinX: Evolving Agent Harnesses Through Natural Selection's WebArena-Infinity arm is the harness-sensitive suite by accident (a coding agent's harness pointed at Chrome), and evolution moves it 43.5% → 93.0% audit-clean on 1,260 held-out real tasks. What is still un-run is the budget-matched version of that experiment: nobody has put parallel sampling and harness evolution on the same suite at the same normalized budget where the harness genuinely binds. #oq/source Partially answered again 2026-09-18, and on the first condition this time — by moving the method off Terminal-Bench entirely. Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents runs a fixed solve-evolve protocol on three non-Terminal-Bench suites and finds real gains where headroom exists: +19.3 pp (Qwen3-235B) and +15.8 pp (GPT-OSS-120B) on SWE-bench Verified, +7.0 pp on MCP-Atlas, +15.1 pp on SkillsBench, whose no-evolution arm starts from an empty skill set and is therefore harness-sensitive by construction rather than by accident. So the negative result is not a property of harness evolution as such. But the same table sharpens the alternative: at the frontier tier those gains collapse to +2.4 to +3.6 pp on SWE and MCP, which is the same near-nothing Wang et al. measure on Terminal-Bench. The un-run experiment therefore changes shape — it is no longer only "run the budget-matched arms on a harness-sensitive suite" but "run them on a mid-tier backbone," because every result in the negative column of this page is a frontier-model result and the only large positives in the corpus outside DarwinX's WAI arm are mid-tier.
• Is the Wang-et-al./DarwinX disagreement actually a search-budget disagreement? The two papers cap the search four orders of magnitude apart and neither reports it in a comparable unit: Wang et al. fix K = 5 rounds of meta-agent edits; DarwinX evolves "over many generations," screening at avg@3 on rotating subsets and confirming steering nodes at avg@5 on the full 89-task suite, and states no generation count, no rollout total and no dollar figure anywhere (the one reconstructible arm is WAI's 26 kept / 36 reverted iterations, each screened at avg@3 over 300 intents). Falsifiable two ways: cap DarwinX's generations at Wang et al.'s K and see whether the gain survives, or have both papers publish total rollouts. #oq/source (Unchanged 2026-09-18, and the premise widened: Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents fixes "the same evolution budget β and per-task turn limit" across all 21 agent-evolver pairs and publishes no value for β, no turn limit, no rollout total, no token count and no dollar figure anywhere in the paper or its appendices. Three of the corpus's four controlled harness-evolution papers now hold a search budget fixed without reporting it.) Partially answered 2026-09-24, by precedent rather than by resolution. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses publishes a complete per-domain hyperparameter table — rounds, trials per task, noise tolerance, edit-budget range, stall window, pruning window, cost-rule coefficients — and runs its four baselines on the same candidate budget it uses itself. That doesn't settle the Wang-et-al./DarwinX gap (different methods, not a shared unit), but it is the first source in this cluster to show a comparable search-budget table can be published at all, which narrows every other source's missing-compute-accounting limitation from "the field doesn't report this" to "this source chose not to."
• How much of the 77.5%→88.8% survives on an agentic benchmark Cline has not been hill-climbing for six months? Every fix is described as a general harness improvement, which is a transfer claim, and no transfer measurement is reported. (HarnessBank does not touch this: it measures within-domain held-out tasks, not cross-suite transfer.)
• How much of every negative result on this page is non-activation rather than a bad patch? Partially answered 2026-09-18 by the primary, and the answer narrows the hypothesis rather than confirming it. Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents (empirical) is the per-model activation measurement this bullet asked for, and it delivers three things. (i) The "bad patch" branch is largely ruled out as the differentiator: harness-updating is flat in base capability — at most a 3.1 pp spread across seven evolvers on any benchmark, with a 9B evolver's skill procedurally isomorphic to Opus 4.6's on the worked case. (ii) Non-activation is real, large and separable from unfaithful use: SkillsBench skill-load rate runs 0.251 (Qwen3-32B) to 0.961 (Qwen3-235B), harness-following rate 0.142 to 0.757, and the two do not track each other — Qwen3-235B loads like Opus 4.6 and follows less than half as often. Full treatment on Harness Activation and Adherence. (iii) And it does not explain the negative results on this page, because all of them are frontier-model results and the frontier tier activates at ≈0.96 and adheres at 0.73–0.76. The mechanism is a weak- and mid-tier phenomenon; this page's corpus has never tested that tier. What remains open is the original falsification, unchanged: HarnessBank still emits a deterministic activation beacon per patch and still reports no rate, and nobody has re-run Wang et al.'s +0.6pp arm with beacons. Publishing either would partition an end-to-end score in the systems this page actually holds, rather than in a third lab's protocol. Sharpened, not answered, 2026-09-24: Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails supplies a case where activation is not the missing variable — a LoRA-SFT'd weak model raises its use of the evolved harness's domain-computation recipe from 30.8% to 76.1% and still regresses -14.9pp on average, because the failure sits in whether the model's own planning cadence still matches the harness's finalization contract, not in whether the harness fires. That is a fourth quantity this page's end-to-end scores conflate, on top of artifact quality, activation and adherence: whether a weight update on the same model preserves its fit to an already-evolved harness.
• Does full-trajectory-imitation's fit-breaking generalize past LoRA-SFT on trajectory imitation specifically? Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails measures exactly one weight-update recipe (LoRA-SFT on complete expert trajectories) against exactly one fix (on-policy per-turn correction), on domain-knowledge harness edits, on two model families. Untested: whether the same collapse appears under full fine-tuning or RL post-training rather than LoRA; whether it appears when the harness edit is a reasoning-budget lever (HarnessBank's solver-fitted class) rather than task-domain knowledge; and whether on-policy correction still composes cleanly after several co-evolution rounds rather than one. Falsifiable directly: run the same evolve-then-adapt protocol with RL post-training in place of SFT, or with a HarnessBank-style pathology-fitted patch in place of a domain-knowledge one.
• Does the semantic quality-diversity archive earn its complexity against a plain greedy keep-the-best loop, holding the significance gate fixed? HarnessBank ablates the gate but never ablates the gene bank; its archive evidence is observational (credited harnesses stack mechanisms from more than one cell), not a controlled arm. Sharpened, not answered, 2026-08-13. DarwinX: Evolving Agent Harnesses Through Natural Selection builds its whole contribution on the archive plus a cross-lineage merge operator and still does not ablate either — §9 says so in its own words ("the contribution of recombination relative to single-lineage mutation still requires controlled ablation"), its archive evidence is the same observational shape (four specialists solve 24/25/26/27 of 41 and the merge solves 28, on a split where one task is 2.4 points), two of those four specialists fail to beat the unevolved base, and on WebArena-Infinity every merge was reverted so the largest gain in the corpus accrued along a single lineage with the recombination machinery contributing nothing. Two papers now rest on the archive and neither has isolated it.

Resolved Questions#

• Is there a stated test that separates narrow scaffold hill-climbing from the recursive self-improvement the term names — e.g. capability transfer to unrelated tasks, or improvement that compounds without a human re-issuing the brief? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? states the test: name what the improvement artifact is fitted to, then measure whether gains survive outside that scope. HarnessBank's two separators (cross-model transplant near-zero off the matched pathology and -15.7 wrong-way; termination at the 10-round floor vs phantom progress in 62–76% of ungated post-convergence rounds), Wang et al.'s compute-side separator (at matched budget the loop does not beat sampling five times), and Caltech's orthogonality result (a domain-fitted artifact compounds portably while the model improves not at all) instantiate it: solver-fitted gains are maintenance, domain-fitted gains are assets, and RSI would require the gains to live in the solver itself. The stated test's un-run instances remain evidence gaps: cross-suite transfer stays in this page's #oq/source items, the two-artifact controlled test on Knowledge-Centric Self-Improvement, and no source yet runs a loop that re-issues its own brief.

Sources#

• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents — Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoit Dumoulin, Cihang Xie, Yuyin Zhou, Suhang Wang & Hanqing Lu (17 authors; The Pennsylvania State University, UC Santa Cruz, Emory University, Northeastern University, UIUC, Amazon), Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents, arXiv 2605.30621, 2026-05-28, 12pp + appendices, empirical. The primary behind the survey summary below, read 2026-09-18. Cited here for §3's harness-updating / harness-benefit formalism and the in-situ scoring rule; §4.1's setup (seven backbones, SWE-bench Verified / MCP-Atlas / SkillsBench, fixed prompts and evolution budget); §4.2 + Tables 5–6 the evolver-side flatness, the extreme-pairing control and the flink-query isomorphism case study; §4.3 + Table 1 the non-monotonic ∆benefit curve; §D.2's own concession that the SkillsBench low-base regime is noisy; and §B.4's writable-scope permissions. The activation and adherence half (Tables 2–3, §D.1, §D.3–D.4) is developed on Harness Activation and Adherence. COI: none — mixed academic and industry authorship, and no author affiliation ships any model in the evaluated set (all seven backbones are third-party). Parse verdict: clean, and unusually so for this corpus: every automated check passed at ingest (0 collapse / 0 shift / 0 split-row / 0 weld), Tables 1, 2 and 3 were verified cell-for-cell against pdftotext -layout, and the one canary-recall soft warning is a confirmed false positive (docling normalized "3–6 tool calls" to "3 - 6"). Tables 4–6 were not page-audited; Table 5's twenty-one ∆update cells were instead re-derived arithmetically from its own pass rates and all twenty-one reproduce, and its anchor columns agree with Table 7 cell-for-cell. Figures 4 and 7 read under the image two-pass rule, both carrying content the prose omits (skill lengths; the paired passing trajectories). Limits: no budget-matched baseline against any non-evolution use of the same compute, and no compute accounting at all — β, the turn limit, rollout counts, tokens and cost are named and never valued; the activation/adherence diagnosis is SkillsBench-only; HFR and the phase-adherence scores are Sonnet-4.6-judge outputs with no agreement statistics; the model set is pre-Claude-5, so the strong-tier ceiling is a 2026-05 ceiling; and the abstract's code-availability sentence reads "publicly available at here" with no URL behind it in the PDF

• The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement — Duan, Liu, Tang, Chen, Zhou et al. (35 authors; SJTU / Theseus Labs / Tsinghua / ByteDance / ModelBest / Xiaohongshu / Shanghai AI Lab / Humanlaya / Agent-Native Research Lab / Frontis.AI), arXiv 2609.11873, 2026-09-10, 79pp (practitioner-opinion). Cited here for §3.5.2's summary of Harness Updating Is Not Harness Benefit (the update-quality versus execution-benefit decomposition, the two failure modes, and the "capability investment belongs in the task-solving agent" consequence), §3.3.2's L2 harness-search roll-call (Self-Harness, Agentic Harness Engineering, ADAS, AFlow, AgentSquare, Microsoft Foundry Agent Optimizer), §3.5's HarnessDev and Evo-Harness protocols, and §3.3.4's adaptive-benchmark warning. This is a secondary source for every one of those results — none of the underlying papers is in the vault, so the findings above are the survey's characterizations, not readings of the primaries. Coverage gap: does not cite HarnessBank, DarwinX or Rethinking the Evaluation of Harness Evolution for Agents. COI on its §5 industry cases (six author affiliations report their own systems) does not bear on the sections cited here, which are literature review. Full treatment on RSI Autonomy Levels (B0–L5)

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Patwardhan et al. (19 authors, OpenAI), arXiv 2510.04374 v1, 2025-10-05, 29pp (empirical). Cited here only for §3.4 and Figure 9: the reasoning-effort ladder (GPT-5 low 32.7% → high 38.8%; o3 29.8% → 34.1%), the prompt-tuning arm (38.8% → 43.1% human-preference win-or-tie, with the multimodal self-inspection and formatting-error deltas), and the scaffolding changes (best-of-N=4 with a GPT-5 judge, GET requests in the container). The prompt itself is reproduced in full in Appendix A.3. Vendor-authored; full treatment and COI on GDPval Benchmark

• Sidekick's continual learning loop — Andrew McNamara & Cody Mazza-Anthony, Sidekick's continual learning loop, Shopify Engineering, 2026-08-05, case-study (first-party account of the authors' own production system; no controlled arm, no held-out split, no gain figure for the harness stage in isolation). Cited for the sixth-instance section only: the "optimization target is the entire harness" framing, the program.md autoresearch config and its editable prompts/ / tools/ / harness/ scope, the propose-evaluate-keep-or-discard loop with a separately-calibrated judge as the metric the optimizer cannot edit, and the plateau statement ("once harness improvements plateau, we begin optimizing in parameter space") — which is an assertion with no score, round count or spend attached. Full source treatment on Agent Quality Flywheel

• Recursive Self Improvement for Coding Agents — Ara Khan, Cline blog, 2026-07-24 (case-study): the five-experiment walkthrough with commit hashes, baseline/candidate/confirmation scores and costs, the Jan-2026 four-engineer contrast, the ~$680/~1B-token accounting, the anti-reward-hacking prompt clause, and the abandoned Fable 5 arm. Vendor-run benchmark on the vendor's own harness; scores self-reported, not third-party replicated

• Rethinking the Evaluation of Harness Evolution for Agents — Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi & Teng Xiao (Allen Institute for AI / University of Washington, arXiv 2607.12227, 2026-07-14, empirical, 13pp; code at github.com/rethinking-harness-evolution): the abstract's two methodological charges (matched feedback and inference budgets; search and evaluation sharing one benchmark); §3.2–3.5 the four-method budget formalism including harness scaling as the paper's own instance-guided control; §4.1 the setup (Terminal-Bench 2.1's 89 tasks, three models, K = 5, m = 1, 128k generation budget, high reasoning effort, two runs averaged); §4.2 + Table 1 the no-unit-test results; §4.3 + Table 2 the unit-test results and the pass@1-versus-pass@5 inference; §4.4 + Table 3 the 45/10/34 generalization split; §5.1 the memorize-versus-distill diagnosis and the context-bloat note; §5.2 the two conditions a fair harness-evolution benchmark must satisfy; Appendix A.1 the minimal seed harness (one bash tool), A.3 the disabled AHE explore agent, A.5 scoring infrastructure exceptions as failures; Appendix B the per-task harness-scaling case studies. Tables 1–3 verified exact against the PDF at ingest — no collapse, no shift — so this document's tables are citable; Figure 1 read per the image two-pass rule and independently reproduces Table 1's Average column (72.3 / 69.3 / 67.4 / 71.8 against a 68.2 baseline). Limits: single benchmark, two runs per cell with no variance, error bars or significance tests reported anywhere, harness evolution instantiated only as AHE with its explore agent disabled (HarnessBank was not run), and the authors' own §5.2 concedes Terminal-Bench may be a poor instrument for the question

• DarwinX: Evolving Agent Harnesses Through Natural Selection — Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, Ran Xu & Zeyuan Chen (Salesforce AI Research / Salesforce Agentforce, arXiv 2608.07545, 2026-07-31, empirical, 33pp / 14 tables / 10 figures): §2 the preserve-and-extend contract, the fitness enabler (g > 0, R ≤ δ), parent selection by cumulative lineage gain, the merge operator's union-coverage acceptance rule, and the three learning signals; §3 the four-regime evaluation ladder and §3.2's own statement that "public leaderboard rows use different models and effort settings, so they provide context rather than controlled comparison"; §4 + Table 2 the TB2.1 leaderboard rows; §4.1 the self-described "effort-controlled comparison" and Figure 5's per-task compute medians; §4.2 the reward-hacking audit (370 rewarded trajectories, two flagged, one confirmed shortcut); §5 + Table 3 TerminalWorld held-out and §5.1 the 31.7-point proxy/truth gap and the McNemar caveats; §6 + Tables 4–5 WebArena-Infinity, the two-stage validity detector and the 293 → 17 invalid-trajectory drop; §7 the SWE-bench Verified transfer; §8 + Table 6 the seven-skill verification/artifact-contract bundle; §9 the limitations, including "inference effort [is] not independently randomized"; Appendix B Table 8 the per-benchmark protocol table (no compute column); Appendix C Table 9 the retry sweep and the pre-TW reference that also reaches 28/41; Appendix D.1/D.3 the synthetic-intent construction and baseline provenance; Appendix E the unmeasured harness-across-model-generations question. COI: Salesforce evaluating Salesforce's own proprietary agent (Monet) — Cline's structural position with an empirical protocol on top; the WAI validity audit is applied only to the authors' own trajectories, which they argue (correctly) makes the external comparison conservative for them. Parse verdict: clean. Tables 1–5 were spot-checked cell-for-cell against pdftotext -f <page> -layout at ingest — captions above all of them, zero collapse, zero shift; Tables 6–14 rest on the automated pass (0 collapse / 0 shift) plus canary-recall 20/20, recall 1.00, and no page was dropped. The only artifact is cosmetic decimal splatter in prose ("2. 2-6. 0-point"), not in any cell. Figures 5 and 6 read under the image two-pass rule and both carry numbers the prose omits: Figure 5's already-solved token medians (125K → 172K) against prose that says compute "barely moves," and Figure 6's specialist bars showing two of four specialists at or below the unevolved base. Limits: single lab, preprint; no effort tier defined anywhere and no dollar cost; TB2.1 reported in-domain; TerminalWorld at n = 41 with McNemar p = 0.45 (vs base) and p = 1.0 (vs Claude Code); no matched-model base arm on SWE-V; no ablation of the archive, the parent selector, the merge operator or the effort setting; the "about 17 points on average" abstract figure is a mean over four heterogeneous benchmarks with different metrics and is not a quantity this wiki cites

• Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution — Razzhigaev, Gritsaev, Kaznacheev, Dragunov, Yampolskiy & Kuznetsov (MSU / Skoltech / Joi Lab / AIRI, arXiv 2608.08311, 2026-08-08, case-study — downgraded from empirical at compile): §3 the commit pipeline and the two modes of core evolution, §4 the 161-day deployment and its two worked evolution traces, Table 4's counters, Appendix C's evolution off scaffold disclosures. Total author COI; baselines cited rather than re-run except on SWE-bench Pro, where the head-to-head is a null (58.2% vs Codex 59.4%, McNemar p = 0.40 on 655 paired tasks after a symmetric contamination filter). Parse warnings (Table 2 collapsed, Table 4 row-shifted past a clean checker verdict) and full treatment on Continuous Self-Modification Under Review

• Knowledge-Centric Self-Improvement — Wang, Yoon, Qu, Wang, Sehgal, Mazumdar & Yue (Caltech, arXiv 2607.19592, 2026-07-21, empirical): §4.4 held-out knowledge transfer (the positive-transfer counterpart to HarnessBank's §4.6 negative), §4.2 the second independent verdict on GEPA, §4.1 the cost comparison against DGM and HyperAgents. Full treatment and parse notes on Knowledge-Centric Self-Improvement

• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution — Luo, Xue, Wang, Hu & Deng (EverMind AI / Shanda Group, arXiv 2607.13683, v1 2026-07-15 / v2 2026-07-30, empirical): §3.2 the semantic gene bank and quality-biased parent selection, §3.3 Gated Harness Screening, §4.2–4.4 the seven-benchmark table and credited held-out gains, §4.3 the GEPA/DGM comparison under a matched rollout budget, §4.5 the LLM-hypothesis caveat on pathology labels, §4.6 the cross-model matching law and the 16× token-budget control, §4.7 the paired-2σ ablation. Tables 1–3 reconciled against the prose on 2026-08-03 (parse clean, no collapse or shift); Figures 3–4 read from the page images. Preprint, single lab, code not released at ingest ("publicly available upon acceptance"), and both baselines were re-run by these authors under their own protocol

• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents — Guo, Cao, Yuan, Wang, Wang & Wang (Chinese Academy of Sciences, arXiv 2607.24300, 2026-07-27, empirical): Finding 1 (self-authored verification loses deployment meaning: 35/35 cells above 0.70 self-score, 15/35 below random reference), Finding 2 (the endo-gate and leaky-anchor arms, the compute-matched pilot separating rollback from exogeneity, and the Breakout trajectory trace). Parse warning: its Table 7 is collapsed and fragmented and is not quotable — cross-game claims come from Finding 3's prose (2026-09-07: since hand-rebuilt in the raw against the PDF). Full treatment and the remaining parse notes on Optimizer–Evaluator Decoupling

• How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes — McManus, Ran & Weight (Bridgewater Associates), LangChain channel, 2026-07-24, 25:44 talk, case-study. The seventh instance (15:24–18:08): agent-written failing benchmark, iterate on context repositories and harness until it passes, regression-check the suite, open a PR to Slack. No efficacy number of any kind — admitted for mechanism only. See Bridgewater Associates and Production-Sourced Evaluation

• Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails — Zhou Yu, Bin Bi, Shiva Kumar Pentyala, Shubham Mehrotra, Sougata Chaudhuri, Shilpa Bhagavath, Zeyuan Chen, Ran Xu, Phil Mui, James Zhu & Sitaram Asur (Salesforce AI Research), arXiv 2609.09134, 2026-09-08, 10pp, empirical. The eighth instance: §3.2 the upward-transfer result (evolved-harness expert 84.4%→93.6%, edit-adoption 93.6–100%); §3.3 + Table 1 the imitation regression (78.0%→63.1%, -14.9 avg, -4.2 to -29.9 per task) and §3.3.1's baseline-harness ablation (29.2%→35.5%, +6.3); §3.3.2 the Gemma replication on Webarena (55.6%→41.1%, below the 46.7% default-harness floor); §3.3.3 + Table 3 (clean parse, reconciled against prose) the failure-composition split (planning 1.1%→14.6% under imitation vs 1.1%→1.8% under correction; knowledge 46.2%→44.5%/43.2%) and the two traced case studies (Appendix D: the 78-step unfinished payroll rollout, the unfiltered-total Webarena answer); §3.4 the on-policy correction pipeline and its result (79.7%, +1.7, gains on 5/7 tasks); Appendix A the per-task harness-adaptation table; Appendix B the LoRA-SFT and correction-data recipes. Table 1 is severely parse-damaged (cells run together into unparseable fraction strings, e.g. "13. 9. 00 67. 5 ± 1. 03") and is not cited anywhere on this page — every Table 1 figure quoted above (the 29.2/78.0/84.4/93.6/63.1/35.5/79.7 headline numbers and every per-task delta) is instead the paper's own prose restatement of that table, cross-checked sentence-by-sentence against §3.2–3.4. Table 2 (harness-edit adoption) has a row-label weld — the gemini-3.1-pro-preview row's label runs into the next row's "qwen3-coder-30b-a3b" text — but its cell values are internally consistent and the only two figures drawn from it here (94.2%, 30.8%, 76.1% on the domain-computation column) are independently confirmed by prose in §3.2 and §3.3.3. Figure 1 (the co-evolution loop diagram) was not opened — its caption fully restates the loop already described in the abstract and §1, with no additional quantitative content. COI: all eleven authors are Salesforce AI Research — Salesforce evaluating a task suite and search framework (Yang et al. 2026, not in the vault) on Salesforce's own harness-evolution pipeline, the same structural position as DarwinX. Limits: single lab, preprint; no budget-matched baseline against spending the evolution or SFT compute on sampling instead (Large-Scale Test-Time Compute's charge is untouched here); the on-policy-correction fix is tested on domain-knowledge harness edits only, not on HarnessBank's solver-fitted reasoning-lever class; and the paper reports no held-out task split — every number is in-domain on the same seven-task suite the harness was evolved against

• RRSI: Regularized Recursive Self-Improvement of Agent Harnesses — Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister & Chen-Yu Lee (Google Cloud AI Research / UNC-Chapel Hill / Stanford University / Washington University in St. Louis), RRSI: Regularized Recursive Self-Improvement of Agent Harnesses, arXiv 2609.24972, 2026-09-21, 24pp docling parse (7 tables, 5 pictures), empirical. The ninth instance: §3.1–3.3 the proposal-side regularizers (annealed L0-style edit-budget cosine schedule Eq. 4, evidence-aware credit assignment, structured exploration during stalls) and selection-side regularizers (leakage screening, noise-adjusted stability floor Eq. 5, Ridge/L2-style cost-gated acceptance Eq. 7, Lasso/L1-style structural pruning); §4.1 the eight-benchmark/three-domain setup and the four matched-budget baselines (Meta-Harness, AHE, TTHE, HarnessX); §4.2 + Figure 3 + Table 1 the main results and the controlled agentic-workspace comparison; §4.3 + Table 2 the proposal/acceptance ablation, Table 3 the two-policy robustness check, Table 4 the cross-model transfer to an unseen smaller policy, and Figure 4 the cost comparison; Appendix D.1 + Table 5 the full per-domain hyperparameter table. COI: none — Google Cloud AI Research and academic co-authors, evaluated with Claude Opus 4.8 (Anthropic) and Gemini 3.5 Flash / 3.1 Flash Lite (Google DeepMind) as policies, on entirely third-party benchmarks (Harvey AI, Merrill et al.'s Terminal-Bench, Jimenez et al.'s SWE-bench, Guo et al.'s EngDesign, Chi et al.'s Frontier-Eng); the proposer/critic/analyst are a competitor's model, not Google's own. Parse verdict: Tables 1–5 read clean and were cross-checked against the prose restatements in §4.2–4.3 with no discrepancies; Table 6 (Appendix E, the qualitative case study) carries a soft table-collapse flag from ingest and is quoted here only for the pattern its surrounding prose narrates, not for individual cell values. Limits: single-lab preprint, not third-party replicated; no significance testing, error bars or confidence intervals on any final reported number, despite an internal noise band δ gating candidate selection during search; no dollar cost reported anywhere, only tokens and steps; frozen-backbone scope only (no setting where weights update during evolution); no budget-matched test-time-scaling baseline (its four baselines are matched to each other's candidate budget, not to a parallel-sampling or sequential-refinement arm); own limitations section concedes hyperparameter sensitivity and that broader validation across different agent architectures and longer-running processes is needed