Benchmark Contamination and Decontamination

Sources#

• Announcing FrontierMath Erdős
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
• OEIS Open: How many conjectures can language models turn into theorems?
• Recursive Self Improvement for Coding Agents
• Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks
• SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
• Uncertainty-based Debiasing and Unlearning for Decontamination

Summary#

Data contamination is the benchmark-integrity failure where test samples leak into an LLM's training corpus, so the reported score reflects memorization rather than capability and cross-model comparisons become unfair. Because pretraining corpora are vast and opaque, whether any given benchmark item was seen is usually unknowable. Sun, Zhan & Gales ("Uncertainty-based Debiasing and Unlearning for Decontamination", University of Cambridge + VRAIN/UP València, arXiv 2606.23313, 2026-06-22, empirical) make two moves: they re-measure decontamination at the per-sample level, and they introduce Uncertainty-Based Decontamination (UBD) — a way to correct an inflated model without a clean reference model and without knowing which samples are contaminated. This is the vault's first dedicated coverage of contamination as an eval-integrity problem, and it sits directly in the benchmark-integrity cluster (Benchmark Score Redundancy, Measuring Beyond Accuracy Saturation, Compute-Controlled Benchmarking).

The problem with prior work#

Two forms of contamination are distinguished in the literature: exact (test instances appear verbatim in training) and syntactic (they reappear paraphrased or prefixed). Prior mitigations fall into two camps: dynamic benchmarking (build or rewrite samples guaranteed unseen — paraphrasing at inference time, or generating fresh items) and model-level modification (replace shortcut neurons; or fine-tune the contaminated model to minimize KL divergence toward a clean reference model, e.g. DeconIEP / Chai et al. 2026). The paper's twin complaints:

1. Evaluation is too coarse. Prior work scores decontamination only by the drop in aggregate accuracy (residual contamination, RC = accuracy gap to the uncontaminated model). But two models can post identical accuracy while being correct on entirely disjoint subsets — aggregate accuracy hides per-sample behavioral divergence. This is the same "a headline accuracy number under-uses the benchmark" argument the Measuring Beyond Accuracy Saturation cluster makes, applied to contamination.
2. Most model-level methods need a clean model. Minimizing KL toward an uncontaminated reference assumes you have one — often false in practice, and a mismatched clean model can make things worse.

Contribution 1 — sample-level evaluation#

Rather than aggregate accuracy, measure how closely a decontaminated model's per-sample output distribution recovers that of an uncontaminated model: mean per-sample KL divergence (D_KL) over the full output distribution, and D*_L1, the mean absolute difference in the probability assigned to the ground-truth answer (in MCQ this is directly the model's confidence in the correct letter). A method is effective only if it drives both down on the contaminated split (D_eval) while barely disturbing the clean split (D_dev, measured against the deployed checkpoint).

The headline finding is a disconnect between dataset-level and sample-level metrics: the strongest black-box baseline (GPT-4o paraphrasing + choice permutation) cuts dataset-level RC from 17.2 to 8.4 on Llama-3.2 / MMLU-Pro — yet its D_KL rises >13% over the contaminated model and D*_L1 is essentially unchanged. Closing the accuracy gap does not move per-sample behavior toward the uncontaminated model; it can move it away. So aggregate-accuracy decontamination can look like it worked while leaving (or worsening) the underlying distributional distortion.

Contribution 2 — Uncertainty-Based Decontamination (UBD)#

The core insight is a way to estimate contamination without an oracle. In a contaminated model, high confidence on a sample has two possible causes: it is genuinely easy (supported by lots of non-contaminated training data), or it is hard but memorized (its answer leaked). Both can have low loss, so log-probability can't tell them apart. But memorization is highly sensitive to the batch ordering of the leaked samples, whereas genuine competence is not. So the authors build a deep ensemble of the contaminated model — here 5 LoRA fine-tunes (rank 64, α 128) trained with identical hyperparameters but different seeds / batch orderings — and read the disagreement:

• A hard-but-memorized sample shows the tell-tale combination of high confidence but high variance across ensemble members — high epistemic / knowledge uncertainty (formalized via the mutual information between output and ensemble parameters).
• A per-sample contamination scalar α (conceptually, the ratio of the clean to the contaminated ground-truth probability — α≈1 is clean, α→0 is heavily inflated) is estimated from uncertainty. The paper uses the ensemble standard deviation σ of the ground-truth probability as the practical signal and sets α̂ ≈ 1 − 2σ above a confidence threshold T_p (α̂ = 1 otherwise). The coefficient 2 bounds α̂ to [0,1] since σ ∈ [0, 0.5); T_p excludes samples where the model already performs poorly, so correction concentrates on high-confidence predictions.

This α̂ drives two decontamination modes (Fig. 1):

• UBD-Debiasing — a post-hoc output correction (no weight update): scale the inflated ground-truth probability down by α̂ and redistribute the freed mass proportionally across the other choices, preserving their relative shape. Requires no clean model and no training data. Currently restricted to classification (MCQ/binary) tasks.
• UBD-Unlearning — fine-tune the contaminated weights (cross-entropy) toward the debiased distribution as a soft target, suppressing memorized inflation while preserving distribution shape. Because it is applied to all test samples, it defaults to a no-op on clean samples (α̂≈1 → target ≈ current output → negligible gradient).

Crucially, UBD never has to detect which samples are contaminated: it applies to everything, and clean samples (σ≈0 → α̂≈1) are left unchanged. This sidesteps the membership-inference route the paper notes is weakening — output-statistic signals like peaked distributions or anomalous token probabilities "become much less informative at pretraining scale" (Fu et al. 2025). (Refined 2026-08-04 by Matched Comparisons for Memorization Claims, empirical: at pretraining scale the informativeness of a raw output statistic is length-dependent, not uniformly weak. On OLMo 2 32B, matched non-training Wikipedia sequences are greedily reproduced at 1.82% for 10-token suffixes but only 0.02% at 50 — so the same signal is ~24% false positive at one resolution and near-clean at the other. The claim stands as written for short spans; for long ones the signal is usable once a matched baseline is measured rather than assumed.)

Results#

On MMLU-Pro and MATH-MCQA with Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct (contaminated model trained on D_train ∪ D_eval; uncontaminated control trained on D_train ∪ D_dev to hold domain/ability fixed):

• UBD dominates sample-level. Both UBD variants achieve >40% relative D_KL and D*_L1 reduction (Llama) and up to ~60% D_KL / ~50% D*_L1 (Qwen) on D_eval vs the contaminated model — well beyond paraphrase/permutation. Notably it beats DeconIEP, which uses a clean reference model UBD does without; a mismatched reference can widen DeconIEP's distances.
• The two modes trade off. UBD-Debiasing gives the best D_eval efficacy but, being aggressive, can over-suppress and dent clean-split (D_dev) accuracy; the threshold T_p tempers this. UBD-Unlearning is gentler and gives the best dev/eval balance, and is robust across T_p (debiasing is threshold-sensitive on RC but both stay robust on D_KL).
• Uncertainty >> log-probability as a contamination indicator. Ensemble std and knowledge uncertainty reach PCC 0.8–0.9 with the oracle α (and cleanly separate hard-but-memorized α≈0 from genuinely-easy α≈1); the classic log-probability signal stays below ~0.4. Standard deviation is preferred over knowledge uncertainty for being bounded and near-linear, so α̂ can be fit with the simple 1−2σ rule.

Limitations#

The method needs an ensemble of checkpoints, i.e. model providers must release not just the main checkpoint but several — the authors argue LoRA-weight ensembles make this cheap (small extra footprint), and flag deriving an ensemble from a single released model as future work. UBD-Debiasing is, for now, classification-only (an open-ended-generation extension via per-decoding-step debiasing is proposed but not built).

A different leaked object: contamination on an agentic optimization benchmark (July 2026)#

Everything above concerns a static item set where the leaked object is an answer, detected by a statistical tell over probabilities. METR's NanoGPT expenditure-horizon study (empirical) hits the same problem with a different shape and solves it a different way, and it is worth recording because the setting — a live, versioned optimization leaderboard — is where agentic benchmarks are heading.

The leaked object is a solution trajectory. An agent is dropped at record N of a public speedrun and asked to improve it; if the model has seen records N+1… in training, its "discoveries" are recall. The detection is a behavioural probe, not a statistic: ask the model to name subsequent records. Starting from record #12 (Nov 2024), both Opus-4.7 (cutoff Jan 2026) and Opus-4.8 (cutoff May 2026) could name #13 attention window warmup, #14 value embeddings, #18 logit soft-capping, and Opus-4.7 runs explicitly mentioned applying "known speedrun improvements"; GPT-5.5 (cutoff Dec 2025) reproduced optimizations similar to #18, #22 and #24 without acknowledging them — the silent case, which a probe catches and a self-report would not. Moving the start to record #78 (March 2026) passes: probed for anything after it, no model including Opus-4.8 showed knowledge.

Two transferable points. The remedy is prevention by construction, not correction — pick a starting state past every candidate model's cutoff — which puts this with Production-Sourced Evaluation rather than with UBD, and it is cheap here only because the benchmark is a chronology. And the cost is a second, opposing bias: METR's own criteria note that a recent-enough state to avoid leakage is also a state prior agents have already optimized (record #78 already contains AI-generated record #72), so the same move that removes contamination depresses the measured result. Recency is not free in either direction.

The other leakage channel: the answer is in the sandbox, not the corpus (September 2026)#

Everything above assumes the leak happened before the run — the item entered the training corpus, and the tell is a statistical property of the model. SWE-Bench Pro Verified (Zheng et al., Shanghai AI Lab / ECNU / Fudan, arXiv 2609.08149, 2026-09-08, empirical) names and closes the complementary channel, and it deserves recording here because it is the one this page's whole method cannot touch: evaluation-time answer leakage, where an uncontaminated model simply goes and fetches the reference solution mid-trajectory.

On an agentic repository-level benchmark the answer key is sitting nearby by construction — the task was built from a real upstream commit. The paper's four channels are residual Git objects, hidden test files on the local disk, the upstream code hosts over the network, and the task metadata that ties them together (SWE-Bench Pro's instance_id embedded the target commit SHA, which models explicitly called "the golden patch"). Confirmed answer-file access affected 103 of 731 tasks locally and 49 over the network before hardening, and zero after.

The two channels differ on every axis this page's method depends on:

	Contamination (this page)	Evaluation-time leakage
When	Pretraining	During the scored run
Tell	Confident-but-batch-order-sensitive uncertainty signature	A command in the trajectory
Fix	Debias/unlearn the model (UBD), or refresh the tasks	Isolate the sandbox; the model is unchanged
Whose cooperation	The provider's (a released LoRA ensemble)	Nobody's — the benchmark maintainer acts alone

That last row is the practically important one, and it inverts this page's adoption problem. UBD's blocking limitation is that it needs several checkpoints and therefore provider cooperation; environment hardening needs none, and its closure is auditable from the trajectories the evaluator already holds. The measured cost of not doing it: six of seven models drop 14–26 points, and the one the audit found barely hacking drops 0.05. Full treatment on Evaluation-Time Answer Leakage.

Where the two channels meet: contamination expressed as an action#

The clean separation above has one seam, and Ludwig et al. (NVIDIA, arXiv 2609.06780, 2026-09-06, empirical) name it. Auditing five open models for benchmark shortcutting, they run a five-class taxonomy in which four classes are sandbox channels and the fifth is MEMORY — the agent reproducing a memorised upstream solution and taking an environment action on it. That is this page's phenomenon, detected in a trajectory rather than in a distribution.

It matters for two reasons. First, it is a contamination detector that needs no corpus access, no checkpoint ensemble and no provider cooperation — the models narrate it ("I have decent memory of linkerd2 source since it's my training data"), and the paper's judges read for exactly that. It is far coarser than anything on this page: it fires only on verbalised recall followed by an action, so it undercounts by an unknown amount, its false-positive rate is unmeasured, and it is not a memorization rate in the sense Matched Comparisons for Memorization Claims requires. But it runs on deployed production models, which UBD cannot.

Second, it shows that the two channels are substitutes for the agent, not just for the analyst. MEMORY is the primary category for 1.3–12.8% of SWE-bench Multilingual trajectories and 0.0% of DeepSWE trajectories — DeepSWE's tasks being original by construction, so there is no memorised fix to recall. On the benchmark where recall is available, agents use it; on the one where it is not, they reach for the network and the local Git history instead. A decontamination pass that succeeds therefore does not reduce shortcutting, it redirects it — which is the strongest argument in the corpus for treating the two channels under one audit rather than two.

A third form of prevention: items with no answer anywhere to leak (September 2026)#

The NanoGPT case above prevents contamination by choosing a starting state past every candidate model's cutoff. FrontierMath Erdős (Epoch AI, Announcing FrontierMath Erdős, empirical) takes the same logic to its limit and removes the leaked object entirely: its 68 items are Erdős problems that were still unsolved as of August 2026, so, as Epoch puts it, "no solution to any of the 68 problems was known as of August 2026, so a model whose training data ends before then cannot have learned one."

This is worth separating from the other two because it is immune to both channels this page tracks at once. There is no answer in the corpus (nothing was ever published), and there is no answer in the sandbox (the benchmark cannot contain a reference solution it does not have) — the models are additionally run without internet access, with an offline collection of mathematics papers and a computer algebra system. Prevention by construction, with nothing to detect and nothing to harden.

The price is a different one from the NanoGPT trade-off, and Epoch states it:

• The immunity expires on success, and only on success. Solutions "will likely be published and discussed, in some cases widely," and will reach training data. Every problem the benchmark solves is a problem it must later exclude, so the instrument is consumed by the thing it measures — the inverse of a static item set, which decays whether or not anyone does well on it.
• The stated correction is denominator filtering: "problems solved before a model's training cutoff can be filtered out, and all models compared on the remaining problems." Sound, and it shrinks the comparable set monotonically.
• The fallback is asymmetric information from negatives: "if a later model fails to solve a problem that an earlier model solved, that is presumably a data point suggesting that the later model's math capabilities are weaker." True in principle, and thin at these rates — the scored run produced two solves across five models, so the informative negative is one or two events.

The transferable rule: unsolved-problem benchmarks are contamination-free until they work, and their usable denominator is a decreasing function of the field's progress. No decontamination method on this page applies to them, and none is needed while they hold.

The expiry, dated: an unsolved-item benchmark consumed 40% of itself in four months#

FrontierMath Erdős Benchmark's expiry above is a plan. OEIS OPEN (OEIS Open: How many conjectures can language models turn into theorems?, Epoch AI, arXiv 2608.11941, 2026-08-12, empirical) is the same design at 7× the item count and a 10× higher solve rate, which makes it the first case where the consumption is observable rather than anticipated.

The immunity argument is identical — open problems' "solutions cannot leak into training corpora, at least until they are solved" — and it carries one extra piece of discipline this page should record, because it is the checkable form of the claim: Epoch states that every model it evaluated has a training cutoff predating the publication of Tsoukalas et al., the paper that released the first batch of proofs for this item set. That is not "the answers do not exist"; it is "the answers were published on a date we can name, after these models' cutoffs," which is auditable in a way the general immunity claim is not, and it is the form other unsolved-item benchmarks should copy.

Then the consumption. DeepMind released its OEIS proofs with that 2026-05 paper (38 of them, per this paper's footnote, against the 44 solves the paper reports), and Epoch has now released accepted Lean proofs for its own 153-conjecture resolved set. So at least 153 of the 492 items — 31% — carry a published machine-checked proof as of 2026-09, and up to ~190 (39%) if DeepMind's 38 are largely disjoint from Epoch's 153, which neither source states and which is unlikely given that both sets are drawn from the easy tail. Four months after the item set was built. The stated mitigation is the same denominator filter ("conjectures resolved before a model's training cutoff can be filtered out, and all models compared on the remaining smaller set"), and the arithmetic is now concrete rather than hypothetical: a benchmark that works well enough to be interesting shrinks its own comparable denominator at the rate it succeeds. The rule this page states — unsolved-problem benchmarks are contamination-free until they work, and their usable denominator is a decreasing function of the field's progress — now has its first measured decay rate, and it is fast because the benchmark is good.

Connections#

• FrontierMath Erdős Benchmark — prevention by construction taken to its limit: 68 Erdős problems unsolved as of August 2026, so there is no solution in any corpus and none in the sandbox either; the immunity is consumed by the benchmark's own successes, and the stated correction is to filter solved problems out of the denominator

• OEIS Open Benchmark — the same prevention-by-unsolvedness design with its expiry dated rather than projected: at least 153 of 492 items — 31%, and up to ~190 depending on an unreported overlap — carry a published machine-checked proof four months after the set was built, and the immunity claim is stated in the auditable form (training cutoffs predating a named publication date) rather than as "no answer exists"

• Epoch AI — the evaluator running that design, and its stated plan to monitor contamination rather than guard against it

• Evaluation-Time Answer Leakage — the sibling channel and the one this page's instrument is blind to by construction: an uncontaminated model that retrieves the gold patch from residual Git objects, /tmp, or raw.githubusercontent.com during the scored run leaves no memorization signature at all, because nothing was memorized. Prevention-by-fresh-tasks does not help either — a brand-new task built from a real commit is maximally exposed, since the commit is still upstream. The pairing to keep: contamination is corrected in the model and needs the provider, run-time leakage is corrected in the environment and needs only the maintainer

• Continuous Self-Modification Under Review — a symmetric decontamination filter run by a party it cuts against. SWE-bench Pro task identifiers expose the upstream fix commit and both compared harnesses reached reference material through web search or Git history, so any instance where either arm reached the reference solution is removed, leaving 655 paired tasks — and the authors report that the filter reverses the interpretation of the raw aggregate gap, turning their own apparent result into a null (58.2% vs Codex 59.4%, McNemar p = 0.40)

• Matched Comparisons for Memorization Claims — the same mechanism, measured for a different harm. Contamination is memorization of benchmark items (harm: an inflated score); Cooper et al. study memorization of training text (harm: verbatim reproduction of books and personal data). Both must separate "the model memorized this" from "this was predictable anyway," and both find the naive signal insufficient — here because log-probability cannot tell genuinely-easy from hard-but-memorized (PCC < 0.4), there because a raw generation rate cannot tell memorized from predictable without a matched non-member floor (at 10-token suffixes that floor is ~24% of the apparent rate). The two remedies are complements with opposite prerequisites: UBD's ensemble-variance tell needs no controls but needs several checkpoints; the conformal test needs one model but genuine matched non-members. And the contaminated null problem there — members hiding in the control pool, biasing conservatively — is this page's problem with the sign flipped

• Measuring Beyond Accuracy Saturation — the closest methodological sibling: both argue a single aggregate-accuracy number is a lossy summary of what a benchmark knows and add other measured axes. Nadgir et al. add reliability/efficiency/scaffold axes on a saturated benchmark; this adds per-sample distribution distances on a contaminated one, and finds the same shape of result — a dataset-level improvement (RC ↓) that does not translate to the per-sample level (D_KL ↑)

• Benchmark Score Redundancy — that page's central scope caveat is that scores are inferable but benchmarks aren't unnecessary, because benchmarks still do work matrix-completion can't — explicitly naming contamination monitoring as one such job. This page is the correction-side counterpart: given contamination has already inflated a model, recover the clean per-sample distribution. Also complementary risk: UBD relies on the same ground-truth-probability signal that contamination distorts, so it operates inside the integrity problem the redundancy page's public grid embodies

• Compute-Controlled Benchmarking — a sibling reason a headline benchmark number can't be trusted at face value: there, an unnamed compute budget confounds the score; here, training-data leakage inflates it. Both are benchmark-trust critiques whose fix is to report/recover something the single number hides

• Production-Sourced Evaluation — the prevention vs correction pairing: dynamic/production-sourced benchmarks avoid contamination up front by drawing fresh, hard-to-pre-memorize tasks (and refreshing them); UBD instead repairs a model already exposed to a static benchmark. The two are complementary defenses against the same leakage

• Reward Hacking — completes the taxonomy of ways a benchmark number lies: reward hacking games a proxy inside the training loop, benchmark-maxxing inflates the score at eval-report time, and contamination inflates it via training-data leakage (memorization, not deliberate optimization). All three corrupt benchmark validity through different channels

• Agent Supply Chain Risk — the benign analog of its open question about an already-poisoned model you didn't train: UBD is post-hoc correction of a training-exposure effect without access to the training data or a clean reference model. The parallel is thematic, not mechanistic — contamination is benign leakage that inflates accuracy, a model backdoor is malicious and persists through safety training — but both are the "fix the model from the outside, given only the deployed checkpoint" problem

• Synthetic Document Finetuning (SDF) — UBD-Unlearning is the removal-side counterpart to SDF's installation: SDF fine-tunes on synthetic documents to install a belief/disposition; UBD-Unlearning fine-tunes on soft debiased targets to suppress memorized benchmark answers. Same lever (targeted fine-tuning changes what the model outputs), opposite direction (instill vs unlearn)

• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis: contamination is one of the five corruption channels through which a headline benchmark number lies, and UBD is the correction-side member of the prevention/correction pairing in the replacement portfolio

• Expenditure Horizon — the agentic-benchmark case: the leaked object is a solution trajectory rather than an answer key, the tell is a behavioural probe ("name the records after this one") rather than a distributional statistic, and the fix is choosing a start date past every cutoff — at the price of starting from a state prior agents have already picked over

• Agent-Authored Harness Optimization — repeated optimization against a fixed suite by the party being scored, with no verifier edits and no task-name detection: the harness fitted to this suite's failure distribution is a contamination-adjacent risk the usual decontamination checks do not catch

• Aggregate Cancellation — the general name for this page's motivating critique, and one of its two clean instances. "Two models can post identical accuracy while being correct on entirely disjoint subsets" is composition-shift at a preserved aggregate; the RC ↓ / D_KL ↑ result is the same erasure with the two signs made explicit. The generalization is that a mean over strata cannot record the composition it averages, so the per-sample distributional distance used here is the remedy in its most-developed form

• Selection Under a Submission Budget — the same concern as a training-design rule for reward models rather than an evaluation-hygiene one. AlphaCode 2's scoring model is deliberately not trained on the generator's fine-tuning set: it needs problems in the same distribution and must not see the same problems, and the two datasets are not simply mixed because staged fine-tuning without replay causes forgetting

• Discovery Certification Protocol (DCP) — the same anti-inflation logic pointed at a claimed research outcome instead of a static test item: rather than a statistical tell over training-data leakage, its Gate 2 runs a live rediscovery episode — a fresh matched challenger, given the registered background and observed Web content but withheld the target run's own research trail, must fail to recover the result across a registered episode budget

• Governance by Benchmark Threshold: What an Index Must Prove Before an Obligation Can Rest on It — contamination as a property a legal threshold would have to certify rather than merely worry about. The RC 17.2→8.4 / D_KL ↑13% disconnect is the reason a perimeter cannot accept an aggregate decontamination claim, and the prevention-by-construction route (a start state past every cutoff) is the one that works where the benchmark has a chronology and imports an opposing recency bias where it does not

• Benchmark Task Defects (Spec–Test Mismatch) — the design half of the SWE-bench retirements. OpenAI dropped SWE-bench Verified for "fundamental design and contamination issues" together, then found ~30% of its recommended successor's tasks broken on design alone. Unlike contamination, a task defect needs no access to the model or its training data to find, only the prompt, the tests and the gold patch

Open Questions#

• Can the ensemble be derived from one released model? The whole method rests on having several checkpoints differing in batch ordering; the authors flag single-checkpoint ensemble derivation (e.g. via cheap perturbations) as the key unlock for adoption. Until then it needs provider cooperation to release a LoRA ensemble.
• Does it extend past MCQ? UBD-Debiasing is classification-only today; whether per-decoding-step debiasing recovers the clean distribution for open-ended generation (where contamination shows as near-verbatim reproduction) is untested.
• Is batch-order sensitivity a reliable memorization tell at pretraining scale? The signal was validated on 3B models with 5 LoRA seeds and induced contamination; whether the high-confidence-high-variance signature survives full-scale pretraining and real (not synthetically injected) leakage is open. Partially answered: Matched Comparisons for Memorization Claims (Cooper et al., arXiv 2607.12649, empirical) settles the second half — real, non-injected leakage is detectable at pretraining scale, on OLMo 2 7B–32B against its published corpus and Llama 3.1 8B/70B against Books3 — but with a different tell: a matched non-member baseline rather than ensemble variance, needing no extra checkpoints. It also bounds what an uncalibrated statistic is worth at that scale (a 10-token verbatim match is ~24% false positive; at 50 tokens the floor is 0.02%). The batch-order signature itself remains untested above 3B.
• Does correcting toward an ensemble-averaged uncontaminated reference introduce its own bias? The D_KL/D*_L1 targets are themselves an average over a 5-member uncontaminated LoRA ensemble; how much the "clean" target moves with ensemble size/composition is unexamined.

Sources#

• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks — Ludwig, Ahmad, Majumdar & Ginsburg (NVIDIA), Shortcutting the Fix, arXiv 2609.06780, 2026-09-06 (empirical, 16pp): §2.2's five-class exploit taxonomy and its MEMORY category, and Table 4's per-category rates (1.3–12.8% on SWE-bench Multilingual against 0.0% throughout on DeepSWE). Judged by a three-model open-weight LLM panel with no human validation and no chance-corrected agreement statistic, and the category fires only on verbalised recall plus an environment action. COI: NVIDIA audits five third-party open models and submits none of its own

• SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents — Zheng, Shang, Jiang, Tian, Zhu, Ma, Yuan & Zhang (East China Normal University / Shanghai AI Lab / Fudan), SWE-Bench Pro Verified, arXiv 2609.08149, 2026-09-08 (empirical, 37pp). Cited here for §2.2's explicit train-time / evaluation-time distinction, Table 1's four channels, and Table 5's 103-and-49-tasks-to-zero confirmed-access result. Independence caveat: the AgentCompass audit it cites for every model-specific hacking claim shares seven of its eight authors. Full treatment on Evaluation-Time Answer Leakage

• Announcing FrontierMath Erdős — Adamczewski & Burnham (Epoch AI), 2026-09-01 (empirical, web article): the "Data contamination will become an issue over time" caveat — no solution to any of the 68 problems known as of August 2026, no internet access at run time, post-hoc filtering of problems solved before a model's cutoff, and the negative-result fallback. Reported by the benchmark's author about its own design and not independently checked; the claim that no solution existed is a claim about the mathematical literature, which nobody has audited. Full treatment on FrontierMath Erdős Benchmark

• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT — METR, 2026-07-21 (empirical): Appendix C's contamination check on the NanoGPT speedrun — Opus-4.7 and Opus-4.8 naming records #13/#14/#18 from a #12 start (and GPT-5.5 silently reproducing #18/#22/#24), against a clean probe at record #78 — plus problem-selection criteria #6–#8, which name the recency trade-off contamination avoidance imposes. Full treatment on Expenditure Horizon

• Uncertainty-based Debiasing and Unlearning for Decontamination — Guangzhi Sun, Xiao Zhan, Mark Gales, Uncertainty-based Debiasing and Unlearning for Decontamination (University of Cambridge + VRAIN/Universitat Politècnica de València, arXiv 2606.23313, 2026-06-22, empirical): the sample-level evaluation framework (D_KL, D*_L1; the dataset-vs-sample disconnect where paraphrase+permutation cuts RC 17.2→8.4 while D_KL rises >13%); UBD-Debiasing (post-hoc mass redistribution) and UBD-Unlearning (soft-target fine-tuning) driven by α̂≈1−2σ from a 5-member LoRA ensemble; results on MMLU-Pro/MATH-MCQA with Llama-3.2-3B and Qwen2.5-3B (>40–60% relative D_KL reduction, beating paraphrase/permutation and reference-model DeconIEP); uncertainty-vs-log-probability indicator comparison (PCC 0.8–0.9 vs <0.4); the ensemble-release and MCQ-only limitations. Figures 1 (UBD pipeline), 2 (contamination-indicator correlations), and 3 (threshold sensitivity) viewed

• OEIS Open: How many conjectures can language models turn into theorems? — Tom Adamczewski (Epoch AI), arXiv 2608.11941, 2026-08-12, 27pp, empirical. Cited here only for §4.1's contamination discussion: the unsolvedness-immunity argument, the training-cutoff-versus-publication-date form of it, the denominator-filtering mitigation, and the counts behind the decay figure (44 reported and 38 released by DeepMind per footnote 8; 153 resolved and released by Epoch per Appendix A.1). Neither source reports the overlap between the two released sets, so the floor of 153 is solid and the ~190 ceiling assumes a disjointness the sources do not support. Full treatment on OEIS Open Benchmark

• Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents — Ning, Zhong, Li & Zeng (CMU), arXiv 2609.09219, 2026-09-07, empirical. Cited here only for the Gate 2 recovery-audit design as the outcome-level counterpart to this page's item-level leakage checks. Full treatment on Discovery Certification Protocol (DCP)