LLM-Judge Validation

Sources#

• A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model
• A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges
• Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
• An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
• Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
• SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization
• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks
• Sidekick's continual learning loop
• User awareness in frontier models
• What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Summary#

Reliability is not validity. A judge can be perfectly reproducible — return the same verdict run after run — and still be systematically wrong: chance-inflated, benchmark-fragile, or deterministically biased toward one answer position. Norman, Rivera & Hughes (UC Berkeley, arXiv 2606.19544, June 2026) ran the largest systematic LLM-as-a-judge evaluation to date — 21 judges from nine providers, three benchmarks (MT-Bench, JudgeBench, RewardBench), three protocols (agreement, consistency, bias-audit), 118 runs, ~541,000 individual judgments, all at temperature 0 over a five-week March–April 2026 window — and found that the way judges are validated in practice (headline exact-match agreement) systematically overstates how good they are. The paper's contribution is not a new judge but a validation discipline, packaged as a five-step Minimum Viable Validation Protocol.

This page is the vault's independent counterweight to DRACO's reassuring "rankings are judge-stable" finding: the two papers measure different invariances, and together they bound how far a judge score can be trusted.

Finding 1 — Kappa deflation (the headline metric lies)#

Practitioners report a judge's exact-match agreement with human labels ("85% agreement!"). That number does not correct for agreement expected by chance. Cohen's κ (and Krippendorff's α) does. The gap — kappa deflation Δκ = EM − κ — is large and universal:

• On MT-Bench, every one of the 21 judges shows Δκ ∈ [33.8, 41.3] pp, cohort mean 38.6 pp. Even the best chance-corrected judge, Gemini 3.1 Pro, posts EM = 0.849 but κ = 0.511 — a 33.8 pp gap. A judge reporting "85% agreement" on MT-Bench has κ ≈ 0.48 — moderate, not the near-perfect band the percentage suggests.
• The deflation tracks the benchmark's label distribution, not the judge's quality. Balanced ternary MT-Bench (A/B/Tie, chance ≈ 1/3) → 38.6 pp mean; JudgeBench (pairwise correctness) → 23.7 pp; binary chosen-vs-rejected RewardBench → 10.2 pp. Balanced labels raise expected-by-chance agreement, which inflates the raw–corrected gap exactly as Cohen's correction predicts. The deflation is a property of metric × benchmark, so exact-match figures used to justify deployment "overstate discriminative ability by an amount that depends on the benchmark, not the judge."

Fix: report κ or α as the headline reliability number, with exact match demoted to a secondary figure.

Finding 2 — Single-benchmark validation doesn't transfer#

Judge rankings are not portable across benchmarks. Validating a judge on one leaderboard tells you little about its standing on another:

• 11 of 21 judges shift ≥ 4 rank positions across the three benchmarks; the abstract's headline is a shift of up to 14 positions. The extreme case is Llama 3.3 70B: MT-Bench #5 → JudgeBench #20 (a collapse); the opposite direction is Minimax M2.7: MT #17 → JB #5 (a jump). Only Gemini 3.1 Pro and Claude Opus 4.6 hold a top-3 position on all three benchmarks.
• Two coupled drivers. First, benchmarks differ wildly in discriminability: MT-Bench compresses all 21 judges into a 13.5 pp κ band (0.376–0.511, ~0.6 pp between adjacent ranks), while JudgeBench spreads the same judges over 60.4 pp (0.271–0.875) — 4.5× wider. Where the band is compressed, tiny κ differences produce huge rank swings. This is the MT-Bench ceiling effect: its preference-style label set can't separate strong judges. Second, the three benchmarks measure different latent constructs — preference alignment (MT-Bench), objective correctness (JudgeBench), chosen-vs-rejected discrimination (RewardBench) — and a judge strong on one can collapse on another.

Fix: validate on ≥ 2 benchmarks spanning the preference-style ↔ correctness-style axis, not the discriminability of any single dataset.

The DRACO reconciliation (two different invariances)#

The vault's prior answer on judge trust came from DRACO: rankings are stable across judge models, absolute magnitudes vary → use ordinal comparisons, distrust cross-paper absolute scores. This paper does not contradict that — it measures a different axis of variation:

• DRACO fixes the tasks + rubric and varies the judge model → the ranking of systems-under-test holds (Gemini-3-Pro, GPT-5.2, Sonnet-4.5 agree on the order). Judge-model-invariant.
• This paper fixes the judge protocol and varies the benchmark → the ranking of judges themselves is fragile (up to 14 positions). Benchmark-variant.

So the practical rule sharpens: a ranking is trustworthy only across the axis you have actually verified it stable on. DRACO earns "use rankings" for judge choice; it says nothing about benchmark choice, which this paper shows is where rankings break. The reassurance and the warning are the same lesson seen from two sides.

A third axis — varying the judge's version rather than its identity or its benchmark — arrives below with Yang et al. (2026).

Finding 3 — The consistency–bias paradox (reliability masking invalidity)#

The paper's sharpest diagnostic. High test-retest reliability (> 0.95) coexists with severe position bias (> 0.10) in two production-deployed judges:

Judge	Test-retest	Position bias	JudgeBench κ
Qwen 3 8B	0.992 (highest in cohort)	0.192 (highest)	0.289 (3rd lowest)
Gemini 2.5 Flash	0.988	0.125	0.578

The mechanism: test-retest measures the stability of a judge's outputs, not the correctness of its decision process. Position bias and within-judge agreement are mathematically orthogonal — a judge that deterministically favors whichever answer sits in position A achieves near-perfect test-retest (it's perfectly reproducible) while exhibiting maximum-possible position bias. The most reproducible judges can be among the least valid. Because reporting test-retest alone remains common practice, current validation "misleads precisely in the cases that matter most for deployment: highly reproducible judges."

This is reliability without validity compressed into a single failure mode — and the whole page's thesis in one number.

Finding 4 — Verbosity bias has largely faded#

A rare "this got better" result. All 21 judges register verbosity bias < 0.011 on MT-Bench (largest: GPT-4o-mini 0.010; 17 of 21 below 0.005) — an order of magnitude below the 20–40% length effects reported in 2023-era studies. Two model generations appear to have wrung most length-preference out of pairwise judging. Scope caveat (the authors are emphatic): this holds under a single pairwise rubric and one length-differential operationalization; it is not a claim that verbosity bias is solved under arbitrary rubrics or scoring tasks.

(Scope narrowed 2026-08-04 by Yang et al. (2026), empirical, and the caveat above is exactly where it lands. Under an active probe — appending a fixed, generic, content-free padding string to one candidate and measuring verdict change — verbosity bias on adversarial LLMBar runs 0.547 for Qwen3-1.7B and ≈0.13 for the MiniMax releases: one to two orders of magnitude above the < 0.011 measured here. The two numbers are not in conflict once the operationalizations are separated: Norman measures a passive length differential between naturally-occurring responses on MT-Bench, Yang measures induced sensitivity to padding on a benchmark built to be adversarial, over a partly-open-weight cohort that reaches down to 1.7B. Neither result generalizes to the other's setup. What survives is the narrower claim — verbosity bias has faded as a property of frontier judges on natural preference data, not as a property of judges — and the practical consequence is that a padding probe belongs in the bias-audit step alongside A/B reversal, because it is the thing that detects what the passive measurement misses.)

The third invariance — vary the judge's version (Yang et al., 2026)#

Yang, Hou & Yang (Imperial College London + Nanchang Institute of Technology, arXiv 2607.08535, July 2026, empirical) name the axis this page and DRACO both leave open. Call it evaluator-replacement ambiguity: when a score moves after you swap the judge for a newer or bigger one, the accuracy number alone cannot say why. The new judge may be more capable, differently biased, failing on a different slice, or simply parsed differently by the harness. Their reframing is the one this page argues for, stated at its most general — an LLM-judge result is a measurement, and replacing the instrument is a measurement-validity event, not a version bump.

Setup. Eight judges on two axes chosen because both are decisions practitioners actually face: a parameter axis (Qwen3 dense 1.7B / 4B / 14B / 32B) and a released-API axis (MiniMax M2 / M2.1 / M2.5 / M2.7, evaluated as shipped — explicitly not a controlled ablation, and the paper makes no causal claim about MiniMax internals). GLM-5.1 and mimo-v2-pro serve as cross-family reference judges outside both axes. Four datasets: LLMBar (419 adversarial pairwise), PandaLM testset-v1 (894 valid after dropping tie-majority items), a seed-42 2,000-example Chatbot Arena sample (1,997 valid), and Judge's Verdict (200 TechQA-derived, three-level pointwise). Near-greedy T = 0.1; exact two-sided McNemar on parse-shared examples (so significance reflects paired verdict changes, not differing parser coverage); Holm correction over the family of 18 adjacent tests.

Finding A — upgrading the judge is not a reliability intervention#

Of 18 adjacent-step tests, one survives Holm correction: Qwen3 1.7B → 4B, on LLMBar and Arena. All nine MiniMax adjacent tests fail to reach even uncorrected p < 0.05, with a maximum adjacent accuracy difference of 0.022.

Table 3 (reconciled against the PDF — see the parse note in wiki/sources.md):

Dataset	Best judge	Qwen3 1.7B → 32B
LLMBar	GLM-5.1, 0.900 [0.868, 0.925]	0.463 [0.416, 0.511] → 0.678 [0.632, 0.721]
PandaLM	MiniMax-M2.7, 0.857 [0.833, 0.878]	0.779 [0.751, 0.805] → 0.769 [0.740, 0.795]
Arena	mimo-v2-pro, 0.742 [0.722, 0.761]	0.625 [0.604, 0.646] → 0.688 [0.667, 0.708]
Judge's Verdict	GLM-5.1, 0.680 [0.612, 0.741]	0.595 [0.526, 0.661] → 0.530 [0.461, 0.598]

Read the two bold cells: 19× the parameters makes the judge worse on two of the four datasets — PandaLM 0.779 → 0.769, Judge's Verdict 0.595 → 0.530. Scaling the judge is not merely a weak intervention, it is not monotone. "Use the strongest model as judge" fails here on its own axis, not just against this page's cheaper-judge caveat.

The headline needs one correction the abstract doesn't make. "MiniMax adjacent releases show no gain" reads as a negative finding about MiniMax. Figure 2's per-model LLMBar accuracies show it is a ceiling effect on the panel, not a failure to improve: Qwen3 runs 0.463 / 0.617 / 0.647 / 0.678 while MiniMax runs 0.829 / 0.839 / 0.832 / 0.832, with GLM-5.1 at 0.900 and mimo-v2-pro at 0.883. The MiniMax series starts above where the entire Qwen3 sweep ends. So the honest form of the result is not "releases don't improve judges" but the reliability gain available from upgrading a judge is concentrated at the bottom of the capability range — the single robust step in the study is also its cheapest (1.7B → 4B, +0.154 on LLMBar) — and adjacent steps near the top buy nothing measurable on these datasets. That is a directly actionable procurement finding: past a threshold, judge-model spend buys you something other than agreement.

No judge wins all four datasets. GLM-5.1 leads LLMBar and Judge's Verdict, MiniMax-M2.7 leads PandaLM, mimo-v2-pro leads Arena. This is the same shape as Finding 2 above with a different thing varied — there, judge identity across benchmarks; here, judge version across benchmarks — and it converges on the same prescription: slice-specific measurement validity, not one-dimensional evaluator ordering.

Finding B — the capability–fairness association is strong and still insufficient#

On LLMBar the position-flip rate falls 0.320 (Qwen3-1.7B) → 0.117–0.147 (MiniMax releases), and across the eight evaluated judges LLMBar accuracy and position-flip rate correlate at Pearson r = −0.957. Granularity sensitivity shrinks too. But MiniMax-M2.7 — the best PandaLM judge in the panel — still changes 14.7% of its verdicts under pure A/B reversal. The authors are careful to call this an association measured on one dataset and one model panel, not a mechanism. Position randomization and slice-level bias reporting survive the upgrade. (The flip-rate metric here is not the same statistic as this page's |P(A wins) − 0.5| position-bias figure, so the numbers are directionally comparable but not on the same scale.)

Finding C — juries are capped by error correlation, and the cap is now a formula#

The jury experiment is a clean negative result with a usable correction attached. For homogeneous juries (one judge sampled K times at T = 0.7, deliberately not T = 0.1, since repeated low-temperature calls would understate juror dependence), the paper estimates the intra-class error correlation ρ from the vote matrix and compares three quantities — the independence prediction, a ρ-corrected beta-binomial prediction (q ~ Beta(α, β), α = ps, β = (1−p)s, s = 1/ρ − 1), and observed accuracy:

• Independence misses observed jury accuracy by median absolute error 0.078 (LLMBar) / 0.093 (PandaLM). The ρ-corrected prediction cuts that to 0.008 / 0.004, max error below 0.02.
• Measured ρ is high: Qwen3 homogeneous juries 0.944–0.972, MiniMax lower but still strongly dependent at 0.664–0.706.
• So jury size barely moves anything: Qwen3-1.7B on LLMBar at K = 1, 3, 5 gives 0.463, 0.475, 0.482.
• Heterogeneous juries also underperform Poisson-binomial independence predictions — mixing model families under a shared prompt does not restore independent errors.

The design question shifts from "how many jurors?" to "how independent are their errors?", and the reporting rule is ρ alongside K, never K alone.

A second, cheaper instrument makes the same rule bite at the hypothesis-test stage, not just the jury-accuracy stage. Hossain, Yousefi & Lim (UCF, arXiv 2609.22512, 2026-09-18, empirical; full measurement treatment on Cross-Model Error Entanglement) measure a ten-judge open-weight bank's mean error correlation (ρ̄ = 0.206) and show it inflates the variance of a pooled-vote significance test by the design-effect factor 1 + 9·ρ̄ ≈ 2.85 — so a test that treats every judge vote as an independent observation is significant in 79% of 5,000 resampled 100-pair panels where a pair-based test (one preference pair, one observation) is significant in only 52%. In up to 28% of panels the two tests disagree about whether one system beats another, on identical votes. This is Yang's ρ-correction and this page's item-as-unit-of-inference rule (MVVP-adjacent, and explicit in Hossain et al.'s own operational checklist: "treat the item, not each judge vote, as the unit of statistical inference") arrived at from the hypothesis-testing side rather than the accuracy side — and it needs no ground-truth accuracy measurement at all, only the preference pairs themselves, which makes it a cheaper diagnostic to run than either ρ or n_eff.

This is the measurement sitting under Zhou's proof. Zhou showed analytically that no monotone aggregation rule can reject a region all judges accept when they threshold a shared latent signal, and measured pairwise acceptance correlation φ = 0.29–0.38 between three judge families — but under deliberate optimization pressure, on manufactured errors. Yang measures the same dependence with nothing optimizing against the judges at all — ordinary pairwise grading, ρ = 0.66–0.97 — and supplies the closed form that converts it into a prediction rather than a warning. Two independent routes to the same conclusion: more judges is not a reliability amplifier, and the family-mixing hedge fails on both.

Finding D — the largest protocol effect in the study is the one that cannot be attributed#

Structured debate (two judges, up to three rebuttal rounds, LLMBar) produces final-vs-round-1 accuracy shifts that dwarf every scaling effect in the paper: Qwen3-1.7B paired with GLM-5.1 / MiniMax-M2.7 / mimo-v2-pro shifts +0.317 / +0.305 / +0.289; Qwen3-4B with GLM-5.1 shifts +0.243 to reach 0.897. Same-family and top-vs-top pairs shift only +0.005 to +0.050, so the effect tracks the capability gap between the paired judges.

And it is uninterpretable. The implementation logged round verdicts and final verdicts but not raw responses or parse-success flags; round-1 parse failures fall back to "A" and later failures retain the previous verdict. Fallback rates are therefore unrecoverable without a full rerun, and the shift cannot be separated from "the weak judge's unparseable outputs got overwritten by the strong judge's verdicts." The authors present it as an auditability case study, not a deliberation result — which is the honest call, and the reason this finding earns a place on a validation page: a protocol intervention larger than any capability intervention in the study, whose mechanism is unrecoverable because six lines of logging were missing.

What this adds to the MVVP#

The MVVP is a pre-deployment checklist for one judge. Yang's Table 4 is a reporting standard for a judge result, and three of its six items are things the MVVP does not ask for:

Reliability risk	Required report item
Dataset-slice shift	Valid N, parseable N, tie handling
Parser artifacts	Parser success rate, fallback rule
Position/verbosity bias	A/B reversal, padding probe
Correlated jury errors	ρ and K, not K alone
Debate protocol artifacts	Raw outputs, parser status, round verdicts
Statistical overclaiming	Paired tests, Holm correction, uncertainty intervals

The parser/slice items (rows 1–2) and the ρ item (row 4) have no MVVP counterpart; row 3 is MVVP step 2 plus the padding probe; row 6 is the statistical hygiene the MVVP leaves implicit. The two documents are complementary rather than overlapping — validate the judge before deploying it (MVVP), then report what the pipeline around it actually did (audit trail).

Robustness checks (which the paper keeps deliberately narrow): human-ceiling calibration shows PandaLM retains headroom — best judge matched leave-one-annotator-out κ = 0.753 against a human ceiling of 0.920 — while Judge's Verdict is at or past its noisy ceiling (best judge κ = 0.620 against human κ = 0.562, i.e. the judges beat the humans, which is a statement about the labels rather than the judges). Arena seed perturbation moves accuracy only 0.004–0.032, smaller than the robust 1.7B→4B Arena gap. Prompt-sensitivity checks preserve cross-family ordering, with stronger judges more prompt-stable.

Weight it accordingly. Two model families, two reference judges, four datasets, a fixed main prompt with prompt sensitivity as a robustness check only, one primary decoding regime per experiment. The MiniMax axis is an observed release sequence, not a controlled ablation — a null result on it is a null result about released APIs, not about training interventions. McNemar tests are adjacent-pair tests and constitute no formal between-axis comparison.

The instrument under the judge — validating rubric items#

Everything above validates a judge. CalibratedRubric (Chen et al., FinStep + StepFun, arXiv 2607.29252, July 2026, empirical) validates the thing the judge is pointed at, and the distinction it draws belongs in this page's vocabulary because the field routinely conflates all three:

Property of a rubric item	Observable	What it does not establish
Measurability	inter-judge agreement rate	that the criterion is worth measuring
Informativeness	IRT item information over the fitted ability range	that graders can apply it consistently
Validity	expert endorsement	— the paper supplies no automated substitute

Their z_j is defined as reproducible judgeability and immediately qualified: "necessary but not sufficient for substantive expert endorsement." That is this page's reliability-is-not-validity thesis restated one level down — an item every judge agrees on can be an item that measures nothing anyone cares about, and agreement is exactly the statistic that will not tell you.

Where it collides with Finding C. CalibratedRubric's measurability posterior is a Beta–Bernoulli over the fraction of instances on which all judges agree, and it requires ≥ 3 judges to carry any signal at all (with two, unanimity is definitionally pairwise agreement — their two-judge HealthBench and HelloBench arms show no gain). Assumption A3 concedes the threat without measuring it: "LLM judges share training data and conventions, so correlated errors can inflate apparent consensus and estimated sensitivity or specificity." Yang's ρ = 0.944–0.972 / 0.664–0.706 is the magnitude of exactly that, measured under no optimization pressure. The two results are not in conflict — they concern different uses of redundancy (Yang: extra jurors as votes, capped by ρ; Chen: extra jurors as an observation about the item), but ρ threatens both. The seam shows in Chen's own data: posterior measurability predicts agreement at r = 0.589 / 0.558 on the LLM-judged blocks and drops to r = 0.127 on JudgmentBench, the single block whose gold labels are human. The mechanism is best-evidenced where LLMs are grading LLMs.

And a directional judge–human bias that agreement filtering does not remove. On JudgmentBench the LLM judges assign positive labels at 55.6–62.9% against the human gold's 47.1% — the authors' own words, "a systematic judge–human mismatch that measurability filtering does not fully eliminate." Filtering for agreement makes the panel more self-consistent without moving it toward the humans. Same direction as Reference-Free Judge Over-Crediting's over-crediting, arriving from rubric grading rather than reference-free QA. The MVVP-shaped addition: when a rubric bank is built from judge agreement, report the panel's base positive rate against the human base rate, because a filter that improves κ and leaves that gap intact has improved reliability without improving calibration.

Who judges well (provider patterns)#

• Frontier/reasoning models reduce position bias but don't eliminate it. Range spans ~two orders of magnitude: Gemini 2.5 Pro 0.002 (best) to Qwen 3 8B 0.192 (worst); within the Gemini family, 2.5 Pro (0.002) vs 2.5 Flash (0.125) differ 70×. The pre-registered prediction that all three thinking-architecture judges (GPT-5.4, Gemini 3.1 Pro, DeepSeek V3.2) would fall below 0.05 held for only Gemini 3.1 Pro (0.038); GPT-5.4 (0.083) and DeepSeek V3.2 (0.094) missed.
• Anthropic judges post the strongest joint performance on hard items — average JudgeBench κ = 0.770 (Opus 4.6 0.875, Sonnet 4.6 0.782, Haiku 4.5 0.653) at the lowest cohort-level position bias of any provider (0.020). OpenAI flagships (GPT-4o/4.1/5.4) average JudgeBench κ = 0.467; generational progress is legible on JudgeBench (0.309 → 0.487 → 0.606) but nearly invisible on the compressed MT-Bench scale (0.451/0.451/0.457).
• Mid-tier can beat frontier on a specific axis. Kimi K2.5 records the lowest position bias of any non-Gemini judge (0.004) and JudgeBench κ = 0.720 at a fraction of frontier cost — so "pick the strongest model as judge" is not a safe default; pick on the dimension you care about.

An eval-hygiene lesson: RewardBench was silently degenerate#

The authors predicted RewardBench would produce κ ≈ 0 because the standard generative loader places every chosen response in position A — making the human label identically "A", collapsing p_e, and degenerating Cohen's κ to 0.000 for every judge. Per-item position randomization (seed 42) restored a valid signal (κ ∈ [0.616, 0.898]), refuting their own hypothesis. The transferable warning: a fixed answer-position convention in a benchmark loader can silently zero out your chance-corrected metric — an artifact of the harness, not the judge.

The Minimum Viable Validation Protocol (MVVP)#

Before deploying an LLM judge:

1. Chance-correct. Report Cohen's κ (or Krippendorff's α) alongside any exact-match figure, and treat the chance-corrected metric as the headline reliability number.
2. Swap positions. Measure position bias via paired AB+BA evaluations; report |P(A wins) − 0.5|.
3. Replicate. Measure test-retest over ≥ 3 independent runs at temperature 0 with response caching disabled.
4. Cross-validate. Evaluate on ≥ 2 benchmarks spanning preference-style and correctness-style label distributions.
5. Audit the paradox. When test-retest exceeds 0.95, verify position bias is below 0.10 before claiming reliability. High stability with high bias is a failure mode, not a strength.

The paper flags its own partial-adoption risk: reporting κ alone (step 1) without the position-swap and consistency checks can manufacture a false sense of having addressed judge reliability.

Scope caveats (from the paper's own Limitations)#

The findings are a snapshot, not a universal law: English-only, text-only, three established benchmarks, a single pairwise rubric template, and a five-week window (hosted endpoints drift silently, un-re-measured here). Thinking channels were suppressed for all reasoning-capable judges to keep them comparable — reasoning-on could change every agreement/consistency/bias profile. And calibration proper (Expected Calibration Error, Brier score) is deferred because most providers don't expose token logprobs — so the question of a judge's confidence calibration remains open.

The same failure outside evaluation: taxonomy classifiers#

Google ATLAS (July 2026) reproduces this page's core structure in a different domain — LLM classifiers mapping conversations onto BLS/O*NET/ATUS categories for economic research rather than scoring model outputs. The shape is identical: a lenient validation metric flatters the classifier, and which metric you report decides whether the pipeline sounds trustworthy. Where this page finds exact-match agreement overstating chance-corrected κ by 33–41pp, ATLAS finds human approval (85.8%) overstating exact accuracy (22.6%) by 63pp at the O*NET task level — because asking a rater "is this label defensible?" is a much easier question than asking them to pick the right one from 18,797.

Two transfers run both ways. ATLAS randomizes classifier option order to defeat position bias — a direct application of the consistency–bias lesson here. And ATLAS supplies a ceiling this page's protocol lacks: human annotators disagree with each other on 42–48% of 3-digit occupation codes (Mellow & Sider 1983; Mathiowetz 1992), so accuracy measured against 100% is measuring the wrong target. A Minimum Viable Validation Protocol arguably needs a human-ceiling estimate alongside its chance correction.

Right metric, wrong axis — and a ratification rate reported as agreement#

A second taxonomy classifier supplies the failure the MVVP has no step for, and it is instructive precisely because the metrics are good. Pahuja, Brokman, Hofman et al. (An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures, arXiv 2608.03735, 2026-08-04, empirical) use Claude Opus 4.8 to sort agent planning failures into six categories, and validate it better than most: chance-corrected κ = 0.860 reported alongside macro-F1 = 0.906 rather than raw agreement (step 1, done properly), uncertain annotations excluded rather than coerced, and a per-category breakdown published including its own weak cells — Operation 83.3%, Answer format 85.7%, Other 88.2% against Entity 94.4% and Source/Temporal 100%, overall 88.9% on 117 decisive verifications.

Two defects survive all of that.

The validation is stratified by category; the claim varies by language. The paper's headline diagnostic result is a per-language gradient across eleven languages spanning five orders of magnitude of Common Crawl share, and no per-language agreement figure appears anywhere — even though the annotation sample was deliberately drawn to span the resource spectrum, so the cut is available and simply was not made. The two lowest-agreement categories are two of the three whose shares move most along the gradient. Its own Limitations concede that "judge calibration may differ across languages." Nothing in the MVVP's five steps catches this: they govern the metric, position bias, replication, benchmark spread and the reliability–bias paradox, and none of them says stratify your validation along the axis your headline claim varies on, and report the per-stratum numbers. The page already praises User Awareness's Appendix G for stratifying and reweighting to the population; this is the counterexample that turns that praise into a rule, and a candidate sixth step. Its coverage is also stated three mutually incompatible ways across §5.2, App. G.4 and Table 11's caption (6 annotators / 122 samples / ~14% versus seven annotators / 280 samples / ~30% versus 117 decisive with 5 annotators and six exclusions) — the kind of accounting slip that only matters because it is the number licensing the judge.

And 88.9% is a ratification rate, not an agreement rate. The annotation guide presents each sample with "all inputs supplied to the judge LLM together with the judge's response, and asks annotators to verify the assigned category." That is the second corpus instance of the asymmetry this page already flags on Automated Failure Attribution, where a human panel ratifies pipeline labels at 94.0 / 90.0 / 90.0 while models predict from scratch — and two instances make it a pattern worth naming rather than a quirk of one benchmark. The reason it matters here is that the mechanism is measured on the sibling page: judging a shown candidate is candidate anchoring, and Zhou's de-anchoring result puts the size of it at FPR 0.719 → 0.012 once the judge must commit its own answer before seeing the candidate. Nobody has run that control on human validators, and the fix is the same shape and nearly free: have a subset of annotators label blind, then reveal the judge's verdict, and report both numbers. Until that exists, a ratification rate is an upper bound on validator-judge agreement of unknown tightness, and every taxonomy-classifier study in this corpus reports the upper bound.

The ceiling as the target, and two construct-validity checks — from a production deployment#

Everything above is measurement of judges by people whose job is measuring judges. Shopify's Sidekick account (McNamara & Mazza-Anthony, Shopify Engineering, 2026-08-05, case-study) is a team validating a judge because it is about to become an RL reward, and its protocol contains the step this page has been asking for plus two nobody here has proposed. Evidence first: this is a first-party account of the authors' own system with no replication and no adversarial review, so it cannot upgrade or contradict any empirical finding above. What it contributes is practice, and two candidate protocol items — checkable on their own terms regardless of who ran them.

The human ceiling, measured in-house rather than borrowed. Before any judge exists, two expert annotators blind-annotate 25 randomly sampled production conversations and their agreement is recorded with Cohen's kappa — chance-corrected, exactly as step 1 prescribes, applied one level up to the humans. The threshold is a rewrite trigger rather than a pass mark: at κ ≈ 0.2 the rubric is declared ambiguous and iterated, on the argument that "if the rubric confuses several product experts who work on this product every day, it will confuse an LLM too." Then the judge is aimed at that number instead of at 100%: "that agreement is the judge's ceiling… The goal is not a judge that is 'perfect,' but one that matches humans about as well as humans match each other." Their reported result, which exists only inside a figure and nowhere in the article's text, is human agreement 83% — annotated in the chart as "perfect: unreachable" — against judge 80%.

This is the ATLAS-derived note above ("a Minimum Viable Validation Protocol arguably needs a human-ceiling estimate alongside its chance correction") arriving as a shipped step, and it improves on ATLAS in one respect worth naming: ATLAS's ceiling is borrowed from the literature (human annotators disagreeing on 42–48% of 3-digit occupation codes, Mellow & Sider 1983; Mathiowetz 1992), where this one is measured on the actual rubric, by the actual annotators, on the actual traffic, before the judge is built. A ceiling estimated off someone else's task tells you a ceiling exists; one estimated on yours tells you where to stop optimizing.

And the defect is this page's own Finding 1, in the wild. The κ discipline is applied to the annotator pair and then dropped: the 83%-versus-80% comparison — the number the whole ceiling argument rests on — is raw agreement, uncorrected, one paragraph after the team correctly reached for chance correction. Finding 1 says the gap between the two tracks the label distribution, and the figure shows a Pass / Borderline / (implied fail) scale, i.e. the multi-category regime where deflation was largest in this study's own cohort. The direction is certain and the magnitude is not computable from what is published (the label distribution is not given), so the honest statement is: the ceiling framing is right and the metric it is expressed in is the one this page demotes to a secondary figure. Note also what the account never says — the κ at which they shipped. Only the rewrite trigger (≈0.2) is published, so the rubric's final chance-corrected agreement is unknown.

The two additions, and why they belong in a validation protocol rather than in a product doc. Every MVVP step and every Yang audit item measures a property of the judge in isolation. These two measure whether it is pointed at the right thing:

Check	Procedure	What it establishes that reliability cannot
Online-metric backtest	Replay the judge against previous A/B tests and ask whether it recovers the direction of known wins and losses in engagement, retention, or whatever the product is built to drive	That the offline proxy is correlated with the outcome it is a proxy for. A judge can be κ-clean, position-unbiased, replicable, and still rank the losing variant first
Targeted degradation test	Deliberately make one behaviour worse — offline or on a controlled traffic slice — and confirm the corresponding criterion falls, and specifically that one ("if the system stops trying to fulfil the user's goal… the goal-fulfilment score should fall specifically")	That the rubric's criteria are separable and each one measures what its name says. This is the per-criterion analogue of a manipulation check

Both are construct-validity instruments, which is precisely the half this page's title says the field leaves out — and the backtest is the only device in the corpus that grounds a judge in an outcome the judge did not define. Their stated precondition is a fourth prescription that reads as an aside and is not one: "keep each judge small and targeted rather than cramming all of your product's behavior into one… focused judges make these tests easier to interpret and the resulting metrics easier to trust." A degradation test on a blended judge is uninterpretable by construction, so judge decomposition is what makes criterion-level falsification possible at all — the same move Agent Quality Flywheel arrives at from the metric side (promote one concern to one stable metric) rather than from the validation side.

What none of this touches is the failure Reference-Free Judge Over-Crediting measures, and this deployment is the corpus's most exposed instance of it: the same calibrated judge becomes the hard-negative selector, the repair gate and the GRPO reward, and nothing in the account re-measures its agreement once a policy is being optimized against it. A one-shot validation, however good — and this is a good one — certifies a judge that will be true only until something starts pushing on it.

Panel composition moves dispersion more than panel size stabilises it (September 2026)#

A judge panel is the standard answer to a single judge's unreliability, and Finding C already prices its ceiling (error correlation ρ ≈ 0.66–0.97 caps a repeated-sample jury). Ludwig et al. (NVIDIA, arXiv 2609.06780, 2026-09-06, empirical) supply an unusually clean natural experiment on the composition axis, as a side-effect of a paper about something else entirely — auditing coding agents for benchmark exploitation (full treatment on Evaluation-Time Answer Leakage).

Their instrument is three open-weight judges — Qwen3.8-27B, DeepSeek-V4-Flash-0731, GLM-5.3-Flash — labelling each agent turn YES/NO plus a category, majority-voted to a trajectory verdict. For one model they re-run the audit with five judges, adding Claude Sonnet 5 and GPT-5.6 Luna, and report it as a robustness check that leaves the conclusion unchanged. It does. What it does not leave unchanged is the agreement structure (Tables 5 and 6, Kimi-K3):

	3 open judges	+2 closed judges
SWE-bench M., vanilla — yes/no disagreement	17.7%	20.8%
SWE-bench M., principled — yes/no disagreement	10.1%	21.9%
SWE-bench M., principled — unanimous negative	83.3%	71.9%
DeepSWE, principled — yes/no disagreement	4.1%	9.1%
DeepSWE, principled — unanimous negative	93.2%	88.2%

Binary disagreement roughly doubles in the low-rate condition when two judges from different training lineages join, and unanimity falls correspondingly. Two of these five judges disagreeing on one trajectory in five is not a robustness result; it is a measurement that the three-judge panel's high agreement was partly shared lineage, exactly the mechanism Finding C's ρ formalises and the reason a jury's members have to be counted as correlated rather than independent. The practical rule: an all-open or all-one-family panel reports its own homogeneity as reliability, and the cheapest test of a panel is to add a judge from outside its lineage and watch the dispersion, not the verdict.

The paper is also a clean instance of the page's central failure, and it is worth naming because the paper is otherwise careful. Every number in it is a judge verdict; the reliability argument is raw inter-judge unanimity percentages, with no human-annotated subset, no precision or recall for the classifier, and no chance-corrected statistic anywhere — the substitution of consistency for validity that Finding 1 measures at 33–41 points of κ inflation and Finding 3 names outright. Format non-compliance is reported to three significant figures (under 0.3% of turns) while the classifier's accuracy against a human is not reported at all.

The ρ you are told to report is not the ρ you can measure (September 2026)#

Finding C's reporting rule — ρ alongside K, never K alone — has a quiet precondition nobody had stated: that ρ is measurable. Sunkavalli (arXiv 2609.08826, 2026-09-08, single independent author; empirical for simulations and diagnostics, and for no real-panel number — see Sources) works out the identification budget, and the answer is that two different quantities have been travelling under one symbol.

Yang's ρ is estimable; the decomposition it suggests is not. Yang's intra-class error correlation is read off a vote matrix — repeated judgments, disagreement rates, no external reference — and it does exactly one job: it converts jury size into predicted jury accuracy (beta-binomial, median absolute error 0.004–0.008 against independence's 0.078–0.093). It does not tell you what the jurors agree about. Write each judge as J_j = t + c + e_j — latent quality t, one common-mode error c shared across the panel, an idiosyncratic residual — and the observable off-diagonal judge covariance is

K = Cov(J_i, J_j)_{i≠j} = σ_t² + σ_c²

signal plus shared error, in one number, with no way to split them from inside the panel at any size. That is the formal statement of why adding judges never resolves the question this page keeps asking: high inter-judge agreement is consistent with a panel that is right and a panel that is uniformly wrong, and the vote matrix cannot distinguish them. Splitting K requires an external reference set — an anchor — and the whole literature that does this assumes the anchor is clean (its error uncorrelated with c). Sunkavalli's ρ_k:= Cov(A_k, c)/(σ_{a_k}σ_c) is that assumption's residual, and his Theorem 1 point-identifies σ_t², σ_c² and every ρ_k in closed form from p ≥ 2 judges and m ≥ 2 anchors with no clean anchor assumed. The full machinery, its boundary and its blind spots are on Weak-Verifier Ensembling; what belongs here is the part that lands on real judges.

Real panels fail the adequacy pre-test, and one failure is a measurement this page can use directly. The battery's Test A is the coefficient of variation of the off-diagonal judge covariances, which the single-common-factor model requires to be equal. On a six-provider LLM panel (Amazon Nova Lite, Llama-3-70B, Mixtral-8x7B, GPT-OSS-120B, Qwen3-Next-80B, DeepSeek-V3.2; temperature 0, 499 complete-case keyed GSM8K/SciQ items, 6,000 calls / 5,996 parsed verdicts) the statistic is 0.175 against a matched-null 95th-percentile threshold of 0.064 — the test fires. The mechanism is the reportable part: the judges' quality loadings span 0.324–0.870, measured as per-judge regression coefficients on a known quality construct, where the model fixes them at 1. Real judges do not load equally on the thing being measured, which is the same heterogeneity this page's panel-composition section sees as dispersion and Finding C sees as a correlation, now measured on the loading itself. On HANNA's three human raters (431 stories, coherence 1–5, marginals extreme enough that one rater puts 62% of mass on category 5) Test A fires far harder: 0.589 against a 0.247 threshold, p < 0.0005. Human raters break the equal-loading structure too, which is worth knowing before treating a human panel as the clean reference in any decomposition.

Two consequences for the protocol. First, an adequacy pre-test belongs before any panel-level correction — the MVVP validates a judge, Yang's Table 4 reports what the pipeline did, and neither asks whether the model under the correction fits. Run in defiance of its own pre-test, the estimator returns (0.887, 0.960, 1.026) against injected (0.0, 0.4, 0.7); the out-of-range value is the misspecification signal, and a correction applied to a panel that fails the pre-test is not a smaller error than no correction. Second, the standard LLM-judge configuration is on the wrong side of two identification walls. With judges and anchors ordinal — Likert judges scored against a Likert reference set, i.e. most of this literature — ρ_k is not identified at any number of anchors, and a known-clean anchor does not restore it; identification needs ordinal judges plus ≥ 3 continuous-scored anchors, at roughly an order of magnitude more variance than the continuous case. And a residual shared panel-wide — one prompt template across every judge, which is how every panel on this wiki is built — is observationally indistinguishable from the common mode, so no within-panel statistic separates them, whatever the diagnostics say.

Weight it as a method paper. The estimator has never been validly applied to a real panel — both panels tested were rejected, no qualifying panel exists, and the paper's Section 8 says the pre-test exists precisely to say so. What is real here is the negative result on real judges and raters, not any contamination number.

Measure the dependence on the answers, not on the verdicts (September 2026)#

Finding C prices a jury's ceiling from its own vote matrix and the section above shows why that matrix cannot say what the jurors agree about. Kuai et al. (Texas A&M / Marquette / Utah, arXiv 2604.07650, v2 2026-08-09, COLM 2026, empirical) take the third route: estimate the dependence somewhere else entirely, on a task with ground truth, and carry it into the judging setting as a covariate. The full machinery is on Cross-Model Error Entanglement; three things land here.

The identification wall is a consequence of having no answer key, not of panels. Judge covariance fuses signal and shared error (K = σ_t² + σ_c²) because two judges agreeing might both be right. Condition on both models being wrong and that ambiguity is gone — what remains is shared error by construction. Kuai et al. therefore run their audit on the failure manifold of 18 models across six families answering MMLU-Pro, with a conditional-independence null given task difficulty (difficulty estimated leave-pair-out from the other 16 models, per-model difficulty-response functions fitted by logistic regression at mean AUC 0.865 / ECE 0.041), Monte Carlo p-values and Benjamini–Hochberg q-values. The cost is scope: the method needs "tasks with objectively verifiable answers," which is the one condition the open-ended generation settings where judge panels actually work do not meet.

A judge's precision is a property of the judge–target pair. Their bias quantity is the deviation in conditional precision, ΔPrec(J_j, M_i) = P(Y=1 | Ŷ_j=1) − P(Y=1 | Ŷ_j=1, M=M_i) — how much worse a judge's precision is for one specific target model than its precision across all of them. That is a reframing this page should adopt: the MVVP validates a judge against a benchmark and returns one κ, one position-bias rate, one agreement figure, and none of them is conditioned on whose output is being graded. Dependence between judge and target predicts that deviation at Spearman ρ = 0.508 (BEI) and 0.520 (CIG) on a disjoint MMLU-Pro subset, and the same estimates transfer to free-form MATH-500 responses at 0.441 and 0.457 where most baseline dependence measures go non-significant. Direction: the more a judge's failures are entangled with a model's, the more it over-endorses that model.

Weight it by what the arithmetic supports, which is less than the table implies. The paper never states the number of points any Spearman runs over; back-solving each column from its own (ρ, p) pairs gives n ≈ 34 on MMLU-Pro and n ≈ 26 on MATH-500, recovered identically from the significant and non-significant rows alike (wiki arithmetic). At n = 34 the 95% interval on ρ = 0.508 is about [0.21, 0.72] and on mutual information's 0.113 about [−0.23, 0.44] — overlapping, so the paper's claimed contrast between its metrics and the conventional ones is not established; what survives is the sign, the significance, and the cross-benchmark transfer. And a plain co-failure Jaccard overlap — no null model, no difficulty conditioning — already reaches ρ = 0.428 on MMLU-Pro, falling to non-significance only on the MATH-500 transfer.

Two additions for this page's protocol, then. First, an entanglement covariate belongs next to κ: "this judge agrees with humans at κ = x" is incomplete without "and its precision drops y points on targets it co-fails with." Second, the instrument itself is cheap where an answer key exists — an error matrix over a shared benchmark is something most eval harnesses already have — which makes it the first dependence measurement in this corpus that a practitioner can run without anchors, human raters, or repeated sampling.

"ρ alongside K" gets a unit, a threshold, and a price in accuracy points (2026-09-22)#

Finding C's reporting rule is a good rule with a missing half: it tells you to report a correlation next to a panel size, and a correlation is not a quantity anyone can act on. Kohli (Apple, arXiv 2605.29800, 2026-05-28, empirical) supplies the conversion — the Kish effective sample size, n_eff = k / (1 + (k−1)·φ̄) — and then spends the paper on what it costs. Full treatment on Cross-Model Error Entanglement; four things land on this page's protocol.

The heterogeneous-jury result that Finding C reported without a coefficient now has one. Yang et al. found that "heterogeneous juries also underperform Poisson-binomial independence predictions" and gave no number. Kohli's nine judges span seven families — OpenAI ×2, Anthropic, Google, Meta ×2, Alibaba, Mistral, DeepSeek — and land at φ̄ = 0.391 ± 0.111, n_eff = 2.18 [2.07, 2.31], an independence ratio of 24.2%. So the missing coefficient is roughly: family mixing takes measured error correlation from the 0.66–0.97 band of a homogeneous jury down to ~0.39, and that is still four judges' worth of redundancy out of nine. Mixing helps and does not come close to fixing.

The cap is now priced in the currency of the result, not of the correlation matrix. A Condorcet null — per-judge confusion matrices estimated within human-entropy tercile, 10,000 Monte Carlo votes per item — predicts 94.0% majority-vote accuracy for the same nine judges voting independently. They score 72.0%. The 22.0pp [19.5, 24.1] shortfall survives the difficulty conditioning almost intact (6.8% of it is attributable to shared item difficulty; 13.5% with ten bins) and a stratified permutation test puts the observed φ̄ at z = 65.6 above a null that never once reached it in 10,000 draws. This is the number the MVVP has no slot for: a panel can pass every check on this page judge-by-judge and still be losing twenty accuracy points to its own redundancy.

A reporting threshold, cheap enough to adopt. The paper's recommendation is one line — "if n_eff/k < 0.5, results should be treated with caution" — and every configuration it tests fails it (24.2% at k = 9; 22% on RewardBench). The input is a panel's own error matrix against whatever gold labels the validation set already has, which is a by-product of any scored run, so this is the first dependence diagnostic in the corpus that costs nothing extra to compute. It belongs in the MVVP beside κ, and the practical corollaries are blunt: on all three NLI datasets and RewardBench the best single judge matches or beats the panel (SNLI −6.5pp, AlphaNLI −2.5pp, RewardBench −2.8pp for the panel), six of nine leave-one-out removals raise accuracy, and the first five judges deliver 90% of the achievable independence against a hard asymptote of 1/φ̄ = 2.56.

And one protocol warning aimed straight at a knob this page has been agnostic about. Re-running the panel with chain-of-thought raises error correlation: φ̄ 0.391 → 0.456, n_eff 2.18 → 1.94, panel accuracy 72.0% → 69.2%. Prompt rewording (2.17), label-order reversal (2.15) and temperature 0.5 (2.17) move nothing; reasoning-on is the only manipulation in the study that moves the dependence, and it moves it the wrong way.

Judge scoring as a method effect across benchmarks (September 2026)#

Everything above validates a judge within a benchmark: agreement with humans, stability across runs, position bias. Desai et al. (arXiv 2609.08812, COLM 2026, empirical) measure something no within-benchmark protocol can see: the variance LLM-judge scoring shares across benchmarks. Across 48 benchmarks and 53 models, a partial Mantel test regresses pairwise ranking correlations on shared concept and shared score format. When format is coded as LLM-judge against everything else, β_format = 0.526 (p < 0.0001) and β_concept = −0.058 (p = 0.998). Knowing that two benchmarks are both judge-scored predicts their agreement, and knowing that they claim the same concept adds nothing. SGBench-mcq, a refusal benchmark scored by multiple choice, clusters with other multiple-choice benchmarks rather than with the judge-scored refusal benchmarks. The effect grows with newer models (format-minus-concept contrast +0.43 before October 2024, +0.84 after).

The paper's judge-swap check shows the effect belongs to the method, not to a judge. Re-scoring four benchmarks with Llama-3.3-70B-Instruct in place of Qwen3-30B moves every statistic by at most 0.06, and the Mantel contrast is unchanged at +0.58. That is reassuring in the way DRACO's "rankings are judge-stable" is reassuring, and limited in the same way: swapping one judge for another keeps whatever LLM judges share. The MVVP above would pass each of these judges and could not have detected the effect.

One confound, noted on the concept page: nearly every judge-scored benchmark in the grid is a refusal or over-refusal benchmark with long free-text responses. "Judge-scored" and "refusal family" are close to collinear, so part of the β may belong to the task family rather than the grader.

A formal checker as the ground truth an LLM judge is graded against (2026-09-29)#

SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization (empirical) scores whether a generated Lean statement matches an informal theorem with, among other metrics, a three-vendor LLM-as-judge (Gemini 3.1 Pro, Claude Opus 4.7, GPT-5.4; three samples each, nine-vote majority, about $480), against two-expert labels on compile-passed outputs from six agentic configurations. The judge's binary agreement is 0.785, which looks respectable and is not: precision 0.235, recall 0.093, F1 0.133, because aligned outputs are rare (compile precision 0.178) and the judge mostly says no. The Lean-checked SA-Pass metric reaches recall 0.930 at precision 1.000. On the system-level ranking of six agents the judge is the worst metric (Spearman 0.72, against 0.99 for bare compile rate). It is a fresh case of this page's first finding, that headline agreement flatters a judge when the label is imbalanced, with the unusual property that a mechanical oracle exists to show what the judge missed. The judge's own weakness is in the concession it shares with Kernel-Level Proof Auditing: whether a statement is the intended one is precisely what a plausible-reading judge is bad at and a forward/backward implication check is good at. Caveat: this is the metric authors' baseline, the judge prompt is the authors', and one panel was run once.

Connections#

• Evaluation Awareness & Grader Gaming — the same free-parameter problem one level up. Every cheating- and misalignment-prevalence figure in the corpus rests on a classifier that drew the cheating/not-cheating line somewhere, none reports sensitivity to where, and a frontier practitioner's own objection is that the binary is an artifact of verifiable-answer domains — "is sycophancy basically reward hacking?" The threshold is unpublished in exactly the way this page finds judge validation under-reported

• Harness Activation and Adherence — a judge design that gets half of this page's checklist right and half of it absent. Right: trajectories blinded by replacing every model-family token with <MODEL>, and a rubric extracted from the artifact and locked before any trajectory is scored, so the judge checks against a fixed list rather than forming an impression. Absent: no human-agreement check, no chance-corrected agreement, no position-bias audit, no second judge. The result is a metric whose ordering across six backbones is usable and whose individual values are not — the exact split this page argues for

• GDPval Benchmark — a field instance of this page's two prescriptions, from a lab with every incentive to skip them. GDPval reports its automated grader's agreement against the human inter-rater ceiling rather than against 100% (65.7% vs 70.8%, defined as E[1 − |H − A|] over ordinal scores in {0, 0.5, 1}, with κ-family statistics explicitly declined as ill-fitted to three-point ordinals) — and then volunteers that the grader, a GPT-5-high model, "shows lower correlation with human expert graders when assessing outputs from capable OpenAI models," citing Panickssery et al. (2024). A vendor measuring and publishing its own judge's self-preference is the strongest form of the disclosure this page asks for; the weaker form is that both agreement metrics are highest on weak models, so a headline agreement number is partly a statement about how bad the outputs were

• Evaluation-Time Answer Leakage — a judge panel doing load-bearing measurement work in a domain with no ground truth, and the source of this page's panel-composition datapoint. Every exploitation rate in that literature is a majority vote of LLM judges reading tool calls and reasoning traces; adding two closed-source judges to three open ones roughly doubles binary disagreement while leaving the majority verdict intact, which is what this page's error-correlation cap looks like from the composition side rather than the sample-count side

• Verbalized-Confidence Soft Scoring for LLM Judges — the step this page's MVVP defers (calibration proper, for lack of logprobs), run without logprobs: stated-confidence AECE and spread on 10 closed flagship judges, with the finding that one prompt sentence moves the calibration number by up to 7pp and that the same judge prompt costs older judges accuracy and newer ones none. That makes the judge prompt a version axis alongside Yang's judge-model axis

• Skill Lift — a live instance of the gap this page names, in a benchmark being used as a distribution gate. NVIDIA publishes five 0–100 rubric dimensions across 300+ skills and never names the grader, reports no human-agreement or chance-corrected statistic, and runs no position-bias check; a +41-point delta measured that way inherits whatever reliability an unvalidated judge has. The Minimum Viable Validation Protocol is exactly the missing disclosure

• Agent Quality Flywheel — the deployment this section is drawn from, and the reason its judge matters more than most: after calibration the judge becomes the offline metric, the hard-negative selector, the replay-repair gate and the RL reward all at once, on a daily training cadence. It also supplies the corpus's clearest statement of why the rubric comes before the judge — "defining quality… begins as a specification of what good looks like and becomes the reward signal that drives learning"

• User Awareness — the same organization doing it right, two days later, which is what makes the contrast worth keeping. Its Appendix G validates the awareness judge on 74 blind hand-labelled items drawn from 11,908 runs, stratified across judge-positives, lexicon-flagged negatives and double negatives, balanced across model families and reweighted to the full population, reporting precision 0.73 [Wilson 0.56–0.85] / recall 1.00 [Jeffreys 0.42–0.99] for evaluation awareness and 0.94 [0.81–0.98] / 0.98 [0.74–0.99] for user awareness, plus an error analysis locating all 9 false positives in one task's vocabulary. Two lessons. The recall intervals are wide enough to matter — correcting the headline rates at the pessimistic corner still leaves them near 1–3%, so the validation is what licenses the paper's central negative claim rather than decorating it. And the same study supplies a validity threat no protocol here tests: a judge's score moves with who it believes is asking

• Misalignment in Production Agent Traffic — this page's failure mode in a result the corpus otherwise leans on. Transluce's entire 8,600-session misalignment rate rests on one unvalidated LLM judge: rubrics drafted by Claude Fable, refined by spot-checking 100 random runs, compared against "a small hand-labeled set" of unstated size, and screened by a GPT-5.6 Sol skeptical verifier — a sensible pipeline that yields no agreement statistic, no kappa, and no reported false-positive or false-negative rate. The stated design bias toward avoiding false-positive severe cases is the only quantitative-ish handle on it. A textbook instance of the under-rigour measured here, on a load-bearing number

• Usage-Telemetry Classifier Validation — the same structural failure in AI-usage economics: a lenient metric (human approval, 85.8%) flattering an LLM classifier whose exact accuracy is 22.6%, plus a human-disagreement ceiling this page's protocol doesn't estimate

• Matched Comparisons for Memorization Claims — the same arithmetic, one field over. Chance-correction is "observed agreement minus the agreement you'd get by chance"; Cooper et al.'s calibrated memorization rate is M(τ) = G(τ) − FPR(τ) — observed generation on training data minus the generation rate matched non-training data reaches. Both correct a raw rate by a baseline that the field had been assuming was zero, and in both the correction is large (κ deflation of 33–41pp there; ~24% of the apparent extraction rate at 10-token suffixes there). The memorization version has the cleaner null — non-membership is definitionally exclusive, so its baseline is measured against something that cannot be a true positive, where a judge's chance baseline must be modeled

• LLM-as-a-Judge — the primitive this page validates; kappa deflation and the consistency–bias paradox are the reliability failures its DRACO-style protocol can hide. It also hosts the upstream instrument: CalibratedRubric's rubric bank, where measurability, informativeness and validity separate and the field's unanimity filter turns out to decay exponentially in leaderboard size for reasons unrelated to rubric quality

• DRACO Benchmark — the judge-model-invariance counterpart; DRACO's "use rankings" reassurance is bounded by this paper's benchmark-variance (two different invariances)

• Production-Sourced Evaluation — the orthogonal axis of eval quality: production-sourcing fixes task representativeness, this page fixes grading validity; a representative task graded by an unvalidated judge is still untrustworthy

• Automated Failure Attribution — a benchmark that follows most of this protocol, applied to label quality rather than judge quality: three independent annotators on 100 stratified traces, majority vote, a chance-corrected Fleiss κ = 0.73 rather than raw agreement, an explicit "no clear decisive error" escape hatch (2.0%), and a published transfer matrix showing which families annotators moved labels between. The gap worth naming is one this page's protocol would catch: the human panel ratifies a supplied label while the models predict from scratch, so its 94.0 / 90.0 / 90.0 is not a like-for-like human ceiling against the models' 73.9 / 57.5 / 22.2 and should not be read as one. Extended 2026-08-13: that page's second source repeats the ratify-don't-predict design (annotators shown the judge's own inputs and its verdict, asked to verify the assigned category) and adds the failure this protocol has no step for — validation stratified by category while the headline claim varies by language, developed in the taxonomy-classifier section above

• Optimizer–Evaluator Decoupling — decoupling makes the evaluator independent but not valid; a decoupled-yet-reproducible judge can be maximally biased, so the MVVP is a concrete answer to that page's "what verifies the verifier?" regress

• Evals as Product Spec — "ten great evals" graded by an LLM judge inherit this validation debt; authoring a good eval and validating the judge that grades it are separate disciplines

• Automated Behavioral Audit — the highest-stakes judge deployment in the vault: a judge model scoring safety behavior across dozens of dimensions, feeding thresholded RSP determinations — exactly where inflated agreement and position bias would corrupt a ship/no-ship call

• Verification as the New Bottleneck — LLM-judge validation is the quality-control layer under one imperfect answer to verification-at-scale

• LLM-Assisted Grey-Literature Theory Building — a research pipeline that practices this page's core prescription: it reports the chance-corrected Cohen's κ = 0.75 (not raw exact-match agreement) against a stronger re-judging model as the reliability number for its relevance judge — the MVVP's step 1 applied to a corpus gate

• Motivated Mislabeling — the failure this protocol structurally cannot surface: a judge that labels for the consequence of the label is content-dependent and direction-consistent, so κ, test-retest and position-randomization all read clean; the only control that catches it is varying the stated consequence

• AI-to-AI Coercion — a design that removes the judge from the headline scoring path entirely: the escalation rung is a required tool argument the model under test emits about its own message, so the primary measure has no validation debt at all (a label-neutralization control rules out a demand effect); judges are kept only for the secondary free-text ladder and the two-judge fabrication adjudication

• Stopping Under a Noisy Verifier — the same statistic doing a different job, and the case where this page's prescription backfires. Youden's J is a judge-selection metric here; there it is a deployment parameter that decides whether an agent loop can be steered on the judge's verdicts at all — a low-J judge's pass rate is ρ₀ + J·Q, mostly its own false-accept rate, so it can rise while true quality falls. Two transfers. It converts the MVVP's binary verdict ("this judge is not valid") into a graded operating decision: above J ≈ 0.18 a calibrated loop tracks the true-parameter reference within 2.8pp, at J = 0.03 it collapses by 58.0pp, and the fix is not a better judge but a rule that does not need the judge calibrated. And it is the corpus's counterexample to "measure more carefully" — the label-free mixture estimator that recovers ρ₀ and ρ₁ is identifiable iff J ≠ 0, so at low J a larger calibration sample makes the estimate strictly worse (ρ̂₁ 0.27 at N = 120 → 0.077 at N = 300, true 0.609). Their held-out separation test exists precisely to avoid diagnosing a broken estimator with its own output, which is step 5 of the MVVP arriving as a deployment gate

• Reference-Free Judge Over-Crediting — the multilingual, reference-axis sibling: a second reliability-without-validity instance (temperature-0, perfectly reproducible judges are systematically invalid without a reference), supplying a concrete answer to this page's "biases beyond position bias" open question and extending the evidence into low-resource languages this English-only study lacked. Its second source adds the axis the MVVP structurally cannot cover — validity is not a property of a judge, it is a property of a judge under a given amount of pressure. The same judges hold usable discrimination (0.21–0.38) when they are only rating, and collapse to 0.05–0.17 once a policy is optimized against them, with true accuracy unmoved. Every step of the protocol is a one-shot measurement of a judge nobody is pushing on, so a judge can pass all five and still be worthless as a reward; the check that catches it is not on the list — a held-out signal the judge never sees, re-measured over the course of the optimization

• Tool-Output Pruning — the case that runs this page's suspicion backwards, and locates the blind spot it shares with the field. Selecting a context pruner by per-line F1 — a deterministic, reference-backed, judge-free label-match metric — picks the wrong head: on both hand-examined cases F1 ranks a signature-only pruner above a usable one (0.53 vs 0.49, and 0.80 vs 0.71) while a GPT-5.4-mini judge separates them by 5-6 points (2/10 vs 8/10; 3/10 vs 8/10), and two loss functions match the winner's F1 while their judge scores collapse to 5.30 and 3.03. The mechanism is that F1 scores set membership on the kept lines while the task is whether the retained skeleton supports the agent's next action. This does not weaken the MVVP — it says the validation debt is not specific to judges: an unvalidated label-match proxy carries the same construct-validity gap, and nobody demands κ, position checks, or a human ceiling from a metric that looks arithmetic. Left open on that page: the judge doing the correcting here is itself unvalidated in this role

• Deep Research Agents — the high end of the agreement range, and what buys it. MisKnow-Agent's FCAR judge reaches 99.7% raw agreement and Cohen's κ = 0.993 against two blinded annotators whose disagreements were adjudicated to a single reference label, on a 300-report sample. The instructive part is the task shape: not "score this report" but "does this report's own conclusion, recommendation or executive summary endorse this specific claim?", with an explicit decision rule listing four ways of surfacing the claim (mention, quotation, attribution, hedging) that score as non-adoption. Reducing an open-ended report grading to one sharply-bounded binary is what moves κ from the 0.4–0.75 band typical of rubric grading to near-unity — a design lever for judge validity that sits upstream of every measurement in the MVVP

• The Verifiability Thesis — the horizon this page's discipline bounds. Karpathy's "council of LLM judges" is the construction that extends the reward signal into soft domains; the MVVP validates one judge, and Yang's ρ = 0.66–0.97 says the council's headroom over that one judge is small by measurement, before anything optimizes against it

• Benchmark Score Redundancy — where this page's debt gets inherited by a proof. DeepMind's CollabEval derives unbiased estimates with asymptotically valid confidence intervals for a model's benchmark mean at a fraction of the annotation budget — and on three of its five datasets the quantity being bounded is an autorater's output (GPT-4 Turbo win-rate on AlpacaEval, AutoAIS on AQA, MetricX on WMT24++). The interval is over sampling uncertainty in the mean of whatever the rater produced, so kappa deflation, position bias and reference-presence sensitivity all pass through it untouched: a 30%-tighter CI around a judge with κ = 0.48 is a more precise estimate of a badly-measured thing. The composition that would close the gap already exists in the same literature (stratified PPI relating cheap autorater labels to expensive human ones), but that paper does not build it — which makes "statistically efficient" and "validated" fully orthogonal properties of an eval pipeline

• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis: unvalidated grading is one of the five corruption channels of the public benchmark number, and the MVVP is the validate-the-judge move in the five-part replacement portfolio

• Deterministic Engineering for Agent Code Review — a same-matcher defense caught in the act. Every figure in OpenCodeReview's twelve-configuration comparison is a Qwen3-235B-A22B-Instruct semantic-match verdict, and §6's mitigation — same matcher, same protocol, so relative comparisons hold — is this page's central distinction (reliability is not validity) restated as though it settled the question. No κ against human matching decisions, no position or order controls, and the five-run-mean variance control is a homogeneous jury: this page's ρ ≈ 0.66–0.97 says repeated sampling of one judge buys far less than independence would. What the paper does have is unusually strong on the other half — AACR-Bench's 1,505 ground-truth comments are expert-verified through three rounds by 80+ engineers — so the failure is squarely the MVVP's missing step 1, not the label quality underneath it

• Weak-Verifier Ensembling — the system this page's Finding C bounds, and the most direct collision in the corpus. Weaver's aggregation is derived from an explicit assumption that "each verifier captures an independent aspect of the correctness"; ρ = 0.944–0.972 (Qwen3) and 0.664–0.706 (MiniMax), both measured on homogeneous juries, say that assumption is false in the direction that costs it, and heterogeneous juries underperforming Poisson-binomial predictions removes the obvious hedge. Three things scope the collision rather than settling it — Weaver's pool mixes trained reward models with judges (never measured here), it filters and weights against real labels rather than voting reference-free, and it runs as a static selector rather than a reward. Its own reported non-monotonicity in ensemble size is the correlated-error signature showing through. Extended 2026-09-10: the identification budget for pricing that dependence is now written down, and it is tight — inter-verifier covariance is σ_t² + σ_c², signal plus shared error, so the split needs external anchors whose own contamination is a free parameter, is unidentified at any anchor count when everything is ordinal, and is undetectable when one prompt template is shared across the whole pool

• What the Instrument Can Resolve: Two Headline Numbers and Their Missing Denominators — the human-ceiling step, promoted from an aside to a reporting standard, and applied outside evaluation. It takes Yang's leave-one-annotator-out calibration and the Shopify pre-judge ceiling as the two answers to Usage-Telemetry Classifier Validation's open question, shows ATLAS already holds the annotations to compute one and stops at an incomparable pair (model-vs-plurality-of-three κ = 0.83 against human-only pairwise κ = 0.66), and reports the resulting (observed − chance) / (ceiling − chance) figures — 73–82% and 85–94% of the human ceiling where the raw numbers read 42.5% and 71.6%. It also supplies a fresh instance of the ratify-versus-predict asymmetry this page names (an approval rate is anchored, a disagreement rate is blind, so the two cannot be divided), and a load-bearing economics result running on a wholly unvalidated ten-way LLM classifier: the 25%/7% screen-out split in Controlled Variance: AI's Edge as Reduced Dispersion, whose codebook thresholds on a variable the treatment moves

• The Price of Mixing Agents, and the Principal Nobody Counted — this page's Finding C doing work outside evals: heterogeneous juries underperforming Poisson-binomial independence is the corpus's only direct test of provider-mixing as a decorrelation intervention, and it reports the direction with no coefficient, which is why no variance-vs-exploitation frontier can be drawn for agent populations. It also flags that Weak-Verifier Ensembling mis-renders the ρ pair as a within-vs-across-family contrast when both ranges are homogeneous-jury values

• Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus — a fourth dependence instrument, and the one that converts ρ̄ directly into a hypothesis-test disagreement rate rather than an accuracy or precision loss: a pooled-vote significance test disagrees with a pair-based one in up to 28% of resampled panels, at the same design-effect factor (1 + 9ρ̄) this page's Finding C reports for jury accuracy. Full treatment on Cross-Model Error Entanglement

• Cross-Model Error Entanglement — the third dependence instrument, and the one that escapes the identification wall above by requiring ground truth instead of anchors: BEI and CIG measure excess co-failure and excess same-distractor collision among 18 models against a difficulty-conditioned null, then predict a judge's over-endorsement of a specific target (ρ = 0.508/0.520, transferring to MATH-500 at 0.441/0.457 on ~34 and ~26 points). Yang's ρ is about the panel, this is about the judge–target pair

• AI-Assisted Error Analysis — the step upstream of anything this page validates. A judge cannot be validated against criteria nobody wrote, and Shankar's argument is that writing them is where automation stops paying: agents may apply a human-authored criterion across a corpus but should not propose the criteria, because validating an agent's proposed taxonomy costs more than authoring your own. The validation burden this page quantifies is the reason

• Benchmark Convergent and Discriminant Validity — the cross-benchmark cost of judge scoring, which no within-benchmark validation measures. Being LLM-judged predicts whether two benchmarks rank models alike (β 0.526) better than claiming the same concept does (−0.058), and swapping one judge for another does not change it. See the section above

• Kernel-Level Proof Auditing — the statement-fidelity section is where a Lean-checked implication metric beat a three-vendor LLM-judge panel on recall (0.930 against 0.093), the case for a mechanical oracle wherever one exists

• Statement Drift — the task on which the three-vendor judge panel fails (recall 0.093): deciding whether a formal statement is the intended one, where a Lean-checked implication reaches 0.930

• Follow-Up Fixes on Agent PRs — a mining-study judge validated close to this page's protocol: binary κ 0.78 against a human-human 0.77 ceiling, Direct-fix precision checked separately on each cohort (27/30 both), rates precision-adjusted, and headline claims kept to the binary label because three-way κ drops to 0.46 judge-human

• Benchmark Task Defects (Spec–Test Mismatch) — agent auditors are judges, validated here the way this page says not to. OpenAI reports a 74% raw category overlap between its Codex investigator agents and five-engineer panels on SWE-Bench Pro, with no chance correction and no inter-rater statistic among the humans. The agent path's biggest miss (low-coverage tests, 4.1% against 9.4%) is a defect that produces no failing trace to read

• Version-Dependent Judge Error — the fourth invariance (evaluated-agent version under a frozen judge), and the first direct audit of a thresholded judge decision in the corpus: Kendall 0.71–0.79 ranking agreement coexists with 8/60 unjustified upgrade declarations, and transported calibration multiplies comparison error 5.2×

Open Questions#

• The MVVP validates reliability and bias; calibration proper (ECE/Brier) is deferred for lack of provider logprobs. How far can a judge's absolute score be trusted for a threshold once confidence calibration is measurable? Partially answered (2026-10-01): calibration is measurable without logprobs, and the measurement belongs to the prompt as well as the judge. Hsiao (Cisco, 2026) (empirical) computes adaptive ECE and a score-spread statistic from a stated 0–100 confidence on 10 closed flagships (GPT, Claude, Gemini) on AggreFact, so the premise of the deferral no longer holds. The answer to "how far" is conditional. With a baseline confidence rubric, flagship AECE runs 8.0–17.7% (Gemini 2.5 Pro worst). One overconfidence-advisory sentence cuts it on 10/10 models, by 1.1–7.1pp, and adding single-call self-debate leaves the full recipe at 3.3–13.5%. So a threshold is trustworthy only against a calibration number measured under the same judge prompt. The prompt also moves the hard verdict, generation-dependently: the same recipe costs pre-2025 judges −1.8pp balanced accuracy and post-2025 judges nothing. Still open because nothing here is a thresholded decision. The tasks are binary faithfulness and quality labels with human gold, and no ship/no-ship or gating threshold is set or audited. AECE is also measured in-distribution, so whether it holds for the items a gate would actually see is untested. Partially answered on the thresholded half (2026-10-01): Li (empirical) audits the ship/no-ship call itself on 20 SWE-bench agent-version pairs against execution labels. Judge-only intervals promote 8 of 60 reference-inconclusive pairs, and old-version calibration is worse than none in 59 of 60 units, so a gating threshold has to be calibrated on the version being gated.
• All judges were run with thinking suppressed. Does reasoning-on flip the consistency–bias paradox, or just move the numbers? Partially answered (2026-09-22) on a neighbouring property, and the answer is unwelcome: Kohli re-runs a nine-judge, seven-family panel on the same 1,000 MNLI items with chain-of-thought and finds mean pairwise error correlation rises — φ̄ 0.391 → 0.456, Kish n_eff 2.18 → 1.94, panel accuracy 72.0% → 69.2% — while prompt rewording, label-order reversal and temperature 0.5 leave all three flat. Shared reasoning amplifies shared error. That does not touch the consistency–bias paradox itself, which is a within-judge reliability-versus-validity claim and would need a position-flip rate measured with reasoning on; it does establish that reasoning-on is not a neutral setting for judge panels, and that a validation protocol which certifies a judge with thinking suppressed is not certifying the configuration a panel would actually be run in. A second neighbouring datum (2026-10-01), on calibration rather than bias: Hsiao varies an in-output reasoning block (not a thinking channel) on 10 flagship judges. Free-form reasoning significantly narrows the confidence distribution on 8/10 judges, which fits a one-directional argument piling probability mass onto the committed answer. Self-debate widens it on 6/10, and it costs pre-2025 judges balanced accuracy (−1.0pp vs the advisory-only prompt) while slightly helping post-2025 ones (+0.6pp). Same direction as Kohli: the reasoning configuration is not neutral, and its effect depends on the judge's generation. The consistency–bias paradox itself is still unmeasured with reasoning on.
• Hosted endpoints drift silently between provider updates. How stable are these agreement/bias profiles over a longer horizon than five weeks — and should judge validation be continuous rather than one-shot? Partially answered (2026-08-04) by Yang et al. (2026), on the announced-upgrade sibling of the question rather than silent drift. Across four released MiniMax generations (M2 → M2.1 → M2.5 → M2.7) on four datasets, adjacent accuracy moves at most 0.022 and not one of the nine adjacent McNemar tests reaches even uncorrected p < 0.05 — and because those tests are paired on parse-shared examples, this is stability of the individual verdicts, not merely of the aggregate. So a deliberate version step at the top of the capability range is a small perturbation to the agreement profile. Three things keep this open. (i) These are version-labeled releases you can pin, not the unannounced same-endpoint drift the question is about — nobody has re-measured a fixed endpoint over months. (ii) Stability of accuracy is not stability of bias: position-flip rates still span 0.117–0.147 across the MiniMax releases, and no one tracked whether that band moves within a single version. (iii) The finding runs the other way on the parameter axis, where a step does move things and can move them down (Qwen3 1.7B → 32B costs 0.065 on Judge's Verdict), so "upgrades are safe" is not the lesson — "upgrades are a measurement event that must be re-validated, and the null case is the lucky case" is. That is an argument for continuous validation, from the direction of the one axis where it was cheap to check.
• Does the paradox generalize beyond position bias — i.e., are there other biases (self-preference, lineage) that high test-retest also masks? Partially answered by Kranti & Vajjala (2026): yes — reference-presence sensitivity is a large one. Their temperature-0 judges (thus perfectly reproducible) systematically over-credit incorrect answers in no-reference settings, an invalidity invisible to any reliability metric until a gold answer is added, which flips up to 85% of verdicts and lifts human-alignment sharply (e.g. Gemma3-27B 0.34→0.85 NR→RV). It does not close the question — self-preference and lineage remain unisolated (their design deliberately overlaps judge and generator but doesn't attribute the effect), and it is a different bias on a different (multilingual QA, three specific judges) setup, not a re-run of this study's position-bias protocol. The self-preference half then largely resolves negatively (2026-08-04), via Zhou (2026): errors optimized against a self-judge transfer to judges from other families that were never in the loop (Llama 0.480 → 0.568, Gemma 0.764 → 0.918) and to same-family judges 3.5× larger (still 77%), with a three-family unanimous-accept ensemble passing 55% and judge acceptances pairwise correlated at φ = 0.29–0.38. So lineage amplifies — the self-judge is the worst single cell at 0.906 — but is not the mechanism; the bias is a shared property of the candidate-conditioned channel. This also adds a bias class no reliability metric on this page can reach, because it is not a property of the judge at all: the same judge, unchanged, is valid before optimization (discrimination 0.31) and invalid after (0.09), so any one-shot validation — including the full MVVP — certifies a judge that will be true only until something starts optimizing against it. That is a direct argument for the continuous-validation question two bullets up. A field datapoint on the same half, 2026-09-10, from GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (empirical): OpenAI's GDPval grader — GPT-5-high, scoring deliverables against a human professional's — is reported by its own authors to agree less with human expert graders on outputs from capable OpenAI models, cited to Panickssery et al. (2024). That is lineage effect surfacing inside a shipped validation number rather than in a lab protocol, and it is invisible to test-retest by construction, since a judge is perfectly self-consistent about preferring its own family. It does not reopen the negative resolution above — the paper reports it observationally with no controlled arm, and agreement also falls with model capability generally, which nothing in the paper separates — but it is the first instance in this corpus of a vendor publishing the effect against itself.

Sources#

• Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus — Elias Hossain, Niloofar Yousefi & Ser-Nam Lim (UCF), arXiv 2609.22512, 2026-09-18, empirical. Cited here for §4.1 (the pooled-vote vs. pair-based significance test, Figure 2, the design-effect variance-inflation factor 1 + (n−1)ρ̄) and its own operational checklist item 2 ("treat the item, not each judge vote, as the unit of statistical inference"). Full source citation, evidence handling and parse note on Cross-Model Error Entanglement

• What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks — Desai et al., arXiv 2609.08812, 2026-09-08, COLM 2026, empirical. Cited here for §4.3 (partial Mantel betas under three-way and binary format coding; SGBench-mcq clustering), Appendix D.2 and Table D.3 (original-paper judges where available, Qwen3-30B default, temperature 0), and Appendix D.5 Tables D.4–D.5 (era split +0.43 / +0.84; judge swap ≤ 0.06, Mantel contrast +0.58 under both; reconciled exact against pdftotext -layout pp. 43–44). Full treatment on Benchmark Convergent and Discriminant Validity

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Patwardhan et al. (19 authors, OpenAI), arXiv 2510.04374 v1, 2025-10-05, 29pp (empirical). Cited here for §2.5 and Appendix A.6 only: the agreement definition E[1 − |H − A|] on ordinal {0, 0.5, 1} scores and the stated reason for declining Cohen's/Fleiss's κ and Krippendorff's α; Figure 4b's 65.7% human–grader agreement against a 70.8% human inter-rater ceiling (figure viewed directly at compile time); A.6.2's self-preference observation on capable OpenAI models with the Panickssery et al. (2024) citation and the "highest agreement on weak models" confound; A.6.3's 12-of-220 ungradable tasks. Vendor-built grader for a vendor-built benchmark — full COI on GDPval Benchmark

• A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges — Kuai, Jiang, Zhu, Wang, Wu, Li, Zhang, Liu, Tu, Fan & Zhou (Texas A&M / Marquette / Utah), A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges, arXiv 2604.07650 v2, 2026-08-09, 19pp, COLM 2026, empirical. Cited here for §3.1 (the leave-pair-out difficulty estimator and the conditional-independence null), §4.1 (the two disjoint 1,000-question MMLU-Pro subsets and the three judges — Llama-3.1-70B-Instruct, GPT-5, GPT-4o-mini), §4.2 (the ΔPrec definition and Table 1's Spearman coefficients) and §C.1 (the difficulty-response calibration diagnostics). The ~34 / ~26 sample sizes and the confidence intervals above are wiki arithmetic, back-solved from the reported (ρ, p) pairs because the paper reports no n anywhere; the paper's own accounting does not close either (three judges × 17 targets = 51 would give p ≈ 1.4 × 10⁻⁴, not 0.002). All seven tables were re-reconciled against pdftotext -layout at compile time and match cell-for-cell. Full treatment, parse notes and COI on Cross-Model Error Entanglement and in wiki/sources.md

• A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model — Veerendra Kumar Sunkavalli (Independent Researcher, single author, no institutional affiliation), A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model, arXiv 2609.08826, 2026-09-08, 11pp, empirical. Cited here for §3 (Assumption 1's judge/anchor moment structure and K = σ_t² + σ_c²; Theorem 1's p ≥ 2 judges / m ≥ 2 anchors identification), §5 (Test A as the coefficient of variation of off-diagonal judge covariances, and the panel-wide-residual blind spot), §7 (the all-ordinal non-identification result; Test A on the HANNA human panel at 0.589 vs a 0.247 matched null, p < 0.0005; the six-provider LLM panel at 0.175 vs 0.064, its 0.324–0.870 quality-loading span, and the (0.887, 0.960, 1.026) estimates produced in defiance of the pre-test) and §8 (nothing established about real judges). Tier scope, and it is narrow: the estimator is validated only in simulation and semi-synthetically under oracle calibration, the diagnostics are validated on real data in the rejecting direction only (specificity on a real adequate panel is untested because no adequate panel exists), and no real panel has passed the pre-test, so no ρ_k figure anywhere is a measurement of the world. What this page takes is the negative half, which is real: two real panels, one human and one six-provider LLM, both detectably violate the equal-loading structure. Full source treatment, parse notes and the estimator machinery on Weak-Verifier Ensembling and in wiki/sources.md

• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks — Ludwig, Ahmad, Majumdar & Ginsburg (NVIDIA), Shortcutting the Fix, arXiv 2609.06780, 2026-09-06 (empirical, 16pp). Cited here for §2.2–2.3 (the turn-level judge protocol, the three-judge open-weight panel, majority-vote aggregation and the DISPUTED_CATEGORY rule), Table 5 (three-judge agreement across five models × two benchmarks × two prompts) and Table 6 (the five-judge panel for Kimi-K3, adding Claude Sonnet 5 and GPT-5.6 Luna). The paper reports no human validation of its own classifier, which is why it appears on this page as both evidence and specimen. Tables 5a/5b and 6 were re-verified against the source PDF at compile time; the paper's §D prose gives a 37.5% lower bound for SWE-bench Multilingual where its own Table 5a minimum is 37.8 (the 37.5 is the DeepSWE figure), immaterial here

• Sidekick's continual learning loop — Andrew McNamara & Cody Mazza-Anthony, Sidekick's continual learning loop, Shopify Engineering, 2026-08-05, case-study (first-party account of the authors' own production system; no replication, no adversarial review, every figure self-reported — it cannot upgrade or contradict any empirical finding on this page). Cited for the pre-judge rubric protocol (two expert annotators, blind, 25 random production samples, Cohen's κ with a ≈0.2 rewrite trigger), the judge's-ceiling framing quoted above, the A/B backtest and per-criterion degradation test, and the keep-judges-small prescription. The 83% / 80% pair exists only inside the article's Judges' agreement figure — grep-confirmed absent from the page text — and was read directly under the image two-pass rule; both are raw agreement, not chance-corrected. Full source treatment and evidence handling on Agent Quality Flywheel and in wiki/sources.md

• An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures — Pahuja, Brokman, Hofman, Nizri, Vishna, Goldfarb-Tarrant, Marchisio, Kojima & Vainshtein (Fujitsu Research of Europe / Cohere / Fujitsu Research), An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures, arXiv 2608.03735, 2026-08-04, empirical. Cited here only for App. G (the six-category judge design, its inputs, and Claude Opus 4.8 as the judge model), App. G.4 + Table 11 (the annotation guide's verify-the-assigned-category instruction, the per-category agreement figures, κ = 0.860 and macro-F1 = 0.906), App. H + Table 12 (the eleven-language Common Crawl spread the unstratified claim runs over) and Limitations (the authors' concession on cross-language judge calibration). Table 11's per-category n sums to its own Overall row and 104/117 reproduces the printed 88.9%; the three mutually inconsistent statements of annotation coverage are quoted from §5.2, App. G.4 and Table 11's caption respectively. Full parse notes and COI on Automated Failure Attribution

• User awareness in frontier models — Zhong, Raghunathan, Laidlaw & Steinhardt, Transluce, 2026-08-06 (empirical): Appendix G — the validation this page prescribes, run on an awareness judge: 74 blind hand-labelled items from 11,908 readable-reasoning runs, stratified across judge-positives, lexicon-flagged negatives and double negatives, balanced across families and reweighted to the population; VEA precision 0.73 [Wilson 0.56–0.85] / recall 1.00 [Jeffreys 0.42–0.99], VUA 0.94 [0.81–0.98] / 0.98 [0.74–0.99]; raw per-setup agreement 32/32, 15/18, 18/24 (VEA) and 31/32, 17/18, 22/24 (VUA); and the error analysis placing all 9 VEA false positives in grading, where the task's own vocabulary was misread as evaluation awareness. Also the identity-conditioned grading result, a validity threat no protocol here tests. Full treatment on User Awareness

• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — Norman, Rivera & Hughes (UC Berkeley, arXiv 2606.19544, June 2026), empirical. §2.1 (metric definitions: kappa deflation Δκ, consistency–bias paradox), §4.1 (kappa deflation universal, Table 2), §4.2–4.3 (position-bias heterogeneity, cross-benchmark rank instability), §4.7 (the paradox — Qwen 3 8B, Gemini 2.5 Flash), §4.8 (verbosity bias < 0.011), §4.9 (provider families), §4.5 + App. E (RewardBench position-randomization fix), §5.3 (MVVP), Limitations + App. H (partial-adoption / temporal-drift risks)

• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability — Zongyou Yang, Yinghan Hou & Xiaokun Yang (Imperial College London + Nanchang Institute of Technology, arXiv 2607.08535, 2026-07-09), empirical. §1 (evaluator-replacement ambiguity, RQ1–RQ3), §3.2–3.3 (eight-judge two-axis panel, four datasets, T = 0.1, parse-shared McNemar, Holm over 18 tests), §4.1 + Table 3 (upgrade non-interchangeability; the 1.7B→4B step as the only Holm-surviving gain; PandaLM and Judge's Verdict regressions; no judge wins all datasets), §4.2 (position flip 0.320 → 0.117–0.147, verbosity 0.547 → ≈0.13 under a padding probe, Pearson r = −0.957, M2.7's residual 14.7%), §4.3 (ρ-corrected beta-binomial; ρ = 0.944–0.972 Qwen3 / 0.664–0.706 MiniMax; K = 1,3,5 → 0.463/0.475/0.482; heterogeneous juries also sub-independence), §4.4 (debate shifts +0.243 to +0.317 with no parser audit trail), §4.5 (human ceilings, Arena seed spread 0.004–0.032), Table 4 (the minimal audit trail), §6 (threats to validity)

• Parse note. Ingest flagged one table-collapse cell; the corpus-wide pattern held — Tables 2, 3 and 4 each had every data row welded into a single grid row (3, 4 and 6 rows respectively), which the checker surfaced for Table 3 only. All three were recovered with pdftotext -f 4/5 -layout against the local PDF and reconciled cell-for-cell: value order inside each collapsed cell is correct, no digit is wrong, nothing was dropped — only the row mapping was lost. Table 3 as quoted above is the recovered version. Table 1 parsed clean.

• Figures read from the page images per the two-pass rule. Figure 2 supplies the per-model LLMBar accuracies (Qwen3 0.463/0.617/0.647/0.678; MiniMax 0.829/0.839/0.832/0.832; GLM-5.1 0.900, mimo-v2-pro 0.883) that reframe the MiniMax null as a ceiling effect — this is not in the prose, which reports only adjacent differences. Figure 5c confirms the per-judge ρ values quoted from prose. Figure 3's McNemar grid gives the uncorrected p-values (Qwen3 1.7B→4B: LLMBar and Arena p <.001, PandaLM 0.090; 14B→32B PandaLM 0.032, which does not survive Holm).

• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation — Mengting Chen et al. (FinStep + StepFun, arXiv 2607.29252, 2026-07-31), empirical. Cited here for the item-level half only (full treatment on LLM-as-a-Judge): §2.1 (z_j as reproducible judgeability, explicitly short of expert endorsement), §3.3 (Beta–Bernoulli measurability posterior; the two-judge degeneracy), §4.2 (κ 0.604 → 0.743; r = 0.589/0.558 vs 0.127; positive-label rates 55.6–62.9% against human gold 47.1%), §2.4 + App. B.5 A3 (correlated LLM judges as an acknowledged, unmeasured threat to the agreement posterior)

• SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization — Han et al., arXiv 2608.29270, 2026-08-29, empirical. Cited here for the LLM-as-judge baseline row only (Table 4 and Table 22, agreeing with the prose: precision 0.235, recall 0.093, agreement 0.785, Spearman 0.72). Full treatment on Kernel-Level Proof Auditing.

• Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models — Yu-Chung Hsiao (Cisco Systems, single author), arXiv 2609.10996, 2026-09-10, empirical. Cited here for Table 2 (per-model AECE under Rubric and Ours), Table 8 (the advisory's 10/10 AECE reduction, −1.1 to −7.1pp) and Table 3 (the era split of the reasoning-block and full-recipe BA effects, recovered from pdftotext -layout because the markdown table is welded). Full treatment and parse notes on Verbalized-Confidence Soft Scoring for LLM Judges and in wiki/sources.md