Trained Calibration

Sources#

• Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
• AI models have likely reached parity with superforecasters on ForecastBench
• Building Prod with Jev and LangGraph
• Claude Opus 5 System Card
• Inkling: Our Open-Weights Model
• Introducing System One Models & Jev
• Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
• User awareness in frontier models
• What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

Summary#

Calibration — expressing the right amount of confidence, including on unsettled questions — treated as a training target with its own reward design, rather than a byproduct of scale or a constitution line. The concrete recipe is TML's "epistemics" stack for Inkling (July 2026, vendor-claim), the first release in this wiki to describe end-to-end how calibration gets trained in:

1. RL against proper scoring rules on a large corpus of resolved real-world questions — the forecasting-theory device (a scoring rule is proper when truthful probability reporting maximizes expected reward) imported directly into the reward function.
2. Abstention-aware short-form QA rewards: answering only pays off when the model is likely to be right, so the optimal policy is to answer when confident and otherwise say "I don't know" or hedge. Some prompts encourage or forbid hedging, so the user's preference for a forced guess versus a calibrated non-answer stays controllable — calibration without hedging-as-a-tic.
3. A dual grader for long-form instruction following: a rubric grader (checklist of what a good answer contains) paired with a claims grader that extracts each factual claim and verifies it by agentic web search, penalizing claims that don't check out.

The dual-grader design is a reward-hacking patch#

TML names the failure mode outright: rubrics "emphasize recall and can be hacked by models spraying plausibly relevant facts hoping to match rubric items." That is Reward Hacking in the instruction-following reward channel — Goodharting a checklist. The claims grader is the counter-pressure: every sprayed fact becomes a liability, because unverifiable claims are penalized individually. TML's claim is that the pair improves helpfulness and reduces hallucination simultaneously, where either grader alone trades one for the other.

The claims grader is also a deployed answer to Reference-Free Judge Over-Crediting: it does not rely on the judge model's own knowledge to score factuality — the exact condition under which reference-free LLM-as-a-Judge evaluation over-credits fluent wrong answers — but grounds each verification in retrieved evidence. The over-crediting result predicts that a knowledge-only claims grader would reward confident confabulation; putting search in the reward loop is the mitigation shape that result implies, moved from eval time into training time.

Forecasting as the resulting evaluation#

If calibration is trained on resolved questions, forecasting benchmarks are its natural exam: integrating conflicting sources into a probability, scored by Brier-type rules where overconfidence is punished symmetrically with underconfidence. TML reports Inkling at 61.1 ForecastBench Brier Index (no search) — level with Gemini 3.1 Pro (61.1) and Grok 4.3 (61.7), above GPT-5.5 (59.1) and well above Claude Opus 4.8 (54.6) — and 0.1617 on Prophet Arena, within noise of the closed frontier. All vendor-claim, on a pre-release checkpoint, tested June 30–July 13, 2026. Taken at face value, the striking part is not the win but the parity: a 41B-active open model matching frontier closed models on calibration while trailing them badly on hard reasoning (HLE 29.7 vs 44–53) suggests calibration is cheap relative to capability — a property of reward design more than of scale.

What the benchmark's own operator reports (2026-07)#

The exam named above now has a primary source, and it changes what a number on it means. ForecastBench's operator (Forecasting Research Institute, 2026-07-16, empirical) reports that several tournament submissions are statistically indistinguishable from the superforecaster median, with one above it on market questions. Three things follow for this page.

The vendor number and the leaderboard number are not the same measurement. TML's figures are a bare model with no search, answering from weights. The tournament leaderboard "is intended to capture the frontier of LLM forecasting ability, so submissions may make use of tools, added context, fine-tuning, ensembling, or other methods" — and the system at the top is a multi-stage pipeline that generates sub-questions and search queries, retrieves current context, filters for relevance and recency, ensembles models, then has another model analyze the reasoning. The frontier of this benchmark is system calibration, not model calibration. Any claim of the form "model X is at the forecasting frontier" needs to say which of the two it means; TML's table and this leaderboard sit on opposite sides of that line.

It supplies the human anchor the vendor table lacked. The superforecaster median scores 63.7 on dataset questions (N=521) and 75.9 on market questions (N=56). TML's quoted figures (Inkling 61.1, Gemini 3.1 Pro 61.1, Grok 4.3 61.7, GPT-5.5 59.1, Opus 4.8 54.6) fall below that dataset figure — if the two are the same index, which cannot be confirmed from either source: TML says "Brier Index" and the leaderboard's columns are labelled only "Dataset / Market / Overall (N)". Treat the band as suggestive, not as a placement.

And the training target is the metric's own family. RL against proper scoring rules on resolved real-world questions is, structurally, training on the thing ForecastBench measures — which is what makes "favorable ground" a live worry rather than a rhetorical one, and what makes the pipeline result above interesting: the top submissions get there by retrieval and ensembling rather than by having been RL'd toward the scoring rule at all.

The mechanism underneath the recipe, and what it does and does not say about this one (2026-08)#

The recipe above is stated as a design; Che, Yuan, Zhao & Yu (arXiv 2608.00301, 2026-07-31, empirical) supply the training-dynamics theory of the design surface it sits on, with a proof, a decay law, and live RL runs at 1.5B/7B. Their result is that error-penalized abstention rewards — +1 correct, −λ wrong, 0 abstain, the rule whose rational optimum is Chow's threshold t* = λ/(1+λ) — can drive a KL-anchored gradient learner to refuse everything while its logged mean reward rises like 1/t, because when abstention is a discrete action the reward gradient and the KL anchor's restoring force carry the same saturation factor σ′(v) and die together. Full mechanism, conditions, estimator analysis and repair on Error-Penalized Abstention Training. Three things follow for this page, and they do not all point the same way.

The paper's prescribed repair is this recipe's item 1. Their fix for the collapse is: answer every question, train a mandatory confidence report under a strictly proper score, and threshold only at deployment — because a report channel has no gate to saturate, so no shared factor can kill its reward gradient and its anchor together, and its calibrated optimum is interior and attracting (Proposition 8). That is structurally TML's proper-scoring-rule RL. The paper cites Damani et al. (2025) as the closest published relative and supplies "the missing half" of why that shape works. So the strongest reading of this source is not a criticism of the recipe — it is an independent derivation of why the recipe's continuous-report half is the right channel, and the corpus's first mechanistic argument that proper-scoring-rule RL on a report is safer than reward-shaping an abstain decision.

Two conditions the recipe's published description does not let us check. The collapse needs three ingredients jointly: (B1) blanket answering loses score in expectation on the training mixture; a shared bounded readout transmitting that loss to profitable prompts; and abstention being a discrete action. TML's item 2 says the model should "answer when confident and otherwise say 'I don't know' or hedge" — a free-form refusal with no designated decision position, which is precisely the case the paper marks untested: every one of its live collapses installs a binary first-token gate in the model's own vocabulary, and Theorem 2's proven cases exclude the generic shared-decoder shape a free-form refusal would occupy. Likewise (B1) is a property of TML's undisclosed training mixture, and the paper notes that curated or retrieval-augmented data "plausibly violate" it. So this paper does not show, or claim, that TML's recipe collapses. What it establishes is that the channel geometry of an abstention reward is load-bearing and that TML's release does not report the two facts (decision-position design, mixture score under blanket answering) that would decide it.

Two transfers that are sharp, and both are about the parts TML did not specify. First, the weights are not free (Proposition 9): scoring α·1{correct} + Brier, pure properness (α = 0) makes the calibrated accuracy gradient 2q − 1 < 0 for every q < 1/2, so the model gains score by degrading accuracy on hard prompts and honestly reporting the degradation — on exactly the prompts the loss condition selects. Any α > 1 fixes it; α = 1 leaves a degenerate stationary point at zero. TML publishes no relative weighting of its scoring-rule and correctness channels, and this says the ordering between them is a correctness condition rather than a tuning knob. Second, properness is not the operative variable (Remark 1): clip a strictly proper Brier signal below t* and the dead zone returns and deployment utility gets weakly worse while the rule stays proper. What matters is whether the training signal has a reachable region of identically zero gradient — which reframes the dual-grader discussion below, because the paper bounds its own properness claims to programmatic grading: with an LLM judge crediting hedged text at rate ν, properness degrades by O(ν). A rubric-and-claims grader is exactly such a judge, and ν is unmeasured anywhere. Hedging that a grader quietly credits is the concrete way this recipe's proper scoring rule could stop being proper.

Weight all of it against the scale gap: 1.5B and 7B Qwen2.5 on TriviaQA and PopQA, against the 41B-active model this page's numbers describe, with no long-form or agentic setting touched.

A third route: calibrate the readout, not the model (July 2026)#

Everything above trains the model's words to be calibrated — a proper scoring rule on a stated probability, an abstention-aware reward on a stated refusal. Sarfati, Tiwari, Boppana, Earls, Varadaraj & Ho (Goodfire / Eternis, arXiv 2607.08046, 2026-07-09, empirical) demonstrate the alternative: leave the weights alone and fit a small linear readout on the frozen activations instead. It is cheaper than either recipe on this page by orders of magnitude, and on the evidence here it is better calibrated than what the recipes produce.

The numbers, on frozen models. Training only probe weights on ~11,800 improved-context forecasting rollouts (3,945 questions × 3 completions), reported on 2,960 held-out test rollouts, with no recalibration of any kind — no temperature, Platt or isotonic scaling:

Model	Probe	Probe ECE	Verbalized ECE
GLM-4.7-Flash	mean-pool L28	0.054	0.287
GLM-4.5-Air	attention-1 L18	0.102	0.255

A five- and two-fold reduction in expected calibration error, from a head that never touches the model. The ranking gains are weaker and the authors say so: GLM-4.7-Flash's leakage-controlled ΔAUROC is +0.055 with a 95% CI of [−0.006, 0.111], so its robust claim is calibration alone; only GLM-4.5-Air's (+0.110 [0.069, 0.149]) clears zero.

Three things this changes for the recipe above.

• The signal is already there; the report is what is broken. On an RLVR-trained forecaster (EF-8B), the probe and the model's own stated probability are indistinguishable as rankers (AUROC 0.756 vs 0.758) and differ only in calibration (ECE 0.044 vs 0.093). The model knows which of its answers are likely right; its verbalized number does not transmit it. The authors' framing is the sharp one: "overconfidence is less a failure of self-knowledge than of self-report." That relocates the target of this page's item 1 — a proper scoring rule on the stated probability is training a channel, not creating information.
• RLVR for the task does not fix the channel. EF-8B was post-trained specifically for forecasting and remains consistently overconfident: accuracy flat at 37 ± 1% while mean stated confidence sits near 50% across ten sampling temperatures, ECE 0.110–0.150, essentially temperature-invariant. Calibration does not arrive as a byproduct of task-specific RL.
• A training-time calibration recipe and a probe are not interchangeable. The one head-to-head against a calibration-trained model splits cleanly. Reproducing Ma et al.'s DCPO (decoupled reasoning and confidence tokens under RLVR) on Qwen3-8B and evaluating on pooled OOD math (AIME24/25 + AMC23/24, n = 580): the frozen probe is the better discriminator (AUROC 0.929 vs 0.865 for whole-response token-logprob confidence) while isotonic-recalibrated verbalized confidence is the better point calibrator (ECE 0.125 vs the probe's 0.143). On the untrained base the probe wins both (0.894 vs 0.752; ECE 0.083 vs 0.270). So the recipe-versus-readout choice depends on whether the target is ranking or point calibration — and the authors flag that the DCPO probe-train pool overlaps the RLVR training distribution through DeepScaleR, which blocks reading this as training-invariant representation.

The deployment consequence, and its precondition. A probe supplies a confidence estimate at every rollout depth, including mid-reasoning before any answer is committed, and costs no generated tokens — where this page's recipes buy their calibration with a stated probability the model must emit and a user must trust. The precondition is white-box access: the technique runs only on weights you hold, which is the same asymmetry recorded on White-Box Activation Monitoring. It is also not a substitute on the abstention half — a probe scores correctness, it does not decide whether to answer.

A fourth route: calibration as the whole objective (TypeSafe, September 2026)#

Every recipe above adds calibration to a model whose main job is producing text. TypeSafe AI's Jev (launch post, 2026-09-15, vendor-claim) removes the text: a non-autoregressive decision model whose outputs are schema-declared bools, scores and choices, each returned with a probability, trained by what TypeSafe calls RL for Calibrated Decisions (RLCD) — "answers with epistemically honest probabilities" as the entire optimization target, framed against RLHF (rater-preferred text) and RLVR (checkable outputs). Structurally this is the recipe Error-Penalized Abstention Training recommends taken to its end: a mandatory confidence report on every output, no abstain action to gate, thresholds left to the caller's code (the vendor's own security-triage workflow acts at P > 0.75 and routes a 0.15–0.60 grey zone to a human). It also sidesteps the verbalization gap the probe result above exposes, since the probability is the output head, not a stated number.

What is missing is any calibration measurement. No reliability diagram, ECE or Brier score is published — the workflow eval scores agreement with a two-LLM reference, not calibration — and no algorithmic detail is given. The one independent datapoint cuts against it: on the shared answer-verification board recorded on Typed Decision Verifiers, Jev's lowest-scoring 20% was 54% already-correct answers and the retry gate built on it netted −0.07pp, despite a co-best AUC. That is ranking-at-a-threshold, not calibration proper, on a task Jev was not pitched for; treat RLCD as a named objective with no public evidence yet that it delivers.

Deployment has run ahead of that evidence. The first partner accounts (LangChain's Jev post, 2026-09-25, vendor-claim) already wire the probability into control flow as a routing threshold — Browserbase's Stagehand sends any browser-action decision under 0.7 confidence to an LLM — and LangSmith ships a trace view labelling the outputs "calibrated." A fixed threshold is only as good as the calibration behind it: if 0.7 does not mean ~70% right, the threshold passes confident errors unescalated and nothing downstream notices. The same post's other property claim, scores that "barely moved across 100 repeated runs," is consistency, which calibration needs but does not imply.

A fifth route, for closed models: instruct the report (September 2026)#

The third route needs weights to probe, and the first two need a training run. Neither is available for a closed judge. [[verbalized-confidence-judge-scoring|Hsiao (Cisco, arXiv 2609.10996, 2026-09-10, empirical)]] calibrates the stated number by prompt alone, on 10 closed flagships (GPT-4o through GPT-5.4, Sonnet 4 through 4.6, Gemini 2.5 and 3.1 Pro) acting as binary faithfulness judges. The lever is one advisory: "You have been consistently overconfident in past evaluations… You are especially overconfident when predicting 'False'." It lowers adaptive ECE on 10/10 models (−1.1 to −7.1pp) and widens the confidence distribution on 10/10 (+6.1 to +30.2pp). Removing it reverses both on 10/10.

Two points bear on this page. First, it is the cheapest evidence yet for Sarfati's "self-report, not self-knowledge" diagnosis on frontier closed models. A single sentence about the channel's bias recovers calibration, so the information was there to report. Second, it shows the reasoning format moves calibration too. Free-form chain of thought narrows the confidence distribution on 8/10 models, and a both-sides self-debate widens it on 6/10. So a proper scoring rule applied to a stated probability (item 1 above) is scoring a quantity that the prompt's reasoning scaffold already shapes. What it does not show: the task is judging with a binary label, not forecasting or abstention. The advisory's direction is hard-coded for a setting where the judge errs strict. Instruction-level calibration is also more fragile than a trained one: it is valid only for the prompt it was measured under, and it cost pre-2025 models 1.8pp of accuracy.

Relation to honesty-as-alignment#

Agentic Honesty & Diligence tracks the same target from the behavioral-audit side: Anthropic's overconfidence evals (admit ignorance about an unseen CLI tool; validate a teammate's wrong example) measure whether calibration shows up in agentic behavior, and Opus 4.8's constitution names "calibrated" as a component of honesty. This page is the training-loop complement: where the audit view asks does the model behave calibratedly, the TML recipe specifies what gradient produces that behavior. The two views meet at an awkward datapoint — the model whose constitution demands calibration scores lowest of the compared frontier models on TML's forecasting table — though a competitor's benchmark of a rival's checkpoint is the weakest possible evidence for that gap.

Connections#

• Diversity Calibration Under SFT — the distributional sense of calibration: output spread (collision ratio) rather than confidence-vs-correctness; SFT on the right target closes that gap from both directions, a different fix from proper-scoring-rule RL
• User Awareness — the input this recipe does not control. A trained self-estimate is only as good as the invariances it holds, and the model's stated probability of solving a hard problem moves with who it believes is asking — 1.5pp at group level, 2.81 population standard deviations for the strongest identity — on questions it does not attempt
• Confident But Unsure — the failure this recipe targets, measured in the wild: Opus 5 states answers its own reasoning cannot support, with accuracy up 11% and hallucination up 6% as its abstention rate falls
• White-Box Activation Monitoring — the readout-side alternative to every reward on this page: a linear head on frozen activations, no weight update, ECE 0.287 → 0.054 and 0.255 → 0.102 on two frozen GLM models, and a confidence estimate available mid-reasoning before any answer is committed. It buys calibration without the channel this page has to train, and pays for it in white-box access
• Pre-Reasoning Commitment — the consequence for routing. A gate on the model's stated confidence underperforms a gate on the spread of its pre-reasoning answer distribution, "because stated confidence tracks how hard a question is rather than whether reasoning would actually change the answer" — a stated calibration failure that costs tokens rather than trust
• Inkling — the model this recipe shipped in
• Verbalized-Confidence Soft Scoring for LLM Judges — the prompt-only route for closed models: an overconfidence advisory plus single-call self-debate lowers stated-confidence ECE on 10/10 flagship judges with no training and no activations, generation-dependently in its accuracy cost
• Reward Hacking — rubric fact-spraying is the named instance; the claims grader is the counter-pressure
• Reference-Free Judge Over-Crediting — why the claims grader must search rather than rely on parametric knowledge
• LLM-as-a-Judge — both graders are judges inside the reward loop; this is judge design under adversarial (Goodhart) pressure
• Agentic Honesty & Diligence — the behavioral-audit view of the same target; calibration as a constitutional dimension vs. as a reward function
• Thinking Machines Lab — the lab; ties to their forecasting fine-tune work on Tinker
• Measuring Beyond Accuracy Saturation — where ForecastBench is treated as an instrument rather than a scoreboard: contamination-proof and non-saturating in score, yet losing discriminating power at the human reference class its headline comparison is built on
• Introspective Coupling — the other trained metacognitive report in this wiki: explanation training rather than calibration training, transferring to behavior it never received labels for, and paying a comparable general-capability toll (TruthfulQA −6.7pp, GSM8K ~−5pp)
• Error-Penalized Abstention Training — the training-dynamics theory of the surface this recipe sits on: why a proper score on a continuous report is the structurally safe channel and a gated abstain action is not, plus the two conditions (decision-position design, mixture score under blanket answering) this recipe's release does not report
• Reasoning–Acting Interleaving (ReAct) — what the alternative looked like. Asked in late 2025 how an agent should decide when to search, CS329A lecture 4 treats calibration as unavailable ("a research problem that is still being solved") and routes around it with unconditional tool use; this page is the recipe that makes the gate buildable instead
• Retrieval Inside the Reasoning Chain — where the late-2025 teaching-grade version of this agenda is recorded. Asked whether output probabilities track correctness, CS329A lecture 7 reports overconfidence and points at exactly this branch without a citation: second-pass confidence estimation, and RL or RLHF aimed at getting the model to not 'emit answers that it has low confidence over'
• Typed Decision Verifiers — the category where calibration is the product: Jev's RLCD makes a calibrated probability the whole output, and the category's one independent board is the only place its gate precision has been measured

Open Questions#

• Does abstention-aware training on short-form QA transfer to calibrated long-form and agentic self-reports (the setting where Agentic Honesty & Diligence shows overconfidence actually bites)? Partially answered (2026-08-12) — and what moved is the question, not the answer. Che, Yuan, Zhao & Yu show that "abstention-aware training" is not one object: a reward shaping a continuous confidence report and a reward shaping a discrete abstain action have different gradient geometry, and only the second has a reachable zero-gradient region where the reward signal and the KL anchor die together. So transfer has to be asked per channel, and a recipe that mixes both (as the one on this page does) can transfer on one half and fail on the other. What is not answered: the paper's live evidence is 1.5B/7B Qwen2.5 on TriviaQA and PopQA with training mixtures deliberately constructed so blanket answering loses score — no long-form, no agentic, no multi-turn, and its own report channel is read out through a linear probe on the prompt state rather than through the model's words, with the verbalized channel explicitly untested. It supplies a mechanism for why short-form results might not carry, and tests transfer nowhere. The settling experiment is unchanged in shape and now has a required control: run the same objective on long-form or agentic self-reports, reporting whether abstention is gated at a designated decision position or emitted free-form.

• The claims grader verifies via agentic web search — importing the search index's coverage, recency, and bias into the reward signal. What does the model learn to claim about facts the index settles wrongly or not at all?

• If calibration parity is achievable at 41B active while reasoning is not, is calibration systematically under-trained in frontier closed models (Opus 4.8's 54.6), or is TML's forecasting table just favorable ground for a model trained on proper scoring rules? Partially answered (2026-08-12) — the dichotomy is missing a third option. ForecastBench's operator shows the benchmark's frontier is held by scaffolded pipelines (retrieval + filtering + ensembling), not by bare models answering from weights, and its tournament rules permit exactly that. So a no-search vendor table and the public leaderboard measure different objects, and the Inkling-vs-Opus-4.8 gap is a bare-model comparison at a point the frontier has already left. Favorable ground is confirmed in one specific sense — the training target and the metric are the same proper-scoring-rule family — while the under-training question is untouched, since no frontier lab has submitted a bare model on a stated protocol. The settling experiment is now nameable: run the same closed models on ForecastBench's question sets with search off and search on, and compare the delta against the pipeline submissions. Partially answered again (2026-09-23), on the under-training half and only for open models. Sarfati et al. measure verbalized calibration directly on three frozen non-TML models and find it poor in all three — expected calibration error 0.287 (GLM-4.7-Flash) and 0.255 (GLM-4.5-Air) on held-out forecasting rollouts, and 0.110–0.150 for an 8B model RLVR-post-trained for forecasting, whose stated confidence sits near 50% against 37% accuracy at every sampling temperature. A linear probe on the same frozen activations reaches 0.054 / 0.102 / 0.044, so the information those models fail to state is present in their own residual streams. That is evidence for under-training rather than favorable ground: the gap is a report failure, not an absent signal, and task-specific RLVR does not close it. What it does not reach: no closed frontier model is measured, ForecastBench is not run, and the metric is ECE on a different question set rather than a Brier Index, so it does not place anyone on this page's table.

• If the probe signal were distilled into the stated probability — training the model to verbalize the confidence its own activations already carry — does the internal signal survive the training, or does optimizing the report degrade the representation the probe was reading? Sarfati et al. name both halves as the natural next step and run neither. It is falsifiable cheaply: refit the same probe after distillation and compare its AUROC and ECE against the pre-distillation head on the same held-out rollouts.

Sources#

• User awareness in frontier models — Zhong, Raghunathan, Laidlaw & Steinhardt, Transluce, 2026-08-06 (empirical): the uncontrolled input under a trained self-estimate: stated probability of solving a hard problem moves with who the model believes is asking (−1.5pp group, −2.81 population SDs for the strongest identity). Full treatment on User Awareness

• Inkling: Our Open-Weights Model — "Epistemics" section (vendor-claim): proper-scoring-rule RL, abstention-aware rewards, rubric+claims dual grader, ForecastBench/Prophet Arena tables

• Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning — Che, Yuan, Zhao & Yu (UNC Charlotte / George Mason / Auburn / North Texas), arXiv 2608.00301, v1 2026-07-31, empirical: the training-dynamics theory of this recipe's design surface. §2–3 (the three joint collapse conditions and the shared-throttling identity), §4 Propositions 8–9 and Remark 1 (why a mandatory report under a proper score is the safe channel, why the correctness weight must strictly exceed the properness weight, and why properness itself is not the operative variable), §7 limitations (free-form refusal untested; verbalized channel untested; the O(ν) LLM-judge degradation that bounds every properness claim to programmatic grading). This source never examines TML's published recipe — the transfer analysis above is this wiki's, stated as such. Full treatment and parse notes on Error-Penalized Abstention Training

• What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness — Sarfati, Tiwari, Boppana, Earls, Varadaraj & Ho (Goodfire / Eternis), arXiv 2607.08046, 2026-07-09, empirical. Used here for §4.1.1 + Appendix B (EF-8B's temperature sweep: accuracy 37 ± 1% against ~50% stated confidence, ECE 0.110–0.150, Brier 0.19–0.21, essentially temperature-invariant), §4.2.2 + Figure 1 (probe vs verbalized confidence at AUROC 0.756 / 0.758 and ECE 0.044 / 0.093, N = 3,020 rollouts), §4.3 + Appendix D + Table 1 (the frozen-GLM probe-only arm, its no-recalibration statement, and the leakage-controlled ΔAUROC intervals), §4.4 + Appendix E + Table 2 (the DCPO head-to-head on pooled OOD math, n = 580) and §5 (the self-knowledge-versus-self-report framing and the probe-distillation proposal). COI: the forecasting models are one author group's own and the probe architecture the other's; experiments were run by Goodfire's agentic research platform under author review. Parse note: ingest verdict warn; Table 2 was a genuine docling row-weld, rebuilt at compile from pdftotext -layout -f 12 -l 12 — the four OOD-math values quoted here are recovered ones and are independently stated in §4.4 prose. Table 1 verified clean cell-for-cell against page 10. Full treatment on White-Box Activation Monitoring

• AI models have likely reached parity with superforecasters on ForecastBench — Forecasting Research Institute (Substack, 2026-07-16, empirical): the tournament leaderboard's stated admission of tools/context/fine-tuning/ensembling, Cassi's pipeline description, the superforecaster median's dataset (63.7, N=521) and market (75.9, N=56) scores, and the 2024 baseline caveat. Leaderboards are PNG screenshots re-read at zoom under the image two-pass rule; the metric is never named in the post's body, so no cross-source score placement is asserted here

• Introducing System One Models & Jev — Diogo Almeida, TypeSafe AI, 2026-09-15, vendor-claim: RLCD named and contrasted with RLHF/RLVR; calibration and consistency claimed, not measured. Full treatment on Typed Decision Verifiers

• Building Prod with Jev and LangGraph — Sydney Runkle & Hunter Lovell, LangChain blog, 2026-09-25, vendor-claim (Jev integration partner): the 0.7 confidence fallback in Browserbase's Stagehand, LangSmith's "calibrated outputs" view, and the 100-run consistency claim. No calibration measurement

• Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models — Yu-Chung Hsiao (Cisco Systems, single author), arXiv 2609.10996, 2026-09-10, empirical. Cited here for Tables 8–9 (the advisory's 10/10 AECE and Spread effects and the 10/10 reversal on removal; free-form reasoning's 8/10 Spread erosion vs self-debate's 6/10 gain), Table 3 (the pre-2025 −1.8pp BA cost, recovered from pdftotext -layout) and Appendix I (advisory text). Full treatment on Verbalized-Confidence Soft Scoring for LLM Judges