Skill Lift

Sources#

• Attackers Target Agents via The Skill Supply Chain
• Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
• EVOMAL: Self-Poisoning in Self-Evolving Coding Agents

Summary#

Skill Lift is the with-skill score minus the without-skill score for a single agent skill, measured in points on a 0–100 rubric, computed once per skill per agent harness and macro-averaged across skill–harness pairs. It comes from NVIDIA SkillEvaluator (open source, August 2026), the first tool in this corpus that treats an individual context artifact as the unit under test, runs a controlled ablation on it, and publishes the delta next to the artifact in the registry that distributes it.

The structural claim is more interesting than the numbers: measurement moves from post-hoc research to a precondition of distribution. NVIDIA's phrasing — "the verified part is the measurement that determines it is ready" — makes a per-skill benchmark run the thing a verified badge means. Skills are packaged, signed capability descriptors for NVIDIA products; each passes three evaluation tiers before publication, and the results ship as benchmarks.json in the nvidia/skills repository.

This is the vendor-side counterpart to Agent Context Files's bounded null. Khatri's ablation found that generic convention context on repositories the agent can already read moves correctness ~0; SkillEvaluator reports +41 points of correctness for task-specific product knowledge the agent probably cannot infer. That is the direction that page's standing open question predicts — and the methodology here cannot settle it, for reasons the "What the design controls" section below sets out.

Evidence note. vendor-claim, assigned at compile — the raw carried no evidence: field. Every number is NVIDIA measuring NVIDIA's own skills against evaluation sets generated from those skills, with a grading procedure the post never names. The methodology is disclosed unusually well for a vendor post (attempt counts, macro-average definition, a pinned snapshot commit, an explicit "we do not report confidence intervals"), which is what makes the specific weaknesses legible rather than hidden. Treat the shape of the finding as informative and the magnitudes as unreplicated.

The three-tier gate#

Each tier answers a different question and can run on its own:

1. Tier 1 — safety and structure. Static checks: schema and frontmatter validation, quality scoring, security scanning for prompt injection and data exfiltration, secret and PII detection, license checks, script linting. No agent runs. This is Agent Supply Chain Risk mitigation applied to the skill artifact rather than to the model or the MCP server.
2. Tier 2 — distinctiveness. Embedding similarity used twice: to find duplicated guidance inside one skill, and overlapping coverage across the catalog. The stated rationale is attention competition — "every skill in an environment competes for the agent's attention, and a skill that loads when it is not relevant can reduce agent performance." Catalog hygiene as a measured property, not a style preference.
3. Tier 3 — live evaluation. The ablation. Built on Harbor (open-source, harbor-framework/harbor) for repeatable isolated agent evaluations; SkillEvaluator converts eval cases into a Harbor task bundle, runs each case twice in separate sandboxes — identical prompt, model, task inputs, and grading criteria, differing only in whether the skill is installed — grades both, and reports the difference.

The workflow is two commands: skillevaluator create-eval-dataset./my-skill --full generates evals/evals.json (explicit, implicit, contextual, and negative cases, each with an ID, prompt, expected output, and optional assertions), then skillevaluator tier3 evaluate./my-skill --agents codex --env-mode docker runs the comparison.

The benchmark#

All figures from the August 12, 2026 snapshot of benchmarks.json at commit 738d79e — 300+ verified skills across 30+ NVIDIA products, evaluated on Claude Code and Codex, macro-averaged so every skill–harness pair carries equal weight.

Dimension	What it measures	Without skill	With skill	Skill Lift
Correctness	Is the final answer correct?	46	87	+41
Discoverability	Does the right skill load when relevant, and stay unloaded when not?	42	82	+40
Effectiveness	Did the agent reach the goal and follow the expected workflow?	39	78	+39
Efficiency	Did it get there without wasted steps or redundant tool calls?	43	78	+35
Security	Does the run avoid unsafe operations, secret leakage, unauthorized access?	97	98	+1
All dimensions				+31
Excluding Security				+39

By harness: Claude Code +34 across all dimensions (+42 excluding Security), Codex +29 (+36). The two harnesses differ by ~5 points, which NVIDIA attributes to different default system prompts, context handling, and tool-calling implementations.

Three readings the post itself supplies, and which the numbers do not support on their own:

• Only Correctness, Effectiveness, and Security are outcome measures comparable across both arms. Discoverability and Efficiency partly score how the skill is used — actions that are unavailable when the skill is absent. The baselines land at 42 and 43 rather than 0 because the without-skill arm still earns credit for productive tool use, clean execution, and correctly leaving the skill unloaded on unrelated tasks. So of the +31 headline, two of five contributors are not lift in the ordinary sense; NVIDIA says to read them as evidence that the skill activates and is used correctly once installed.
• Security's +1 is a regression check, not a gain. The 97 baseline means the objective was to verify that installing a skill introduces no security regression.
• Scores are not pass probabilities. 46 → 87 on Correctness is a higher average rubric score on the evaluated specialized tasks, not a 41-point pass-rate jump.

Statistical hygiene#

85% of published skills ran one attempt per task; 15% ran two. No confidence intervals are reported anywhere. Live agent runs vary between runs, so individual skill scores are noise-dominated and only the catalog-wide averages (aggregating thousands of trials) are defensible readouts. The post says all of this in plain language — a disclosure standard Compute-Controlled Benchmarking would like more vendors to meet — but the consequence stands: no per-skill number in benchmarks.json should be treated as an estimate with a known error bar, which is awkward given the per-skill numbers are the ones surfaced to adopters in a marketplace UI.

What the design controls, and what it doesn't#

Within a harness, the ablation is clean: same prompt, same model, same inputs, same grader, separate sandboxes, one variable. That is a real controlled comparison and better than most vendor benchmarking.

What it does not control is where the tasks come from. create-eval-dataset./my-skill generates the evaluation set from the skill under test. The skill's own content therefore fixes the task distribution and the expected outputs, and the without-skill arm is graded on an exam derived from the answer key of the arm it is being compared against. For the question NVIDIA is actually asking — does this skill make an agent better at the job this skill claims to cover? — that is defensible and arguably the only tractable design. For the question a reader will take away — do skills make agents better? — it is not, because a skill that describes an idiosyncratic workflow will generate tasks only it can satisfy, and score high for that reason alone. NVIDIA's own first key finding says as much from the friendly direction: "a skill can only be measured as precisely as its evaluation set describes the job." The measurement is a self-consistency check on the artifact, upgraded to a distribution gate.

The same circularity shows up inside the harness comparison. NVIDIA reports that per-product Skill Lift ranges from roughly +2 to +46 while the two harnesses differ by ~5 points, and concludes "the product matters more than the agent — the domain, the task, and the evaluation design mattered more than the harness." True, and note the last clause: the evaluation design varies with the product, because each product's eval set is generated from its own skill. So a large part of "the product matters more" is "the eval set matters more," which is the circularity restated as a finding.

The grader is never named. Rubric dimensions phrased as "did the agent reach the user's goal and follow the expected workflow" and scored 0–100 imply an LLM judge, but the post reports no judge model, no human-agreement check, no chance-corrected agreement, and no position-bias audit. LLM-Judge Validation's central result is that unvalidated judges routinely overstate reliability by 33–41pp on chance correction alone; a 41-point delta measured by an unvalidated judge is a 41-point delta of unknown provenance. This is the largest single gap in the published methodology.

Token savings are not automatic#

The most useful empirical detail in the post, because it is the one that goes against the vendor's own interest. SkillEvaluator tracks token usage separately from the Efficiency rubric, and two single-attempt examples run in opposite directions:

Skill	Tokens	Wall clock
jetson-optimize-memory	617,306 → 142,540 (−76.9%)	474.9s → 220.0s (−53.7%)
cuopt-install	25,227 → 55,582 (+120.3%)	34.0s → 41.1s (+20.8%)

A skill can more than double the token cost of the task it was written to help with. NVIDIA files this as "identifying an opportunity for further optimization," which is the right response, but the general point is stronger: installing a context artifact has a sign-indeterminate effect on cost, and you find out which sign by measuring.

This is a direct tension with Agent Context Files's Khatri result, which is worth keeping rather than averaging. Khatri found the only effects that survived a correctness null were on cost and latency — a significant cache-creation reduction under selective retrieval, and a ~24% wall-clock drop driven dose-dependently by a runtime warning in the file. That is the one place the context-file literature had a reliable win. SkillEvaluator says the sign of that win is not guaranteed even for purpose-built skills. Both are weakly powered on this axis (Khatri: exploratory, n=5, one repo, one agent; NVIDIA: two single-attempt anecdotes), so the honest reading is that nobody has measured context-artifact token cost with enough power to state a sign, and two independent efforts have now produced results pointing opposite ways.

Where the measurement is being distributed#

Two pilots put eval output at the point of adoption rather than in a paper:

• ClawHub is piloting SkillEvaluator for official organizations: Tier 3 runs, with with-skill and without-skill results shown in an Evals tab so developers see the evaluation signal where they discover and install skills. A nutrition label on a marketplace listing.
• Nous Research's Hermes Agent tested it as an optional advisory scan at install time — SkillSpector (NVIDIA/SkillSpector) checks for PII, Unicode smuggling, script-lint issues, license problems and security findings, surfacing file-line results before installation. 29 passing tests; ~1.4–1.5 seconds per skill scan.

The Hermes integration is the notable one for Agent Supply Chain Risk: it is the first case in this corpus of a skill registry running static analysis inside the install flow, at a per-scan cost cheap enough to be unconditional. It is advisory rather than blocking, and it is scanning artifacts that are, by NVIDIA's description, already signed capability descriptors — signature plus content scan is the posture that page asks for from MCP and does not get.

NVIDIA publishes the skills as plugins for Claude Code, Codex and Cursor, and through Skills.sh, ClawHub, and Hermes Hub — the same artifact distributed across six channels, which is Agentic Work Systematization's cross-vendor distribution observation with a measurement layer bolted on.

The same channel, with no gate on it (August 2026)#

One of those six channels supplied the counterfactual a month later. Zenity Labs (Michael Bargury, 2026-08-06, case-study, vendor-authored) documents an in-the-wild campaign on skills.sh: typosquatted publisher identities, a family of trojanized markdown skills, and a displayed aggregate above 1.7M installs (a platform counter that the marketplace reports as installs and not unique users — the author is explicit that "we don't know the number of unique victims") accumulated over four weeks of trending before a third party found it by detonating a batch of marketplace skills in a sandbox. Two readings for this page.

It is the strongest available argument for Tier 1 as a precondition of distribution, and for SkillSpector's install-time placement over a publication-time one, since the campaign's artifacts were installed by agents whose users never saw a scan. The registry that carried it published a monotonically rising install counter and nothing about content.

And it names the gate's untested edge. Tier 1 and SkillSpector both describe checks on a skill — schema and frontmatter validation, prompt-injection and exfiltration scanning, secrets, PII, licenses, script lint. The campaign's malicious text was never in the skill file: it sat in a secondary setup-installation.md the agent was told to open only at install time, reached through cross-references from sibling skills, so a scan that opens the front file and stops finds a clean artifact. Neither NVIDIA's post nor the Hermes pilot says whether the scan resolves a skill's reference tree, and it is the one question the field case makes load-bearing. Filed as an open question on Agent Supply Chain Risk; the composition problem in its detection framing is on MCP Tool Poisoning.

The term this ablation multiplies by, and nobody was reporting it (September 2026)#

Skill Lift is a difference between two arms: the agent with the skill and the agent without it. Lin et al. (arXiv 2605.30621, empirical) measure the quantity that determines whether the first arm is what it says it is — the skill-load rate, the fraction of trajectories in which the agent actually brings a skill into context — and it is not close to 1.

On SkillsBench, across six backbones: 0.251 for Qwen3-32B, 0.446 for GPT-OSS-120B, 0.794 for Haiku 4.5, and 0.957–0.961 for Qwen3-235B, Sonnet 4.6 and Opus 4.6. Read off the runner's action log, not judged. At 0.251 the with-skill arm is three-quarters a without-skill arm wearing a label, and the measured lift is a diluted estimate of the skill's effect on the trajectories where it fired.

Three consequences for this page's instrument.

• The lift number carries a silent activation term, and its size is model-dependent. The same skill measured on two backbones can differ in reported lift entirely because of how often it loads, with no difference in the artifact at all. SkillEvaluator's finding that "the product matters more than the agent — the domain, the task, and the evaluation design mattered more than the harness" is measured on two harnesses and a fixed model; it has not been measured across capability tiers, which is where this term bites.
• The ceiling on lift is not 1 − baseline, it is the adherence rate. Loading is necessary and not sufficient: harness-following rate on the same six models runs 0.142 to 0.757, and it does not track the load rate — Qwen3-235B loads as often as Opus 4.6 (0.961 vs 0.957) and follows less than half as often (0.350 vs 0.757). A publication gate that passes a skill on lift alone cannot tell a skill that is hard to follow from a skill that is wrong.
• It sharpens the eval-set circularity critique above rather than softening it. Tasks generated from the skill under test should, if anything, make activation easier — the task text is drawn from the artifact that is supposed to fire on it. Lin et al.'s activation failures happen on tasks where the agent has already correctly identified which skill it needs, and fails at the action-protocol layer instead.

Caveats that belong with the numbers: the diagnosis is SkillsBench-only, the model set is pre-Claude-5, and the harness-following rate is an LLM-judge output with no agreement statistics — the same LLM-Judge Validation discount this page already applies to SkillEvaluator's own unnamed grader. Full treatment on Harness Activation and Adherence.

Connections#

• Harness Configuration Defects — the static half of a skill gate, validated, and its stated limit. A model-free scanner whose findings are re-derived at pinned commits finds 3.5% of published collections shipping a skill outside the specification and 3.7% shipping a shell pre-approval in allowed-tools — the checks Tier 1 and SkillSpector run. The same paper cites ACES (arXiv 2608.20614) for a 0.14 correlation between structural scans and measured skill effect, which is the case for pairing a static gate with this page's live ablation rather than choosing one
• Human-Governed Skill Maintenance — the opposite sign from a design with the opposite weakness: a pre-registered, powered comparison of maintained vs earliest skill versions on 143 transfer tasks finds −0.09 (CI [−0.28, +0.10]) against this page's +41 Correctness — with judges below their own reliability gate on author-and-model-constructed tasks, where this page's eval set is generated from the skill under test. Neither has an independently sourced task set, which is the experiment this page's first open question asks for
• Harness Activation and Adherence — the precondition this instrument assumes and never measures. A with-skill/without-skill ablation is only an ablation if the skill loads; at a 0.251 load rate it is mostly a without-skill arm, and the adherence rate above it caps how much lift the artifact could have produced even when it did load. The two instruments compose — Skill Lift measures the artifact, load and following rates measure the consumer — and neither is interpretable alone
• Agent Context Files — the counterpart experiment and the reason this page matters. Khatri's controlled two-agent ablation finds a bounded null on correctness for naturalistic convention-and-style context on repositories the agent can read in full, with the surviving effects on cache and wall clock; SkillEvaluator reports +41 correctness for task-specific proprietary product context. The artifact classes are different in exactly the way that page's open question isolates — inferable vs not-inferable context — so the direction is confirmatory and the magnitude is not usable, since NVIDIA's tasks are generated from the skill and Khatri's come from merged PRs with a hidden gold-test oracle
• Agentic Work Systematization — the adoption curve; this is the efficacy measurement that curve never had. Codex telemetry shows skill use rising 5.4%→26.6% of weekly-active users and 53% of reused skills never modified; Skill Lift is the missing second axis, and the two together frame the real question: adoption is measured, quality is now measurable, and the maintenance discipline that connects them is still the gap
• Agent Supply Chain Risk — Tier 1 and SkillSpector are supply-chain scanning moved onto the skill artifact: signed descriptors, prompt-injection and exfiltration scans, secret/PII detection, license checks, script linting, and a ~1.4s advisory scan wired into a third-party install flow. The 2026-08 Zenity campaign is that page's first in-the-wild instance and this page's counterfactual — the same distribution channel with nothing scanning it, and a payload placed one dereference past where a per-skill scan looks
• MCP Tool Poisoning — the detection literature this gate sits downstream of, and where the same evasion is stated formally. ShareLock's threshold fragmentation defeats per-tool scanning with information-theoretic secrecy; the Zenity campaign gets a weaker version of the same property for free by putting the loader in a referenced file. The distinction matters for what a gate can be asked to do: the first is impossible to scan, the second is merely not scanned, so extending Tier 1 to a skill's transitive reference closure is a specification rather than a research problem
• Harness Build-vs-Buy — rung 3 of the customization ladder (skills and plugins) acquires an instrument. The ladder's argument is to customize at the highest layer that works; Skill Lift is how you find out whether rung 3 worked before descending to rung 4
• LLM-Judge Validation — the unaddressed dependency. Five 0–100 rubric dimensions with no named grader, no agreement statistics, and no bias audit; every headline number inherits whatever reliability that judge has
• Production-Sourced Evaluation — the opposite sourcing choice, and the sharper contrast in the corpus. DRACO draws tasks from real production traffic independent of the system being scored; SkillEvaluator generates tasks from the artifact being scored. Representativeness versus perfect topical alignment, with the failure modes exactly mirrored
• Evals as Product Spec — the same idea one step further along: here the eval set is generated from the spec, so the spec's own precision becomes the ceiling on measurement precision, which is NVIDIA's first key finding stated as a design constraint
• Compute-Controlled Benchmarking — the disclosure standard this post partly meets (pinned snapshot commit, stated attempt counts, explicit no-CI caveat) and partly misses (no cost or effort budget per arm, and the with-skill arm demonstrably spends different token counts than the without-skill arm)
• Evaluation Awareness & Grader Gaming — the adjacent failure mode this design leaves open: when the exam is generated from the artifact under test, high scores are consistent with the artifact having specified an easy exam, and nothing in Tier 3 distinguishes the two
• Harness Shrinkage as Models Improve — the durability question. A skill supplying proprietary API facts should hold its lift as models improve; one supplying workflow scaffolding should erode. benchmarks.json is re-run continuously, so this is one of the few places in the corpus where shrinkage is directly observable over time
• Verification as the New Bottleneck — measurement as a gate on distribution rather than a report after it; the skill registry version of shifting verification left
• AI-Native Organization — Tan's mapping makes trigger evals the skill file's performance review, but a trigger eval only tests that the right skill loads; Skill Lift is the other half of the review — whether the loaded skill changed the outcome — and the per-product spread (+2 to +46 for the same skill) is the case the org metaphor has no slot for
• Agent Self-Poisoning (the CREATE-Path) — the pre-publication gate tested at exactly the stage it was built for, and past it. Tier 1 is static safety-and-structure scanning of a submitted skill before catalog entry. EvoMal (Wu, Shi et al., Queen's University, arXiv 2608.25776, empirical) runs four mainstream detectors at that admission stage against eight planted skills that read as ordinary SE helpers: a name blocklist catches 0%, Llama-Guard-3-8B 25%, and Prompt-Guard-86M catches every seed but at a 47% false-positive rate on the 232 benign entries — the only detector that would have held the line is the one that would reject nearly half the catalog. Bandit catches all eight through a single urllib.request.urlopen signature that a one-line swap removes. The deeper point for this page is structural rather than a matter of tuning: once the consuming agent writes new skills back into the same store, Corollary 2 says a submission-side screen reduces infection only through what it keeps out at admission and provides nothing after it. A publication gate remains necessary and stops being sufficient the moment the catalog is also a write target
• Write-Then-Trusted — the failure class the install-time placement answers, and the limit of the answer. A marketplace listing is RC2 at population scale: the trust decision is made against an install count and a trending rank, both monotonic and backward-looking, and the bytes behind the identity are swapped later. An install-time scan is a re-check at the last point where one can run — but it is still checked once, and the 53% of reused skills never modified after adoption are the population no later re-check reaches
• Harness Value Is a Product, Not a Score — Why the Artifact-Payoff Questions Keep Returning Partially Answered — generalizes the activation term this page identifies into the full product an end-to-end score composes (activation × adherence × artifact quality × task headroom), and sets Skill Lift among the corpus's interventional instruments — the with/without arms are what make it informative, and the reason no observational proxy substitutes for them

Open Questions#

• Does Skill Lift survive an evaluation set the skill's author did not generate? The discriminating experiment is cheap and entirely within NVIDIA's reach: build a held-out task set from product documentation independently of the skill, re-run the same Tier 3 ablation, and publish both numbers. If the lift collapses, benchmarks.json measures skill–eval agreement rather than agent capability.
• Who or what grades the five dimensions, and has that grader been validated? Nothing in the post names the judge model or reports agreement statistics; LLM-Judge Validation's Minimum Viable Validation Protocol is the bar. Falsifiable by NVIDIA publishing the grading configuration alongside benchmarks.json.
• Does per-skill lift decay as models improve? The catalog is re-evaluated continuously against a pinned-commit history, so a second snapshot on a newer model generation would show whether skills supplying not-inferable product facts hold their lift while workflow-scaffolding skills erode — the first direct measurement of Harness Shrinkage as Models Improve on context artifacts. Trigger: a benchmarks.json snapshot on a subsequent frontier model release.

Sources#

• Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator — Michelle Horton, NVIDIA Technical Blog, 2026-08-20, ~1.8k words. vendor-claim (assigned at compile; raw untagged). All figures from the 2026-08-12 benchmarks.json snapshot at commit 738d79e in NVIDIA/skills. HTML-derived, not PDF — _system/pdf-table-parsing.md does not apply; the three tables are clean markdown and internally reconcile (the reported +31 and +39 averages match the per-dimension lifts, and the two harness rows average to them)
• Attackers Target Agents via The Skill Supply Chain — Michael Bargury (Zenity Labs), Attackers Target Agents via The Skill Supply Chain, labs.zenity.io, 2026-08-06, case-study (vendor-authored — Zenity sells agent security and the piece previews a Black Hat USA talk on agent detonation; the OSV / Amazon Inspector corroboration, commit SHAs, archive captures and published hashes are treated as fact, the detonation results are the vendor's own instrument, and the install counters are platform-displayed and explicitly not unique-user). Cited here for the skills.sh distribution channel, the displayed install figures, and “Hiding in progressive discovery” — the referenced-file placement that a per-skill scan does not reach. Full treatment on Agent Supply Chain Risk
• EVOMAL: Self-Poisoning in Self-Evolving Coding Agents — Wu, Shi, Q. Li, Zhao, X. Li, Adams, Hassan & Ni (Queen's University), arXiv 2608.25776, 2026-08-26, empirical. Cited here for §9.1 with Table 5 (the four detectors evaluated at admission on the planted seeds and after authoring, with false-positive rates on a 232-skill benign pool) and Theorem 1 with Corollary 2 (why submission-side screening cannot reach the artifact the agent authors). Full treatment on Agent Self-Poisoning (the CREATE-Path)