NVIDIA SkillEvaluator: Three-Tier Agent Skill Quality Gates (Open Repo Guide)

NVIDIA/SkillEvaluator is the open CLI behind NVIDIA's push to treat Agent Skills like tested software — static gates first, overlap detection second, live agent runs last. The repo trended with ~130 new GitHub stars in a day and ships on PyPI as skillevaluator, with docs at docs.nvidia.com/skills/skillevaluator.

If you read our August coverage of NVIDIA ACES — where a large share of skill runs failed to beat baseline — SkillEvaluator is the tooling layer NVIDIA wants maintainers to run before those embarrassing trajectories hit production.

TL;DR: three tiers at a glance

Why NVIDIA open-sourced another skills repo

Agent Skills folders are deceptively simple — a SKILL.md, a few scripts, maybe a checklist. But agents treat them as executable policy. NVIDIA's README frames SkillEvaluator as part of the Verified Skills pipeline tied to the Agent Skills specification.

The problem ACES quantified still stands: document scans correlate weakly with runtime lift. SkillEvaluator operationalizes that insight into repeatable gates teams can wire into pull requests, not slide decks.

Support level is Experimental per README — community best-effort, no enterprise SLA. That matters when you compare it to hosted eval vendors; here you own the runners, scanners, and sandbox costs.

Reproducible benchmark stamps representing SkillEvaluator tier-one static checks on agent skills

Tier 1: validation without burning tokens

Tier 1 is where most teams should start. The README's keyless walkthrough clones the repo (or uses PyPI install) and runs skillevaluator quality-check against the bundled text-analyzer reference skill. Example output in docs lands near 86.2/100 (B) with default pass threshold 70.

Checks cover harness compatibility, static structure, and security scanners. README notes SkillSpector and Gitleaks fetches when missing; Semgrep is optional. A scanner that cannot run marks INCOMPLETE and exits non-zero — so "no API key" does not mean "no dependencies."

For builders already shipping skills to Claude Code or Codex, Tier 1 is the fastest feedback loop: catch broken frontmatter, unsafe shell patterns, or missing resources before you ask an LLM to judge prose quality.

Evaluation contamination metaphor for overlapping duplicate agent skill instructions

Tier 2: duplication is a product bug

Skills proliferate — every vendor ships a "write better prompts" skill. Tier 2 embeddings compare intra-skill and optional catalog overlap so maintainers see when two skills teach the same workflow.

The combined validate command blocks on dedup by default unless you pass --no-block-on-dedup. That design nudges monorepos toward a single canonical skill per task, which is healthier for agent context budgets than ten near-duplicate folders.

Tier 3: live agent evaluation with Harbor

Tier 3 answers the ACES question directly: with vs without the skill installed, does success rate move? README points at Harbor for sandboxed agent runs and documents autopilot modes that synthesize starter eval datasets when you lack traces.

Costs are real: model calls, Docker sandboxes, and managed endpoints can bill per run. README's doctor command checks readiness without guaranteeing inference success — run it before you wire nightly CI.

Optional NVIDIA Build keys unlock hosted endpoints for prototyping under published rate limits — useful for rubric scoring (skillevaluator rubric-eval) when you want model-judged feedback on documentation quality without standing up your own judge model.

CI and the SkillEvaluator skill meta-layer

Two install paths matter for teams:

1. skillevaluator ci init — generates GitHub (or GitLab template) CI that runs validation on changed skills.
2. npx skills add NVIDIA/SkillEvaluator — drops a meta-skill into Claude Code, Codex, or other skill-aware agents so you can ask qualitative questions in natural language.

The meta-skill does not replace Tier 3 sandboxes; it helps authors interpret reports and fix failing gates interactively.

How this connects to explainx.ai's ACES coverage

Our ACES post reported NVIDIA's finding that a substantial fraction of skill runs did not beat baseline and that static scans alone correlated near zero with runtime outcomes. SkillEvaluator is the engineering response: encode tiers, ship CLI, publish arXiv 2608.20614, and document report JSON schemas.

If you maintain internal skills for loop engineering or customer agents, treat ACES statistics as the why and SkillEvaluator as the how.

Practical workflow for a new skill repo

1. Scaffold with the Agent Skills layout (SKILL.md, scripts/, resources/, examples/).
2. Run Tier 1 locally in CI on every pull request — cheap, deterministic.
3. Add Tier 2 once you have more than a handful of skills in a catalog JSON.
4. Promote to Tier 3 only for skills that change agent behavior on high-risk tools (payments, prod deploys, customer data).
5. Archive reports (skillevaluator-quality.json, rubric outputs) next to version tags so regressions are auditable.

Reading SkillEvaluator reports without drowning in JSON

Each tier emits structured artifacts — skillevaluator-quality.json for static passes, rubric outputs when you enable LLM judges, and Tier 3 traces when Harbor finishes a paired run. The README's performance dashboard documentation describes how to collapse those files into a single pass/fail gate for release managers who will never open a terminal.

For engineering leads, the useful habit is to attach reports to skill version tags the same way you attach SBOMs to container images. When ACES-style lift drops after a innocuous README edit, you want a diffable JSON trail proving whether Tier 2 overlap crept in or Tier 3 success rates moved. NVIDIA's experimental support tier means you should not wait for a ticket queue — fork the templates in ci/bundles/ if you need custom thresholds per business unit.

Synthetic datasets and --force eval generation

Tier 3's create-eval-dataset command deserves its own planning meeting. The README warns that autopilot starters are limited coverage and that --force replaces autogenerated cases — meaning you can accidentally bake in trivial positives that make a harmful skill look helpful. The documented pattern is: write substantive evals/EVAL.md guidance first, optionally supply a --prompt file with realistic task requests, then review negatives before you trust --block-on-agent-eval.

That workflow mirrors how mature ML teams build golden sets for RAG — except the unit under test is procedural text an agent might follow literally. If your organization already runs Claude Code review pipelines, port the same "human approves eval set" rule before CI starts blocking merges on Tier 3.

Security scanners inside Tier 1

SkillSpector and Gitleaks integration is not decorative. Skills are attractive supply-chain targets because they ride inside trusted agent contexts. A malicious scripts/ hook could exfiltrate environment variables the next time a developer asks Claude to "run the skill's setup." Tier 1's incomplete scanner state exits non-zero so you cannot silently skip security stages in CI — plan runner images that allow those binaries or accept that local laptops may need one-time downloads pinned by SkillEvaluator.

Pair scanner gates with organizational policy: only install skills from namespaces you track, pin commits on first install, and re-run skillevaluator validate after every upstream pull — the same hygiene we recommend for trending agent browser CLIs.

When not to run Tier 3 on every pull request

Harbor sandboxes and judge models cost money and queue time. A sensible maturity curve keeps Tier 1 on all PRs, Tier 2 on nightly main builds, and Tier 3 on release candidates or skills tagged production. NVIDIA documents --tiers 1,2 shortcuts on the combined validate command for that reason.

Teams maintaining dozens of micro-skills for internal SOPs may discover Tier 2 dedup matters more than live agent eval — two departments writing nearly identical "how to file an expense" skills will confuse routers long before runtime lift shows up in metrics.

Comparison with rolling your own evals

Related reading

• NVIDIA ACES skill lift — why runtime eval matters
• What are Agent Skills? — folder format SkillEvaluator expects
• Google Stitch Skills — another trending skills marketplace
• eStack vs gStack agent skills — how teams package skill stacks
• Claude Code token efficiency — why duplicate skills hurt context

Official sources: NVIDIA/SkillEvaluator README · SkillEvaluator docs · NVIDIA developer blog · arXiv 2608.20614

Livestream and paper trail

NVIDIA's README links a YouTube livestream titled "Ask the Experts: Evaluating Agent Skills" and arXiv 2608.20614 with DOI 10.48550/arXiv.2608.20614. When you cite SkillEvaluator in internal RFCs, cite both the tool version and the paper — reviewers will ask whether your lift numbers came from static scans or Harbor-backed Tier 3 runs. The developer blog walkthrough mirrors the CLI quickstart and is the fastest onboarding link for managers who will not clone the repo.

CLI flags and tier behavior reflect the main-branch README as of November 15, 2026.