# SkillEvaluator
[](LICENSE)
[](https://www.python.org/)
[](https://docs.nvidia.com/skills/skillevaluator/)
[](https://arxiv.org/abs/2608.20614)
[](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/)
> 📺 **Livestream:** [Ask the Experts: Evaluating Agent Skills](https://www.youtube.com/watch?v=4rI0zATFxYI)
SkillEvaluator is an open-source tool that checks AI agent skills and measures their effect on task results:
start with quality checks, then compare live runs with and without a skill.
Agent skills are folders of instructions and supporting files defined by the
[Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of
the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills).
## Bring skill evaluation to your repository
Check changed skills in GitHub or GitLab with quality checks, SkillSpector
security scanning and reports. Choose folders and thresholds; no evaluator code
to import.
**GitHub guided setup:** run inside your checked-out repository:
```bash
pip install --upgrade skillevaluator
skillevaluator ci init
```
Choose checks and optional hosted feedback, then review the setup PR.
[CLI setup guide](docs/cli-onboarding.mdx).
**Manual setup:** copy a filled-in template from
[CI templates](https://github.com/NVIDIA/SkillEvaluator/tree/main/ci/bundles/0.8.0). No local CLI needed.
| Your repository | Ready-to-use template | Copy destination |
| --- | --- | --- |
| GitHub | [Skill checks](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/0.8.0/skill-checks.yml) | `.github/workflows/skill-checks.yml` |
| GitLab | [Skill checks](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/0.8.0/gitlab-skill-checks.yml) | `.gitlab-ci.yml`, or merge into your existing pipeline |
| GitHub reports and PR comments | [Optional publisher](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/0.8.0/publish-skill-reports.yml) | Second workflow; complete publisher setup |
See [GitHub](docs/github-actions.mdx) and [GitLab](docs/gitlab-ci.mdx) setup,
[template reference](ci/templates/README.md) and [report examples](ci/examples/README.md).
## Quickstart
Both paths use the bundled `text-analyzer` skill.
| Start here | Requires | First result |
| --- | --- | --- |
| [Run without an LLM](#run-without-an-llm) | Git and uv | A deterministic quality score and findings |
| [Use NVIDIA Build](#use-nvidia-build) | Git, uv, and a Build API key | Model-backed rubric scores and feedback |
Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and Git,
then get the example:
```bash
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
```
uv provisions Python 3.13 and dependencies. Run from this checkout; for a standalone CLI, see
[Installation](https://docs.nvidia.com/skills/skillevaluator/installation).
### Run without an LLM
```bash
uv run --python 3.13 skillevaluator quality-check \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-quality
```
No API key, scanner, GPU or Docker is needed. The example scores **84.5/100 (B)**,
exits `0` and saves `reports/first-quality/skillevaluator-quality.json`. The
passing threshold defaults to 70.
This checks static quality; publication evidence stays `INCOMPLETE`. Security
needs scanners; agent impact needs live evaluation. Replace the example path
with your skill directory. See the
[keyless walkthrough](docs/quickstart.mdx#run-without-an-llm).
### Use NVIDIA Build
Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for
**free prototyping**, subject to [access terms and rate limits](https://docs.api.nvidia.com/nim/docs/product).
1. Sign in at [build.nvidia.com](https://build.nvidia.com/) and complete enrollment.
2. Open a model page, select **Get API Key**, and generate your key.
3. Set the provider and key in the same terminal, then run the rubric check:
```bash
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-build
```
The command sends skill documentation to the model and saves feedback in
`reports/first-build/skillevaluator-rubric.json`. No agent runtime or Docker is
needed. Completed evaluations can still fail rubric thresholds.
The same key supports Tier 2 embeddings and supported Tier 3 agents. See the
[Build walkthrough](docs/quickstart.mdx#use-nvidia-build) or [provider configuration](docs/configuration.mdx).
## Three-tier overview
Each tier is independent. Add the relevant extra to your checkout command:
| Tier | Question | Example after setup | Additional setup |
| --- | --- | --- | --- |
| Tier 1: Validation | Is the skill safe and well-formed? | `uv run --extra security skillevaluator tier1 ./my-skill --no-llm` | External scanners for full static coverage; a provider and model clients enable LLM checks |
| Tier 2: Deduplication | Does guidance repeat or overlap? | `uv run --extra tier2 skillevaluator tier2 ./my-skill` | Embeddings and chat; add `--catalog ./skill-catalog.json` for inter-skill comparison |
| Tier 3: Live Evaluation | Does the skill help the agent? | `uv run --extra tier3 skillevaluator tier3 ./my-skill` | Provider access, a supported agent, and a running sandbox (Docker by default) |
Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they
are missing; Semgrep is optional. See [scanners](docs/installation.mdx#system-tools).
`--no-llm` does not remove scanner requirements; a scanner that cannot run is
`INCOMPLETE` and exits nonzero.
[SkillSpector](https://github.com/NVIDIA/SkillSpector) provides specialized skill
security scanning; [Harbor](https://github.com/harbor-framework/harbor), the
open-source agent evaluation framework, powers sandboxed agent runs.
Tier 2 reports when inter-skill comparison did not run. Tier 3 reuses evaluation
sources or creates one starter case with limited coverage. See the
[tier guides](https://docs.nvidia.com/skills/skillevaluator/) for details.
## Run the combined pipeline
After setting up the provider, scanners, and agent sandbox, run all three tiers:
```bash
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill
```
`doctor` checks readiness, not inference success. `validate` runs all tiers with
autopilot for skills. Use `--tiers 1,2` to skip live evaluation or `--tiers 1` for
static checks. `--no-autopilot` requires an existing source. Other artifact types
retain their applicable stages.
Tier 1 always gates `validate`. Tier 2 gates unless `--no-block-on-dedup` is set;
Tier 3 gates only with `--block-on-agent-eval`. Model and managed-sandbox usage
can incur charges; see [runtime and cost](docs/tier3-live-evaluation.mdx#plan-for-cost).
For broader coverage, generate and review additional cases. Templates omit
negatives unless authored in `EVAL.md`:
```bash
uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full
```
## Run from a source checkout
Use `uv run --python 3.13` with the relevant extra. See
[Installation](docs/installation.mdx#choosing-extras) for extras and standalone upgrades.
## Research and citation
Resources: [paper](https://arxiv.org/abs/2608.20614),
[PDF](https://arxiv.org/pdf/2608.20614),
[DOI](https://doi.org/10.48550/arXiv.2608.20614),
[BibTeX](https://arxiv.org/bibtex/2608.20614), and the
[NVIDIA Developer blog](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/).
Cite the paper and record the SkillEvaluator release or commit; see [CITATION.cff](CITATION.cff).
## Documentation
See [the documentation](https://docs.nvidia.com/skills/skillevaluator/) for
setup, tier guides, reports, CI integration, CLI reference, and
[performance dashboard](docs/reports.mdx#performance-dashboard). To let an agent run
SkillEvaluator, install [`skills/skillevaluator/`](skills/skillevaluator/).
## Installation and third-party software
Follow the [installation guide](https://docs.nvidia.com/skills/skillevaluator/installation)
to choose the full installation or a smaller per-tier setup.
This project will download and install additional third-party open source
software projects. Review the license terms of these open source projects before
use.
## Contributing
Read [CONTRIBUTING.md](CONTRIBUTING.md) and run:
```bash
make lint && make test && make build
```
See [project governance](GOVERNANCE.md) and the [Code of Conduct](CODE_OF_CONDUCT.md).
## Support
Support level: **Experimental**. Community-supported, best-effort, with no SLA
or NVIDIA enterprise support entitlement. Report bugs and feature requests through
[GitHub Issues](https://github.com/NVIDIA/SkillEvaluator/issues); see
[SUPPORT.md](SUPPORT.md).
## Security
Report suspected vulnerabilities using the private process in
[SECURITY.md](SECURITY.md). Do not disclose security issues in a public GitHub
issue.
## Releases
Release changes are recorded in [CHANGELOG.md](CHANGELOG.md) and
[GitHub Releases](https://github.com/NVIDIA/SkillEvaluator/releases).
## License
Apache License 2.0 — see [LICENSE](LICENSE), [NOTICE](NOTICE), and
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.