NVIDIA-NeMo/eval-author

Letzte erfasste Quellaktivität: 30. Oktober 2026 um 21:00

SkillsMP-Katalog aktualisiert: 5. November 2026 um 19:57

gesammelte Skills: 13

GitHub-Stars: 10
GitHub-Forks: 1

Mit einem KI-Assistenten installieren

Diesen Prompt kopieren und in den verwendeten KI-Assistenten einfügen.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/NVIDIA-NeMo/eval-author.

README des Repositorys

README-Quelldatei: README.md

README-Quelldatei lesen

Vollständige README anzeigen

# NeMo Eval Author ![Status: Research Preview](https://img.shields.io/badge/Status-Research%20Preview-orange) Turning your agent's intended behavior into representative custom evaluation cases is hard. Each case needs checks that distinguish success from failure, and passing cases can still leave important behavior untested. NeMo Eval Author is a set of tools that guides your coding assistant to help you create evaluations and investigate gaps in behavior coverage for your custom evals. It provides skills that work with your agent's repository, its intended behavior, and any available execution traces (records of the steps your agent took). You can build a first evaluation suite, plan new cases, or define an audit specification without traces. A full coverage audit requires saved traces from completed evaluation runs. For example, the results of an audit might report: > **Recovery from rejected tool arguments lacks trace evidence.** The agent is > expected to correct an invalid argument, but the measured traces do not cover > that behavior. This does not establish that a recovery test is missing. Review > the existing cases to decide whether to add a test or collect more evidence. > [!IMPORTANT] > **Research preview** > > NeMo Eval Author is an early-stage applied research project. > Workflows and interfaces may change. Review generated cases, graders (the > checks that score an agent's work), and coverage findings before relying on > their results. See [known issues](KNOWN_ISSUES.md) for current limitations and practical guidance on task difficulty and authoring prompts. # Set up ## Prerequisites NeMo Eval Author has tested support for [NeMo Gym](https://docs.nvidia.com/nemo/gym/latest/) and [Harbor](https://docs.harborframework.com/) eval formats. Authoring defaults to Harbor when neither is specified. NeMo Gym is compatible with Harbor, and the user has the final say. NeMo Gym authoring needs NeMo Gym v0.6.0+ in its separate Python 3.13.14+ environment; Harbor authoring needs its CLI and supported Python environment. See the [first-eval guide](docs/first-evals.md) and [Harbor setup guide](skills/eval-author/references/harbor-setup.md). Full creation and improvement onboarding verifies the selected runtime before Ethos. Explicit inventory or planning-only requests do not require an evaluation runtime. All three journeys review or create an **ETHOS.md** describing your agent's purpose, boundaries, and success criteria. An existing document can stay in your agent's directory; see [document selection](docs/getting-started.md#load-the-skills). Install the [Ethos skills](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use) to confirm an existing document or create one with you: ```bash npx skills@latest add nvidia/skills \ --skill ethos-orchestrate --skill ethos-explore --skill ethos-write \ --agent claude-code --yes ``` Run from your agent's repository, replace `claude-code` with your assistant's identifier, and add `--global` for a global installation. The discovery helper requires Python 3.11+. Audit generation, validation, and aggregation require Python 3.11+, PyYAML, and jsonschema. Measuring coverage from ATIF traces requires Python 3.12+ and the [audit dependencies](skills/eval-author-audit/requirements.txt), including Harbor. Task execution may need Docker, application access, and model credentials, depending on the task and agent. See the [full requirements](docs/getting-started.md#requirements) for each workflow, including local trace conversion and experimental environment preparation. ## Start here To use NeMo Eval Author, you need a coding assistant that can read and write project files and run shell commands, [Node.js 22.20+](https://nodejs.org/en/download) (including npm and `npx`), and [Git](https://git-scm.com/downloads). In a terminal, change to the root of the repository containing **your agent**, then install for your assistant: ```bash cd /path/to/your-agent-repository npx skills add NVIDIA-NeMo/eval-author --skill '*' --agent claude-code --yes ``` Replace `claude-code` with your assistant's identifier: | Coding assistant | `--agent` value | | --- | --- | | Claude Code | `claude-code` | | Codex | `codex` | | Cursor | `cursor` | | Pi | `pi` | See the [Skills CLI documentation](https://github.com/vercel-labs/skills#supported-agents) for other assistants. Installation adds ten skill directories with their instructions, scripts, schemas, templates, and references. Harbor and Python dependencies are separate [workflow requirements](docs/getting-started.md#requirements). Verify the installation from the same directory, using the same assistant value: ```bash npx skills list --agent claude-code ``` For a global installation, add `--global` to this check too. Look for `eval-author` and its supporting skills. The CLI may show other assistants that share a skills directory; it has not installed those assistants. See the [installation guide](docs/getting-started.md#install-eval-author-with-npx) for scope, file locations, and troubleshooting. To install without Node.js, follow the [manual installation steps](docs/getting-started.md#manual-installation-without-nodejs). ## Choose your journey Eval Author supports three journeys: **build your first evaluations**, **audit your existing coverage**, and **improve your evaluation suite**. Each starts by checking the prerequisites for your goal and confirming how your agent should behave using `ETHOS.md`. Open your agent's repository in your coding assistant and copy the prompt for your goal. Replace the bracketed values; for creation, choose NeMo Gym or Harbor. ### Build your first evaluations Start a suite for an agent that does not have evaluations yet. Have the selected NeMo Gym or Harbor runtime available; no existing suite, traces, or audit is required. ```text Use eval-author to build my first evaluations for [agent name] using [NeMo Gym or Harbor]. I don't have evaluations yet. ``` See [Build your first evaluations](docs/first-evals.md). ### Audit coverage, identify gaps, and recommend improvements Review your evaluations against your agent's intended behavior, identify coverage gaps, and get recommendations grounded in the findings and evidence limits. You need an existing suite and traces from completed evaluation runs. Eval Author confirms trace availability during prerequisites; you select the traces within the audit. ```text Use eval-author to audit coverage of the existing evaluations for [agent name]. The suite is in [path/to/evals]. I have traces from completed evaluation runs. ``` See [Audit coverage](docs/existing-evals.md#audit-coverage). ### Improve your evaluation suite Add or strengthen evaluations in an existing NeMo Gym or Harbor suite. No prior audit or traces are required to start; the selected evaluation runtime must be available. ```text Use eval-author to help me improve the existing evaluations for [agent name]. The suite is in [path/to/evals]. ``` Current authoring routes extend Eval Author starter suites or propose additions from reviewed audit findings. Broader suite expansion is still in development. ### Other requests You can also ask for a specific operation: | Say to your coding assistant | Guide | | --- | --- | | "Find the evals in this repository and explain how to run them." | [Discover existing evaluations](docs/getting-started.md#find-existing-evaluations) | | "Check whether my NeMo Gym or Harbor evals are ready to run." | [Check readiness](docs/existing-evals.md#check-readiness) | | "Create a NeMo Gym evaluation from this audit gap." | [Author and prove a NeMo Gym task](skills/eval-author-task-create/references/gym-tasks.md) | | "Use this audit report to propose the next evals to add." | [Plan new or improved evaluations](docs/existing-evals.md#propose-new-evaluations) | | "Explain Intake trace TRACE_ID in workspace WORKSPACE." | [Inspect a trace](docs/traces.md#inspect-an-intake-trace) | | "Turn this local trace into a Harbor eval task." | **Experimental** · [Build a task from a trace](docs/traces.md#build-an-evaluation-task) | | "Explain this eval report and what I should check next." | [Read your results](docs/results.md) | | "Review this task derived from a trace before I share it." | **Experimental** · [Review the task for publication](docs/traces.md#sharing-generated-tasks) | Include the relevant file paths, trace IDs, or workspace names in your request. ### Experimental trace-environment loop The [trace-environment workflow](docs/traces.md#build-an-evaluation-task) turns trace evidence into a private candidate task, validates it with Harbor's `nop` and `oracle` or NeMo Gym's native validation and repeated controls, and uses the findings to refine the task. NeMo Gym output uses a private native report; automated publication and readiness reporting remain Harbor-specific. It can start directly from a trace or be paired with [dataset proposals](docs/existing-evals.md#propose-new-evaluations): propose a useful scenario, construct it from supporting trace evidence, then validate and refine it. ```text Use this audit report to propose an eval, then use eval-author-trace-environment to build a private candidate from the supporting trace. Validate it when the required runtime is available, and report any remaining blockers. ``` A proposal does not establish that a task can be built or that its environment works. The loop may produce no candidate or an unproven candidate; successful validation still does not measure your agent's performance. Proposal-only requests stop at recommendations. See the [trace guide](docs/traces.md#build-an-evaluation-task) for evidence requirements and validation, and the [publication review](docs/traces.md#sharing-generated-tasks) before sharing a task. [Development](DEVELOPMENT.md) · [Skill reference](DEVELOPMENT.md#skill-reference) · [Support](https://github.com/NVIDIA-NeMo/eval-author/issues) # Research Topics ## Questions we are exploring - How can intended behavior become representative cases with meaningful success checks and execution environments? - How can generated tasks challenge an agent in realistic ways and distinguish stronger behavior from weaker behavior? - How can evaluations and traces reveal gaps without overstating coverage? - How can developers decide which evaluations to add or improve next? ## Help shape the research Try Eval Author with your agent and tell us where the approach needs work: - Where do proposed cases miss behavior that matters to you? - Where do graders judge outcomes incorrectly? - Which coverage findings leave you unsure what to test or investigate next? Generated tasks can be too easy. Where do your agents pass without exercising the behavior you wanted to test? An anonymized example of the task, how the agent passed, and what would make it meaningfully challenging would help us improve task generation. If an NVIDIA contact shared this preview with you, send your feedback through that contact and ask them to route it to the Eval Author research team. Otherwise, use the [NVIDIA Developer contact form](https://developer.nvidia.com/contact) with **Eval Author research preview feedback** in the subject. Include your workflow, what you expected, what happened, and a small anonymized example. Please avoid sending raw traces, prompts, credentials, or user data in an initial message. For code contributions, see the [current contribution policy](CONTRIBUTING.md). ## License Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. Eval Author is licensed under the [Apache License, Version 2.0](LICENSE). See [NOTICE](NOTICE) for attributions and the external-materials disclaimer, [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for dependency license disclosures, and [third_party/licenses.jsonl](third_party/licenses.jsonl) for the machine-readable inventory. The bundled JSON examples and synthetic test fixtures are NVIDIA-authored and covered by the project license. This project is currently not accepting contributions. Internal maintenance is limited to the NVIDIA ASE team. See [CONTRIBUTING.md](CONTRIBUTING.md) for the current policy, sign-off, and review requirements.

README des Repositorys wird geladen…

Skills in diesem Repository

Klassifizierung ausstehend