NVIDIA-NeMo/eval-author

Latest recorded source activity: October 30, 2026 at 21:00

SkillsMP catalog refreshed: November 5, 2026 at 19:57

skills collected: 13

GitHub stars: 10
GitHub forks: 1

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/NVIDIA-NeMo/eval-author.

Repository README

Repository README source: README.md

Read the source README

Expand full README

# NeMo Eval Author ![Status: Research Preview](https://img.shields.io/badge/Status-Research%20Preview-orange) Turning your agent's intended behavior into representative custom evaluation cases is hard. Each case needs checks that distinguish success from failure, and passing cases can still leave important behavior untested. NeMo Eval Author is a set of tools that guides your coding assistant to help you create evaluations and investigate gaps in behavior coverage for your custom evals. It provides skills that work with your agent's repository, its intended behavior, and any available execution traces (records of the steps your agent took). You can build a first evaluation suite, plan new cases, or define an audit specification without traces. A full coverage audit requires saved traces from completed evaluation runs. For example, the results of an audit might report: > **Recovery from rejected tool arguments lacks trace evidence.** The agent is > expected to correct an invalid argument, but the measured traces do not cover > that behavior. This does not establish that a recovery test is missing. Review > the existing cases to decide whether to add a test or collect more evidence. > [!IMPORTANT] > **Research preview** > > NeMo Eval Author is an early-stage applied research project. > Workflows and interfaces may change. Review generated cases, graders (the > checks that score an agent's work), and coverage findings before relying on > their results. See [known issues](KNOWN_ISSUES.md) for current limitations and practical guidance on task difficulty and authoring prompts. # Set up ## Prerequisites NeMo Eval Author has tested support for [NeMo Gym](https://docs.nvidia.com/nemo/gym/latest/) and [Harbor](https://docs.harborframework.com/) eval formats. Authoring defaults to Harbor when neither is specified. NeMo Gym is compatible with Harbor, and the user has the final say. NeMo Gym authoring needs NeMo Gym v0.6.0+ in its separate Python 3.13.14+ environment; Harbor authoring needs its CLI and supported Python environment. See the [first-eval guide](docs/first-evals.md) and [Harbor setup guide](skills/eval-author/references/harbor-setup.md). Full creation and improvement onboarding verifies the selected runtime before Ethos. Explicit inventory or planning-only requests do not require an evaluation runtime. All three journeys review or create an **ETHOS.md** describing your agent's purpose, boundaries, and success criteria. An existing document can stay in your agent's directory; see [document selection](docs/getting-started.md#load-the-skills). Install the [Ethos skills](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use) to confirm an existing document or create one with you: ```bash npx skills@latest add nvidia/skills \ --skill ethos-orchestrate --skill ethos-explore --skill ethos-write \ --agent claude-code --yes ``` Run from your agent's repository, replace `claude-code` with your assistant's identifier, and add `--global` for a global installation. The discovery helper requires Python 3.11+. Audit generation, validation, and aggregation require Python 3.11+, PyYAML, and jsonschema. Measuring coverage from ATIF traces requires Python 3.12+ and the [audit dependencies](skills/eval-author-audit/requirements.txt), including Harbor. Task execution may need Docker, application access, and model credentials, depending on the task and agent. See the [full requirements](docs/getting-started.md#requirements) for each workflow, including local trace conversion and experimental environment preparation. ## Start here To use NeMo Eval Author, you need a coding assistant that can read and write project files and run shell commands, [Node.js 22.20+](https://nodejs.org/en/download) (including npm and `npx`), and [Git](https://git-scm.com/downloads). In a terminal, change to the root of the repository containing **your agent**, then install for your assistant: ```bash cd /path/to/your-agent-repository npx skills add NVIDIA-NeMo/eval-author --skill '*' --agent claude-code --yes ``` Replace `claude-code` with your assistant's identifier: | Coding assistant | `--agent` value | | --- | --- | | Claude Code | `claude-code` | | Codex | `codex` | | Cursor | `cursor` | | Pi | `pi` | See the [Skills CLI documentation](https://github.com/vercel-labs/skills#supported-agents) for other assistants. Installation adds ten skill directories with their instructions, scripts, schemas, templates, and references. Harbor and Python dependencies are separate [workflow requirements](docs/getting-started.md#requirements). Verify the installation from the same directory, using the same assistant value: ```bash npx skills list --agent claude-code ``` For a global installation, add `--global` to this check too. Look for `eval-author` and its supporting skills. The CLI may show other assistants that share a skills directory; it has not installed those assistants. See the [installation guide](docs/getting-started.md#install-eval-author-with-npx) for scope, file locations, and troubleshooting. To install without Node.js, follow the [manual installation steps](docs/getting-started.md#manual-installation-without-nodejs). ## Choose your journey Eval Author supports three journeys: **build your first evaluations**, **audit your existing coverage**, and **improve your evaluation suite**. Each starts by checking the prerequisites for your goal and confirming how your agent should behave using `ETHOS.md`. Open your agent's repository in your coding assistant and copy the prompt for your goal. Replace the bracketed values; for creation, choose NeMo Gym or Harbor. ### Build your first evaluations Start a suite for an agent that does not have evaluations yet. Have the selected NeMo Gym or Harbor runtime available; no existing suite, traces, or audit is required. ```text Use eval-author to build my first evaluations for [agent name] using [NeMo Gym or Harbor]. I don't have evaluations yet. ``` See [Build your first evaluations](docs/first-evals.md). ### Audit coverage, identify gaps, and recommend improvements Review your evaluations against your agent's intended behavior, identify coverage gaps, and get recommendations grounded in the findings and evidence limits. You need an existing suite and traces from completed evaluation runs. Eval Author confirms trace availability during prerequisites; you select the traces within the audit. ```text Use eval-author to audit coverage of the existing evaluations for [agent name]. The suite is in [path/to/evals]. I have traces from completed evaluation runs. ``` See [Audit coverage](docs/existing-evals.md#audit-coverage). ### Improve your evaluation suite Add or strengthen evaluations in an existing NeMo Gym or Harbor suite. No prior audit or traces are required to start; the selected evaluation runtime must be available. ```text Use eval-author to help me improve the existing evaluations for [agent name]. The suite is in [path/to/evals]. ``` Current authoring routes extend Eval Author starter suites or propose additions from reviewed audit findings. Broader suite expansion is still in development. ### Other requests You can also ask for a specific operation: | Say to your coding assistant | Guide | | --- | --- | | "Find the evals in this repository and explain how to run them." | [Discover existing evaluations](docs/getting-started.md#find-existing-evaluations) | | "Check whether my NeMo Gym or Harbor evals are ready to run." | [Check readiness](docs/existing-evals.md#check-readiness) | | "Create a NeMo Gym evaluation from this audit gap." | [Author and prove a NeMo Gym task](skills/eval-author-task-create/references/gym-tasks.md) | | "Use this audit report to propose the next evals to add." | [Plan new or improved evaluations](docs/existing-evals.md#propose-new-evaluations) | | "Explain Intake trace TRACE_ID in workspace WORKSPACE." | [Inspect a trace](docs/traces.md#inspect-an-intake-trace) | | "Turn this local trace into a Harbor eval task." | **Experimental** · [Build a task from a trace](docs/traces.md#build-an-evaluation-task) | | "Explain this eval report and what I should check next." | [Read your results](docs/results.md) | | "Review this task derived from a trace before I share it." | **Experimental** · [Review the task for publication](docs/traces.md#sharing-generated-tasks) | Include the relevant file paths, trace IDs, or workspace names in your request. ### Experimental trace-environment loop The [trace-environment workflow](docs/traces.md#build-an-evaluation-task) turns trace evidence into a private candidate task, validates it with Harbor's `nop` and `oracle` or NeMo Gym's native validation and repeated controls, and uses the findings to refine the task. NeMo Gym output uses a private native report; automated publication and readiness reporting remain Harbor-specific. It can start directly from a trace or be paired with [dataset proposals](docs/existing-evals.md#propose-new-evaluations): propose a useful scenario, construct it from supporting trace evidence, then validate and refine it. ```text Use this audit report to propose an eval, then use eval-author-trace-environment to build a private candidate from the supporting trace. Validate it when the required runtime is available, and report any remaining blockers. ``` A proposal does not establish that a task can be built or that its environment works. The loop may produce no candidate or an unproven candidate; successful validation still does not measure your agent's performance. Proposal-only requests stop at recommendations. See the [trace guide](docs/traces.md#build-an-evaluation-task) for evidence requirements and validation, and the [publication review](docs/traces.md#sharing-generated-tasks) before sharing a task. [Development](DEVELOPMENT.md) · [Skill reference](DEVELOPMENT.md#skill-reference) · [Support](https://github.com/NVIDIA-NeMo/eval-author/issues) # Research Topics ## Questions we are exploring - How can intended behavior become representative cases with meaningful success checks and execution environments? - How can generated tasks challenge an agent in realistic ways and distinguish stronger behavior from weaker behavior? - How can evaluations and traces reveal gaps without overstating coverage? - How can developers decide which evaluations to add or improve next? ## Help shape the research Try Eval Author with your agent and tell us where the approach needs work: - Where do proposed cases miss behavior that matters to you? - Where do graders judge outcomes incorrectly? - Which coverage findings leave you unsure what to test or investigate next? Generated tasks can be too easy. Where do your agents pass without exercising the behavior you wanted to test? An anonymized example of the task, how the agent passed, and what would make it meaningfully challenging would help us improve task generation. If an NVIDIA contact shared this preview with you, send your feedback through that contact and ask them to route it to the Eval Author research team. Otherwise, use the [NVIDIA Developer contact form](https://developer.nvidia.com/contact) with **Eval Author research preview feedback** in the subject. Include your workflow, what you expected, what happened, and a small anonymized example. Please avoid sending raw traces, prompts, credentials, or user data in an initial message. For code contributions, see the [current contribution policy](CONTRIBUTING.md). ## License Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. Eval Author is licensed under the [Apache License, Version 2.0](LICENSE). See [NOTICE](NOTICE) for attributions and the external-materials disclaimer, [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for dependency license disclosures, and [third_party/licenses.jsonl](third_party/licenses.jsonl) for the machine-readable inventory. The bundled JSON examples and synthetic test fixtures are NVIDIA-authored and covered by the project license. This project is currently not accepting contributions. Internal maintenance is limited to the NVIDIA ASE team. See [CONTRIBUTING.md](CONTRIBUTING.md) for the current policy, sign-off, and review requirements.

Loading the repository README…

Skills in this repository

classification pending

skill

occupation

description

updated

eval-author-audit

unclassified

Audit evaluation coverage, find gaps, and recommend improvements against ETHOS.md. Use for "audit my existing evals", "generate audit.md from ETHOS.md", schema validation, measuring an ATIF trace, or aggregating coverage reports. Full audits first read…

2026-10-30

eval-author

unclassified

Use for "help me with my evals", "help me onboard to expand my evals", "what's the state of the eval suite here?", "what happened in this trace?", or requests to create, discover, audit, or improve evaluations. For fresh full workflows, read this skill, show…

2026-10-30

eval-author-first-eval

unclassified

For a fresh start, read the eval-author core skill at ../eval-author/SKILL.md, use its pure conversation renderer for orientation and end the turn before repository inspection or prerequisite work. Confirm ETHOS.md, plan evaluation tasks, and build first…

2026-10-30

eval-author-environment

unclassified

Required by Eval Author's first-eval and task-create flows before they write a Harbor or NeMo Gym task's tests, and available to trace-environment: builds the environment the task runs in, meaning the agent's databases, files, services, and tools, seeded…

2026-10-27

eval-author-trace-environment

unclassified

Experimental compatibility workflow for an explicit trace-environment request or continuation of an existing trace-environment workspace. New requests to derive tasks from recorded interactions go to task-create ATIF source mode or first-eval. Supports NeMo…

2026-10-27

ethos-explore

unclassified

Captures what an AI agent is supposed to do — its purpose, boundaries, and success criteria — as a durable snapshot for anyone building, evaluating, reviewing, or changing it. Explores the user's codebase and docs first, then asks one intent question at a…

2026-10-27

ethos-write

unclassified

Writes an AI agent's ETHOS.md — a durable snapshot of its purpose, intended behavior, boundaries, and success criteria — from the answers gathered by ethos-explore. Renders the schema-v1 template, validates the front matter and required markdown sections,…

2026-10-27

ethos-orchestrate

unclassified

Entry point for an AI agent's ETHOS.md: a durable snapshot of what the agent is for, what it must not do, and what success looks like. Routes a request to explore and capture intent (ethos-explore), or write or edit the file (ethos-write). Owns the standard…

2026-10-26

eval-author-discover

unclassified

Record whether a repository's Harbor and NeMo Gym evaluations are ready to run, proven in four phases rather than guessed. Probe finds the installed runtimes, Explore finds job configs, datasets, task directories, and NeMo Gym manifests, Judge has Harbor and…

2026-10-26

eval-author-inspect-trace

unclassified

Use when eval-author has routed here to inspect one Intake trace, or to prepare and privacy-review a local ATIF source for task construction. Do not use for instrumenting agents, ingesting telemetry, importing a trace store, verifying ingest, or querying…

2026-10-26

eval-author-task-create

unclassified

Create a task from a recorded interaction in ATIF source mode, without requiring an audit gap. Or propose dataset improvements from Eval Author audit findings, then optionally create one NeMo Gym or Harbor task from one actionable uncovered tool. Prove it…

2026-10-26

mlflow-to-atif

unclassified

Convert bounded live MLflow traces or exported Trace.to_dict() JSON into one canonical ATIF trajectory per trace for Harbor or Eval Author audit coverage. Also load complete local exports through NeMo Compass for normalized span evidence. Use for…

2026-10-26

gym-to-atif

unclassified

Convert one bounded NeMo Gym Responses rollout record, or a retained Harbor ATIF from a NeMo Gym run, into one canonical ATIF trajectory for Harbor or Eval Author's experimental environment derivation. Offline only: no NeMo Gym runtime, model invocation, or…

2026-10-23

Showing 13 of 13 collected skills.