NVIDIA-NeMo/eval-author

ethos-orchestrate

Entry point for an AI agent's ethos.md: a durable snapshot of what the agent is for, what it must not do, and what success looks like. Routes a request to explore and capture intent (ethos-explore), or write or edit the file (ethos-write), including renaming…

Idioma do texto original: inglês

eval-author-discover

Record whether a repository's Harbor and NeMo Gym evaluations are ready to run, proven in four phases rather than guessed. Probe finds the installed runtimes, Explore finds job configs, datasets, task directories, and Gym manifests, Judge has Harbor and Gym…

Idioma do texto original: inglês

eval-author

Guide broad eval-help requests into onboarding to build or expand evals. Build first evals from a required Ethos, work on existing evaluation suites in a user's repository, derive an environment from trace evidence (experimental), or understand an agent run…

Idioma do texto original: inglês

eval-author-audit

Guide coverage audits with milestone check-ins and user-confirmed eval and trace sources. Generate, validate, measure, and report on an audit-spec coverage denominator for Eval Author. Use when the user wants a hand-editable audit.md file derived from Ethos,…

Idioma do texto original: inglês

eval-author-environment

Required by Eval Author's first-eval and task-create flows before they write a Harbor or Gym task's tests, and available to trace-environment: builds the environment the task runs in, meaning the agent's databases, files, services, and tools, seeded realistic…

Idioma do texto original: inglês

eval-author-first-eval

Establish a required Ethos, plan evaluation cases, and build first evals or expand a starter Gym or Harbor suite while explaining its parts. No prior traces or coverage reports are required. Missing selected runtime blocks scaffolding and execution, not…

Idioma do texto original: inglês

eval-author-task-create

Propose dataset improvements from Eval Author audit findings, then optionally create one Gym or Harbor task from one actionable uncovered tool. Prove it with Gym controls and native execution or Harbor's Oracle, then run it repeatedly with the repository's…

Idioma do texto original: inglês

eval-author-trace-environment

Experimental workflow. Use Gym, MLflow, Intake, OpenTelemetry, or ATIF trace evidence to derive a private, reproducible Harbor or native NeMo Gym task candidate.

Idioma do texto original: inglês

gym-to-atif

Convert one bounded Gym Responses rollout record, or a retained Harbor ATIF from a Gym run, into one canonical ATIF trajectory for Harbor or Eval Author's experimental environment derivation. Offline only: no Gym runtime, model invocation, or image download.…

Idioma do texto original: inglês

mlflow-to-atif

Convert bounded live MLflow traces or exported Trace.to_dict() JSON into one canonical ATIF trajectory per trace for Harbor or Eval Author audit coverage. Also load complete local exports through NeMo Compass for normalized span evidence. Use for…

Idioma do texto original: inglês

eval-author-inspect-trace

Use when eval-author has routed to this sub-flow for one Intake trace. Do not use for instrumenting agents, ingesting telemetry, importing a trace store, verifying ingest, or querying Intake outside Eval Author; those belong to nemo-intake.

Idioma do texto original: inglês