NVIDIA-NeMo/eval-author

ethos-orchestrate

Entry point for an AI agent's ethos.md: a durable snapshot of what the agent is for, what it must not do, and what success looks like. Routes a request to explore and capture intent (ethos-explore), or write or edit the file (ethos-write), including renaming…

원문 언어: 영어

eval-author-discover

Record whether a repository's Harbor and NeMo Gym evaluations are ready to run, proven in four phases rather than guessed. Probe finds the installed runtimes, Explore finds job configs, datasets, task directories, and Gym manifests, Judge has Harbor and Gym…

원문 언어: 영어

eval-author

Guide broad eval-help requests into onboarding to build or expand evals. Build first evals from a required Ethos, work on existing evaluation suites in a user's repository, derive an environment from trace evidence (experimental), or understand an agent run…

원문 언어: 영어

eval-author-audit

Guide coverage audits with milestone check-ins and user-confirmed eval and trace sources. Generate, validate, measure, and report on an audit-spec coverage denominator for Eval Author. Use when the user wants a hand-editable audit.md file derived from Ethos,…

원문 언어: 영어

eval-author-environment

Required by Eval Author's first-eval and task-create flows before they write a Harbor or Gym task's tests, and available to trace-environment: builds the environment the task runs in, meaning the agent's databases, files, services, and tools, seeded realistic…

원문 언어: 영어

eval-author-first-eval

Establish a required Ethos, plan evaluation cases, and build first evals or expand a starter Gym or Harbor suite while explaining its parts. No prior traces or coverage reports are required. Missing selected runtime blocks scaffolding and execution, not…

원문 언어: 영어

eval-author-task-create

Propose dataset improvements from Eval Author audit findings, then optionally create one Gym or Harbor task from one actionable uncovered tool. Prove it with Gym controls and native execution or Harbor's Oracle, then run it repeatedly with the repository's…

원문 언어: 영어

eval-author-trace-environment

Experimental workflow. Use Gym, MLflow, Intake, OpenTelemetry, or ATIF trace evidence to derive a private, reproducible Harbor or native NeMo Gym task candidate.

원문 언어: 영어

gym-to-atif

Convert one bounded Gym Responses rollout record, or a retained Harbor ATIF from a Gym run, into one canonical ATIF trajectory for Harbor or Eval Author's experimental environment derivation. Offline only: no Gym runtime, model invocation, or image download.…

원문 언어: 영어

mlflow-to-atif

Convert bounded live MLflow traces or exported Trace.to_dict() JSON into one canonical ATIF trajectory per trace for Harbor or Eval Author audit coverage. Also load complete local exports through NeMo Compass for normalized span evidence. Use for…

원문 언어: 영어

eval-author-inspect-trace

Use when eval-author has routed to this sub-flow for one Intake trace. Do not use for instrumenting agents, ingesting telemetry, importing a trace store, verifying ingest, or querying Intake outside Eval Author; those belong to nemo-intake.

원문 언어: 영어