| name | eval-author |
| description | Work on evaluation suites in a user's repository or understand an agent run from NeMo Intake. Owns the evidence standard that every Eval Author sub-flow follows. Use when the user asks "help me with my evals", "what's the state of the eval suite here?", "what happened in this trace?", or when you need to pick between the Eval Author sub-flows. Routes to a sub-flow and changes none of the user's source. The selected sub-flow uses the provider's supported tools and saves its findings under `.eval-author/`. |
| triggers | ["help me with the evals in this repo","what is the state of the eval suite here","I inherited a repo with Harbor tasks in it","work on my evaluation suite","what happened in this agent trace","which eval author step do I need"] |
| not-for | ["eval-author-discover (use to run the discovery pass and get a runnable verdict)","eval-author-audit (use to generate and validate audit.md coverage denominators)","eval-author-inspect-trace (use after this skill selects the trace sub-flow)","nemo-intake (use to instrument agents, ingest telemetry, or query Intake outside Eval Author)","nemo-experimentalist (use to run insight-driven optimization end to end, which drives the Eval Author agent itself)","nemo-evaluator (use to run an existing benchmark rather than work on a repository's own suite)"] |
| compatibility | Reading only. Discovery and audit use the local checkout. Trace inspection requires the nemo CLI, an explicit workspace, and read access to configured Intake. |
| maturity | alpha |
| license | Apache-2.0 |
| user-invocable | true |
| allowed-tools | ["Read","Grep","Glob"] |
Eval Author
Work on repository-owned evaluation suites and understand agent traces. Route
each request to the narrow sub-flow that owns it.
A report that a downstream model trusts has to be right. A plausible report is
worse than no report when somebody acts on it.
The standard
Every fact you record comes from authoritative evidence, not a guess.
The authority depends on the sub-flow:
- For suite discovery, Harbor's validators judge runnability. A file's presence
doesn't prove that Harbor accepts it.
- For audit-spec validation, the bundled schema and validator judge the finite
audit.md coverage denominator.
- For trace inspection, Intake establishes what happened. Local source code can
explain behavior, but it can't replace recorded trace evidence.
No sub-flow reimplements a provider's rules. When evidence can't settle a claim,
the report marks the claim unproven or uncertain.
Vocabulary
The sub-flows share this language, and reports use it verbatim.
| Term | Meaning |
|---|
| Check | One named result: pass, warn, or fail. Carries a message and, when it fails, a hint |
| Required | A failing required check blocks the suite. Report the suite as not ready |
| Advisory | A warning worth surfacing that blocks nothing |
| Rung | One step of a provider's validation ladder, ordered so a lower rung's failure often clears once a higher one is fixed |
| Proven | A provider judged this check. An unproven check is an observation and never evidence |
| Provider | The evaluation framework that owns the rules. Harbor today |
| Finding | One trace claim categorized as behavior, issue, recovery, or uncertainty, with evidence IDs |
| Outcome | The trace assessment: success, failure, or unknown |
Sub-flows
Read the sub-flow's own SKILL.md and follow it. This file carries the standard
and the boundaries; the sub-flow carries the steps.
