| name | eval-author-discover |
| description | Record whether a repository's Harbor evaluations are ready to run, and prove it with Harbor's own validators instead of guessing. Finds every repository-owned job config, dataset, and task directory, then makes Harbor judge each config: schema, job resolution, agent, environment backend, per-task validity, tasks Harbor silently dropped, and required host variables. Use when the user wants to run an eval suite they did not write, hand a suite to a cheaper model, or asks "can I run these evals?", "why won't my Harbor config resolve?", "which env vars does this suite need?", "where are the evals in this repo?", or "why did Harbor skip my task?". Changes none of your source, and leaves behind `.eval-author/discovery.md` so your team and the next model read the verdict without Harbor and without discovering again. |
| triggers | ["can I run the evals in this repo","where are the Harbor evals in this repository","why won't my Harbor job config resolve","which environment variables does this eval suite need","why did Harbor skip one of my tasks","check whether this eval suite is ready to run"] |
| not-for | ["eval-author (use for the standard, the boundaries, and to pick a sub-flow)","nemo-experimentalist (use to run insight-driven optimization end to end, which drives the Eval Author agent itself)","nemo-evaluator (use to run an existing benchmark rather than establish that a Harbor suite is runnable)"] |
| compatibility | Python 3.11 or later. Harbor must be importable by the interpreter that runs the script for any finding to be proven; without it the script reports an unproven inventory and exits 1. Docker is needed only for the environment backend check. |
| maturity | alpha |
| license | Apache-2.0 |
| user-invocable | true |
| allowed-tools | ["Bash","Read","Write","Grep","Glob"] |
Eval Author: discover
The Eval Author discovery pass. Read eval-author for the standard this follows,
the shared vocabulary, and the boundaries that apply throughout. In short: Harbor
judges every fact recorded here, and anything Harbor did not judge is marked
unproven and is not evidence.
Three phases, in order. The bundled script runs all three in one invocation.
- Probe. Is Harbor importable by this interpreter?
- Inventory. Which config files, datasets, and task directories does the
repository own?
- Judge. Only when Harbor is importable: run Harbor's full validation ladder.
Without Harbor, phase 3 cannot run and no claim about runnability is possible. The
report still comes back, every finding marked unproven, with a required failure
naming what to install.
Before you start
Run the script with the interpreter that has Harbor installed. This is the one step
people get wrong, and getting it wrong voids the whole report.
A harbor command on your PATH does not mean Harbor is importable by the Python
you are about to run. A repository with its own virtual environment usually needs
that environment's interpreter. Try these in order until one prints a version:
for py in .venv/bin/python ./venv/bin/python python3; do
"$py" -c "import harbor, sys; print(sys.executable, harbor.__version__)" 2>/dev/null && break
done
Nothing prints a version when Harbor is not installed anywhere. Do not install it
yourself; in the user's repository the missing environment is the finding. Tell
them what you found and ask how they want to proceed.
The report records which mode produced it either way, in runtime.harbor_importable
and the top-level proven field.
Step 1: run discovery
Point the script at the repository root, not at a suite directory. It searches for
configs to a depth of four directories and finds datasets at any depth.
.venv/bin/python <skill_dir>/scripts/discover.py --repo .
One JSON object goes to stdout, and --compact puts it on one line. The script
writes no files; you save the report in Step 5.
The exit code carries the verdict, so check it:
