| name | nexus-eval-harness |
| description | Work on the Nexus LLM eval harness in tests/eval/ — author or fix a scenario fixture, write an eval config, change the executors, assertions or reports, or explain a run that produced nothing, everything-fails, or numbers that disagree. Use when an eval scenario is wrong, a run behaves oddly, or the harness itself needs to change. To grade a model rather than change the harness, use nexus-model-eval. |
Nexus Eval Harness
Context: the harness under tests/eval/ drives the real production path —
StreamingOrchestrator plus tool continuation — with a mock or live tool
executor swapped in, and grades the tool calls a model emits. It is a fixture
system, and almost every surprising result is the fixture talking, not the
model. This skill owns the fixtures, the configs, the executors and the
reports.
Workflow
-
Pick the job and open its protocol. Work from the protocol; this router
names procedures, it does not contain them.
| Job | Protocol |
|---|
| Add a scenario, or fix one that grades wrongly | protocols/add-a-scenario.md |
| Write or change a config, choose targets, mode, retries | protocols/configure-a-run.md |
| A run produced nothing, all-fails, a hang, or odd numbers | protocols/debug-a-run.md |
| Change the executors, assertions, loader or reports | protocols/extend-the-harness.md |
-
Derive every list from the tree, never from this skill. It names no
scenarios, no configs, no models and no env-var table on purpose, and you
MUST NOT add one — the harness gains knobs faster than a document survives.
ls tests/eval/scenarios/ tests/eval/configs/
grep -rhoE "get(Number|List)?Env\('[A-Z_]+'\)|process\.env\.[A-Z_]+" tests/eval/ \
| grep -oE "[A-Z][A-Z_]{3,}" | sort -u
-
Before calling any scenario change done, run the checker from the repo root
and fix everything it prints:
python3 .claude/skills/nexus-eval-harness/scripts/check_scenarios.py
-
NEVER trust jest's exit code as the verdict on a run, and never report a
pass rate you read from stdout. The saved reports under the configured
artifacts dir are the only source of truth, and they are written even when
the run times out — see references/run-behavior.md.
-
At the end of a session that used this skill, run protocols/self-refine.md.
Map
protocols/ the procedures named in step 1, plus self-refine.md.references/ read on demand: harness-map.md (what each file owns and how a
run is assembled), scenario-contract.md (what a scenario fixture means and
the traps in it), run-behavior.md (config resolution, concurrency, retries,
artifacts, what the numbers count).scripts/check_scenarios.py the mechanical check from step 3. Run it; do not
reimplement it.refinement-log.md what past sessions changed here and why.agents/openai.yaml an interface manifest several nexus-* skills carry.
Not a subagent prompt; nothing in this skill reads it.
Siblings — name them, do not duplicate them
nexus-model-eval — grading models. Which models to run, whether a slug
resolves, how to read a leaderboard, and whether a failure indicts the model.
That skill consumes the harness; this one changes it. If the question is "how
good is model X", stop here and use it.nexus-testing — the gate that keeps this suite from running (and billing) in
CI, and how to watch a run in flight.nexus-agents — the real getTools/useTools contract the fixtures imitate.
When a fixture and production disagree, production wins, and that skill says
what production does.nexus-llm-adapters — provider adapters. A run that fails inside streaming
for one provider only is an adapter problem, not a harness problem.nexus-tool-schemas — the live tool catalog, for checking a fixture's tool
slugs against the registry instead of guessing.
