| name | nexus-model-eval |
| description | Grade how well a model drives the Nexus two-tool protocol (getTools/useTools) and decide whether a low score is the model's fault or the harness's. Use when asked to grade, benchmark, rank or compare models on Nexus tool use, when picking a default model, or when an eval report needs interpreting. |
Nexus model eval
Context: the harness in tests/eval/ shows a model the same two tools the app
does — getTools for discovery, useTools for execution — and grades the calls
it makes, not the prose it writes. This skill owns the verdict: which models to
run, and what a FAIL actually means. Running, configuring and extending the
harness itself belongs to nexus-eval-harness. This file routes; detail loads
when you take the path.
Workflow
- Get current truth before running anything. A model cannot be graded on a
fixture no model can satisfy, and the fixture set moves:
ls tests/eval/scenarios/ tests/eval/configs/
python3 .claude/skills/nexus-eval-harness/scripts/check_scenarios.py
python3 .claude/skills/nexus-model-eval/scripts/check_advertised_tools.py
A non-zero exit from the scenario checker means some scenario can never pass;
resolve that first, and the fix belongs to nexus-eval-harness, not to this
run. The advertised-tools gap is not a defect — it is the list of correct
model behaviors this harness punishes, and you will need it in step 3.
- Run the grade:
protocols/grade-models.md. Read it before you start; a
summarized procedure is one you will improvise, and every scenario in the
matrix costs live, billed API calls.
- You MUST attribute every failure before you report a number:
protocols/attribute-failures.md. The harness fails models for things the
model did not do, so a raw pass rate with unread failures is not a grade.
scripts/summarize_eval.py --labels refuses to sign off while any failure is
unlabelled.
- Report both numbers — raw pass rate and the attributed rate that charges only
model-failure verdicts — plus what the excluded failures actually were. One
number alone is either unfair to the model or unfair to the reader.
- At the end of a session that used this skill, run
protocols/self-refine.md.
Map
protocols/ the procedures: grade-models.md (target list → run → artifacts),
attribute-failures.md (FAIL → verdict → defensible grade), self-refine.md.references/ read on demand: what-is-graded.md (what makes a scenario pass,
what a "turn" counts, how retries and exclusions move the number),
harness-artifacts.md (symptom → cause → proof for failures the model did not
cause — read this before blaming any model).scripts/ run them, do not reimplement:
scripts/check_advertised_tools.py — the commands the eval system prompt
tells the model to use that the executor cannot run, so obeying the prompt
scores as a hallucination.scripts/preflight_models.py — do these slugs exist, before the run spends
money proving they do not.scripts/summarize_eval.py — report JSON → per-model rollup, bucketed
failures, and an attribution that is checked rather than asserted.
refinement-log.md what past sessions changed here and why.
Siblings
The boundary with nexus-eval-harness: it owns the instrument, this skill owns
the verdict. Anything that changes the harness or its inputs — env knobs,
target syntax, live mode and the headless vault, config YAML, scenario authoring,
harness code — is that skill's. Anything that changes what you conclude about a
model is this one's. When a run reveals a fixture defect, hand it over rather
than fixing it here.
Also: nexus-model-updates owns provider model definitions and whether a model
ID works at all (grade nothing until it does); nexus-testing owns Jest lanes
and what a mock can prove; nexus-agents owns the two-tool contract the harness
is imitating; nexus-llm-adapters owns the adapter a stream error comes from.
