| name | agentic-suite |
| description | Use when driving the agentic tool-calling benchmark specifically — running harness/agentic/run.py against an endpoint, choosing the cell matrix, reading the multi-turn degradation track, or recovering partial results after a crash. For the general five-suite flow use the run-benchmark skill. |
Agentic tool-calling suite
Measures whether a served model produces valid structured tool calls while
carrying a 22-tool registry under load. Full description:
docs/agentic.md. Runner:
harness/agentic/run.py — a standalone
PEP 723 script (uv run resolves its one dependency, httpx).
Run it
uv run harness/agentic/run.py --base-url http://localhost:11434/v1 \
--model <served-id> --api-key-env OPENAI_API_KEY
Never run against a busy Studio without asking — one actor per host.
- Matrix: thinking on/off × concurrency 1/4 × context small/large ×
stream/nostream, plus the multi-turn track. Narrow with
--cells (a
comma-separated substring filter; include multiturn to keep that track).
- Run both thinking tracks — thinking changes the tool-call formatting path.
- Pass signal:
valid_tool_call_rate (function name in the registry,
arguments parse as JSON with all required keys, finish_reason == tool_calls).
- Primary quant discriminator: the multi-turn
first_degraded_round —
stock 4-bit Qwen3.x quants fall back to plain-text [Tool call: ...] around
round 5 (mlx-lm #1011); single-shot cells cannot catch this.
Warm-up and crash recovery
Each cell fires one untimed warm-up request excluded from every statistic
(first-request cold-start would skew the cell). The canonical results JSON is
written only after the whole matrix finishes; if a run dies mid-matrix, every
completed cell survives in <output>.partial.jsonl beside the output — one
JSON object per line, tagged kind: cell or kind: multiturn. A clean run
deletes it.
Publish
mlx-bench-publish <output.json> --kind agentic --suite tool-calling
See the publish-results skill for the dry-run + write-token flow.
