| name | performance-eval-author |
| description | Use when the team needs a repeatable benchmark or evaluation harness before parallel optimization work begins, so every candidate change is tested with the same command, data shape, and success criteria. |
Performance Eval Author
Author the benchmark before the fixes multiply.
The goal is a small, repeatable harness that makes branch comparison fair. It does not need to be fancy. It needs to be stable.
Deliverables
- One benchmark command or script
- Input data or request shape for the target flow
- Reported metrics and win criteria
- Instructions for warmup, sample size, and reruns
Workflow
- Lock the target.
Use the exact endpoint or workflow chosen in intake and instrumentation.
- Choose the simplest repeatable tool.
Prefer a tool already in the repo. If none exists, use a tiny script or a standard CLI benchmark tool.
- Define the measurement protocol.
Record enough information to compare branches fairly: latency, error rate, and any obvious resource metric if easy to collect.
- Remove ambiguity.
State warmup count, sample count, concurrency, seed data, and any required env vars.
- Publish the evaluation contract.
Make it easy for engineers and the evaluator to run the same test without interpretation.
Recommended Metrics
- Mean latency
- P50 and P95 if available
- Error rate
- Throughput if the tool exposes it
Guardrails
- Prefer repeatability over realism if time is tight.
- Keep the harness small enough to run multiple times during iteration.
- Do not change the benchmark between candidate branches unless there is a documented flaw.
- If the data setup is noisy, freeze a fixture or a seed path first.
