| name | agentic-bench |
| description | Autonomous model validation and benchmarking. Investigates any ML model (LLM, image gen, TTS, time series, etc.), runs it on GPU cloud, evaluates quality and performance, and generates HTML reports. Use when user asks to verify, benchmark, evaluate, or test a model. Triggers on "verify model", "benchmark", "evaluate model", "test model", "run benchmark", "model evaluation", "モデルを検証", "ベンチマーク", "モデルを試して". |
agentic-bench: Autonomous Model Validation
You are an experienced ML engineer. When a user asks you to verify or benchmark a model,
you autonomously research it, execute it on GPU cloud, evaluate the results, and produce
a publication-ready report.
Workflow Overview
User: "Verify {model_name}"
|
v
Phase 1: Research (model-researcher)
→ Read model card, estimate VRAM, select provider, plan evaluation
|
v
Phase 2: Execute (gpu-runner)
→ Write inference code, run on GPU cloud, collect outputs
|
v
Phase 3: Report (eval-reporter)
→ Generate metrics.json + HTML report, commit to results/
Phase 1: Research
Consult the model-researcher skill knowledge:
- Run
python .claude/skills/model-researcher/scripts/hf_model_info.py MODEL_ID --json
- Run
python .claude/skills/model-researcher/scripts/hf_inference_check.py MODEL_ID --json
- Run
python .claude/skills/model-researcher/scripts/gpu_estimator.py --params PARAMS --model-type TYPE --check-env --json
- Read the appropriate eval guide from
.claude/skills/model-researcher/references/
Produce a research summary including estimated cost before proceeding.
The research summary (URL, description, features) will also be used in the Phase 3 report.
Cost Gate
Always present the cost estimate to the user before starting Phase 2.
Example output:
## Cost Estimate
- Provider: Modal (A100-40GB)
- Estimated duration: ~15 min
- Estimated cost: ~$0.53
- Alternative: HF Inference API (free, if available)
Proceed with execution? [y/N]
