llm-evals-benchmarks

name	llm-evals-benchmarks
description	Creates task-specific evaluations for model quality, latency, reliability, and regressions. Use this when the work involves models, inference, training, evaluation, or LLM system design or a task in the "AI / LLM Runtime and Integration Skills" family needs repeatable procedure rather than ad hoc prompting. Do not use for ordinary software tasks with no model, inference, evaluation, or agent-runtime concerns.
license	Apache-2.0
compatibility	{"clients":["openai-codex","gemini-cli","opencode","github-copilot"]}
metadata	{"owner":"codex","domain":"llm-evals-benchmarks","maturity":"draft","risk":"low","tags":["llm","evals","benchmarks"]}

Purpose

Creates task-specific evaluations for model quality, latency, reliability, and regressions.

When to use this skill

Use this skill when:

• the work involves models, inference, training, evaluation, or LLM system design
• a task in the "AI / LLM Runtime and Integration Skills" family needs repeatable procedure rather than ad hoc prompting
• a plan, ticket, or repo state would benefit from explicit guardrails around llm evals and benchmarks

Do not use this skill when

• the task is really about ordinary software tasks with no model, inference, evaluation, or agent-runtime concerns
• If the task is more specifically about tool-use-agents or multimodal-ai, prefer that skill instead.
• the relevant files, runtime, or deliverable type are already covered by a more specific active skill

Operating procedure

1. Clarify the runtime goal, model boundaries, and interfaces involved in LLM Evals and Benchmarks.
2. Make schemas, prompt contracts, and tool surfaces explicit before iterating on behavior.
3. Constrain costs, latency, and failure fallbacks alongside quality goals.
4. Use representative eval or review cases instead of relying on one attractive demo.
5. Document the tradeoffs and next experiments needed to improve the system safely.

Decision rules

• Make schemas and prompts serve the product boundary, not the other way around.
• Prefer measurable eval cases over intuition when runtime boundaries or eval coverage matter.
• Handle fallback and refusal paths explicitly.
• Do not hide cost or latency regressions behind quality anecdotes.

Output requirements

1. Runtime Context
2. Interfaces and Schemas
3. Safety or Cost Controls
4. Evaluation Plan

References

Read these only when relevant:

• references/runtime-contracts.md
• references/eval-cases.md
• references/risk-controls.md

Related skills

• tool-use-agents
• multimodal-ai
• context-management-memory
• safety-guardrails

Failure handling

• If the scope is ambiguous, restate the decision boundary before proceeding.
• If the evidence is weak, say so explicitly and lower confidence instead of smoothing it over.
• If the task would be better served by a narrower skill, redirect to it rather than stretching this one.