菜单
Enforce "evals over vibes" — before any prompt, agent, or model change, require an eval set with success criteria and baseline measurement. Blocks vibe-only changes. Triggers on "tune prompt", "improve agent", "change model" without prior eval set.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | eval-first |
| description | Enforce "evals over vibes" — before any prompt, agent, or model change, require an eval set with success criteria and baseline measurement. Blocks vibe-only changes. Triggers on "tune prompt", "improve agent", "change model" without prior eval set. |
Karpathy: "evals are everything". No change without measurement.
eval-set-builder skill.EVAL GATE: <change description>
- Eval set: <path or BLOCKED>
- Metric: <name>
- Baseline: <number>
- Post-change: <number>
- Delta: <signed %>
- Decision: SHIP / BLOCK / NEEDS_REVIEW
Pair with eval-set-builder, regression-eval.
Visualize an agent's perceive→think→act loop. Inspects tool calls, latency per step, decision branches, retry patterns. Surfaces bottlenecks. Triggers on "trace agent", "why is agent slow", "agent loop analysis", "what did the agent do".
Compute context window budget for a prompt/agent — sums system prompt, tool descriptions, examples, conversation history; flags overruns; suggests cuts. Triggers on "context budget", "how much context", "will this fit", "budget my prompt".
Treat dataset / eval JSONL as code — lint, validate schema, detect duplicates, leakage between train/eval, label imbalance, PII. Karpathy "Software 2.0" lens. Triggers on "audit dataset", "lint jsonl", "check for leakage", "validate eval data".
Build a 10-30 case eval dataset from a PRD, skill, or prompt — covering golden path, edge cases, and adversarial inputs. Output as JSONL with input/expected/category. Triggers on "build eval set", "create evals", "generate test cases for prompt".
Break down any ML/AI concept to first principles — strip jargon, derive from basics, build physical/mathematical analogy. Karpathy intuition-building style. Triggers on "first principles", "explain like I'm 5", "derive from scratch", "why does X actually work".
Generate minimal, runnable from-scratch implementation of LLM concepts (attention, BPE tokenizer, backprop, RoPE, KV-cache, MoE, LoRA, RLHF). Karpathy-style pedagogy — code first, theory in comments. Triggers on "explain X from scratch", "minimal X implementation", "show me how X works in code", "build X from zero".