agentsop-prompt-compilation

# prompt-compilation — The Compile-Readiness Gate > "It's unproductive to launch optimization runs using a poorly designed program or a bad metric." > — DSPy core team [dspy.ai/learn/optimization/overview/] > "Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out." > — this skill's operating principle (synthesized from the line above + DSPy Case C) This is an **enhancement-overlay decision skill**. It answers exactly one question the broad `[[dspy]]` library skill buries under API surface: **have you earned the right to run an optimizer yet, and which one?** It produces a go / no-go gate plus an optimizer pick. It defers *every* implementation detail — Signature syntax, module choice, `compile()` calls, save/deploy — to `[[dspy]]` and the full workflow in `[[agentsop-dspy]]`. It defers metric *construction* to `[[agentsop-metric-design]]`; this skill only checks the metric *exists and is validated*, then uses it as the gate. The trap it removes: people reach for `MIPROv2(auto="heavy")` because the API is right there, before they have a metric worth maximizing or enough data to avoid memorization. Compilation is a hyperparameter search costing hundreds-to-thousands of LM calls ($2–$40+, minutes-to-hours) [dspy.ai/faqs/]. Spending that on an un-validated metric or 8 examples is pure waste. --- ## 1. 何时激活 (When to Activate) Activate when **all three** of these are plausibly true (the gate then *confirms* them): - **A hand-tuned prompt has plateaued.** The team has manually iterated few-shot examples / wording past the point of obvious returns. Symptom from `[[agentsop-dspy]]` §1: "the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design." - **A metric exists (or can be built).** There is a `metric(example, pred) -> bool|float` — or one can be written and human-validated. Without this, do not activate; the optimizer has nothing to maximize. - **Labeled examples exist.** There is a dev set. The *count* determines which optimizer is even legal (§3, §4.2). Concrete triggers in intent or codebase: | Trigger | Signal | |---|---| | Spend intent | "auto-tune this prompt", "should I run MIPRO?", "is it worth compiling?", "GEPA vs MIPROv2", "optimize prompts for our metric" | | API reach | `MIPROv2(`, `BootstrapFewShot(`, `dspy.GEPA(`, `teleprompter`, `optimizer.compile(` about to be called | | Symptom | hand-tuned prompt stuck; few-shot examples curated by hand; metric written but only used for reporting, not optimization | **Do NOT activate when:** - The task is one-shot or the Signature/I-O contract is still churning daily — compile only after it stabilizes [dspy.ai/learn/optimization/overview/]; otherwise you pay compile cost for prompts you'll throw away. - No metric is possible and none will be built — then this is verbose prompting, not compilation. Route to `[[agentsop-metric-design]]` first; if the user refuses any success criterion, the gate stays closed. - Compliance requires verbatim human-authored prompts — optimized prompts are machine-generated artifacts. - You only need *parse safety* (a typed Signature), not *quality optimization* — that is `[[agentsop-dspy]]` Stage 1 and the `signature-design` overlay. Promoting prose to a typed Signature and *compiling* it are two different gates. --- ## 2. 核心心智模型 (Core Mental Model) ### **"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."** An optimizer (MIPROv2, GEPA, BootstrapFewShot) is a black-box search over prompt instructions + few-shot demos that maximizes `metric(pred, example)`. It has no taste. It will faithfully chase *whatever the metric rewards*, biases and all. From `[[agentsop-dspy]]` Case C: "DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale." Two corollaries make this a *gate*, not a step: 1. **The metric is the precondition, not a tunable.** Before any compute is spent, the metric must (a) exist and (b) agree with human judgment on ≥20 spot-checks. An un-validated metric means the expensive search optimizes the metric's blind spot. This is non-negotiable [dspy.ai/learn/evaluation/metrics/; Case C]. Construction is `[[agentsop-metric-design]]`'s job; this skill only *checks the receipt*. 2. **Data scale is a hard floor, not a preference.** Below the optimizer's example floor you are not training, you are memorizing. The DSPy 20/80 train/val split exists *because* "prompt-based optimizers often overfit to small training sets" [dspy.ai/learn/optimization/overview/]. The floor differs per optimizer (§4.2). ### The two-gate picture ``` ┌──────────────────────────────────────────────┐ Gate 1 │ METRIC: exists? AND human-validated ≥20? │ ── No ─► STOP. Build/validate metric ([[agentsop-metric-design]]). (measure) └──────────────────────────────────────────────┘ │ Yes ▼ ┌──────────────────────────────────────────────┐ Gate 2 │ EXAMPLES: ≥ floor for the optimizer I want? │ ── No ─► Pick a lower-floor optimizer, collect data, (data) └──────────────────────────────────────────────┘ or STOP (use LabeledFewShot as a floor). │ Yes ▼ COMPILE (cheap probe first: auto="light") ``` ### The cost reality (why the gate is worth having) Compilation runs hundreds-to-thousands of LM calls. Reference run: ~3.2k API calls, $2–$3, 6–20 min for `auto="light"`; `auto="heavy"` on 1000+ examples can hit tens of dollars and hours [dspy.ai/faqs/]. Cost scales with `num_trials × |trainset| × |program LM calls|`. The gate's entire job is to stop you spending that on a metric or dataset that cannot pay it back. --- ## 3. SOP (Standard Operating Procedure) ``` 0. Confirm activation (§1): plateaued hand-tuning + metric + examples. 1. GATE 1 — metric exists AND validated ≥20 human spot-checks? [hard] 2. GATE 2 — example count ≥ floor for the chosen optimizer? [hard] 3. PICK — choose optimizer by data scale + feedback signal. (§4.2) 4. BUDGET — estimate cost; cap it; choose a cheap optimizer LM. 5. PROBE — run auto="light" (or smallest config) as a signal. 6. DECIDE — gain ≥ threshold → escalate; else go back, don't escalate. ``` ### Gate 1 — Metric (the "can you measure" gate) - Is there a `metric(example, pred, trace=None) -> bool|float`? If not → **STOP**, route to `[[agentsop-metric-design]]`. - Has it been validated against a human on ≥20 spot-checks (≥30 if open-ended), with ≥80% agreement? If not → **STOP and validate first.** "Never compile against a metric you haven't human-validated on ≥20 spot-checks" [`[[agentsop-dspy]]` Case C; dspy.ai/learn/evaluation/metrics/]. - Is the metric a *single* holistic LLM-judge? Then it inherits length / self-preference / position bias and the optimizer will chase those biases — decompose it (`[[agentsop-metric-design]]`) before compiling [arxiv.org/pdf/2506.02592]. **Exit:** a validated metric callable + a calibration receipt. Otherwise the gate is closed; do not proceed. ### Gate 2 — Examples (the "enough data" gate) Count labeled examples. The DSPy-documented thresholds: **≥30 = minimum useful, ~300 = recommended, 200+ required for MIPROv2** to avoid overfitting [dspy.ai/learn/optimization/overview/]. The floor is *per optimizer*: - `< 10` → only `LabeledFewShot(k=8)` (no search, weakest) — usually means **STOP, collect data**. - `~10+` **with textual feedback** → `GEPA` is legal and sample-efficient (this inverts the usual "more data" rule). - `~30–50` → `BootstrapFewShot` / `BootstrapFewShotWithRandomSearch`. - `200+` → `MIPROv2`. **Exit:** the example count clears the floor of the optimizer you intend to run. If not, either drop to a lower-floor optimizer or stop. ### Step 3 — Pick optimizer by data scale + signal Use the table in §4.2. The single most important branch: **do you have textual error feedback** (test diffs, schema violations, judge rationales like "answer was verbose")? If yes, `GEPA` needs only ~10 examples and converges faster because it reflects on the *text* of the feedback, not just a scalar [dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. If no, fall back to the data-scale ladder. ### Step 4 — Budget Estimate before launching (§4.4). Cap spend. Use a **cheap optimizer LM** (e.g. gpt-4o-mini) even when the *task* LM is expensive — community-reported parity at a fraction of cost [github.com/stanfordnlp/dspy/issues/1596]. Set `dspy.configure(track_usage=True)` to log actual spend. ### Step 5 — Probe cheap first Run `auto="light"` (MIPROv2) or the smallest config first. Docs: "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy/issues/1596]. **Never start at `auto="heavy"`.** ### Step 6 — Decide on the probe result - `light` gives **<2% lift** → do **not** escalate. The bottleneck is the program/metric, not the optimizer. Loop back to `[[agentsop-dspy]]` Stage 1 (signature ambiguous? wrong decomposition?) [dspy.ai/learn/optimization/overview/]. - `light` gives **2–10% lift** → escalate to `medium`; go to `heavy` only if data ≥300 *and* you have a held-out test set distinct from val. - Always confirm the final gain on a **held-out test set**, not the val set used during optimization. --- ## 4. 操作模型 (Operations) ### 4.1 — Operation registry (Trigger / Action / Output / Evidence) #### OP-1 — ReadinessGate (entry, hard gate) - **Trigger**: Anyone about to call an optimizer's `compile()`. - **Action**: Run Gate 1 (metric exists + validated ≥20) then Gate 2 (examples ≥ optimizer floor). Any failure → STOP. - **Output**: Decision token `compile-ready` | `not-ready` + the failing gate. - **Evidence**: [dspy.ai/learn/optimization/overview/]; `[[agentsop-dspy]]` Case C, §3 three-stage gate. #### OP-2 — MetricExistsCheck - **Trigger**: Gate 1. - **Action**: Confirm a `metric(example, pred, trace=None) -> bool|float` exists. If absent → route to `[[agentsop-metric-design]]`, gate stays closed. - **Output**: Metric callable or a STOP. - **Evidence**: Anti-pattern "compiling without a metric" [dspy.ai/learn/optimization/overview/]. #### OP-3 — MetricValidatedCheck - **Trigger**: Metric exists (OP-2). - **Action**: Require ≥20 (open-ended ≥30) human spot-checks at ≥80% agreement, plus a calibration receipt. If a single holistic LLM-judge, require decomposition first. - **Output**: `validated` flag + receipt reference. - **Evidence**: `[[agentsop-dspy]]` Case C step 4; [dspy.ai/learn/evaluation/metrics/]; [arxiv.org/pdf/2506.02592]. #### OP-4 — ExampleFloorCheck - **Trigger**: Gate 2. - **Action**: Count labeled examples; compare to the floor of the intended optimizer (LabeledFewShot any; GEPA ~10+feedback; Bootstrap ~30–50; MIPROv2 200+). - **Output**: `cleared` | `below-floor` + the legal optimizer set. - **Evidence**: [dspy.ai/learn/optimization/overview/] (30 min / 300 rec / 200+ MIPROv2). #### OP-5 — OptimizerByDataScale - **Trigger**: Both gates passed. - **Action**: Pick from the §4.2 table by example count and feedback signal. Prefer GEPA when textual feedback exists. - **Output**: Optimizer choice + config. - **Evidence**: [dspy.ai/learn/optimization/optimizers/]; [dspy.ai/api/optimizers/GEPA/overview/]. #### OP-6 — CostBudgetGuard - **Trigger**: Before launching compile. - **Action**: Estimate `num_trials × |trainset| × calls`; cap spend; set a cheap optimizer LM; enable `track_usage=True`. - **Output**: A budget ceiling + chosen optimizer LM. - **Evidence**: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596]. #### OP-7 — CheapProbeFirst - **Trigger**: Compile-ready, budget set. - **Action**: Run `auto="light"` (or smallest config) first. Never start at `heavy`. - **Output**: A cheap lift signal. - **Evidence**: [github.com/stanfordnlp/dspy/issues/1596]; `[[agentsop-dspy]]` Case A. #### OP-8 — EscalateOrReturn - **Trigger**: Probe finished. - **Action**: <2% lift → return to program/metric design (do NOT escalate). 2–10% → `medium`. `heavy` only if data ≥300 + held-out test set. Confirm gain on held-out test. - **Output**: Escalate / iterate-back decision. - **Evidence**: `[[agentsop-dspy]]` Case A; [dspy.ai/learn/optimization/overview/]. ### 4.2 — Optimizer-by-data-scale table | Examples | Feedback signal | Optimizer | Why / floor | |---|---|---|---| | `<10` | any | `LabeledFewShot(k=8)` | No search; weakest. Usually means STOP and collect data. [dspy.ai/cheatsheet/] | | `~10+` | **textual** (diffs, schema violations, judge rationales) | `dspy.GEPA(metric=m_with_feedback)` | Inverts the data-scale rule — reflection on text feedback is sample-efficient. [arxiv.org/abs/2507.19457] | | `~30–50` | scalar | `BootstrapFewShot` / `…WithRandomSearch` | Self-bootstrapped demos; minimum useful regime. [dspy.ai/learn/optimization/optimizers/] | | `200+` | scalar | `MIPROv2(metric=m, auto="light")` then escalate | Joint instruction + demo Bayesian search; 200 is the documented floor. [dspy.ai/api/optimizers/MIPROv2/] | | any (post-MIPRO, ship smaller model) | — | chain `BootstrapFinetune(student=small, teacher=optimized)` | Distills prompts into weights. [dspy.ai/api/optimizers/BootstrapFinetune/] | ### 4.3 — Readiness checklist (copy/paste) ``` GATE 1 — MEASURE [ ] metric(example, pred, trace=None) -> bool|float exists [ ] validated vs human: >=20 spot-checks (>=30 open-ended), >=80% agreement [ ] receipt saved {n_spot_checks, agreement, judge_model, task_model, date} [ ] NOT a lone holistic LLM-judge (else decompose via [[agentsop-metric-design]] first) GATE 2 — DATA [ ] labeled examples counted: N = ____ [ ] N clears the floor of the optimizer chosen below [ ] (prompt-based optimizers) 20/80 train/val split planned; GEPA uses standard split PICK + BUDGET [ ] optimizer chosen by §4.2 (GEPA if textual feedback) [ ] cost ceiling set; cheap optimizer LM chosen; track_usage=True [ ] plan: probe auto="light" first, escalate only on >=2% lift, confirm on held-out test ``` ### 4.4 — Cost budget heuristics | Config | Rough cost | Use when | |---|---|---|