agentsop-prompt-compilation

name	agentsop-prompt-compilation
version	0.1.0
phase	D
tier	core
frequency	medium
status	opinionated
layer	enhance-overlay
decision_layer_only	true
defers_implementation_to	["dspy","dspy-sop","metric-design"]
description	The compile-readiness gate for prompt auto-optimization. Decide whether you have earned the right to run an optimizer (DSPy MIPROv2 / GEPA / BootstrapFewShot) before spending compute. Two preconditions only — a real metric, and enough examples for the optimizer you picked. Garbage metric in, garbage prompt out. Pick the optimizer by data scale; GEPA inverts the scale assumption (~10 examples + textual feedback).

prompt-compilation — The Compile-Readiness Gate

"It's unproductive to launch optimization runs using a poorly designed program or a bad metric." — DSPy core team [dspy.ai/learn/optimization/overview/]

"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out." — this skill's operating principle (synthesized from the line above + DSPy Case C)

This is an enhancement-overlay decision skill. It answers exactly one question the broad [[dspy]] library skill buries under API surface: have you earned the right to run an optimizer yet, and which one? It produces a go / no-go gate plus an optimizer pick. It defers every implementation detail — Signature syntax, module choice, compile() calls, save/deploy — to [[dspy]] and the full workflow in [[agentsop-dspy]]. It defers metric construction to [[agentsop-metric-design]]; this skill only checks the metric exists and is validated, then uses it as the gate.

The trap it removes: people reach for MIPROv2(auto="heavy") because the API is right there, before they have a metric worth maximizing or enough data to avoid memorization. Compilation is a hyperparameter search costing hundreds-to-thousands of LM calls ($2–$40+, minutes-to-hours) [dspy.ai/faqs/]. Spending that on an un-validated metric or 8 examples is pure waste.

---

1. 何时激活 (When to Activate)

Activate when all three of these are plausibly true (the gate then confirms them):

• A hand-tuned prompt has plateaued. The team has manually iterated few-shot examples / wording past the point of obvious returns. Symptom from [[agentsop-dspy]] §1: "the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design."
• A metric exists (or can be built). There is a metric(example, pred) -> bool|float — or one can be written and human-validated. Without this, do not activate; the optimizer has nothing to maximize.
• Labeled examples exist. There is a dev set. The count determines which optimizer is even legal (§3, §4.2).

Concrete triggers in intent or codebase:

Trigger	Signal
Spend intent	"auto-tune this prompt", "should I run MIPRO?", "is it worth compiling?", "GEPA vs MIPROv2", "optimize prompts for our metric"
API reach	MIPROv2(, BootstrapFewShot(, dspy.GEPA(, teleprompter, optimizer.compile( about to be called
Symptom	hand-tuned prompt stuck; few-shot examples curated by hand; metric written but only used for reporting, not optimization

Do NOT activate when:

• The task is one-shot or the Signature/I-O contract is still churning daily — compile only after it stabilizes [dspy.ai/learn/optimization/overview/]; otherwise you pay compile cost for prompts you'll throw away.
• No metric is possible and none will be built — then this is verbose prompting, not compilation. Route to [[agentsop-metric-design]] first; if the user refuses any success criterion, the gate stays closed.
• Compliance requires verbatim human-authored prompts — optimized prompts are machine-generated artifacts.
• You only need parse safety (a typed Signature), not quality optimization — that is [[agentsop-dspy]] Stage 1 and the signature-design overlay. Promoting prose to a typed Signature and compiling it are two different gates.

---

2. 核心心智模型 (Core Mental Model)

"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."

An optimizer (MIPROv2, GEPA, BootstrapFewShot) is a black-box search over prompt instructions + few-shot demos that maximizes metric(pred, example). It has no taste. It will faithfully chase whatever the metric rewards, biases and all. From [[agentsop-dspy]] Case C: "DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale." Two corollaries make this a gate, not a step:

1. The metric is the precondition, not a tunable. Before any compute is spent, the metric must (a) exist and (b) agree with human judgment on ≥20 spot-checks. An un-validated metric means the expensive search optimizes the metric's blind spot. This is non-negotiable [dspy.ai/learn/evaluation/metrics/; Case C]. Construction is [[agentsop-metric-design]]'s job; this skill only checks the receipt.

2. Data scale is a hard floor, not a preference. Below the optimizer's example floor you are not training, you are memorizing. The DSPy 20/80 train/val split exists because "prompt-based optimizers often overfit to small training sets" [dspy.ai/learn/optimization/overview/]. The floor differs per optimizer (§4.2).

The two-gate picture

            ┌──────────────────────────────────────────────┐
 Gate 1     │  METRIC: exists?  AND  human-validated ≥20?   │  ── No ─► STOP. Build/validate metric ([[agentsop-metric-design]]).
 (measure)  └──────────────────────────────────────────────┘
                              │ Yes
                              ▼
            ┌──────────────────────────────────────────────┐
 Gate 2     │  EXAMPLES: ≥ floor for the optimizer I want?  │  ── No ─► Pick a lower-floor optimizer, collect data,
 (data)     └──────────────────────────────────────────────┘           or STOP (use LabeledFewShot as a floor).
                              │ Yes
                              ▼
                   COMPILE (cheap probe first: auto="light")

The cost reality (why the gate is worth having)

Compilation runs hundreds-to-thousands of LM calls. Reference run: ~3.2k API calls, $2–$3, 6–20 min for auto="light"; auto="heavy" on 1000+ examples can hit tens of dollars and hours [dspy.ai/faqs/]. Cost scales with num_trials × |trainset| × |program LM calls|. The gate's entire job is to stop you spending that on a metric or dataset that cannot pay it back.

---

3. SOP (Standard Operating Procedure)

0. Confirm activation (§1): plateaued hand-tuning + metric + examples.
1. GATE 1 — metric exists AND validated ≥20 human spot-checks?   [hard]
2. GATE 2 — example count ≥ floor for the chosen optimizer?      [hard]
3. PICK    — choose optimizer by data scale + feedback signal.   (§4.2)
4. BUDGET  — estimate cost; cap it; choose a cheap optimizer LM.
5. PROBE   — run auto="light" (or smallest config) as a signal.
6. DECIDE  — gain ≥ threshold → escalate; else go back, don't escalate.

Gate 1 — Metric (the "can you measure" gate)

• Is there a metric(example, pred, trace=None) -> bool|float? If not → STOP, route to [[agentsop-metric-design]].
• Has it been validated against a human on ≥20 spot-checks (≥30 if open-ended), with ≥80% agreement? If not → STOP and validate first. "Never compile against a metric you haven't human-validated on ≥20 spot-checks" [[[agentsop-dspy]] Case C; dspy.ai/learn/evaluation/metrics/].
• Is the metric a single holistic LLM-judge? Then it inherits length / self-preference / position bias and the optimizer will chase those biases — decompose it ([[agentsop-metric-design]]) before compiling [arxiv.org/pdf/2506.02592].

Exit: a validated metric callable + a calibration receipt. Otherwise the gate is closed; do not proceed.

Gate 2 — Examples (the "enough data" gate)

Count labeled examples. The DSPy-documented thresholds: ≥30 = minimum useful, ~300 = recommended, 200+ required for MIPROv2 to avoid overfitting [dspy.ai/learn/optimization/overview/]. The floor is per optimizer:

• < 10 → only LabeledFewShot(k=8) (no search, weakest) — usually means STOP, collect data.
• ~10+ with textual feedback → GEPA is legal and sample-efficient (this inverts the usual "more data" rule).
• ~30–50 → BootstrapFewShot / BootstrapFewShotWithRandomSearch.
• 200+ → MIPROv2.

Exit: the example count clears the floor of the optimizer you intend to run. If not, either drop to a lower-floor optimizer or stop.

Step 3 — Pick optimizer by data scale + signal

Use the table in §4.2. The single most important branch: do you have textual error feedback (test diffs, schema violations, judge rationales like "answer was verbose")? If yes, GEPA needs only ~10 examples and converges faster because it reflects on the text of the feedback, not just a scalar [dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. If no, fall back to the data-scale ladder.

Step 4 — Budget

Estimate before launching (§4.4). Cap spend. Use a cheap optimizer LM (e.g. gpt-4o-mini) even when the task LM is expensive — community-reported parity at a fraction of cost [github.com/stanfordnlp/dspy/issues/1596]. Set dspy.configure(track_usage=True) to log actual spend.

Step 5 — Probe cheap first

Run auto="light" (MIPROv2) or the smallest config first. Docs: "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy/issues/1596]. Never start at auto="heavy".

Step 6 — Decide on the probe result

• light gives <2% lift → do not escalate. The bottleneck is the program/metric, not the optimizer. Loop back to [[agentsop-dspy]] Stage 1 (signature ambiguous? wrong decomposition?) [dspy.ai/learn/optimization/overview/].
• light gives 2–10% lift → escalate to medium; go to heavy only if data ≥300 and you have a held-out test set distinct from val.
• Always confirm the final gain on a held-out test set, not the val set used during optimization.

---

4. 操作模型 (Operations)

4.1 — Operation registry (Trigger / Action / Output / Evidence)

OP-1 — ReadinessGate (entry, hard gate)

• Trigger: Anyone about to call an optimizer's compile().
• Action: Run Gate 1 (metric exists + validated ≥20) then Gate 2 (examples ≥ optimizer floor). Any failure → STOP.
• Output: Decision token compile-ready | not-ready + the failing gate.
• Evidence: [dspy.ai/learn/optimization/overview/]; [[agentsop-dspy]] Case C, §3 three-stage gate.

OP-2 — MetricExistsCheck

• Trigger: Gate 1.
• Action: Confirm a metric(example, pred, trace=None) -> bool|float exists. If absent → route to [[agentsop-metric-design]], gate stays closed.
• Output: Metric callable or a STOP.
• Evidence: Anti-pattern "compiling without a metric" [dspy.ai/learn/optimization/overview/].

OP-3 — MetricValidatedCheck

• Trigger: Metric exists (OP-2).
• Action: Require ≥20 (open-ended ≥30) human spot-checks at ≥80% agreement, plus a calibration receipt. If a single holistic LLM-judge, require decomposition first.
• Output: validated flag + receipt reference.
• Evidence: [[agentsop-dspy]] Case C step 4; [dspy.ai/learn/evaluation/metrics/]; [arxiv.org/pdf/2506.02592].

OP-4 — ExampleFloorCheck

• Trigger: Gate 2.
• Action: Count labeled examples; compare to the floor of the intended optimizer (LabeledFewShot any; GEPA ~10+feedback; Bootstrap ~30–50; MIPROv2 200+).
• Output: cleared | below-floor + the legal optimizer set.
• Evidence: [dspy.ai/learn/optimization/overview/] (30 min / 300 rec / 200+ MIPROv2).

OP-5 — OptimizerByDataScale

• Trigger: Both gates passed.
• Action: Pick from the §4.2 table by example count and feedback signal. Prefer GEPA when textual feedback exists.
• Output: Optimizer choice + config.
• Evidence: [dspy.ai/learn/optimization/optimizers/]; [dspy.ai/api/optimizers/GEPA/overview/].

OP-6 — CostBudgetGuard

• Trigger: Before launching compile.
• Action: Estimate num_trials × |trainset| × calls; cap spend; set a cheap optimizer LM; enable track_usage=True.
• Output: A budget ceiling + chosen optimizer LM.
• Evidence: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596].

OP-7 — CheapProbeFirst

• Trigger: Compile-ready, budget set.
• Action: Run auto="light" (or smallest config) first. Never start at heavy.
• Output: A cheap lift signal.
• Evidence: [github.com/stanfordnlp/dspy/issues/1596]; [[agentsop-dspy]] Case A.

OP-8 — EscalateOrReturn

• Trigger: Probe finished.
• Action: <2% lift → return to program/metric design (do NOT escalate). 2–10% → medium. heavy only if data ≥300 + held-out test set. Confirm gain on held-out test.
• Output: Escalate / iterate-back decision.
• Evidence: [[agentsop-dspy]] Case A; [dspy.ai/learn/optimization/overview/].

4.2 — Optimizer-by-data-scale table

Examples	Feedback signal	Optimizer	Why / floor
<10	any	LabeledFewShot(k=8)	No search; weakest. Usually means STOP and collect data. [dspy.ai/cheatsheet/]
~10+	textual (diffs, schema violations, judge rationales)	dspy.GEPA(metric=m_with_feedback)	Inverts the data-scale rule — reflection on text feedback is sample-efficient. [arxiv.org/abs/2507.19457]
~30–50	scalar	BootstrapFewShot / …WithRandomSearch	Self-bootstrapped demos; minimum useful regime. [dspy.ai/learn/optimization/optimizers/]
200+	scalar	MIPROv2(metric=m, auto="light") then escalate	Joint instruction + demo Bayesian search; 200 is the documented floor. [dspy.ai/api/optimizers/MIPROv2/]
any (post-MIPRO, ship smaller model)	—	chain BootstrapFinetune(student=small, teacher=optimized)	Distills prompts into weights. [dspy.ai/api/optimizers/BootstrapFinetune/]

4.3 — Readiness checklist (copy/paste)

GATE 1 — MEASURE
[ ] metric(example, pred, trace=None) -> bool|float exists
[ ] validated vs human: >=20 spot-checks (>=30 open-ended), >=80% agreement
[ ] receipt saved {n_spot_checks, agreement, judge_model, task_model, date}
[ ] NOT a lone holistic LLM-judge (else decompose via [[agentsop-metric-design]] first)

GATE 2 — DATA
[ ] labeled examples counted: N = ____
[ ] N clears the floor of the optimizer chosen below
[ ] (prompt-based optimizers) 20/80 train/val split planned; GEPA uses standard split

PICK + BUDGET
[ ] optimizer chosen by §4.2 (GEPA if textual feedback)
[ ] cost ceiling set; cheap optimizer LM chosen; track_usage=True
[ ] plan: probe auto="light" first, escalate only on >=2% lift, confirm on held-out test

4.4 — Cost budget heuristics

Config	Rough cost	Use when
LabeledFewShot / BootstrapFewShot	cents–~$1	Floor; ≤50 examples
MIPROv2(auto="light")	~$2–3, 6–20 min, ~3.2k calls	First probe, always
MIPROv2(auto="medium")	single–low-tens of $	Light showed ≥2% lift
MIPROv2(auto="heavy")	tens of $, hours	Only if data ≥300 + held-out test + budget
GEPA (~10+ examples)	low, sample-efficient	Textual feedback available

Source: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596]; [arxiv.org/abs/2507.19457].

---

5. 困境决策案例 (Dilemma Cases)

Dilemma 1 — Optimizer cost vs gain: is compiling even worth it?

困境: A 3-stage RAG pipeline already hits 72% on dev after manual prompt tuning. MIPROv2(auto="heavy") would cost ~$40 and 4 hours. Worth it? (Adapted from [[agentsop-dspy]] Case A.)

约束:

• 250 labeled examples (above the MIPROv2 200-floor) [dspy.ai/learn/optimization/optimizers/].
• Prompts already hand-iterated — diminishing returns suspected.
• Pipeline LM = GPT-4o; per-call cost compounds at trial scale.

决策步骤:

1. Gate 1 first: were the prompts ever scored against the metric, or just eyeballed? If only eyeballed, even auto="light" (~$2) often yields large lifts over hand-tuned baselines (paper reports 25%/65% over standard few-shot) [arxiv.org/abs/2310.03714] — the metric was never actually driving design.
2. Probe light (~$2), never heavy (OP-7). It is a cheap signal for whether more compute helps.
3. <2% lift → do NOT escalate to heavy. Return to program/metric: signature ambiguous? is 3-stage the right decomposition? (OP-8) [github.com/stanfordnlp/dspy/issues/1596].
4. 2–10% lift → medium; heavy only if data ≥300 + a held-out test set distinct from val.
5. Use gpt-4o-mini as the optimizer LM even though the task LM is gpt-4o (OP-6) — community parity.

结果: The $40 heavy run is almost never the right first move. The $2 probe tells you whether to spend more or to go fix the program. Often the answer is "fix the program/metric first."

可提取的操作: OP-1, OP-6 CostBudgetGuard, OP-7 CheapProbeFirst, OP-8 EscalateOrReturn. Lesson: never open at heavy. Probe light with a cheap optimizer LM, and treat <2% as a signal to fix the program, not to add compute.

Dilemma 2 — Only 12 examples: GEPA inverts the data-scale assumption

困境: A code-fix agent has just 12 labeled examples, but each failing run produces rich textual feedback: the failing test diff, a schema-violation message, a linter error. The data-scale ladder says 12 < 30 → "STOP, collect data, you can't optimize." Is that right?

约束:

• 12 examples is below the BootstrapFewShot (~30) and MIPROv2 (200) floors.
• Collecting 200+ labeled examples is weeks of effort.
• But the failures emit machine-readable text feedback, not just a pass/fail scalar.

决策步骤:

1. Gate 1: a metric exists — pass/fail on the test plus a textual feedback string. Validate it (the test is ground truth; spot-check the feedback strings are accurate). Pass.
2. Gate 2 — do NOT apply the scalar-data ladder. The presence of textual feedback changes which optimizer is legal. GEPA needs only ~10 examples because it reflects on the content of the feedback, not a scalar gradient [dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. 12 ≥ ~10 → GEPA is legal.
3. Pick GEPA (OP-5): return dspy.Prediction(score=..., feedback="failing assert: expected X got Y") from the metric; this is GEPA's superpower [dspy.ai/api/optimizers/GEPA/overview/].
4. Probe cheap (OP-7), confirm lift on the held-out examples (OP-8).

结果: A dataset that is far too small for MIPROv2/Bootstrap is sufficient for GEPA. The "you need 200 examples" intuition is specific to scalar-feedback optimizers; textual feedback buys an order-of-magnitude in sample efficiency. GEPA reported beating MIPROv2 by 10–13% on benchmarks while being more sample-efficient [arxiv.org/abs/2507.19457].

可提取的操作: OP-4 ExampleFloorCheck, OP-5 OptimizerByDataScale. Lesson: the example floor is per-optimizer. If you can express why an output failed as text, GEPA inverts the data-scale assumption — ~10 examples suffice. Do not reflexively gate out small datasets that carry rich feedback.

---

6. 反模式与边界 (Anti-Patterns & Boundaries)

Anti-patterns

#	Anti-pattern	Why it's wrong	Fix
AP-1	Compiling without a metric	The optimizer has nothing to maximize; DSPy degenerates to verbose prompting	Refuse the compile; build a metric (OP-2, [[agentsop-metric-design]])
AP-2	Compiling against an un-validated single LLM-judge	The search faithfully chases the judge's length / self-preference / position bias [arxiv.org/pdf/2506.02592]	Validate ≥20 spot-checks; decompose the judge (OP-3)
AP-3	Compiling on <10 examples (scalar feedback)	Below the floor you memorize, not train; "prompt-based optimizers overfit small sets" [dspy.ai/learn/optimization/overview/]	Collect data, or use LabeledFewShot as a floor — or GEPA if textual feedback (OP-4)
AP-4	Starting at auto="heavy"	Tens of $ / hours before you know if compute even helps	Probe auto="light" first (OP-7)
AP-5	Escalating after a <2% probe lift	The bottleneck is the program/metric; more compute won't fix it	Return to [[agentsop-dspy]] Stage 1 (OP-8)
AP-6	Using the expensive task LM as the optimizer LM	Multiplies trial-scale cost for no documented gain	Cheap optimizer LM (gpt-4o-mini) (OP-6)
AP-7	Applying the scalar data-floor to a feedback-rich task	Wrongly gates out GEPA-legal small datasets	Check for textual feedback before counting examples (OP-5)
AP-8	Reporting the gain on the val set used in optimization	Overfit signal; not a real held-out improvement	Confirm on a held-out test set distinct from val (OP-8)
AP-9	Compiling while the Signature/I-O contract is still churning	You pay compile cost for prompts you'll throw away	Stabilize the contract first (signature-design / [[agentsop-dspy]] §1)

Boundaries (when this skill cannot help)

• No willingness to define any success criterion — no metric can exist; the gate stays closed. Problem definition first ([[agentsop-metric-design]], then scientific-critical-thinking).
• Parse-safety, not quality — if you only need a typed Signature so code can consume the output, that is the promote gate (signature-design / [[agentsop-dspy]] Stage 1), a different decision from compile.
• HOW to write/run the optimizer — class syntax, compile() args, save/deploy: that is [[dspy]] and [[agentsop-dspy]], not this overlay.
• Constructing the metric — decomposition, bias probes, calibration receipts: that is [[agentsop-metric-design]]. This skill only checks the receipt exists.
• Non-DSPy auto-optimization — e.g. OpenAI fine-tuning is a weights path, not prompt compilation; the readiness logic (metric + data floor) still applies but the operators differ (§7).

---

7. 跨框架对照 (Cross-Framework Mapping)

The readiness gate (metric + data floor) is framework-agnostic; the operators differ.

Concept	DSPy optimizers	Manual few-shot tuning	OpenAI fine-tuning
What is optimized	prompt instructions + demos (and optionally weights via BootstrapFinetune)	the prompt string, by hand	model weights
Metric required?	Yes — metric(ex, pred) -> bool|float drives the search	implicit / eyeballed (the failure mode)	a held-out eval set + loss; eval suite recommended
Example floor	~10 (GEPA+feedback) / ~30 (Bootstrap) / 200+ (MIPROv2)	none, but no guarantees	OpenAI guidance: ~50–100+ examples minimum, more is better
Cost shape	`num_trials ×	trainset	× calls` ($2–$40+) [dspy.ai/faqs/]
Readiness gate (this skill)	both gates apply directly	Gate 1 is exactly what's missing — you're "tuning" with no validated metric	both gates apply; "metric validated" = your eval set is trustworthy
When to prefer	metric + ≥10 examples + want a portable, recompilable artifact	one-shot, unstable contract, or audit-mandated verbatim prompts	task is stable, latency/cost matters at high volume, prompt optimization plateaued

Decision summary: If you have a validated metric and examples clearing a floor, compile (DSPy). If you have a metric but it's never been validated, you are doing manual tuning dressed up — close Gate 1 first. If prompt optimization has plateaued and you have hundreds of stable examples and volume justifies it, consider fine-tuning (or BootstrapFinetune to distill an already-compiled program into a smaller model).

Combination patterns:

• prompt-compilation + [[agentsop-metric-design]]: this skill's Gate 1 is satisfied by a [[agentsop-metric-design]] calibration receipt. No receipt → gate closed.
• prompt-compilation + [[agentsop-dspy]]: this overlay is the sharpened version of [[agentsop-dspy]] §3 Stage 2→3 transition (the "are you allowed to compile yet?" boundary). After the gate opens, hand off to [[agentsop-dspy]] for the full compile/save/deploy workflow.
• prompt-compilation + [[dspy]]: [[dspy]] provides the optimizer APIs; this skill decides whether and which.

---

References

• references/R1-source-evidence.md — every claim traced to the local dspy-sop skill and the upstream DSPy docs it cites; overlap check vs [[dspy]] and [[agentsop-metric-design]].
• intermediate/operation_candidates.json — the 8 operations + tables in machine-readable form.

Citations: [dspy.ai/learn/optimization/overview/], [dspy.ai/learn/optimization/optimizers/], [dspy.ai/api/optimizers/MIPROv2/], [dspy.ai/api/optimizers/GEPA/overview/], [dspy.ai/api/optimizers/BootstrapFinetune/], [dspy.ai/learn/evaluation/metrics/], [dspy.ai/faqs/], [dspy.ai/cheatsheet/], [dspy.ai/tutorials/gepa_ai_program/], [arxiv.org/abs/2310.03714], [arxiv.org/abs/2507.19457], [arxiv.org/pdf/2506.02592], [github.com/stanfordnlp/dspy/issues/1596]. Primary source: /Users/5imp1ex/Desktop/Skill-Workplace/output/dspy-sop-skill/SKILL.md.