The compile-readiness gate for prompt auto-optimization. Decide whether you have earned the right to run an optimizer (DSPy MIPROv2 / GEPA / BootstrapFewShot) before spending compute. Two preconditions only — a real metric, and enough examples for the optimizer you picked. Garbage metric in, garbage prompt out. Pick the optimizer by data scale; GEPA inverts the scale assumption (~10 examples + textual feedback).
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Der Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
Datei-Explorer
4 Dateien
SKILL.md wird angezeigt
SKILL.md
Quellanweisungen · Schreibgeschützte Vorschau
name
agentsop-prompt-compilation
version
0.1.0
phase
D
tier
core
frequency
medium
status
opinionated
layer
enhance-overlay
decision_layer_only
true
defers_implementation_to
["dspy","dspy-sop","metric-design"]
description
The compile-readiness gate for prompt auto-optimization. Decide whether you have earned the right to run an optimizer (DSPy MIPROv2 / GEPA / BootstrapFewShot) before spending compute. Two preconditions only — a real metric, and enough examples for the optimizer you picked. Garbage metric in, garbage prompt out. Pick the optimizer by data scale; GEPA inverts the scale assumption (~10 examples + textual feedback).
prompt-compilation — The Compile-Readiness Gate
"It's unproductive to launch optimization runs using a poorly designed program or a bad metric."
— DSPy core team [dspy.ai/learn/optimization/overview/]
"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."
— this skill's operating principle (synthesized from the line above + DSPy Case C)
This is an enhancement-overlay decision skill. It answers exactly one question the broad [[dspy]] library
skill buries under API surface: have you earned the right to run an optimizer yet, and which one? It produces a
go / no-go gate plus an optimizer pick. It defers every implementation detail — Signature syntax, module choice,
compile() calls, save/deploy — to [[dspy]] and the full workflow in [[agentsop-dspy]]. It defers metric construction
to [[agentsop-metric-design]]; this skill only checks the metric exists and is validated, then uses it as the gate.
The trap it removes: people reach for MIPROv2(auto="heavy") because the API is right there, before they have a
metric worth maximizing or enough data to avoid memorization. Compilation is a hyperparameter search costing
hundreds-to-thousands of LM calls ($2–$40+, minutes-to-hours) [dspy.ai/faqs/]. Spending that on an un-validated
metric or 8 examples is pure waste.
1. 何时激活 (When to Activate)
Activate when all three of these are plausibly true (the gate then confirms them):
A hand-tuned prompt has plateaued. The team has manually iterated few-shot examples / wording past the point
of obvious returns. Symptom from [[agentsop-dspy]] §1: "the team manually tunes few-shot examples; a metric exists
but isn't being used to drive prompt design."
A metric exists (or can be built). There is a metric(example, pred) -> bool|float — or one can be written
and human-validated. Without this, do not activate; the optimizer has nothing to maximize.
Labeled examples exist. There is a dev set. The count determines which optimizer is even legal (§3, §4.2).
Concrete triggers in intent or codebase:
Trigger
Signal
Spend intent
"auto-tune this prompt", "should I run MIPRO?", "is it worth compiling?", "GEPA vs MIPROv2", "optimize prompts for our metric"
API reach
MIPROv2(, BootstrapFewShot(, dspy.GEPA(, teleprompter, optimizer.compile( about to be called
Symptom
hand-tuned prompt stuck; few-shot examples curated by hand; metric written but only used for reporting, not optimization
Do NOT activate when:
The task is one-shot or the Signature/I-O contract is still churning daily — compile only after it stabilizes
[dspy.ai/learn/optimization/overview/]; otherwise you pay compile cost for prompts you'll throw away.
No metric is possible and none will be built — then this is verbose prompting, not compilation. Route to
[[agentsop-metric-design]] first; if the user refuses any success criterion, the gate stays closed.
You only need parse safety (a typed Signature), not quality optimization — that is [[agentsop-dspy]] Stage 1 and
the signature-design overlay. Promoting prose to a typed Signature and compiling it are two different gates.
2. 核心心智模型 (Core Mental Model)
"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."
An optimizer (MIPROv2, GEPA, BootstrapFewShot) is a black-box search over prompt instructions + few-shot demos that
maximizes metric(pred, example). It has no taste. It will faithfully chase whatever the metric rewards, biases
and all. From [[agentsop-dspy]] Case C: "DSPy will optimize toward whatever the metric rewards. A bad metric becomes a
bad program at scale." Two corollaries make this a gate, not a step:
The metric is the precondition, not a tunable. Before any compute is spent, the metric must (a) exist and
(b) agree with human judgment on ≥20 spot-checks. An un-validated metric means the expensive search optimizes
the metric's blind spot. This is non-negotiable [dspy.ai/learn/evaluation/metrics/; Case C]. Construction is
[[agentsop-metric-design]]'s job; this skill only checks the receipt.
Data scale is a hard floor, not a preference. Below the optimizer's example floor you are not training, you
are memorizing. The DSPy 20/80 train/val split exists because "prompt-based optimizers often overfit to small
training sets" [dspy.ai/learn/optimization/overview/]. The floor differs per optimizer (§4.2).
The two-gate picture
┌──────────────────────────────────────────────┐
Gate 1 │ METRIC: exists? AND human-validated ≥20? │ ── No ─► STOP. Build/validate metric ([[agentsop-metric-design]]).
(measure) └──────────────────────────────────────────────┘
│ Yes
▼
┌──────────────────────────────────────────────┐
Gate 2 │ EXAMPLES: ≥ floor for the optimizer I want? │ ── No ─► Pick a lower-floor optimizer, collect data,
(data) └──────────────────────────────────────────────┘ or STOP (use LabeledFewShot as a floor).
│ Yes
▼
COMPILE (cheap probe first: auto="light")
The cost reality (why the gate is worth having)
Compilation runs hundreds-to-thousands of LM calls. Reference run: ~3.2k API calls, $2–$3, 6–20 min for
auto="light"; auto="heavy" on 1000+ examples can hit tens of dollars and hours [dspy.ai/faqs/]. Cost scales with
num_trials × |trainset| × |program LM calls|. The gate's entire job is to stop you spending that on a metric or
dataset that cannot pay it back.
3. SOP (Standard Operating Procedure)
0. Confirm activation (§1): plateaued hand-tuning + metric + examples.
1. GATE 1 — metric exists AND validated ≥20 human spot-checks? [hard]
2. GATE 2 — example count ≥ floor for the chosen optimizer? [hard]
3. PICK — choose optimizer by data scale + feedback signal. (§4.2)
4. BUDGET — estimate cost; cap it; choose a cheap optimizer LM.
5. PROBE — run auto="light" (or smallest config) as a signal.
6. DECIDE — gain ≥ threshold → escalate; else go back, don't escalate.
Gate 1 — Metric (the "can you measure" gate)
Is there a metric(example, pred, trace=None) -> bool|float? If not → STOP, route to [[agentsop-metric-design]].
Has it been validated against a human on ≥20 spot-checks (≥30 if open-ended), with ≥80% agreement? If not →
STOP and validate first. "Never compile against a metric you haven't human-validated on ≥20 spot-checks"
[[[agentsop-dspy]] Case C; dspy.ai/learn/evaluation/metrics/].
Is the metric a single holistic LLM-judge? Then it inherits length / self-preference / position bias and the
optimizer will chase those biases — decompose it ([[agentsop-metric-design]]) before compiling [arxiv.org/pdf/2506.02592].
Exit: a validated metric callable + a calibration receipt. Otherwise the gate is closed; do not proceed.
Gate 2 — Examples (the "enough data" gate)
Count labeled examples. The DSPy-documented thresholds: ≥30 = minimum useful, ~300 = recommended, 200+ required
for MIPROv2 to avoid overfitting [dspy.ai/learn/optimization/overview/]. The floor is per optimizer:
< 10 → only LabeledFewShot(k=8) (no search, weakest) — usually means STOP, collect data.
~10+with textual feedback → GEPA is legal and sample-efficient (this inverts the usual "more data" rule).
Exit: the example count clears the floor of the optimizer you intend to run. If not, either drop to a
lower-floor optimizer or stop.
Step 3 — Pick optimizer by data scale + signal
Use the table in §4.2. The single most important branch: do you have textual error feedback (test diffs, schema
violations, judge rationales like "answer was verbose")? If yes, GEPA needs only ~10 examples and converges faster
because it reflects on the text of the feedback, not just a scalar [dspy.ai/tutorials/gepa_ai_program/;
arxiv.org/abs/2507.19457]. If no, fall back to the data-scale ladder.
Step 4 — Budget
Estimate before launching (§4.4). Cap spend. Use a cheap optimizer LM (e.g. gpt-4o-mini) even when the task LM
is expensive — community-reported parity at a fraction of cost [github.com/stanfordnlp/dspy/issues/1596]. Set
dspy.configure(track_usage=True) to log actual spend.
Step 5 — Probe cheap first
Run auto="light" (MIPROv2) or the smallest config first. Docs: "start with moderate values, observe behavior, and
scale up only if you see clear gains" [github.com/stanfordnlp/dspy/issues/1596]. Never start at auto="heavy".
Step 6 — Decide on the probe result
light gives <2% lift → do not escalate. The bottleneck is the program/metric, not the optimizer. Loop
back to [[agentsop-dspy]] Stage 1 (signature ambiguous? wrong decomposition?) [dspy.ai/learn/optimization/overview/].
light gives 2–10% lift → escalate to medium; go to heavy only if data ≥300 and you have a held-out
test set distinct from val.
Always confirm the final gain on a held-out test set, not the val set used during optimization.
Trigger: Anyone about to call an optimizer's compile().
Action: Run Gate 1 (metric exists + validated ≥20) then Gate 2 (examples ≥ optimizer floor). Any failure → STOP.
Output: Decision token compile-ready | not-ready + the failing gate.
Evidence: [dspy.ai/learn/optimization/overview/]; [[agentsop-dspy]] Case C, §3 three-stage gate.
OP-2 — MetricExistsCheck
Trigger: Gate 1.
Action: Confirm a metric(example, pred, trace=None) -> bool|float exists. If absent → route to [[agentsop-metric-design]], gate stays closed.
Output: Metric callable or a STOP.
Evidence: Anti-pattern "compiling without a metric" [dspy.ai/learn/optimization/overview/].
OP-3 — MetricValidatedCheck
Trigger: Metric exists (OP-2).
Action: Require ≥20 (open-ended ≥30) human spot-checks at ≥80% agreement, plus a calibration receipt. If a single holistic LLM-judge, require decomposition first.
Output: validated flag + receipt reference.
Evidence: [[agentsop-dspy]] Case C step 4; [dspy.ai/learn/evaluation/metrics/]; [arxiv.org/pdf/2506.02592].
OP-4 — ExampleFloorCheck
Trigger: Gate 2.
Action: Count labeled examples; compare to the floor of the intended optimizer (LabeledFewShot any; GEPA ~10+feedback; Bootstrap ~30–50; MIPROv2 200+).
Output: cleared | below-floor + the legal optimizer set.
Evidence: [dspy.ai/learn/optimization/overview/] (30 min / 300 rec / 200+ MIPROv2).
OP-5 — OptimizerByDataScale
Trigger: Both gates passed.
Action: Pick from the §4.2 table by example count and feedback signal. Prefer GEPA when textual feedback exists.
Action: Run auto="light" (or smallest config) first. Never start at heavy.
Output: A cheap lift signal.
Evidence: [github.com/stanfordnlp/dspy/issues/1596]; [[agentsop-dspy]] Case A.
OP-8 — EscalateOrReturn
Trigger: Probe finished.
Action: <2% lift → return to program/metric design (do NOT escalate). 2–10% → medium. heavy only if data ≥300 + held-out test set. Confirm gain on held-out test.
Output: Escalate / iterate-back decision.
Evidence: [[agentsop-dspy]] Case A; [dspy.ai/learn/optimization/overview/].
4.2 — Optimizer-by-data-scale table
Examples
Feedback signal
Optimizer
Why / floor
<10
any
LabeledFewShot(k=8)
No search; weakest. Usually means STOP and collect data. [dspy.ai/cheatsheet/]
Dilemma 1 — Optimizer cost vs gain: is compiling even worth it?
困境: A 3-stage RAG pipeline already hits 72% on dev after manual prompt tuning. MIPROv2(auto="heavy") would
cost ~$40 and 4 hours. Worth it? (Adapted from [[agentsop-dspy]] Case A.)
约束:
250 labeled examples (above the MIPROv2 200-floor) [dspy.ai/learn/optimization/optimizers/].
Pipeline LM = GPT-4o; per-call cost compounds at trial scale.
决策步骤:
Gate 1 first: were the prompts ever scored against the metric, or just eyeballed? If only eyeballed, even
auto="light" (~$2) often yields large lifts over hand-tuned baselines (paper reports 25%/65% over standard
few-shot) [arxiv.org/abs/2310.03714] — the metric was never actually driving design.
Probe light (~$2), never heavy (OP-7). It is a cheap signal for whether more compute helps.
<2% lift → do NOT escalate to heavy. Return to program/metric: signature ambiguous? is 3-stage the right
decomposition? (OP-8) [github.com/stanfordnlp/dspy/issues/1596].
2–10% lift → medium; heavy only if data ≥300 + a held-out test set distinct from val.
Use gpt-4o-mini as the optimizer LM even though the task LM is gpt-4o (OP-6) — community parity.
结果: The $40 heavy run is almost never the right first move. The $2 probe tells you whether to spend more or
to go fix the program. Often the answer is "fix the program/metric first."
可提取的操作: OP-1, OP-6 CostBudgetGuard, OP-7 CheapProbeFirst, OP-8 EscalateOrReturn. Lesson: never
open at heavy. Probe light with a cheap optimizer LM, and treat <2% as a signal to fix the program, not to add
compute.
Dilemma 2 — Only 12 examples: GEPA inverts the data-scale assumption
困境: A code-fix agent has just 12 labeled examples, but each failing run produces rich textual feedback:
the failing test diff, a schema-violation message, a linter error. The data-scale ladder says 12 < 30 → "STOP,
collect data, you can't optimize." Is that right?
约束:
12 examples is below the BootstrapFewShot (~30) and MIPROv2 (200) floors.
Collecting 200+ labeled examples is weeks of effort.
But the failures emit machine-readable text feedback, not just a pass/fail scalar.
决策步骤:
Gate 1: a metric exists — pass/fail on the test plus a textual feedback string. Validate it (the test is
ground truth; spot-check the feedback strings are accurate). Pass.
Gate 2 — do NOT apply the scalar-data ladder. The presence of textual feedback changes which optimizer is
legal. GEPA needs only ~10 examples because it reflects on the content of the feedback, not a scalar gradient
[dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. 12 ≥ ~10 → GEPA is legal.
Pick GEPA (OP-5): return dspy.Prediction(score=..., feedback="failing assert: expected X got Y") from the
metric; this is GEPA's superpower [dspy.ai/api/optimizers/GEPA/overview/].
Probe cheap (OP-7), confirm lift on the held-out examples (OP-8).
结果: A dataset that is far too small for MIPROv2/Bootstrap is sufficient for GEPA. The "you need 200 examples"
intuition is specific to scalar-feedback optimizers; textual feedback buys an order-of-magnitude in sample
efficiency. GEPA reported beating MIPROv2 by 10–13% on benchmarks while being more sample-efficient
[arxiv.org/abs/2507.19457].
可提取的操作: OP-4 ExampleFloorCheck, OP-5 OptimizerByDataScale. Lesson: the example floor is per-optimizer.
If you can express why an output failed as text, GEPA inverts the data-scale assumption — ~10 examples suffice.
Do not reflexively gate out small datasets that carry rich feedback.
6. 反模式与边界 (Anti-Patterns & Boundaries)
Anti-patterns
#
Anti-pattern
Why it's wrong
Fix
AP-1
Compiling without a metric
The optimizer has nothing to maximize; DSPy degenerates to verbose prompting
Refuse the compile; build a metric (OP-2, [[agentsop-metric-design]])
AP-2
Compiling against an un-validated single LLM-judge
The search faithfully chases the judge's length / self-preference / position bias [arxiv.org/pdf/2506.02592]
Validate ≥20 spot-checks; decompose the judge (OP-3)
AP-3
Compiling on <10 examples (scalar feedback)
Below the floor you memorize, not train; "prompt-based optimizers overfit small sets" [dspy.ai/learn/optimization/overview/]
Collect data, or use LabeledFewShot as a floor — or GEPA if textual feedback (OP-4)
AP-4
Starting at auto="heavy"
Tens of $ / hours before you know if compute even helps
Probe auto="light" first (OP-7)
AP-5
Escalating after a <2% probe lift
The bottleneck is the program/metric; more compute won't fix it
Return to [[agentsop-dspy]] Stage 1 (OP-8)
AP-6
Using the expensive task LM as the optimizer LM
Multiplies trial-scale cost for no documented gain
Cheap optimizer LM (gpt-4o-mini) (OP-6)
AP-7
Applying the scalar data-floor to a feedback-rich task
Wrongly gates out GEPA-legal small datasets
Check for textual feedback before counting examples (OP-5)
AP-8
Reporting the gain on the val set used in optimization
Overfit signal; not a real held-out improvement
Confirm on a held-out test set distinct from val (OP-8)
AP-9
Compiling while the Signature/I-O contract is still churning
You pay compile cost for prompts you'll throw away
Stabilize the contract first (signature-design / [[agentsop-dspy]] §1)
Boundaries (when this skill cannot help)
No willingness to define any success criterion — no metric can exist; the gate stays closed. Problem
definition first ([[agentsop-metric-design]], then scientific-critical-thinking).
Parse-safety, not quality — if you only need a typed Signature so code can consume the output, that is the
promote gate (signature-design / [[agentsop-dspy]] Stage 1), a different decision from compile.
HOW to write/run the optimizer — class syntax, compile() args, save/deploy: that is [[dspy]] and
[[agentsop-dspy]], not this overlay.
Constructing the metric — decomposition, bias probes, calibration receipts: that is [[agentsop-metric-design]].
This skill only checks the receipt exists.
Non-DSPy auto-optimization — e.g. OpenAI fine-tuning is a weights path, not prompt compilation; the
readiness logic (metric + data floor) still applies but the operators differ (§7).
7. 跨框架对照 (Cross-Framework Mapping)
The readiness gate (metric + data floor) is framework-agnostic; the operators differ.
Concept
DSPy optimizers
Manual few-shot tuning
OpenAI fine-tuning
What is optimized
prompt instructions + demos (and optionally weights via BootstrapFinetune)
the prompt string, by hand
model weights
Metric required?
Yes — metric(ex, pred) -> bool|float drives the search
implicit / eyeballed (the failure mode)
a held-out eval set + loss; eval suite recommended
OpenAI guidance: ~50–100+ examples minimum, more is better
Cost shape
`num_trials ×
trainset
× calls` ($2–$40+) [dspy.ai/faqs/]
Readiness gate (this skill)
both gates apply directly
Gate 1 is exactly what's missing — you're "tuning" with no validated metric
both gates apply; "metric validated" = your eval set is trustworthy
When to prefer
metric + ≥10 examples + want a portable, recompilable artifact
one-shot, unstable contract, or audit-mandated verbatim prompts
task is stable, latency/cost matters at high volume, prompt optimization plateaued
Decision summary: If you have a validated metric and examples clearing a floor, compile (DSPy). If you
have a metric but it's never been validated, you are doing manual tuning dressed up — close Gate 1 first. If
prompt optimization has plateaued and you have hundreds of stable examples and volume justifies it, consider
fine-tuning (or BootstrapFinetune to distill an already-compiled program into a smaller model).
Combination patterns:
prompt-compilation + [[agentsop-metric-design]]: this skill's Gate 1 is satisfied by a [[agentsop-metric-design]]
calibration receipt. No receipt → gate closed.
prompt-compilation + [[agentsop-dspy]]: this overlay is the sharpened version of [[agentsop-dspy]] §3 Stage 2→3
transition (the "are you allowed to compile yet?" boundary). After the gate opens, hand off to [[agentsop-dspy]] for
the full compile/save/deploy workflow.
prompt-compilation + [[dspy]]: [[dspy]] provides the optimizer APIs; this skill decides whether and
which.
References
references/R1-source-evidence.md — every claim traced to the local dspy-sop skill and the upstream DSPy docs it cites; overlap check vs [[dspy]] and [[agentsop-metric-design]].
intermediate/operation_candidates.json — the 8 operations + tables in machine-readable form.
