agentsop-dspy

# DSPy SOP — Programming, Not Prompting > *"DSPy isn't a prompt-optimization agent framework. It's the LLM compiler for the shortest, cleanest code."* > — Eito Miyamura [eito.substack.com/p/dspy-the-most-misunderstood-agent] > > *"Prompts are effectively the weights of an LLM application."* > — Core philosophy [arxiv.org/abs/2310.03714] --- ## 1. 何时激活 (When to activate) Activate this skill when **any** of the following triggers are present in the user's intent or codebase: | Trigger | Signal | |---|---| | Imports / mentions | `import dspy`, `dspy.Signature`, `dspy.ChainOfThought`, `dspy.ReAct`, `Predict`, `MIPROv2`, `BootstrapFewShot`, `GEPA`, `teleprompter`, `compile(` on an LM program | | Tasks | "auto-tune this prompt", "I want to swap GPT-4 for a smaller model without re-engineering prompts", "I have 50/200/1000 labeled examples — optimize this", "compile a pipeline for our metric", "distill GPT-4 into Llama-3-8B" | | Symptoms | Hand-written prompts grow past ~50 lines; brittleness on model swap; the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design | | Cross-skill bridges | LangGraph node calls an LLM and needs better prompts → wrap the node body in a DSPy module. LlamaIndex retriever feeds a reranker → DSPy-compile the reranker against a labeled set | **Do NOT activate** when: - The task is one-shot ("just answer this question once") — use raw `client.messages.create`. - No evaluation metric is possible and none is willing to be built — DSPy without a metric is just verbose prompting. - Prompts must remain human-authored verbatim for compliance, audit, or stylistic reasons. - The team is in *rapid exploration* mode where the task signature itself is changing daily — compile only after the signature stabilizes [dspy.ai/learn/optimization/overview/]. --- ## 2. 核心心智模型 (Core mental model) DSPy's full name is **D**eclarative **S**elf-improving **Py**thon. The three primitives form a PyTorch-like compile chain [arxiv.org/abs/2310.03714]: ``` ┌─────────────┐ ┌──────────┐ ┌──────────────┐ ┌─────────┐ │ Signature │ → │ Module │ → │ Teleprompter │ → │ Compile │ │ (what) │ │ (how) │ │ (optimizer) │ │ (tune) │ └─────────────┘ └──────────┘ └──────────────┘ └─────────┘ I/O spec Predict/CoT/ MIPROv2/GEPA/ Bake demos field names ReAct/PoT BootstrapFewShot + instructions = semantic = strategy = search algorithm into JSON ``` **Three mental shifts** the agent must internalize: 1. **Prompts are weights.** The prompt string is not the artifact you ship — the *compiled program* (a JSON of demonstrations + instructions + structural choices) is. You ship `program.json`, not a `.txt` prompt [dspy.ai/tutorials/saving/]. 2. **Signatures carry semantic load.** `question -> answer` is not the same as `query -> response`. DSPy uses the *field names* as the only natural-language hint the optimizer has about intent before it sees data. Name them like you'd name function parameters in well-written code [dspy.ai/learn/programming/signatures/]. 3. **Compile is a hyperparameter search, not a one-shot call.** Compilation runs hundreds-to-thousands of LM calls (typically $2–$3 USD, 6–20 minutes, 3.2k API calls in the reference run). Costs scale with `num_trials × |trainset| × |program LM calls|` [dspy.ai/faqs/]. **The PyTorch analogy** is load-bearing. Signatures ≈ `nn.Module.forward()` shape contract. Modules ≈ `nn.Linear` / `nn.Transformer`. Teleprompters ≈ `torch.optim.Adam`. `compile()` ≈ training loop. `save()/load()` ≈ checkpoint. --- ## 3. SOP 工作流 (SOP workflow) The DSPy team is explicit about a **three-stage gate** [dspy.ai/learn/]: > "It's unproductive to launch optimization runs using a poorly designed program or a bad metric." Do not skip stages. Each stage has an exit criterion. ### Stage 1 — Programming (no optimizer yet) 1. **Pin the task as a Signature.** Start inline (`"question -> answer"`); upgrade to a class-based `dspy.Signature` with `InputField(desc=...)` / `OutputField(desc=...)` when types matter or fields need disambiguation. 2. **Pick the lowest-power Module that works.** Default to `dspy.ChainOfThought`. Use `dspy.Predict` for trivial classification, `dspy.ReAct` only when tools are needed, `dspy.ProgramOfThought` for arithmetic-heavy tasks [dspy.ai/learn/programming/modules/]. 3. **Compose with plain Python control flow.** Subclass `dspy.Module`, instantiate sub-modules in `__init__`, call them in `forward()`. No special DSL. 4. **Run zero-shot on 5–10 hand-picked examples.** Look at outputs with `dspy.inspect_history(n=3)`. **Exit criterion:** the un-optimized program produces *plausible* outputs on 5+ examples. Not great — plausible. ### Stage 2 — Evaluation (no optimizer yet) 5. **Build a dev set.** Documented sweet spot: **30 examples = minimum useful, 300 = recommended, 200+ required for MIPROv2** to avoid overfitting [dspy.ai/learn/optimization/overview/]. 6. **Write a metric**: `def metric(example, pred, trace=None) -> float|bool`. Start with exact-match; only escalate to LLM-as-judge when the task demands it (open-ended generation, multi-criteria). 7. **Run `dspy.Evaluate(devset=dev, metric=metric, num_threads=16)`** and record a baseline score. **Exit criterion:** baseline score is stable across two runs (cache-free) AND the metric agrees with human judgment on 10 spot-checks. ### Stage 3 — Optimization (compile) 8. **Pick optimizer by data + signal regime** (Section 4 table). Decide *which model optimizes vs. which model is the task model* — they can differ. 9. **Use the unusual 20/80 split** (20% train, 80% val) for prompt-based optimizers. GEPA uses standard ML splits (maximize train) [dspy.ai/learn/optimization/overview/]. 10. **Start `auto="light"`.** Only escalate to `"medium"`/`"heavy"` if dev-set gains flatten and budget allows. 11. **Save the compiled program**: `compiled.save("v1.json")` for state, or `compiled.save("./v1/", save_program=True)` for whole-program (preferred for production with metadata) [dspy.ai/tutorials/saving/]. 12. **Deploy** via FastAPI (`dspy.asyncify`) or MLflow (`mlflow.dspy.log_model`) [dspy.ai/tutorials/deployment/]. **Exit criterion:** compiled program beats baseline on a *held-out* test set (not the val set used in optimization) by ≥ task-relevant delta. ### When to iterate back Loop to Stage 1 if optimization plateaus. Per the docs: "Is your task well-defined? Do you need more data? Should your evaluation metric change?" — these are the questions to re-ask, not "should I try a different optimizer?" [dspy.ai/learn/optimization/overview/]. --- ## 4. 操作模型 (Trigger / Action / Output / Evidence) ### 4.1 Choose the optimizer | Trigger | Action | Output | Evidence | |---|---|---|---| | ≤10 labeled examples | `BootstrapFewShot(metric=m, max_bootstrapped_demos=4, max_rounds=1)` | Compiled program with self-generated demos | [dspy.ai/learn/optimization/optimizers/] | | 30–50 examples | `BootstrapFewShotWithRandomSearch` | Best-of-N candidate programs | [dspy.ai/learn/optimization/optimizers/] | | 200+ examples, willing to spend compute | `MIPROv2(metric=m, auto="light")` then escalate | Jointly-tuned instructions + few-shot demos via Bayesian optimization | [dspy.ai/api/optimizers/MIPROv2/] | | Need zero-shot prompts (no demos in final) | `MIPROv2(..., max_bootstrapped_demos=0, max_labeled_demos=0)` | Instruction-only optimization | [dspy.ai/learn/optimization/optimizers/] | | Have textual error feedback (test diffs, schema violations, judge rationales) | `dspy.GEPA(metric=m_with_feedback)` | Reflection-evolved prompts; sample-efficient | [dspy.ai/tutorials/gepa_ai_program/], [arxiv.org/abs/2507.19457] | | Already optimized with MIPROv2 / want to ship a smaller model | Chain into `BootstrapFinetune(student=small_lm, teacher=optimized)` | Finetuned weights (not just prompts) | [dspy.ai/api/optimizers/BootstrapFinetune/] | | Just want labeled demos in prompt (no search) | `LabeledFewShot(k=8)` | Trivial — fastest, cheapest, weakest | [dspy.ai/cheatsheet/] | ### 4.2 Module selection | Trigger | Action | Why | |---|---|---| | Simple input → output | `dspy.Predict(Sig)` | Lowest overhead | | Reasoning helps | `dspy.ChainOfThought(Sig)` | **Default choice** per docs | | Math / counting / parsing | `dspy.ProgramOfThought(Sig)` | Code execution grounds the answer | | Tools (search, calc, API) | `dspy.ReAct(Sig, tools=[...])` | Built-in tool loop | | Ensemble for hard cases | `dspy.MultiChainComparison` or `dspy.majority` | Vote across N CoT samples | ### 4.3 Metric design | Trigger | Action | Caveat | |---|---|---| | Exact answer expected | `lambda ex, pred: ex.answer.lower() == pred.answer.lower()` | Cheap, deterministic | | Open-ended generation | LLM-as-judge with `dspy.ChainOfThought(JudgeSig)` | Watch for self-preference bias, recency bias, score-ID bias [arxiv.org/pdf/2509.26072] | | Multi-criteria (factuality + tone + length) | Sub-judge each dim, return **`bool` during optimization (`trace is not None`) and `float` during evaluation** | Documented pattern [dspy.ai/learn/evaluation/metrics/] | | Have rich error context | Return `dspy.Prediction(score=..., feedback="missing field X")` and use **GEPA** | Textual feedback is GEPA's superpower [dspy.ai/api/optimizers/GEPA/overview/] | ### 4.4 Cost guardrails | Trigger | Action | Reference | |---|---|---| | Before any `MIPROv2` call | Estimate: `auto="light"` ≈ a few $; `auto="heavy"` on 1000+ examples can hit tens of $ | [dspy.ai/faqs/] | | Budget tight | Use a **cheap optimizer LM** (e.g. gpt-4o-mini) to optimize prompts for a more expensive task LM — community-reported parity [github.com/stanfordnlp/dspy/issues/1596] | | Compile stuck mid-trial | Check issue #1970 pattern; reduce `minibatch_size` or kill and restart with smaller `num_trials` | | Need reproducibility | `dspy.configure(track_usage=True)` + log `program.get_lm_usage()` | --- ## 5. 困境决策案例 (Dilemma cases — ≥3) ### Case A — "Optimizer cost vs gain: when is it worth compiling?" **困境 (Dilemma):** User has a 3-stage RAG pipeline. Hand-tuned prompts already hit 72% on dev. MIPROv2 `auto="heavy"` would cost ~$40 and 4 hours. Worth it? **约束 (Constraints):** - 250 labeled examples (above MIPROv2 200-example floor) [dspy.ai/learn/optimization/optimizers/]. - Prompts already manually iterated — diminishing returns suspected. - Pipeline LM = GPT-4o ($-per-call adds up at trial scale). **决策步骤 (Decision steps):** 1. Check whether the prompts were ever validated against the metric, or just eye-balled. If eye-balled, even `auto="light"` (~$2) typically yields 10–30%+ on hand-tuned baselines per the paper's GPT-3.5/Llama2 results (25%/65% lift over standard few-shot) [arxiv.org/abs/2310.03714]. 2. Run **`auto="light"` first** as a cheap signal. The docs explicitly recommend "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy issue #1596]. 3. If `light` gives <2% lift, do **not** escalate to `heavy`. Instead, revisit Stage 1: is the signature ambiguous? Is the program structure (3 stages) actually right? 4. If `light` gives 5–10% lift, run `medium`. Only escalate to `heavy` if data ≥ 300 *and* you have a held-out test set distinct from val. 5. Use **gpt-4o-mini as the optimizer LM** even when the task LM is gpt-4o. Community evidence: parity at fraction of cost [github.com/stanfordnlp/dspy issue #1596]. **结果 (Outcome):** Typical: `light` exposes whether more compute helps. Often the answer is "no — fix the program/metric first." **可提取的操作 (Extractable operation):** **Never start compilation at `auto="heavy"`. Always probe with `light` and use a cheap optimizer LM.** --- ### Case B — "Swap the underlying LM: recompile, transfer, or both?" **困境:** Compiled program for GPT-4o works at 85%. Need to switch to Llama-3-8B for cost. Re-use the GPT-4o-compiled `program.json` or recompile? **约束:** - Compiled program contains demos + instructions that may exceed the smaller model's context coherence. - Recompile cost ≈ another $2–5. - The DSPy doctrine: "If you optimize a complex pipeline for GPT-4, it usually breaks on a smaller model like Llama-3-8b" [acldigital.com — Death to Prompting]. **决策步骤:** 1. **Always recompile when changing the task LM family.** This is the headline value prop of DSPy: "swap the LLM definition and re-compile your program" [ganeshkedari.substack.com/p/stop-writing-prompts-a-guide-to-dspy]. 2. The optimizer will discover the smaller model needs **more examples / simpler reasoning steps** and adjusts automatically — you do not edit prompts. 3. Use **`BootstrapFinetune`** as a follow-on: optimize prompts on the big model, then distill into a 1B–7B student. Typical setup: `student=Llama-3.2-1B-Instruct`, `teacher=gpt-4o-mini` [dspy.ai/api/optimizers/BootstrapFinetune/]. 4. If demos in the saved program reference GPT-4o-style verbose CoT, the small model may parrot length without reasoning. Recompile is mandatory, not optional. 5. Keep both `program.gpt4o.json` and `program.llama8b.json` checked in; A/B in production. **结果:** Recompiled programs typically recover 70–90% of the larger-model performance at 1/10–1/50 the per-call cost. The "transfer without recompile" path is reliably worse. **可提取的操作:** **Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile.** --- ### Case C — "Metric design: proxy correctness vs ground truth" **困境:** Open-ended customer-support response task. No exact-match metric possible. LLM-as-judge "feels right" but the team worries the judge will be biased toward verbose, hedged outputs. **约束:** - 400 labeled examples with a *reference response* (not the unique correct response — one acceptable response). - Production users penalize verbosity. - DSPy will optimize *toward whatever the metric rewards*. A bad metric becomes a bad program at scale. **决策步骤:** 1. **Refuse to ship a single-LLM-judge as the optimization metric.** Document evidence: LLM judges exhibit self-preference, recency, rubric-order, and provenance biases [arxiv.org/pdf/2506.02592, arxiv.org/pdf/2509.26072]. 2. **Decompose the judge** into orthogonal sub-judges, each a `dspy.Predict(Assess)` call with a single yes/no question (factual? on-topic? concise? non-hedging?). Documented pattern [dspy.ai/learn/evaluation/metrics/]. 3. **Use `trace is not None` to return bool during compile, float during eval** — same metric function, two modes. Avoids the optimizer overfitting to score noise. 4. **Spot-check the metric on 20 examples with a human judge first.** If sub-judges disagree with human on >20% of cases, fix the metric before compiling. Garbage metric → garbage compiled program.