Operating SOP for DSPy (Stanford NLP) — the declarative framework for "programming, not prompting" language models.
Activate when the user says any of: "use DSPy", "compile a prompt", "optimize prompts/programs", "MIPRO/MIPROv2",
"BootstrapFewShot", "GEPA", "Signatures + Modules", "teleprompter", "auto-tune prompts for a different LM",
or whenever a brittle hand-crafted prompt pipeline needs to be turned into a *compiled*, measurable, swappable program.
Do NOT activate for one-shot prompt tweaks, no-metric exploratory work, or pipelines where prompts must remain
human-authored verbatim — use raw prompting or LangChain templates instead.
Operating SOP for DSPy (Stanford NLP) — the declarative framework for "programming, not prompting" language models.
Activate when the user says any of: "use DSPy", "compile a prompt", "optimize prompts/programs", "MIPRO/MIPROv2",
"BootstrapFewShot", "GEPA", "Signatures + Modules", "teleprompter", "auto-tune prompts for a different LM",
or whenever a brittle hand-crafted prompt pipeline needs to be turned into a *compiled*, measurable, swappable program.
Do NOT activate for one-shot prompt tweaks, no-metric exploratory work, or pipelines where prompts must remain
human-authored verbatim — use raw prompting or LangChain templates instead.
DSPy SOP — Programming, Not Prompting
"DSPy isn't a prompt-optimization agent framework. It's the LLM compiler for the shortest, cleanest code."
— Eito Miyamura [eito.substack.com/p/dspy-the-most-misunderstood-agent]
"Prompts are effectively the weights of an LLM application."
— Core philosophy [arxiv.org/abs/2310.03714]
1. 何时激活 (When to activate)
Activate this skill when any of the following triggers are present in the user's intent or codebase:
Trigger
Signal
Imports / mentions
import dspy, dspy.Signature, dspy.ChainOfThought, dspy.ReAct, Predict, MIPROv2, BootstrapFewShot, GEPA, teleprompter, compile( on an LM program
Tasks
"auto-tune this prompt", "I want to swap GPT-4 for a smaller model without re-engineering prompts", "I have 50/200/1000 labeled examples — optimize this", "compile a pipeline for our metric", "distill GPT-4 into Llama-3-8B"
Symptoms
Hand-written prompts grow past ~50 lines; brittleness on model swap; the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design
Cross-skill bridges
LangGraph node calls an LLM and needs better prompts → wrap the node body in a DSPy module. LlamaIndex retriever feeds a reranker → DSPy-compile the reranker against a labeled set
Do NOT activate when:
The task is one-shot ("just answer this question once") — use raw client.messages.create.
No evaluation metric is possible and none is willing to be built — DSPy without a metric is just verbose prompting.
Prompts must remain human-authored verbatim for compliance, audit, or stylistic reasons.
The team is in rapid exploration mode where the task signature itself is changing daily — compile only after the signature stabilizes [dspy.ai/learn/optimization/overview/].
2. 核心心智模型 (Core mental model)
DSPy's full name is Declarative Self-improving Python. The three primitives form a PyTorch-like compile chain [arxiv.org/abs/2310.03714]:
Prompts are weights. The prompt string is not the artifact you ship — the compiled program (a JSON of demonstrations + instructions + structural choices) is. You ship program.json, not a .txt prompt [dspy.ai/tutorials/saving/].
Signatures carry semantic load.question -> answer is not the same as query -> response. DSPy uses the field names as the only natural-language hint the optimizer has about intent before it sees data. Name them like you'd name function parameters in well-written code [dspy.ai/learn/programming/signatures/].
Compile is a hyperparameter search, not a one-shot call. Compilation runs hundreds-to-thousands of LM calls (typically $2–$3 USD, 6–20 minutes, 3.2k API calls in the reference run). Costs scale with num_trials × |trainset| × |program LM calls| [dspy.ai/faqs/].
The PyTorch analogy is load-bearing. Signatures ≈ nn.Module.forward() shape contract. Modules ≈ nn.Linear / nn.Transformer. Teleprompters ≈ torch.optim.Adam. compile() ≈ training loop. save()/load() ≈ checkpoint.
3. SOP 工作流 (SOP workflow)
The DSPy team is explicit about a three-stage gate [dspy.ai/learn/]:
"It's unproductive to launch optimization runs using a poorly designed program or a bad metric."
Do not skip stages. Each stage has an exit criterion.
Stage 1 — Programming (no optimizer yet)
Pin the task as a Signature. Start inline ("question -> answer"); upgrade to a class-based dspy.Signature with InputField(desc=...) / OutputField(desc=...) when types matter or fields need disambiguation.
Pick the lowest-power Module that works. Default to dspy.ChainOfThought. Use dspy.Predict for trivial classification, dspy.ReAct only when tools are needed, dspy.ProgramOfThought for arithmetic-heavy tasks [dspy.ai/learn/programming/modules/].
Compose with plain Python control flow. Subclass dspy.Module, instantiate sub-modules in __init__, call them in forward(). No special DSL.
Run zero-shot on 5–10 hand-picked examples. Look at outputs with dspy.inspect_history(n=3).
Exit criterion: the un-optimized program produces plausible outputs on 5+ examples. Not great — plausible.
Stage 2 — Evaluation (no optimizer yet)
Build a dev set. Documented sweet spot: 30 examples = minimum useful, 300 = recommended, 200+ required for MIPROv2 to avoid overfitting [dspy.ai/learn/optimization/overview/].
Write a metric: def metric(example, pred, trace=None) -> float|bool. Start with exact-match; only escalate to LLM-as-judge when the task demands it (open-ended generation, multi-criteria).
Run dspy.Evaluate(devset=dev, metric=metric, num_threads=16) and record a baseline score.
Exit criterion: baseline score is stable across two runs (cache-free) AND the metric agrees with human judgment on 10 spot-checks.
Stage 3 — Optimization (compile)
Pick optimizer by data + signal regime (Section 4 table). Decide which model optimizes vs. which model is the task model — they can differ.
Use the unusual 20/80 split (20% train, 80% val) for prompt-based optimizers. GEPA uses standard ML splits (maximize train) [dspy.ai/learn/optimization/overview/].
Start auto="light". Only escalate to "medium"/"heavy" if dev-set gains flatten and budget allows.
Save the compiled program: compiled.save("v1.json") for state, or compiled.save("./v1/", save_program=True) for whole-program (preferred for production with metadata) [dspy.ai/tutorials/saving/].
Deploy via FastAPI (dspy.asyncify) or MLflow (mlflow.dspy.log_model) [dspy.ai/tutorials/deployment/].
Exit criterion: compiled program beats baseline on a held-out test set (not the val set used in optimization) by ≥ task-relevant delta.
When to iterate back
Loop to Stage 1 if optimization plateaus. Per the docs: "Is your task well-defined? Do you need more data? Should your evaluation metric change?" — these are the questions to re-ask, not "should I try a different optimizer?" [dspy.ai/learn/optimization/overview/].
Return dspy.Prediction(score=..., feedback="missing field X") and use GEPA
Textual feedback is GEPA's superpower [dspy.ai/api/optimizers/GEPA/overview/]
4.4 Cost guardrails
Trigger
Action
Reference
Before any MIPROv2 call
Estimate: auto="light" ≈ a few $; auto="heavy" on 1000+ examples can hit tens of $
[dspy.ai/faqs/]
Budget tight
Use a cheap optimizer LM (e.g. gpt-4o-mini) to optimize prompts for a more expensive task LM — community-reported parity [github.com/stanfordnlp/dspy/issues/1596]
Compile stuck mid-trial
Check issue #1970 pattern; reduce minibatch_size or kill and restart with smaller num_trials
Pipeline LM = GPT-4o ($-per-call adds up at trial scale).
决策步骤 (Decision steps):
Check whether the prompts were ever validated against the metric, or just eye-balled. If eye-balled, even auto="light" (~$2) typically yields 10–30%+ on hand-tuned baselines per the paper's GPT-3.5/Llama2 results (25%/65% lift over standard few-shot) [arxiv.org/abs/2310.03714].
Run auto="light" first as a cheap signal. The docs explicitly recommend "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy issue #1596].
If light gives <2% lift, do not escalate to heavy. Instead, revisit Stage 1: is the signature ambiguous? Is the program structure (3 stages) actually right?
If light gives 5–10% lift, run medium. Only escalate to heavy if data ≥ 300 and you have a held-out test set distinct from val.
Use gpt-4o-mini as the optimizer LM even when the task LM is gpt-4o. Community evidence: parity at fraction of cost [github.com/stanfordnlp/dspy issue #1596].
结果 (Outcome): Typical: light exposes whether more compute helps. Often the answer is "no — fix the program/metric first."
可提取的操作 (Extractable operation):Never start compilation at auto="heavy". Always probe with light and use a cheap optimizer LM.
Case B — "Swap the underlying LM: recompile, transfer, or both?"
困境: Compiled program for GPT-4o works at 85%. Need to switch to Llama-3-8B for cost. Re-use the GPT-4o-compiled program.json or recompile?
约束:
Compiled program contains demos + instructions that may exceed the smaller model's context coherence.
Recompile cost ≈ another $2–5.
The DSPy doctrine: "If you optimize a complex pipeline for GPT-4, it usually breaks on a smaller model like Llama-3-8b" [acldigital.com — Death to Prompting].
决策步骤:
Always recompile when changing the task LM family. This is the headline value prop of DSPy: "swap the LLM definition and re-compile your program" [ganeshkedari.substack.com/p/stop-writing-prompts-a-guide-to-dspy].
The optimizer will discover the smaller model needs more examples / simpler reasoning steps and adjusts automatically — you do not edit prompts.
Use BootstrapFinetune as a follow-on: optimize prompts on the big model, then distill into a 1B–7B student. Typical setup: student=Llama-3.2-1B-Instruct, teacher=gpt-4o-mini [dspy.ai/api/optimizers/BootstrapFinetune/].
If demos in the saved program reference GPT-4o-style verbose CoT, the small model may parrot length without reasoning. Recompile is mandatory, not optional.
Keep both program.gpt4o.json and program.llama8b.json checked in; A/B in production.
结果: Recompiled programs typically recover 70–90% of the larger-model performance at 1/10–1/50 the per-call cost. The "transfer without recompile" path is reliably worse.
可提取的操作:Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile.
Case C — "Metric design: proxy correctness vs ground truth"
困境: Open-ended customer-support response task. No exact-match metric possible. LLM-as-judge "feels right" but the team worries the judge will be biased toward verbose, hedged outputs.
约束:
400 labeled examples with a reference response (not the unique correct response — one acceptable response).
Production users penalize verbosity.
DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale.
决策步骤:
Refuse to ship a single-LLM-judge as the optimization metric. Document evidence: LLM judges exhibit self-preference, recency, rubric-order, and provenance biases [arxiv.org/pdf/2506.02592, arxiv.org/pdf/2509.26072].
Decompose the judge into orthogonal sub-judges, each a dspy.Predict(Assess) call with a single yes/no question (factual? on-topic? concise? non-hedging?). Documented pattern [dspy.ai/learn/evaluation/metrics/].
Use trace is not None to return bool during compile, float during eval — same metric function, two modes. Avoids the optimizer overfitting to score noise.
Spot-check the metric on 20 examples with a human judge first. If sub-judges disagree with human on >20% of cases, fix the metric before compiling. Garbage metric → garbage compiled program.
If sub-judge feedback is rich (e.g. "answer was verbose"), pipe textual feedback into dspy.GEPA instead of MIPROv2 — GEPA leverages text feedback for faster, more sample-efficient convergence [dspy.ai/api/optimizers/GEPA/overview/, arxiv.org/abs/2507.19457].
Add a length penalty as a separate scalar in the metric — don't rely on the judge to penalize verbosity (judges over-prefer length).
结果: Multi-dimension metric with explicit length penalty + GEPA's textual feedback typically beats single-judge + MIPROv2 by 10–13% on AIME-style benchmarks [arxiv.org/abs/2507.19457] and is the empirically robust path.
可提取的操作:Never compile against a metric you haven't human-validated on ≥ 20 spot-checks. Decompose multi-criteria metrics. Prefer GEPA when you can express textual feedback.
Case D — "Compile-time hang / stuck trial — abort or wait?"
困境: MIPROv2 compile stuck mid-trial (no progress logs for 30 min). Reported pattern in issue #1970 [github.com/stanfordnlp/dspy/issues/1970]. Abort and restart, or wait?
约束:
$15 spent so far on the run.
Sunk cost vs. wasted further spend.
Possible causes: a single example triggers rate limits / context-length overflow / a tool call hangs.
决策步骤:
Check dspy.inspect_history(n=3) — does the last LM call show truncation or rate-limit error?
If context-length: reduce max_bootstrapped_demos and max_labeled_demos (default 4 each); the docs explicitly cite this as the #1 context-length fix [dspy.ai/faqs/].
If rate-limit: lower num_threads in the underlying Evaluate; add retry/backoff in the LM client.
If neither, abort. Restart with smaller minibatch_size (default 35; try 16) and smaller num_trials. Hanging is a known failure mode without graceful resume.
Save partial progress: even mid-compile, student retains best demo candidates — check compiled._predictors state.
可提取的操作:Compile is not atomic. Treat long hangs as failure. The cost of restart < cost of indefinite wait.
6. 反模式与边界 (Anti-patterns & boundaries)
Anti-patterns
Compiling without a metric. Without a metric, DSPy collapses to verbose prompt templating. The Predict module requires nothing; the optimizers require a metric. If you cannot write a metric, you cannot optimize, period [dspy.ai/learn/optimization/overview/].
Compiling on 5 examples. Below ~30 examples, you're not training — you're memorizing. The 20/80 train/val split exists because "prompt-based optimizers often overfit to small training sets" [dspy.ai/learn/optimization/overview/].
Editing the compiled JSON by hand. It's plain JSON and readable — but human edits invalidate the assumption that the artifact was metric-optimized. Re-compile or don't touch.
Using LLM-as-judge as the only metric for any open-ended task. See Case C. Biases are documented and reproducible [arxiv.org/pdf/2506.02592].
Starting at auto="heavy". Always probe with light first [Case A].
Treating DSPy modules as agents. ReAct is a thin tool-loop, not a multi-agent framework. For long-running, stateful, branching workflows: combine DSPy with LangGraph (see Section 7).
Ignoring program structure when optimization stalls. If MIPROv2 light + medium both flatline, the bottleneck is almost always the program graph (wrong decomposition, wrong module choice) not the optimizer [dspy.ai/learn/optimization/overview/].
Re-using GPT-4-compiled programs on Llama-8B. See Case B.
Skipping the unusual 20/80 split. The reversed ratio is intentional and prevents prompt-overfitting [dspy.ai/learn/optimization/overview/].
Forgetting cache=False in Lambda / stateless deploys. Caches default to a writable dir and break in serverless [dspy.ai/faqs/].
Boundaries (when NOT to use DSPy)
One-shot tasks. "Summarize this email once" → raw API call. The compile loop has no payoff.
The task signature is still changing daily. Compile only after the I/O contract stabilizes; otherwise you're paying compile cost for prompts you'll throw away.
Compliance/audit requires verbatim human-authored prompts. Optimized prompts are machine-generated artifacts; some regulated contexts disallow this.
No labeled data and no labelable proxy. Without a metric, the framework can't help you. (Note: even 30 examples can work — but you need some signal.)
You need rich agent observability with LangFuse-style traces today. Native integration is limited; bolt-on via MLflow tracing works but is not first-class [eito.substack.com].
Streaming partial outputs is essential. DSPy supports dspy.streamify from 2.6.0+ but it's newer than the rest of the stack — verify your version [dspy.ai/tutorials/deployment/].
7. 生态对照 (Ecosystem context)
Layer positioning
DSPy is not the same layer as LangChain / LlamaIndex / LangGraph. It sits underneath them as a compiler for the individual LM calls inside those orchestration layers [langwatch.ai/blog/best-ai-agent-frameworks-in-2025-...].
LangChain: orchestration, batteries-included, hand-authored prompts. 71 lines for a typical agent.
DSPy: compile-the-prompt, 30 lines for the same task [eito.substack.com — code comparison].
Together: rare. LangChain's prompt templates collide with DSPy's compilation model. Pick one per pipeline; if you must combine, isolate DSPy modules behind clean Python interfaces inside LangChain chains.
Together: highly complementary. Pattern: each LangGraph node's body invokes a compiled DSPy program. LangGraph handles state and routing; DSPy handles prompt quality [rajapatnaik.com/blog/2025/10/23/langgraph-dspy-gepa-researcher]. Quote from acldigital.com: "DSPy fits into the Prompt Management & Optimization layer—bringing software engineering discipline to prompting" while LangGraph fits the orchestration layer.
vs LlamaIndex
LlamaIndex = data ingestion + retrieval + query engines.
DSPy = optimizes the generator / reranker / synthesizer component.
Together: standard pattern. LlamaIndex retrieves passages → DSPy-compiled Predict(context, question -> answer) synthesizes. JetBlue's chatbot uses exactly this split — retrieval quality + answer quality as separate metrics, DSPy optimizes both [databricks.com/blog/optimizing-databricks-llm-pipelines-dspy].
vs Guidance / LMQL / Outlines
These control one LM call at the token level (grammars, regex, JSON schema).
DSPy controls multi-call programs at the optimization level.
Together: orthogonal. Use Outlines for "force valid JSON"; use DSPy for "make the JSON-emitting prompt good." DSPy's typed OutputField already pushes the LM toward structure but doesn't guarantee grammar conformance [dspy.ai/faqs/].
vs raw prompt engineering
Raw prompts win when: one-shot, signature unstable, no metric, audit constraints (Section 6).
DSPy wins when: pipeline ≥ 2 LM calls, you have ≥ 30 labeled examples, you'll swap models or scale, you have a metric (even an LLM-judge one — but harden it per Case C).
Production case study evidence
JetBlue + Databricks: RAG chatbot. Before DSPy = manual prompt tuning on retrieval/answer quality metrics. After = DSPy directly optimizes those metrics, faster development cycle [tastytechbytes.com/databricks-dspy-jetblue-ai-chatbot].
Haize Labs: automated LLM red-teaming.
In production at: Shopify, Databricks, Dropbox, JetBlue, Moody's, AWS, Sephora, VMware [dspy.ai].
Tobi Lütke (Shopify CEO): "Both DSPy and (especially) GEPA are currently severely under hyped in the AI context engineering world" [eito.substack.com].
Quick-reference appendix
Minimal end-to-end (verbatim from cheatsheet) [dspy.ai/cheatsheet/]
import dspy
# 1. SignatureclassBasicQA(dspy.Signature):
"""Answer questions with short factoid answers."""
question: str = dspy.InputField()
answer: str = dspy.OutputField(desc="often between 1 and 5 words")
# 2. Module
qa = dspy.ChainOfThought(BasicQA)
# 3. Metricdefmetric(ex, pred, trace=None):
return ex.answer.lower() in pred.answer.lower()
# 4. Compilefrom dspy.teleprompt import MIPROv2
optimizer = MIPROv2(metric=metric, auto="light")
compiled = optimizer.compile(qa, trainset=trainset)
# 5. Save / load
compiled.save("v1.json")
Decision tree (one screen)
Have a metric? ─── No ──► Stop. Build a metric first. (Or skip DSPy.)
│
Yes
│
Have ≥ 30 examples? ─── No ──► Stop. Collect more data, or use LabeledFewShot(k=8) as floor.
│
Yes
│
Have textual error feedback? ─── Yes ──► dspy.GEPA
│
No
│
≤ 10 examples? ──► BootstrapFewShot
30–50? ──► BootstrapFewShotWithRandomSearch
50–200? ──► MIPROv2(auto="light", max_bootstrapped_demos=4)
200+? ──► MIPROv2(auto="light" → "medium" if gains; "heavy" only if 300+ and budget)
Need to ship small model? ──► chain BootstrapFinetune after MIPROv2
Anatomy of a compiled program.json
After compiled.save("v1.json"), the file is plain JSON. Per-predictor it contains [dspy.ai/tutorials/saving/]:
{"predictor_name":{"signature_instructions":"Given the context, answer the question with a short factoid...","signature_prefix":"Answer:","extended_signature_instructions":"...","demos":[{"question":"...","reasoning":"...","answer":"..."},
...
],"signature":{"instructions":"...","fields":[{"prefix":"Question:","description":"..."}, ...]}}}
What changes when you compile:
Instructions are rewritten by the optimizer (MIPROv2 proposes; Bayesian search picks).
Demos are bootstrapped: the teacher program is run on trainset, metric filters keep the good ones.
Signature shape does NOT change — that's your code's job.
What does NOT change between LMs (so you can read across artifacts):
The signature field names.
The metric.
The program's Python structure (which modules call which).
What DOES change between LMs (so you can't reuse):
The instructions (smaller LMs need simpler, more explicit wording).
The demos (smaller LMs benefit from more, simpler demos; larger LMs benefit from fewer, richer ones).
Constraint primitives — dspy.Assert vs dspy.Suggest
For self-refining pipelines [dspy.ai/learn/programming/7-assertions/, arxiv.org/pdf/2312.13382]:
# Hard: halts after max retries with dspy.AssertionError
dspy.Assert(len(pred.answer) < 100, "Answer must be < 100 chars")
# Soft: retries with feedback in prompt, logs failure, continues
dspy.Suggest(is_valid_json(pred.output), "Output must be valid JSON")
When a constraint fails, DSPy backtracks to the previous module and re-runs with the error message injected into the prompt. This is self-refinement at inference time — distinct from compile-time optimization.
Use Assert during development (catch bugs hard). Use Suggest in production (degrade gracefully).
Key citations
DSPy paper (ICLR 2024): [arxiv.org/abs/2310.03714]
GEPA paper (ICLR 2026 oral): [arxiv.org/abs/2507.19457]
