Run dspy.Evaluate to score DSPy programs against labeled examples using custom or built-in metrics. Use when you need to measure how well your DSPy program performs — writing metrics, scoring against a dev set, or comparing before/after optimization. Common scenarios - measuring accuracy before and after optimization, writing custom metrics for your task, scoring a program against a held-out dev set, comparing two prompt strategies, building a test suite for AI quality, or running regression tests on AI outputs. Related - ai-improving-accuracy, ai-scoring, ai-monitoring. Also used for dspy.Evaluate, dspy.evaluate, write DSPy metric function, measure AI accuracy, evaluate DSPy program, dev set evaluation, before and after optimization comparison, custom scoring function, test AI quality systematically, AI regression testing, metric-driven development, how to know if my DSPy program improved, score predictions against labels, evaluation harness for LLM, CI/CD for AI quality.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Run dspy.Evaluate to score DSPy programs against labeled examples using custom or built-in metrics. Use when you need to measure how well your DSPy program performs — writing metrics, scoring against a dev set, or comparing before/after optimization. Common scenarios - measuring accuracy before and after optimization, writing custom metrics for your task, scoring a program against a held-out dev set, comparing two prompt strategies, building a test suite for AI quality, or running regression tests on AI outputs. Related - ai-improving-accuracy, ai-scoring, ai-monitoring. Also used for dspy.Evaluate, dspy.evaluate, write DSPy metric function, measure AI accuracy, evaluate DSPy program, dev set evaluation, before and after optimization comparison, custom scoring function, test AI quality systematically, AI regression testing, metric-driven development, how to know if my DSPy program improved, score predictions against labels, evaluation harness for LLM, CI/CD for AI quality.
Evaluate Your DSPy Program
Guide the user through measuring AI quality with DSPy's Evaluate class. The pattern: pick a metric, prepare a devset, run the evaluator, interpret results, then feed the same metric into an optimizer.
Step 1 — Gather context
Before recommending a metric or writing evaluation code, clarify:
What fields does your program output? (answer, response, custom fields — metric field names must match exactly; SemanticF1 requires response, not answer)
Do you have labeled gold answers, or do you need an LM judge to assess open-ended quality?
What does "correct" mean for your task? Exact string match, semantic overlap, factual groundedness, safety, or a weighted combination?
Is this metric feeding into an optimizer? If yes, you may want trace-aware logic — stricter requirements during optimization than during bare evaluation.
What is dspy.Evaluate
dspy.Evaluate runs your program on every devset example, scores each with a metric, and reports the aggregate score. It handles threading and progress display. Returns an EvaluationResult; access .score for the percentage (0-100) and .results for per-example (example, prediction, score) tuples.
Built-in metrics
DSPy provides answer_exact_match (normalized string equality — compares example.answer with prediction.answer) and answer_passage_match (RAG retrieval hit check — returns True if any passage in prediction.context contains example.answer). They do not share the same field expectations.
Choosing a metric
Metric
Example fields
Prediction fields
Best for
LM cost
answer_exact_match
answer
answer
Factoid QA, structured output
Free
answer_passage_match
answer
context (list of passages)
RAG retrieval hit rate
Free
SemanticF1
question, response
response
Open-ended QA with near-miss answers
1 LM call
CompleteAndGrounded
question, response
response, context
RAG faithfulness + completeness
2 LM calls
Custom LM-as-judge
Any
Any
Subjective quality, style, safety
1+ LM calls
Start with the cheapest metric that captures your quality signal. Each LM-judge call costs one LM invocation per example — on a 100-example devset with 4 optimizer rounds this adds thousands of extra calls. Only reach for LM-as-judge when simpler metrics genuinely cannot distinguish good from bad outputs.
SemanticF1
Measures token-level overlap between the predicted and expected answer using an F1 score. More forgiving than exact match — it gives partial credit for answers that are close but not identical:
SemanticF1 is a good default metric for open-ended QA tasks where exact match is too strict. It expects a question field on the example plus a response field on both the example and prediction (not answer). Constructor: SemanticF1(threshold=0.66, decompositional=False). The threshold controls the minimum score during optimization (when trace is set).
CompleteAndGrounded
Checks whether the predicted answer is both complete (covers all key claims in the gold answer) and grounded (doesn't hallucinate facts not in the gold answer):
This is an LM-based metric — it uses the configured LM to judge completeness and groundedness. It expects response and context fields on the prediction, and question and response on the example. Constructor: CompleteAndGrounded(threshold=0.66). Useful for RAG tasks where you care about both recall and precision of facts.
Custom metrics
A metric is def metric(example, prediction, trace=None) returning bool, int, or float. The trace parameter is None during evaluation but set during optimization (use this to apply stricter requirements during training).
For open-ended tasks (summaries, creative writing, complex QA), use an LM to judge quality. Define a signature for the judge, then call it inside your metric:
classAssessAnswer(dspy.Signature):
"""Assess if the predicted answer correctly addresses the question."""
question: str = dspy.InputField()
gold_answer: str = dspy.InputField(desc="The reference answer")
predicted_answer: str = dspy.InputField(desc="The answer to evaluate")
is_correct: bool = dspy.OutputField(desc="True if the prediction is correct and complete")
defllm_judge_metric(example, prediction, trace=None):
judge = dspy.Predict(AssessAnswer)
result = judge(
question=example.question,
gold_answer=example.answer,
predicted_answer=prediction.answer,
)
return result.is_correct
Use a separate LM for the judge
To avoid the model grading its own work, use a different (often stronger) LM for the judge:
judge_lm = dspy.LM("openai/gpt-4o") # or "anthropic/claude-sonnet-4-5-20250929", etc.defllm_judge_metric(example, prediction, trace=None):
judge = dspy.Predict(AssessAnswer)
with dspy.context(lm=judge_lm):
result = judge(
question=example.question,
gold_answer=example.answer,
predicted_answer=prediction.answer,
)
return result.is_correct
Graded judge (float scores)
Return a float instead of a bool for partial credit:
classGradeAnswer(dspy.Signature):
"""Grade the predicted answer on a scale of 0 to 5."""
question: str = dspy.InputField()
gold_answer: str = dspy.InputField()
predicted_answer: str = dspy.InputField()
score: int = dspy.OutputField(desc="Score from 0 (completely wrong) to 5 (perfect)")
justification: str = dspy.OutputField(desc="Why this score was given")
defgraded_metric(example, prediction, trace=None):
judge = dspy.ChainOfThought(GradeAnswer)
result = judge(
question=example.question,
gold_answer=example.answer,
predicted_answer=prediction.answer,
)
return result.score / 5.0# normalize to 0.0-1.0
Composite metrics
Combine multiple signals into a single score with weights:
defhybrid_metric(example, prediction, trace=None):
# Fast exact checkif prediction.answer.strip().lower() == example.answer.strip().lower():
return1.0# Fall back to LM judge for partial credit
judge = dspy.Predict(AssessAnswer)
result = judge(
question=example.question,
gold_answer=example.answer,
predicted_answer=prediction.answer,
)
return0.5if result.is_correct else0.0
Debugging with per-example scores
Evaluate returns an EvaluationResult with .score (aggregate percentage) and .results (list of (example, prediction, score) tuples). Use .results to find failing examples and understand failure patterns.
Common patterns
Trace-aware metrics for optimization
The trace parameter is None during evaluation but set during optimization. Use this to apply stricter requirements during training:
defmetric(example, prediction, trace=None):
correct = prediction.answer.strip().lower() == example.answer.strip().lower()
if trace isnotNone:
# During optimization: also require good reasoning
has_reasoning = len(getattr(prediction, "reasoning", "")) > 50return correct and has_reasoning
# During evaluation: only check correctnessreturn correct
This makes the optimizer filter for traces where the model both got the answer right and showed its work. The result is more robust few-shot demonstrations.
Before-and-after comparison
A common workflow for measuring the impact of optimization:
Claude uses return_all_scores=True or return_outputs=True which no longer exist.Evaluate now returns an EvaluationResult object. Access .score for the aggregate percentage and .results for per-example (example, prediction, score) tuples. Do not pass return_all_scores or return_outputs.
Claude uses SemanticF1 with answer fields but it expects response.SemanticF1 looks for response on both the example and prediction, not answer. If your signature uses answer, either rename the field or write a wrapper metric.
Claude uses CompleteAndGrounded without providing context.CompleteAndGrounded expects response and context on the prediction, and question and response on the example. Without context, it cannot check groundedness.
Metrics must return a float or bool, not a string -- returning a string silently breaks scoring.
Small dev sets (<30 examples) give unreliable scores -- results can swing 10-20% between runs. Aim for 50+ examples for stable evaluation.
Do not start with an LM-as-judge if answer_exact_match or SemanticF1 is sufficient. Each judge call costs one LM invocation per example per evaluation pass. During optimizer compilation with hundreds of candidate traces, costs multiply fast. Benchmark the simple metrics first; add a judge only if they cannot distinguish your good from bad outputs.
For constructor signatures and method reference, see reference.md
For worked examples (exact match, LM judge, composite), see examples.md
Cross-references
Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>
Need to prepare training and evaluation data? Use /dspy-data
Ready to optimize with few-shot examples? Use /dspy-bootstrap-few-shot
Want the best prompt optimization? Use /dspy-miprov2
For the full measure-improve-verify loop, see /ai-improving-accuracy
For decomposed RAG evaluation (faithfulness, context precision/recall) see /dspy-ragas
For scoring AI outputs outside of optimization loops, see /ai-scoring
For worked examples (exact match, LM judge, composite), see examples.md
Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do
