Measure and improve how well your AI works. Use when AI gives wrong answers, accuracy is bad, responses are unreliable, you need to test AI quality, evaluate your AI, write metrics, benchmark performance, optimize prompts, improve results, or systematically make your AI better. Also used for spent hours tweaking prompts, trial and error prompt engineering is not working, quality plateaued early, stale prompts everywhere in your codebase, my AI is only 60% accurate, how to measure AI quality, AI evaluation framework, benchmark my LLM, prompt optimization not working, systematic way to improve AI, AI accuracy plateaued, DSPy optimizer tutorial, MIPROv2 optimization, how to go from 70% to 90% accuracy.
Measure and improve how well your AI works. Use when AI gives wrong answers, accuracy is bad, responses are unreliable, you need to test AI quality, evaluate your AI, write metrics, benchmark performance, optimize prompts, improve results, or systematically make your AI better. Also used for spent hours tweaking prompts, trial and error prompt engineering is not working, quality plateaued early, stale prompts everywhere in your codebase, my AI is only 60% accurate, how to measure AI quality, AI evaluation framework, benchmark my LLM, prompt optimization not working, systematic way to improve AI, AI accuracy plateaued, DSPy optimizer tutorial, MIPROv2 optimization, how to go from 70% to 90% accuracy.
Measure and Improve Your AI
Guide the user through measuring how well their AI works, then systematically improving it. This is a loop: define "good" -> measure -> improve -> verify.
The Workflow
Define what "good" means — write a metric
Measure current quality — run an evaluation
Improve — choose an optimizer, run it
Verify — re-evaluate to confirm improvement
Iterate or ship
Step 1: Understand the problem
Ask the user:
What does your AI get wrong? (wrong answers, wrong format, inconsistent, too slow?)
Do you have labeled examples? (how many? what format?)
How do you know when an answer is good? (exact match, partial credit, human judgment?)
Have you tried optimization before? (if yes, what and what happened?)
If the user does not have labeled data, point them to /ai-generating-data first.
Step 2: Define what "good" means (write a metric)
A metric takes an expected answer and the AI answer, and returns a score.
Claude writes metrics that return strings instead of floats or bools. DSPy metrics must return a numeric score (float 0.0-1.0) or a boolean. Returning a string like "correct" silently breaks evaluation — the score will be 0 for every example.
Claude forgets .with_inputs() on evaluation Examples. Every dspy.Example must call .with_inputs("field1", ...) to mark input fields. Without this, the evaluator passes all fields (including the expected output) to the program, inflating scores because the model sees the answer.
Claude uses the same data for training and evaluation. Always split into trainset and devset. Evaluating on training data gives misleadingly high scores — the optimizer may have memorized those exact examples.
AI-as-judge metrics are slow and expensive during optimization. Each training example triggers a separate LM call for the judge. For a 200-example trainset with MIPROv2 auto="medium", this can add thousands of extra LM calls. Use exact-match or F1 metrics during optimization, then validate with AI-as-judge on the final result.
display_table reveals metric bugs that the score hides. A 75% score looks reasonable, but display_table=10 might show the metric gives credit for completely wrong answers that happen to match on whitespace. Always inspect individual predictions before trusting aggregate scores.
Cross-references
Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>
See optimization progress live -- see /ai-watching-optimization
Cost reduction once quality is good -- see /ai-cutting-costs
Production monitoring to track quality after deployment -- see /ai-monitoring
Experiment tracking to log and compare optimization runs -- see /ai-tracking-experiments
Generating data when you need more training examples -- see /ai-generating-data
Fixing errors when the AI crashes or throws exceptions -- see /ai-fixing-errors
Signatures for defining typed input/output contracts -- see /dspy-signatures
Optimizers for detailed API on MIPROv2 and BootstrapFewShot -- see /dspy-optimizers
Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do
Additional resources
For worked examples and end-to-end workflows, see examples.md
For optimizer details and metric patterns, see reference.md
