dspy-best-of-n

name

dspy-best-of-n

description

Runs any DSPy module N times and returns the highest-scoring output via a reward function using dspy.BestOfN. Use when output quality varies across runs and you want to sample multiple completions and pick the best — trading latency for reliability on high-stakes outputs. Common scenarios - generating multiple candidate answers and picking the highest-scoring one, improving reliability on high-stakes classification, reducing variance in creative generation, getting better summaries by sampling several and selecting the best, or trading latency for quality on critical decisions. Related - ai-improving-accuracy, ai-making-consistent, dspy-refine. Also used for sample multiple completions, pick the best of several LLM outputs, majority voting for LLM, self-consistency decoding, reduce LLM output variance, generate and select pattern, best candidate selection, how to make AI more reliable by trying multiple times, brute force better quality, retry and pick best, dspy.BestOfN, quality vs latency tradeoff, n=5 completions pick best.

Pick the Best Output with dspy.BestOfN

Guide the user through using DSPy's BestOfN module to run a program multiple times and keep the highest-scoring result. This is rejection sampling -- generate N candidates, score each one, return the winner.

Before generating code, confirm

If the user has not already answered these, ask before writing BestOfN code:

1. Do you have an existing DSPy module to wrap, or do you need to build one first? (BestOfN wraps any dspy.Module — Predict, ChainOfThought, ReAct, a custom class, etc.)
2. Can you score outputs automatically? BestOfN requires a reward_fn — a function that scores each output without human judgment. If the user has no scoring idea, suggest simple heuristics (length check, regex match, test execution) before reaching for an LM judge.
3. What is the latency or cost budget? N attempts at temperature=1.0 means N full LM calls. Help the user pick N that fits their constraints.

What is BestOfN

dspy.BestOfN wraps any DSPy module and calls it up to N times with temperature=1.0 (each attempt uses a different rollout ID to get diverse outputs). A reward function scores every result, and BestOfN returns the single best prediction.

If any attempt hits a score threshold you set, execution stops early -- no need to burn through all N attempts when you already have a great result.

Your module ──> Run N times ──> Score each with reward_fn ──> Return best

When to use BestOfN

• You have a cheap, fast metric that can score outputs automatically (test suite passes, regex match, word count check, etc.)
• Quality variance is high -- the same prompt sometimes produces great output and sometimes doesn't
• You'd rather spend tokens than engineering time -- BestOfN is the simplest way to boost quality without optimization
• You need a quick quality boost before investing in full prompt optimization with MIPROv2 or BootstrapFewShot

Do not use BestOfN when:

• You have no way to automatically score outputs (you need a metric)
• Latency matters more than quality (N calls take N times longer, unless you can parallelize)
• Cost is a hard constraint and N is large

Basic usage

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# 1. Define your module
qa = dspy.ChainOfThought("question -> answer")

# 2. Define a reward function
def short_answer(args, pred):
    """Prefer concise single-word answers."""
    return 1.0 if len(pred.answer.split()) == 1 else 0.0

# 3. Wrap with BestOfN
best_qa = dspy.BestOfN(
    module=qa,
    N=3,
    reward_fn=short_answer,
    threshold=1.0,
)

# 4. Call it like any module
result = best_qa(question="What is the capital of Belgium?")
print(result.answer)

Constructor parameters

dspy.BestOfN(
    module,       # Any dspy.Module to run repeatedly
    N,            # Number of attempts (int)
    reward_fn,    # Scoring function: (args_dict, prediction) -> float
    threshold,    # Early-stop threshold: stop as soon as a score >= threshold
    fail_count=None,  # Max failures before raising an error (defaults to N)
)

Parameter	Type	Description
module	dspy.Module	The module to run N times
N	int	Maximum number of attempts
reward_fn	Callable[[dict, Prediction], float]	Scores each prediction; higher is better
threshold	float	If any attempt scores >= this value, return immediately
fail_count	int | None	How many attempts can fail (raise exceptions) before BestOfN itself raises. Defaults to N (all can fail before error)

The reward function

The reward function is the core of BestOfN. It receives two arguments:

def reward_fn(args: dict, prediction: dspy.Prediction) -> float:
    # args: the keyword arguments you passed to the BestOfN call
    # prediction: the output from one attempt of the wrapped module
    # Return: a scalar score (higher = better)
    ...

Key differences from a dspy.Evaluate metric:

• Signature: (args_dict, prediction) not (example, prediction, trace)
• No gold labels: args contains only the inputs you passed, not expected outputs
• No trace parameter: BestOfN doesn't use traces

Reward function examples

Binary pass/fail:

def passes_tests(args, pred):
    """Score 1.0 if generated code passes all tests, 0.0 otherwise."""
    try:
        exec(pred.code)
        return 1.0
    except Exception:
        return 0.0

Graded score:

def quality_score(args, pred):
    """Score summaries on length and keyword coverage."""
    score = 0.0
    # Prefer summaries under 100 words
    if len(pred.summary.split()) <= 100:
        score += 0.5
    # Reward covering key topics
    keywords = ["revenue", "growth", "forecast"]
    covered = sum(1 for kw in keywords if kw in pred.summary.lower())
    score += 0.5 * (covered / len(keywords))
    return score

Using an LM as judge inside the reward:

class JudgeQuality(dspy.Signature):
    """Rate the answer quality from 0.0 to 1.0."""
    question: str = dspy.InputField()
    answer: str = dspy.InputField()
    score: float = dspy.OutputField(desc="Quality score from 0.0 to 1.0")

judge = dspy.Predict(JudgeQuality)

def llm_reward(args, pred):
    result = judge(question=args["question"], answer=pred.answer)
    return result.score

Note: Using an LM as judge inside the reward function costs additional tokens per attempt. Reserve this for cases where programmatic scoring isn't feasible.

Tuning N

N	Trade-off
2-3	Low cost, modest quality gain. Good starting point.
5	Solid improvement for tasks with high variance. Sweet spot for most uses.
10+	Diminishing returns unless your metric is very selective (e.g., <10% pass rate).

Rule of thumb: if your base module succeeds ~50% of the time, N=3 gives you a ~87.5% chance of at least one success. If it succeeds ~20% of the time, you need N=8 for ~83%.

The math: probability of at least one success in N tries = 1 - (1 - p)^N where p is the single-attempt success rate.

How selection works internally

1. BestOfN calls your module with temperature=1.0 and a unique rollout ID for each attempt
2. Each attempt produces a dspy.Prediction
3. The reward function scores the prediction
4. If the score >= threshold, return immediately (early stopping)
5. If the attempt raises an exception, increment the failure counter
6. After all N attempts (or early stopping), return the prediction with the highest score
7. If failures exceed fail_count, raise an exception

The unique rollout IDs ensure the LM produces diverse outputs even with the same input. Temperature is fixed at 1.0 to maximize diversity.

Cost considerations

BestOfN multiplies your token usage by up to N times (fewer if early stopping kicks in). Budget accordingly:

Base cost per call	N	Max cost
$0.01	3	$0.03
$0.01	5	$0.05
$0.01	10	$0.10

Ways to manage cost:

• Set a tight threshold so good results stop early (often after 1-2 attempts)
• Use a cheap model as the base module and a stronger model only for the reward function
• Start with N=3 and increase only if your metric shows it helps
• Use programmatic reward functions (regex, test execution, length checks) instead of LM-based judges to avoid extra LM calls per attempt

BestOfN vs MultiChainComparison

Both BestOfN and dspy.MultiChainComparison aim to pick the best output from multiple candidates, but they work differently:

	BestOfN	MultiChainComparison
Selection method	Your reward function scores each candidate	An LM reads all candidates and picks the best
Metric required	Yes -- you must provide a reward_fn	No -- the LM decides what "best" means
Token cost	N calls to your module (+ reward fn)	Multiple chain calls + one comparison call
Best when	You have a clear, automatable scoring criterion	Quality is subjective or hard to score programmatically
Optimizable	The wrapped module can be optimized	The comparison module can be optimized

Use BestOfN when you can write a reward function. Use MultiChainComparison when you want the LM to judge quality using its own understanding.

Combining BestOfN with optimization

BestOfN works well as a complement to DSPy optimizers. Optimize your module first, then wrap the optimized version with BestOfN for an additional quality boost:

# Optimize the base module
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# Wrap the optimized module with BestOfN
best_qa = dspy.BestOfN(
    module=optimized_qa,
    N=3,
    reward_fn=my_reward,
    threshold=1.0,
)

This stacks two quality improvements: better prompts from the optimizer, and rejection sampling from BestOfN.

Gotchas

• Claude writes the reward function with (example, prediction, trace=None) signature. BestOfN reward functions take (args_dict, prediction), not the (example, prediction, trace) signature used by dspy.Evaluate metrics. The args dict contains only the inputs you passed to the call, not labeled examples with gold outputs.
• Claude sets N too high without considering cost. Each attempt is a full LM call at temperature=1.0. N=10 means 10x the token cost. Start with N=3 and increase only if your metric shows improvement — diminishing returns kick in quickly above N=5.
• Claude uses BestOfN when the reward function is as expensive as the module itself. If your reward function calls an LM (e.g., LM-as-judge), each BestOfN attempt costs 2x tokens (one for the module, one for the judge). For N=5, that is 10 LM calls total. Use programmatic reward functions (test execution, regex, length checks) whenever possible.
• Claude forgets to set threshold to enable early stopping. Without a meaningful threshold, BestOfN always runs all N attempts even when the first one is perfect. Set threshold to a value that represents "good enough" (e.g., 1.0 for binary pass/fail, 0.9 for graded metrics) to save tokens on easy inputs.
• Claude wraps an already-optimized module but does not evaluate the incremental gain. BestOfN on top of an optimized module costs N times more per call at inference time. Always measure the quality gain from BestOfN separately to confirm the extra cost is justified — if the optimized module already hits 95%+, BestOfN may not add enough to be worth it.
• Claude skips a baseline check before recommending BestOfN. Before adding BestOfN, run the base module on 20 inputs and count what fraction score >= threshold with your reward function. If >85% already pass, BestOfN adds little value at N times the cost. If <50% pass, BestOfN provides significant lift. Skip this check and you might add N=5 calls to a module that was already succeeding 90% of the time.

Additional resources

• dspy.BestOfN API docs
• reference.md — constructor parameters, forward() method, key behaviors
• examples.md — code generation with test-based selection, summarization with graded metric

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• MultiChainComparison for LM-based candidate selection -- see /dspy-multi-chain-comparison
• Refine for iterative improvement with LM feedback between attempts -- see /dspy-refine
• Evaluate for measuring quality with metrics and devsets -- see /dspy-evaluate
• Improving accuracy for the full optimization workflow -- see /ai-improving-accuracy
• Making consistent for reducing variance in AI output -- see /ai-making-consistent
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do