Ralph Loop — Iterative Quality Improvement
Generate → Evaluate → Feedback → Re-generate until quality threshold met.
Pattern Origin
Inspired by the Sensei iterative quality improvement patterns from GitHub Copilot for Azure:
"LLMs improve dramatically when given structured feedback about what went wrong and specific guidance on how to fix it."
— Shayne Boyer (@spboyer), Microsoft
Core Loop
┌─────────────────────────────────────────────────────────────┐
│ RALPH LOOP │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Generate │────▶│ Evaluate │────▶│ Analyze │ │
│ └─────────┘ └──────────┘ └──────────┘ │
│ ▲ │ │
│ │ ┌──────────────────────┘ │
│ │ ▼ │
│ │ ┌──────────┐ ┌───────────┐ │
│ └────│ Feedback │◀────│ Threshold │ │
│ └──────────┘ │ Met? │ │
│ └───────────┘ │
│ │ Yes │
│ ▼ │
│ ┌────────┐ │
│ │ Done │ │
│ └────────┘ │
└─────────────────────────────────────────────────────────────┘
Configuration
| Parameter | Default | Description |
|---|
maxIterations | 5 | Maximum iterations before stopping |
qualityThreshold | 80 | Score (0-100) to consider quality met |
improvementThreshold | 5 | Minimum improvement required per iteration |
earlyStopOnPerfect | true | Stop immediately when score reaches 100 |
includeFeedback | true | Include feedback in re-generation prompts |
Stop Conditions
- Perfect score — Score reaches 100
- Quality threshold met — Score ≥
qualityThreshold
- Improvement plateau — Improvement <
improvementThreshold for 2+ iterations
- Max iterations — Reached
maxIterations
Acceptance Criteria
Criteria define what "good" looks like:
{
"skillName": "cosmos-db-patterns",
"language": "typescript",
"correctPatterns": [
{
"code": "container.items.query()",
"description": "Use container.items.query() for parameterized queries",
"section": "query-patterns"
}
],
"incorrectPatterns": [
{
"code": "SELECT * FROM c",
"description": "Avoid SELECT * — specify required fields",
"section": "query-patterns"
}
],
"rules": [
{
"name": "error-handling",
"requiredPatterns":
Feedback Builder
Transforms evaluation findings into LLM-actionable feedback:
Input (Evaluation Result)
{
"score": 65,
"passed": false,
"findings": [
{
"severity": "error",
"rule": "pattern:query-patterns",
"message": "Incorrect pattern found: Avoid SELECT * — specify required fields"
}
],
"matchedIncorrect": ["query-patterns"]
}
Output (Feedback)
## Issues Found in Generated Content
### CRITICAL ERRORS (Must Fix)
- **pattern:query-patterns**: Incorrect pattern found: Avoid SELECT * — specify required fields
- Suggestion: Review acceptance criteria for correct usage
### INCORRECT PATTERNS DETECTED
Found incorrect patterns in sections: **query-patterns**
Review the acceptance criteria for these sections and use correct patterns instead.
### SUGGESTED CORRECTIONS
Based on the acceptance criteria, consider:
- Use: Use container.items.query() for parameterized queries
CLI Usage
node ralph-loop.cjs --evaluate --content "SELECT * FROM c" --criteria criteria.json
node ralph-loop.cjs --evaluate --content-file output.ts --criteria criteria.json
node ralph-loop.cjs --extract-criteria --skill .github/skills/cosmos-db/SKILL.md
node ralph-loop.cjs --feedback --findings '[{"severity":"error","rule":"syntax","message":"Missing semicolon"}]'
echo '{"content":"...","criteria":{...}}' | node ralph-loop.cjs --stdin
Integration Patterns
VS Code Extension
import { ContentEvaluator, FeedbackBuilder, RalphLoopController } from './ralph-loop';
const criteria = loadSkillCriteria('cosmos-db-patterns');
const controller = new RalphLoopController(criteria, {
maxIterations: 3,
qualityThreshold: 85,
});
const result = await controller.run(
async (prompt) => await copilot.generate(prompt),
"Write a Cosmos DB query for user lookup",
"user-query-scenario"
);
console.log(`Final score: ${result.finalScore}`);
console.log(`Converged: ${result.converged}`);
console.log(`Iterations: ${result.iterations.length}`);
Standalone Evaluation
const evaluator = new ContentEvaluator(criteria);
const feedbackBuilder = new FeedbackBuilder();
const result = evaluator.evaluate(generatedCode, 'my-scenario');
const feedback = feedbackBuilder.buildFeedback(result, criteria);
if (!result.passed) {
console.log(feedback);
}
Quality Metrics
Track these across iterations:
| Metric | Description |
|---|
score | Overall quality score (0-100) |
errorCount | Number of critical errors |
warningCount | Number of warnings |
matchedCorrect | Correct patterns found |
matchedIncorrect | Incorrect patterns found |
improvement | Score change from first to last iteration |
converged | Whether quality threshold was met |
When to Use
- Code generation — Validate SDK usage, patterns, security
- Documentation — Check structure, completeness, accuracy
- Configuration — Validate schema compliance, best practices
- Any LLM output — Iteratively improve until quality threshold met
Anti-Patterns
| Don't | Do |
|---|
Set maxIterations too high (>5) | Usually 3-5 iterations sufficient |
Set qualityThreshold at 100 | 80-90 is realistic for most use cases |
| Include vague criteria | Be specific with patterns and rules |
| Skip feedback on failure | Always provide structured feedback |
Related
- testing-strategies — Test design and coverage
- code-review — Manual review patterns
- debugging-patterns — Root cause analysis
- skill-creator — Creating acceptance criteria for skills
