| license | BSL-1.1 |
| name | dag-quality |
| description | Validates agent outputs against schemas and quality criteria, scores confidence, detects hallucinations, monitors convergence, decides when to iterate, and synthesizes actionable feedback. Use when checking if a node's output is acceptable, scoring confidence, detecting fabricated content, deciding whether to re-execute, or generating improvement feedback. Activate on "validate output", "check quality", "confidence score", "hallucination check", "should we iterate", "improvement feedback". NOT for executing DAGs (use dag-runtime), planning DAGs (use dag-planner), or matching skills (use dag-skills-matcher). |
| allowed-tools | Read,Write,Edit,Grep,Glob |
| metadata | {"category":"DAG Framework","tags":["dag","quality","validate-output","check-quality","confidence-score"]} |
| category | Agent & Orchestration |
| tags | ["dag","quality","validation","assessment","metrics"] |
DAG Quality
Validates outputs, scores confidence, detects hallucinations, monitors convergence, decides on iteration, and synthesizes feedback. The quality gate between DAG nodes. Consolidates dag-output-validator, dag-confidence-scorer, dag-hallucination-detector, dag-convergence-monitor, dag-iteration-detector, and dag-feedback-synthesizer.
When to Use
✅ Use for:
- Validating node output against declared schema
- Scoring confidence on agent outputs (0-1)
- Detecting fabricated content, false citations, unverifiable claims
- Deciding whether to iterate (re-execute a node or loop)
- Generating structured improvement feedback for re-execution
- Monitoring quality trends across iterations
❌ NOT for:
- Executing nodes (use
dag-runtime)
- Planning DAG structure (use
dag-planner)
- Grading skills themselves (use
skill-grader)
Quality Pipeline
flowchart TD
O[Node output] --> SV[Schema validation]
SV -->|Invalid| REJ[Reject + specific errors]
SV -->|Valid| CV[Content validation]
CV --> CS[Confidence scoring]
CS --> HD[Hallucination detection]
HD --> D{Quality above threshold?}
D -->|Yes| ACC[Accept → pass to downstream]
D -->|Below threshold, iteration < max| FB[Generate feedback]
FB --> RE[Re-execute with feedback]
D -->|Below threshold, iteration = max| ESC[Escalate to human]
Schema Validation
Structural check: does the output match the node's declared output contract?
- Required fields present
- Types correct (string, number, array, object)
- Constraints met (min/max length, ranges, enums)
- Nested structures valid
Content Validation
Semantic check: is the content reasonable?
- Non-empty meaningful content (not just filler)
- Length within expected range for the task
- Internal consistency (no contradictions)
- References exist (cited files, URLs, identifiers)
Confidence Scoring
Aggregate four evaluator signals (see skill-lifecycle.md for full architecture):
| Evaluator | Weight | Signal |
|---|
| Self-evaluation | 0.15 | Agent's own assessment (sycophancy-biased) |
| Peer evaluation |
