dag-quality

license	BSL-1.1
name	dag-quality
description	Validates agent outputs against schemas and quality criteria, scores confidence, detects hallucinations, monitors convergence, decides when to iterate, and synthesizes actionable feedback. Use when checking if a node's output is acceptable, scoring confidence, detecting fabricated content, deciding whether to re-execute, or generating improvement feedback. Activate on "validate output", "check quality", "confidence score", "hallucination check", "should we iterate", "improvement feedback". NOT for executing DAGs (use dag-runtime), planning DAGs (use dag-planner), or matching skills (use dag-skills-matcher).
allowed-tools	Read,Write,Edit,Grep,Glob
metadata	{"category":"DAG Framework","tags":["dag","quality","validate-output","check-quality","confidence-score"]}
category	Agent & Orchestration
tags	["dag","quality","validation","assessment","metrics"]

DAG Quality

Validates outputs, scores confidence, detects hallucinations, monitors convergence, decides on iteration, and synthesizes feedback. The quality gate between DAG nodes. Consolidates dag-output-validator, dag-confidence-scorer, dag-hallucination-detector, dag-convergence-monitor, dag-iteration-detector, and dag-feedback-synthesizer.

---

When to Use

✅ Use for:

• Validating node output against declared schema
• Scoring confidence on agent outputs (0-1)
• Detecting fabricated content, false citations, unverifiable claims
• Deciding whether to iterate (re-execute a node or loop)
• Generating structured improvement feedback for re-execution
• Monitoring quality trends across iterations

❌ NOT for:

• Executing nodes (use dag-runtime)
• Planning DAG structure (use dag-planner)
• Grading skills themselves (use skill-grader)

---

Quality Pipeline

flowchart TD
  O[Node output] --> SV[Schema validation]
  SV -->|Invalid| REJ[Reject + specific errors]
  SV -->|Valid| CV[Content validation]
  CV --> CS[Confidence scoring]
  CS --> HD[Hallucination detection]
  HD --> D{Quality above threshold?}
  D -->|Yes| ACC[Accept → pass to downstream]
  D -->|Below threshold, iteration < max| FB[Generate feedback]
  FB --> RE[Re-execute with feedback]
  D -->|Below threshold, iteration = max| ESC[Escalate to human]

Schema Validation

Structural check: does the output match the node's declared output contract?

• Required fields present
• Types correct (string, number, array, object)
• Constraints met (min/max length, ranges, enums)
• Nested structures valid

Content Validation

Semantic check: is the content reasonable?

• Non-empty meaningful content (not just filler)
• Length within expected range for the task
• Internal consistency (no contradictions)
• References exist (cited files, URLs, identifiers)

Confidence Scoring

Aggregate four evaluator signals (see skill-lifecycle.md for full architecture):

Evaluator	Weight	Signal
Self-evaluation	0.15	Agent's own assessment (sycophancy-biased)
Peer evaluation