harness-engineering

name	harness-engineering
description	Design, audit, and improve the agent harness — the environment, constraints, context management, evaluation gates, and feedback loops that surround and guide AI agents. Inspired by OpenAI and Anthropic harness engineering best practices. Triggers: "harness", "harness audit", "improve harness", "agent environment", "context management", "evaluation gates", "feedback loop", "harness engineering".
allowed-tools	["Bash","Read","Write","Edit","Grep","Glob"]
when_to_use	Use to audit or improve the agent harness: context efficiency, tool space, evaluation gates, feedback loops. Triggers on 3-strike agent failure signal or explicit harness audit request.
produces	docs/superomni/harness-audits/harness-audit-[branch]-[session]-[date].md
consumes	[]

Preamble (Core)

Status protocol — end every session with one of: DONE (evidence provided) · DONE_WITH_CONCERNS (list each) · BLOCKED (state what blocks you) · NEEDS_CONTEXT (state what you need).

Auto-advance — pipeline: THINK → PLAN → REVIEW → BUILD → VERIFY → RELEASE. Only human gate is spec approval at THINK. On DONE at other stages, print [STAGE] DONE -> advancing to [NEXT-STAGE] and invoke the next skill. On any non-DONE status at any stage, STOP.

Output directory — all artifacts go in docs/superomni/<kind>/<kind>-[branch]-[session]-[date].md. See CLAUDE.md for the full directory map.

TACIT-DENSE — before high-tacit decisions, classify D1 (domain expertise) · D2 (user-facing UX) · D3 (team culture) · D4 (novel pattern). On hit, output TACIT-DENSE [D#]: [question] — My default: [recommendation]. See reference for actions.

Anti-sycophancy — take a position on every significant question. Name flaws directly. No filler ("that's interesting", "you might consider", "that could work").

Telemetry (local only) — at session end, log bin/analytics-log. Nothing leaves the machine.

See preamble-ref.md for detailed protocols.

Harness Engineering

Goal: Design and maintain the agent harness — the scaffolding of environment, context, tools, constraints, evaluation gates, and feedback loops that determine how well agents perform.

"Engineers design the system. Agents execute." — OpenAI Harness Engineering

Iron Law

THE HARNESS IS THE PRODUCT. CODE IS ITS OUTPUT.

A well-designed harness produces reliable, high-quality agent output without requiring manual intervention on every task. When agents fail repeatedly, the correct response is to improve the harness — not to keep retrying the same prompt.

Core Principles (From OpenAI + Anthropic)

Principle	What it means in superomni
Context is everything	Agents can only work with what they can see — keep docs, specs, and constraints in-repo and up-to-date
Fewer, more expressive tools	Prefer composable skills over sprawling tool menus
Evaluate relentlessly	Judgment gates must exist at every major transition point
Signal-driven iteration	Agent failures are design signals — update the harness, not just the prompt
Boring > clever	Prefer simple, composable patterns over novel abstractions
Garbage collection	Periodically audit for drift, stale docs, and architectural decay

---

Phase 1: Harness Inventory

Take stock of the current harness state:

# Skill count + structure
ls skills/ | wc -l
ls skills/

# Agent count
ls agents/

# Command count
ls commands/

# Preamble size (context overhead)
wc -l lib/preamble.md

# Skill template sizes (larger = more context pressure)
wc -l skills/*/SKILL.md.tmpl | sort -n | tail -10

# Validation status
npm test 2>/dev/null || bash lib/validate-skills.sh 2>/dev/null

# Recent harness changes
git log --oneline -10 -- lib/ skills/ agents/ commands/

# Any stale/out-of-date docs
find docs/ -name "*.md" -older /tmp 2>/dev/null | head -10

Document findings:

• Total skills: ___
• Total agents: ___
• Total commands: ___
• Preamble size (lines): ___
• Validation status: PASS / FAIL
• Largest skill (lines): ___

Phase 2: Context Window Audit

Context window pressure is one of the most common causes of agent degradation. Audit the harness context load:

2a. Preamble Efficiency Analysis

Review lib/preamble.md:

• Every section earns its inclusion — remove dead protocols
• No repetition with individual skill instructions
• Status protocol is concise and actionable
• Performance checkpoint is lightweight (3 questions max)
• Telemetry block is at the end (lowest priority)

Target preamble size: < 150 lines. Flag if > 200 lines.

2b. Skill Bloat Detection

For each skill > 200 lines, ask:

• Are all phases strictly necessary?
• Are examples too verbose? (truncate to headers only)
• Can any phase be replaced with a reference to another skill?
• Is this skill actually used? (check telemetry)

2c. Progressive Disclosure Check

Does the framework expose only necessary context at each stage?

Stage	Context needed	Currently loaded
Planning	spec, constraints
Implementation	plan, code context
Review	diff, standards
Debug	error, minimal repro

Good harnesses load context on demand, not all at once.

Phase 3: Tool Action Space Audit

Per Anthropic's principle: fewer, more expressive tools outperform large menus of narrow ones.

Review the agent's tool access:

# Check allowed-tools across all skills
grep "allowed-tools" skills/*/SKILL.md.tmpl

For each skill, evaluate:

• Are the allowed tools the minimum needed for that skill?
• Are any rarely-used tools creating confusion or unnecessary options?
• Do composable skill combinations cover the same ground as single complex tools?

Recommended tool sets by role:

Role	Minimal tool set
Planning / Brainstorming	Read, Write, Glob
Implementation	Bash, Read, Write, Edit, Grep, Glob
Review / Audit	Read, Grep, Glob
Debug	Bash, Read, Grep, Glob

Flag any skill whose tool set exceeds its role's minimum.

Phase 4: Evaluation Gate Audit

"Evaluation is the load-bearing part of agent harness design." — OpenAI/Anthropic harness engineering principles

Map every major workflow transition and verify an evaluation gate exists:

Transition	Evaluation gate	Present?
Spec → Plan	plan-review skill or planner-reviewer agent (planning mode)
Plan → Execution	dependency analysis wave plan
Execution Wave → Next Wave	wave verification step
Implementation → Review	code-review skill or planner-reviewer agent
Review → Ship	production-readiness skill
Ship → Done	verification skill
Sprint → Next Sprint	self-improvement skill

Any gap = harness deficiency. Add missing gates.

Phase 5: Feedback Loop Audit

A healthy harness converts agent failures into harness improvements:

Agent fails → Signal captured → Harness updated → Agent retries → Improvement
     ↑                                                                   |
     └───────────────────────────────────────────────────────────────────┘

Check the current feedback paths:

5a. Error → Harness Signal

When an agent fails a task repeatedly (3+ attempts), is there a defined process to:

• Record the failure pattern (systematic-debugging skill)?
• Identify the harness gap causing the failure?
• Update the relevant skill, doc, or constraint?

5b. Performance → Improvement

Does the self-improvement skill output get consumed?

ls docs/superomni/improvements/ 2>/dev/null | head -5

• Improvement reports exist?
• Action items from last report applied to current sprint?
• workflow skill reads improvement reports at sprint start?

5c. Documentation Garbage Collection

Is there a regular cadence for cleaning up:

• Stale skill instructions that no longer match agent behavior?
• Outdated docs that contradict current implementation?
• Dead commands that are registered but never invoked?
• Agent definitions whose Iron Laws conflict with updated skills?

Recommended: Schedule a harness GC pass after every 5 sprints.

Phase 6: Harness Health Score

Score the harness on each dimension (1-5):

Dimension	Score	Key Finding
Context efficiency	/5
Tool space minimalism	/5
Evaluation gate coverage	/5
Feedback loop completeness	/5
Documentation freshness	/5

Total: __ / 25

Scoring guide:

• 21-25 — World-class harness. Focus on new capabilities.
• 16-20 — Good harness. Address identified gaps.
• 11-15 — Fair harness. Significant drift or missing gates.
• < 11 — Harness needs major refactor before next sprint.

Phase 7: Harness Improvement Plan

For each finding from Phases 2-5 with a score < 4:

HARNESS IMPROVEMENT [N]: [TITLE]
Dimension:   [context | tools | evaluation | feedback | docs]
Finding:     [specific issue identified]
Impact:      [how this degrades agent performance]
Fix:         [concrete change to harness — specific file, section, or process]
Priority:    [P0 — blocks agent / P1 — degrades quality / P2 — nice to have]

Generate a prioritized backlog. P0 items must be fixed before the next sprint.

Phase 8: Save Harness Audit Report

HARNESS_DIR="docs/superomni/harness-audits"
mkdir -p "$HARNESS_DIR"
BRANCH=$(git branch --show-current 2>/dev/null | tr '/' '-' || echo "main")
TIMESTAMP=$(date +%Y-%m-%d-%H%M%S)
REPORT_FILE="$HARNESS_DIR/harness-audit-${BRANCH}-${TIMESTAMP}.md"
echo "Saving harness audit to $REPORT_FILE"

Save the full audit report including all scores, findings, and improvement backlog.

Report

HARNESS AUDIT REPORT
════════════════════════════════════════
Branch:             [branch]
Date:               [date]
Skills / Agents:    [N] skills, [N] agents, [N] commands
Preamble size:      [N] lines ([OK / BLOATED])
Validation:         [PASS / FAIL]
Health score:       [N]/25 ([rating])
Top finding:        [single most important issue]
P0 improvements:    [N]
P1 improvements:    [N]
P2 improvements:    [N]
Report saved:       [docs/superomni/harness-audits/...]
Status: DONE | DONE_WITH_CONCERNS | BLOCKED
════════════════════════════════════════