ai-harness-audit

name	ai-harness-audit
description	Use when reviewing, tightening, or repairing the repository AI harness in VS Code. Trigger on requests to audit instructions, prompts, agents, hooks, skills, or memory for drift, overlap, missing layers, wrong ownership, or instruction bloat.
argument-hint	[scope or issue]
user-invocable	true
disable-model-invocation	false

AI Harness Audit

Audit the full repository AI harness and improve it with the smallest effective fixes.

Use the same layer model as ai-harness-bootstrap, but focus on diagnosis, ownership, and targeted repair.

Support three audit-oriented maintenance operations:

• ingest: inspect repo state and fold important changes into harness or memory
• query: answer using the maintained harness and memory structure
• lint: detect drift, contradictions, stale context, orphaned files, excessive top-layer bloat, and misassigned responsibilities
• verify governance mode clarity, verifier-first fit, and machine-readable audit provenance
• verify runtime harness role clarity, runtime handoff fit, and SEA-vs-runtime control-plane boundaries

Use just harness-audit-report "summary" --reflection "..." [--note "..."] [--pattern "..."] to record meaningful audit outcomes.

If the repo contains plugin or external agent packs, audit whether their useful primitives have been absorbed into the harness cleanly or are lingering as a parallel customization system.

For imported skill packs, prefer auditing for architectural absorption rather than wholesale duplication.

ACE Application

Audit whether the harness is behaving like an evolving playbook:

• are durable harness lessons being curated into memory/playbook.md
• is execution feedback being captured in memory/reflections.md
• are updates incremental, or are contexts collapsing into brief rewrites
• is the harness learning from natural execution feedback, or only restating static rules

AutoHarness Audit Lens

Audit whether the harness has adopted the useful AutoHarness ideas coherently:

• is there a small explicit governance profile instead of hidden mode decisions
• is the default governance verifier-first rather than over-blocking
• are stronger controls reserved for risky, multi-agent, or deployment-sensitive work
• is there a machine-readable audit trail alongside human-readable maintenance notes
• does harness maintenance use a critic-refiner loop instead of broad rewrites after every failure

Environment-Tuned Audit Lens

Apply the useful parts of environment tuning during audit:

• check whether harness maintenance is staged or whether every update is treated as an all-at-once rewrite
• check whether failure feedback is actionable:
  • does it identify root cause
  • does it reveal a missing prerequisite or violated rule
  • does it preserve exploration space instead of dictating the whole fix
• check whether the harness exposes fine-grained progress signals instead of only a binary pass or fail outcome
• prefer tuning the surrounding environment of the agent:
  • prompts
  • checks
  • examples
  • reports
  • hooks before adding more always-on policy

Implicit Context Update Lens

Apply the useful parts of in-context learning theory during audit:

• check whether new context is being added as narrow deltas or as monolithic rewrites
• check whether prompts, examples, and playbook entries are high-signal enough to act like useful behavioral updates
• trim redundant or low-value examples that add noise without changing behavior
• prefer small scoped additions over broad edits to .github/copilot-instructions.md or AGENTS.md

Audit Workflow

1. Run just harness-snapshot.
2. Inventory the active harness layers:
   • .github/copilot-instructions.md
   • AGENTS.md
   • .github/instructions/
   • .github/prompts/
   • .github/agents/
   • .github/hooks/
   • .github/harness/
   • .github/skills/ and .sea/skills/
   • memory/
3. Map each file to one primary responsibility.
4. Flag overlap, drift, missing layers, stale files, and instruction bloat.
5. Recommend or apply the smallest change that restores clean ownership.
6. Run just harness-check after edits.
7. When evaluating scaffold quality, compare the live base files against the bootstrap templates in ../ai-harness-bootstrap/assets/templates/.
8. Explicitly check whether the harness is missing any easy-to-miss operational details.

What To Check

Ownership drift

• top-level instructions carrying volatile task state
• prompts carrying stable repo policy
• agents carrying giant workflow manifests
• skills duplicating top-level instructions
• memory files storing durable docs that belong elsewhere, or failing to use the right Markdown, YAML, and JSON split

Missing layers

• no thin always-on operating contract
• no router or priority file
• missing scoped rules for materially different code areas
• no explicit harness prompts or harness agent
• no safe hook definitions
• no volatile memory subsystem
• no explicit governance profile despite meaningful mode or risk differences
• no machine-readable audit trail where provenance would materially help
• no independent coherence evaluator or reflector when coherence measurement is claimed
• no route for architecture synthesis despite an architecture-synthesis agent existing
• no guarded conversation-to-spec path for domain-expert requirement work
• no domain-expert communication overlay despite repeated non-technical requirement capture
• no case-orchestration or cognitive-artifact entrypoints when those work modes are repeatedly needed
• no generator-first route despite spec-first and generated-zone constraints
• no generator-builder, generator-validate, or last-mile entrypoints despite generator-centric workflows
• no SEA-DSL authoring route for .sea work
• no semantic-anchoring route where ontology-linked artifacts or semanticRefs are used
• no deploy or release route despite environment-sensitive operational work
• no runtime harness layer despite multi-runtime integration requirements
• no explicit runtime-role or handoff profiles despite differentiated runtime responsibilities

Missing operational details

• no explicit authority hierarchy
• no inspect-before-ask rule
• no edit discipline
• no validation tiers
• no definition of done
• no memory update protocol
• no staleness policy
• no secrets or environment safety rule
• no task-routing hints by task type
• no promotion rules for repeated patterns
• no staged rollout strategy for harness upgrades
• no actionable-but-non-leaky corrective feedback model
• no fine-grained progress checkpoints for harness maintenance

Redundancy

• repeated rules across .github/copilot-instructions.md, AGENTS.md, and broad .instructions.md files
• prompt or skill pairs that duplicate the same workflow with no division of labor
• multiple skill roots not resolving to the canonical source
• too many examples, hints, or memory notes that do not change agent behavior
• imported plugin state duplicated beside existing harness memory surfaces

Staleness

• obsolete .chatmode.md or old workflow files
• prompt or agent references to moved paths or tools
• memory entries that no longer match the codebase
• hook configs that no longer point to valid scripts
• missing or stale entries in memory/index.md, memory/index.json, or memory/log.md
• missing or stale curation in memory/playbook.md or memory/reflections.md
• stale coherence state or open-break artifacts
• plugin-local docs still treated as authoritative after a harness-native assimilation exists

Fix Strategy

Prefer these fixes, in order:

1. trim duplication
2. move content to the correct layer
3. split oversized files by responsibility
4. improve the environment around the agent with better diagnostics, examples, or checks
5. add missing minimal files only when a real gap remains
6. preserve working patterns rather than normalizing everything for its own sake

If the repo already has good content for a missing detail in another file, move or reference it instead of rewriting it.

Append a short reviewable entry to memory/log.md after meaningful audit runs. Capture raw lessons in memory/reflections.md, and only add --pattern when the lesson is durable enough for memory/playbook.md.

If the harness is missing the generation-reflection-curation loop, add the smallest possible implementation of it. If the harness feedback is vague, replace it with more actionable but still non-leaky guidance. If an imported coherence pack created a second state system, collapse it into the harness with explicit mapping and minimal new files. If imported skills remain disconnected from prompts, agents, or scoped rules, connect them through the smallest fitting harness layer. If imported generator or semantic skills remain un-routed, prefer prompt entrypoints plus a narrow scoped instruction before adding more agents.

Output Contract

Always end with:

## Good
- [...]

## Weak
- [...]

## Applied
- [...]

## Change Next
- [...]

References

Load references/audit-checklist.md for:

• a compact ownership matrix
• common repair patterns
• validation checks