deep-audit

name	deep-audit
description	Repository-wide consistency audit for skills, hooks, rules, and docs.

Deep Audit — Repository Infrastructure Audit

Run a comprehensive consistency audit across the entire repository, fix all issues found, and loop until clean.

When to Use

• After broad changes (new skills, rules, hooks, guide edits)
• Before releases or major commits
• When the user asks to "find inconsistencies", "audit", or "check everything"

Workflow

PHASE 1: Run 4 Audit Passes

Run these 4 audit passes. If the user explicitly requested delegated or parallel audit work, you may launch them as parallel audit subagents; otherwise execute the same passes locally:

Agent 1: Guide Content Accuracy

Focus: README.md, rules/workflow-overview.md, .codex/hooks.json, and the three active hook scripts

• All numeric claims match reality (skill count, agent count, rule count, repo hook command count)
• All file paths mentioned actually exist on disk
• All skill/agent/rule names match actual directory names
• No stale counts from previous versions

Agent 2: Hook Code Quality

Focus: hooks/*.py and hooks/*.sh

• No remaining /tmp/ usage for persistent state (prefer plugin-relative storage or ~/.codex/sessions/ only when session-scoped persistence is intentional)
• Hash length consistency ([:8] across all hooks)
• Proper error handling (fail-open pattern: top-level try/except with sys.exit(0))
• JSON input/output correctness (stdin for input, stdout/stderr for output)
• Exit code correctness (0 for non-blocking, non-zero only when intentionally blocking)
• from __future__ import annotations for Python 3.8+ compatibility
• Correct field names from hook input schema (source not type for SessionStart)
• Repo-local hook wiring in .codex/hooks.json resolves to the three plugin hook scripts under plugins/social-science-research/hooks/

Agent 3: Skills, Agents, and Rules Consistency

Focus: skills/*/SKILL.md, agents/*.md, and rules/*.md

• Valid YAML frontmatter in all files
• No stale legacy Claude-only frontmatter fields in Codex skills
• Rule paths: reference existing directories
• No contradictions between rules
• Documentation tables match the actual skill directories 1:1
• Skills that invoke subagents point to the correct files in agents/
• Template references use the Codex resolution rule (plugins/social-science-research/templates/..., templates/..., then embedded fallback)

Agent 4: Cross-Document Consistency

Focus: README.md, rules/workflow-overview.md, skills/research-setup/SKILL.md

• All feature counts agree across the user-facing docs
• All links point to valid targets
• Directory tree matches actual structure
• No stale counts from previous versions

PHASE 2: Triage Findings

Categorize each finding:

• Genuine bug: Fix immediately
• False alarm: Discard (document WHY it's false for future rounds)

Common false alarms to watch for:

• Quarto callout ## Title inside ::: divs — this is standard syntax, NOT a heading bug
• Prompt blocks and Markdown tables inside skills — not a syntax bug if the YAML frontmatter parses cleanly
• Counts in old session logs — these are historical records, not user-facing docs

PHASE 3: Fix All Issues

Apply fixes in parallel where possible. For each fix:

1. Read the file first (required by Edit tool)
2. Apply the fix
3. Verify the fix (grep for stale values, check syntax)

PHASE 4: Documentation Check

This plugin does not maintain a Quarto source file. If README.md, rules/workflow-overview.md, or the project metadata generation logic were modified, verify they are still consistent — no render step is needed.

PHASE 5: Loop or Declare Clean

After fixing, rerun the 4 audit passes to verify.

• If new issues found → fix and loop again
• If zero genuine issues → declare clean and report summary

Max loops: 5 (to prevent infinite cycling)

Key Lessons from Past Audits

These are real bugs found across 7 rounds — check for these specifically:

Bug Pattern	Where to Check	What Went Wrong
Stale counts ("19 skills" → "21")	Guide, README, landing page	Added skills but didn't update all mentions
Hook exit codes	All Python hooks	Exit 2 in PreCompact silently discards stdout
Hook field names	post-compact-restore.py	SessionStart uses source, not type
State in /tmp/	All Python hooks	Should use plugin-local storage or ~/.codex/sessions/<hash>/ only when session persistence is intentional
Hash length mismatch	All Python hooks	Some used [:12], others [:8]
Missing fail-open	Python hooks __main__	Unhandled exception → exit 1 → confusing behavior
Python 3.10+ syntax	Type hints like `dict	None`
Missing directories	quality_reports/specs/	Referenced in rules but never created
Always-on rule listing	Guide + README	meta-governance omitted from listings
macOS-only commands	Skills, rules	open without xdg-open fallback
Hook discovery path	hook wiring	Codex reads <repo>/.codex/hooks.json, not plugin-bundled hooks.json

Output Format

After each round, report:

## Round N Audit Results

### Issues Found: X genuine, Y false alarms

| # | Severity | File | Issue | Status |
|---|----------|------|-------|--------|
| 1 | Critical | file.py:42 | Description | Fixed |
| 2 | Medium | file.qmd:100 | Description | Fixed |

### Verification
- [ ] No stale counts (grep confirms)
- [ ] All hooks have fail-open + future annotations
- [ ] All Codex skill frontmatter parses cleanly
- [ ] README.md, workflow docs, and project metadata template agree

### Result: [CLEAN | N issues remaining]