audit-documentation

name	audit-documentation
description	Audit Fabrik docs against recently shipped features — files gap issues for undocumented behavior, closes issues whose gaps are now covered
user_invocable	true

Audit Documentation

Compare recently shipped features against USER_GUIDE.md, README.md, and docs/index.md. File GitHub issues for documentation gaps and close existing documentation issues whose gaps are now covered.

Usage

/audit-documentation                  # Scan issues closed in the last 30 days
/audit-documentation --since v0.0.20  # Scan issues referenced in commits since a tag

Known limitation: Rate limits may apply for repos with hundreds of closed issues. The tag-based mode may miss issues if their PRs used Fixes or bare URLs instead of Closes #NNN.

Steps

Step 1 — Parse arguments and pre-flight

Detect whether --since <tag> was provided:

# Tag-based mode
TAG="v0.0.20"  # extracted from --since argument

# Validate tag exists locally
git rev-parse --verify "refs/tags/$TAG" 2>/dev/null
if [ $? -ne 0 ]; then
  # Try fetching tags and retry
  git fetch --tags origin
  git rev-parse --verify "refs/tags/$TAG" 2>/dev/null || {
    echo "Error: tag '$TAG' not found locally or on origin. Aborting."
    exit 1
  }
fi

For the default 30-day mode, compute the cutoff date using Python (portable across macOS and Linux):

CUTOFF_DATE=$(python3 -c "from datetime import datetime,timedelta; print((datetime.now()-timedelta(days=30)).strftime('%Y-%m-%d'))")
echo "Scanning issues closed since: $CUTOFF_DATE"

Step 2 — Discover source issues

Tag-based mode (two-pass):

Pass 1 — extract issue numbers from commit messages since the tag:

git log "$TAG..HEAD" --format="%s%n%b" | grep -oE '#[0-9]+' | grep -oE '[0-9]+' | sort -u

Pass 2 — get the tag's commit date, then find merged PRs since that date and extract Closes #NNN references:

TAG_DATE=$(git log -1 --format="%ci" "refs/tags/$TAG" | cut -d' ' -f1)
gh pr list -R handarbeit/fabrik --state merged \
  --search "merged:>=$TAG_DATE" \
  --json number,body \
  --limit 500 \
  --jq '.[] | .body' \
  | grep -oiE '(closes|fixes|resolves) #[0-9]+' \
  | grep -oE '[0-9]+' \
  | sort -u

Combine and deduplicate the issue numbers from both passes into a list (e.g. 200 201 205).

Then fetch their full details for use in subsequent steps:

for NUM in $ISSUE_NUMBERS; do
  gh issue view "$NUM" -R handarbeit/fabrik --json number,title,labels,body
done

Date-based mode (default 30 days):

Fetch closed issues with their full bodies in one call to avoid rate-limit issues from per-issue view calls:

gh issue list -R handarbeit/fabrik \
  --state closed \
  --search "closed:>$CUTOFF_DATE" \
  --json number,title,labels,body \
  --limit 200

Step 3 — Filter issues

From the discovered set, exclude:

1. Issues labeled documentation — avoid recursive analysis of documentation issues as source material
2. Issues that are infrastructure/tooling-only — issues with no user-facing feature content (e.g., CI fixes, dependency bumps, internal refactors with no observable behavior change)

For each excluded issue, note: issue number, title, and reason for exclusion.

Log the exclusions in your working notes so you can include them in the final summary.

Step 4 — Read documentation

Read the three documentation targets in full:

cat USER_GUIDE.md
cat README.md
cat docs/index.md

As you read, note which sections are most likely to be affected by recently shipped features — these are the sections to scrutinize during gap analysis.

Step 5 — Check existing open documentation issues

Fetch all currently open issues labeled documentation:

gh issue list -R handarbeit/fabrik \
  --state open \
  --label documentation \
  --json number,title,body

Keep this list for two purposes:

1. Deduplication — don't file a new gap issue if an equivalent one already exists
2. Deck clearing — close issues whose gaps are now covered by current docs

Step 6 — Analyze gaps

For each filtered source issue (or group of related issues), compare the feature/behavior described against the three documentation files.

Rules for gap analysis:

• A feature is adequately documented if a user reading the docs would know the feature exists, how to use it, and what behavior to expect
• A feature is inadequately documented if it's missing entirely, mentioned without explanation, or only documented internally (e.g., in ADRs, not in USER_GUIDE)
• Err on the side of filing — if uncertain whether a feature is adequately covered, file a gap issue. False positives are cheap (the Specify stage will refine them); false negatives cause documentation drift
• Group related issues that describe the same undocumented feature into a single gap entry — don't file one issue per source issue if they all relate to the same gap

For each gap, record:

• Gap title (what's undocumented)
• Gap description (what the docs are missing)
• Source issue numbers
• Whether a matching open documentation issue already exists (from Step 5)

Step 7 — Clear the deck

For each existing open documentation issue from Step 5 where the current docs clearly and completely address the described gap:

gh issue close -R handarbeit/fabrik <number> \
  --comment "Closed by /audit-documentation: gap is now covered in current documentation."

Conservatism rule: Only close if the documentation section clearly describes this specific behavior — if in doubt, leave open. Partial coverage does not justify closure.

Step 8 — File new gap issues

For each gap from Step 6 that does NOT already have a matching open documentation issue:

# 1. Create the issue, capture the URL
ISSUE_URL=$(gh issue create -R handarbeit/fabrik \
  --title "<gap title>" \
  --label "documentation" \
  --label "fabrik:yolo" \
  --body "<issue body — see format below>" \
  --json url --jq '.url')

# 2. Add to the Fabrik PM project board (org project #1), capture item ID
ITEM_ID=$(gh project item-add 1 --owner handarbeit \
  --url "$ISSUE_URL" \
  --format json --jq '.id')

# 3. Set status to Specify
gh project item-edit \
  --id "$ITEM_ID" \
  --field-id PVTSSF_lADOA8vIBc4BTew-zhAtcsQ \
  --project-id PVT_kwDOA8vIBc4BTew- \
  --single-select-option-id 29d94fed

Issue body format:

## Documentation Gap

<Clear description of what is missing or inadequately documented in USER_GUIDE.md, README.md, or docs/index.md>

## Problem

<Excerpt of the relevant Problem/Summary from the source issue(s) — keep concise, 2-5 sentences>

## Scope

- USER_GUIDE.md — <which sections need updating>
- README.md — <if relevant>
- docs/index.md — <if relevant>

## Source issues

<list of source issue numbers as #N references>

Keep the body concise — the Specify stage will elaborate as needed.

Step 9 — Report summary

Print a structured summary of everything the audit did:

## Audit Documentation — Summary

**Period**: <date range or tag range scanned>

**Issues scanned**: <N>
**Issues excluded**: <N>
  - <issue#>: <title> — <reason>
  - ...

**Documentation gaps found**: <N>

**New gap issues filed** (<N>):
  - #<number>: <title>
  - ...

**Existing documentation issues closed** (<N>):
  - #<number>: <title>

**Existing documentation issues left open** (<N>):
  - #<number>: <title> — <reason left open>

If no gaps were found, say so explicitly: "No documentation gaps found — docs appear current."