validate-artifacts

name	validate-artifacts
description	Comprehensive artifact maintenance — validates log naming, decision format, plan syntax compliance, staleness detection, and ensures the artifact corpus is parseable.

Validate Artifacts

Mental Model

Artifacts are the project's institutional memory. Logs capture what happened, decisions capture why, plans capture what's next. They must be well-formed so future agents can parse and learn from them. Malformed artifacts are technical debt — they look like documentation but can't be consumed by tools.

Artifacts live alongside code. When code changes and artifacts don't, staleness accumulates. A stale artifact is worse than no artifact — it actively misleads.

Coverage

Documented: Log naming conventions, decision structure, plan syntax validation, staleness detection, auto-fix capabilities.

Last updated: 2026-06-24

---

What This Skill Checks

1. Logs (artifacts/logs/)

Naming: YYYY-MM-DD_slug.md

• Must match pattern: \d{4}-\d{2}-\d{2}_[a-z0-9_-]+\.md
• Slug should be lowercase, hyphen-separated

Content structure (per TEMPLATE.md):

• Must have H1 title (e.g., # Experiment Log: ...)
• Must have **Date:** YYYY-MM-DD field
• Must have **Category:** field (valid values: run, discovery, dead-end, uncertainty)
• Should have **Notebook:** field referencing the source notebook
• Configuration section with parameter table (optional but expected for run category)
• Results section with metrics table (optional but expected for run category)

Integrity:

• Append-only (no deletion of past entries)
• No empty files
• No binary content

Ignore: .log.jsonl files are agent activity logs, not experiment logs — skip them during validation.

2. Decisions (artifacts/decisions/)

Naming: DEC-NNN_slug.md

• Must match pattern: DEC-\d+_[a-z0-9_-]+\.md
• Number should be sequential (gaps allowed, but flag them)

Content structure (per TEMPLATE.md and actual decisions):

• # DEC-NNN: <Title> (H1 with DEC number)
• **ID:** DEC-NNN
• **Date:** YYYY-MM-DD
• **Status:** — valid values (lowercase): accepted, rejected, superseded, proposed
  • Note: superseded decisions should reference their replacement
  • Note: Non-standard statuses (e.g., open — architectural fork, not yet decided) should be flagged as warnings, not errors
• ## Context — what situation prompted this decision
• ## Decision — what was decided
• ## Rationale — why this option over alternatives
• ## Alternatives Considered — table of alternatives with rejection reasons
• ## Consequences — what this decision enables and constrains
• ## References — links to paper sections, log entries, related decisions

Integrity:

• No duplicate DEC numbers
• Status should be valid enum value
• Superseded decisions should reference their replacement

3. Plans (artifacts/plans/)

Structure:

artifacts/plans/
├── research-plan.md        # Top-level research plan
├── music-sim-plan.md       # Feature-specific plan
├── pending/                # Plans being worked on
│   └── TASK-*.md           # Active task plans
└── completed/              # Archived plans (optional directory)

Naming: Descriptive names for top-level plans (research-plan.md, music-sim-plan.md). Pending plans use TASK-*.md convention.

Content structure — validated against plan_md.py parser:

• Must have # Task: <title> header (or # Research Plan: ... for legacy plans)
• Must have ## Problem Statement section
• Must have ## Phases section
• Must have ## Completion Criteria section
• Steps MUST be flat lists (no nesting)
• Phase numbers must be sequential starting from 1
• Annotations (**Notes:**, **Warning:**, **Blocked:**) go after steps, not inside them

Critical violations (will break parser):

• Nested checkboxes (- [ ] subtask under another checkbox)
• Non-sequential phase numbers (e.g., Phase 1 then Phase 3)
• Bullets or numbered lists inside annotation text
• Invalid phase format (e.g., ### Phase One: instead of ### Phase 1:)

Integrity:

• No orphaned plans (plan references non-existent files)
• Completion state is consistent (all steps marked done = plan should be archived)

4. README (artifacts/README.md)

• Must exist
• Should describe the artifact structure
• Should be up-to-date with actual directory contents

5. Staleness Detection

Compare artifact age against code changes to detect drift:

• Log staleness: If a log references a notebook that has been modified since the log was written, flag it
• Decision staleness: If a decision references a file path that no longer exists, flag it
• Plan staleness: If a plan references files or steps that have changed, flag it
• Cross-reference integrity: If a decision references logs/YYYY-MM-DD_slug.md and that file doesn't exist, flag it

---

Auto-Fix Capabilities

Fix log naming

# Find misnamed logs
for f in artifacts/logs/*.md; do
  basename=$(basename "$f")
  if ! echo "$basename" | grep -qP '^\d{4}-\d{2}-\d{2}_[a-z0-9_-]+\.md$'; then
    echo "MISNAMED: $basename"
    # To fix: rename to YYYY-MM-DD_slug.md
  fi
done

Fix decision naming

# Find misnamed decisions
for f in artifacts/decisions/*.md; do
  basename=$(basename "$f")
  if ! echo "$basename" | grep -qP '^DEC-\d+_[a-z0-9_-]+\.md$'; then
    echo "MISNAMED: $basename"
  fi
done

Fix plan syntax (manual)

• Nested checkboxes → flatten to separate steps or move to **Notes:**
• Non-sequential phases → renumber sequentially
• Invalid phase format → change ### Phase One: to ### Phase 1:

Fix empty files

# Find empty log files
find artifacts/logs/ -name "*.md" -empty -type f

---

Validation Checklist

Run these checks in order:

Step 1: Log validation

• List all *.md files in artifacts/logs/
• Verify each filename matches YYYY-MM-DD_slug.md
• Read each file and verify it has H1 title, Date, Category
• Verify Category is one of: run, discovery, dead-end, uncertainty
• Skip .log.jsonl files (agent activity logs, not experiment logs)

Step 2: Decision validation

• List all *.md files in artifacts/decisions/
• Verify each filename matches DEC-NNN_slug.md
• Extract DEC numbers and check for duplicates
• Read each file and verify required sections: ID, Date, Status, Context, Decision, Rationale, Alternatives Considered, Consequences, References
• Verify Status is lowercase and valid: accepted, rejected, superseded, proposed
• Flag non-standard statuses as warnings

Step 3: Plan validation

• List all *.md files in artifacts/plans/ and artifacts/plans/pending/
• For each plan, use plan_read(plan_name) to validate parseability
• Check for nested checkboxes (must be flat)
• Check phase numbering (must be sequential starting from 1)
• Check annotations (must not contain bullets or numbered lists)

Step 4: Staleness validation

• For each log, check if referenced notebook exists and if log is older than last notebook modification
• For each decision, check if referenced file paths exist
• For each plan, check if referenced files exist
• Cross-reference: verify decision references to logs point to existing files

Step 5: README validation

• Read artifacts/README.md
• Verify it describes the current artifact structure
• Update if stale

---

Anti-Patterns to Fix

Pattern	Where to Look	Fix
Bad log filename	artifacts/logs/	Rename to YYYY-MM-DD_slug.md
Log missing Category	artifacts/logs/*.md	Add **Category:** run|discovery|dead-end|uncertainty
Missing decision sections	artifacts/decisions/	Add Rationale, Alternatives Considered, Consequences
Non-lowercase status	artifacts/decisions/	Change to lowercase (accepted, rejected, etc.)
Nested checkboxes	artifacts/plans/	Flatten to separate steps
Non-sequential phases	artifacts/plans/	Renumber sequentially
Empty log	artifacts/logs/	Delete if truly empty
Stale README	artifacts/README.md	Update to match current structure
Broken cross-references	All artifacts	Update or remove references to deleted files

---

Verification

After fixing issues, verify:

# Check log filenames (should produce no output):
for f in artifacts/logs/*.md; do
  basename=$(basename "$f")
  if ! echo "$basename" | grep -qP '^\d{4}-\d{2}-\d{2}_[a-z0-9_-]+\.md$'; then
    echo "STILL BAD: $basename"
  fi
done

# Check decision filenames (should produce no output):
for f in artifacts/decisions/*.md; do
  basename=$(basename "$f")
  if ! echo "$basename" | grep -qP '^DEC-\d+_[a-z0-9_-]+\.md$'; then
    echo "STILL BAD: $basename"
  fi
done

# Check plan parseability:
# Use plan_read() tool for each plan file

# Check for empty files:
find artifacts/ -name "*.md" -empty -type f

Report Format

STATUS: [no_work | fixed | stale | errors]

CHANGES:
- artifacts/logs/2026-06-24_experiment.md: Renamed from bad-name.md
- artifacts/decisions/DEC-001_use-sam.md: Added missing Rationale section
- artifacts/plans/research-plan.md: Flattened nested checkboxes

STALENESS:
- artifacts/logs/2026-04-13_aversion-layer-run1.md: References notebooks/01_train_aversion_layer_local.py (modified since log)
- artifacts/decisions/DEC-003.md: References logs/2026-04-15_foo.md (file not found)

ERRORS:
- artifacts/decisions/DEC-006.md: Missing required section "Alternatives Considered"
- artifacts/plans/pending/TASK-foo.md: Phase numbers non-sequential (1, 3)

DETAILS:
{Detailed explanation of each issue and fix}

---

Tool Integration

Use these tools when working with artifacts:

• plan_read(plan_name) — Validate plan parseability
• plan_complete_step(plan_name, step_id) — Mark plan steps complete
• plan_archive(plan_name) — Archive completed plans
• log_write(agent, title, category, body) — Write new log entries
• log_read(agent, category, tag) — Read existing log entries
• adr_search(query) — Search decisions by topic
• adr_read(name) — Read a specific decision

---

When to Run This Skill

• Before starting work: Check for relevant ADRs and logs in the area you're about to touch
• After completing work: Log your findings, update decisions if needed
• Periodically: Run staleness checks to catch drift
• When creating new artifacts: Ensure they follow the conventions documented here