data-effectiveness-audit

name	data-effectiveness-audit
description	Comprehensive audit of data system effectiveness — lifecycle scoring across Capture, Storage, Recall, and Action dimensions for all data systems in the project.

Document Version: 1.0 Last Updated: 2026-03-13 Status: ACTIVE

Data Effectiveness Audit

Systematic audit of all data systems to identify write-only data, broken feedback loops, and enforcement gaps. Uses lifecycle scoring (Capture/Storage/Recall/Action, 0-3 each, 0-12 total) to quantify effectiveness and route gaps through the learning-to-automation pipeline.

Invocation: /data-effectiveness-audit

Critical Rules (MUST follow)

1. READ lifecycle-scores.jsonl first (MUST) — never generate findings without current data.
2. Present findings interactively (MUST) — one system at a time, worst first, with user decisions.
3. Use conversational Q&A for decisions (MUST) — NEVER use AskUserQuestion.
4. Update scores after decisions (MUST) — write to lifecycle-scores.jsonl immediately.
5. Regenerate dashboard after updates (MUST) — run node scripts/generate-lifecycle-scores-md.js.
6. Route gaps through pipeline (MUST) — Action<2 gaps routed via scripts/route-lifecycle-gaps.js.
7. Create TDMS entries (MUST) — for deferred findings via /add-debt.

---

When to Use

• User explicitly invokes /data-effectiveness-audit
• Periodic health check on data pipeline effectiveness
• After adding new JSONL files or data systems
• When health score Data Effectiveness dimension drops
• When /alerts surfaces data staleness warnings

When NOT to Use

• For individual file rotation — use scripts/rotate-jsonl.js
• For learning routing — use scripts/lib/learning-router.js directly
• For general health monitoring — use /health-ecosystem-audit
• For code quality auditing — use /audit-code
• Running all ecosystem audits — use /comprehensive-ecosystem-audit

Routing Guide

Concern	Use
Data system lifecycle scoring	/data-effectiveness-audit
Health monitoring system audit	/health-ecosystem-audit
Hook internals, pre-commit	/hook-ecosystem-audit
PR review workflow	/pr-ecosystem-audit
Session lifecycle	/session-ecosystem-audit
Technical debt management	/tdms-ecosystem-audit
All of the above	/comprehensive-ecosystem-audit

---

Process Overview

WARM-UP    Orientation     -> Effort estimate, process overview
PHASE 1    Load & Score    -> Read lifecycle-scores.jsonl, check staleness
PHASE 2    Re-Score        -> Score all systems across 4 dimensions (if stale)
PHASE 3    Dashboard       -> Composite grades, domain breakdown (worst first)
PHASE 4    Walkthrough     -> Interactive finding-by-finding decisions
PHASE 5    Route Gaps      -> Run gap routing scripts
PHASE 6    Summary         -> Decision aggregate + remediation plan
PHASE 7    Verification    -> Re-run scoring to confirm improvements
PHASE 8    Closure         -> Retro, invocation tracking, artifact list

---

Warm-Up (MUST)

Data Effectiveness Audit
Phases: Load scores -> Re-score -> Dashboard -> Walkthrough -> Route gaps -> Summary
Estimated time: 15-25 min (depends on system count + finding severity)

Done when: User confirms proceed.

---

Audit Domains (8)

D1. Capture Completeness

Score systems on data capture quality (0-3):

Score	Meaning
0	No structured capture
1	Ad-hoc/manual capture
2	Systematic capture with schema
3	Automated capture with validation

D2. Storage Health

Score systems on data storage (0-3):

Score	Meaning
0	No rotation or management
1	Exists but unbounded
2	Rotation policy applied
3	Rotation + backup + versioned schema

D3. Consumer Coverage (Recall)

Score systems on data consumption (0-3):

Score	Meaning
0	Write-only (no consumer)
1	One informational consumer
2	Multiple active consumers
3	Consumers trigger automated actions

D4. Enforcement (Action)

Score systems on automated enforcement (0-3):

Score	Meaning
0	No enforcement
1	Alerting/informational
2	Semi-automated (gates + manual)
3	Fully automated (gates + auto-fix)

D5. Feedback Loop Closure

Check that data flows form closed loops: capture -> store -> recall -> act -> improve capture.

D6. Orphan Detection

Identify files with zero consumers (Recall = 0).

D7. Growth Management

Verify all JSONL files have rotation policies (Storage >= 2).

D8. Cross-System Integration

Count cross-skill data flows and identify isolated systems.

---

Scoring Rubric

Score	Grade	Meaning
10-12	A	Excellent — fully wired and automated
8-9	B	Good — most loops closed, minor gaps
6-7	C	Adequate — functional but gaps exist
4-5	D	Poor — significant gaps, action needed
0-3	F	Critical — write-only or broken

Threshold: Systems below 6/12 are flagged for immediate remediation.

---

Phase 1: Load & Score (MUST)

1. Read .claude/state/lifecycle-scores.jsonl for current scores.
2. Check staleness: if newest entry is > 7 days old, mark as stale.
3. List all known data systems (JSONL files, state files, metrics).
4. Identify any new systems not yet scored.

Done when: Current scores loaded and staleness assessed.

---

Phase 2: Re-Score (CONDITIONAL)

If scores are stale or new systems found:

1. For each data system, score across all 4 dimensions (C/S/R/A).
2. Compute total (0-12) and letter grade.
3. Write updated scores to .claude/state/lifecycle-scores.jsonl.
4. Run node scripts/generate-lifecycle-scores-md.js to regenerate dashboard.

Done when: All systems scored, dashboard regenerated.

Skip if: All scores are fresh (< 7 days) and no new systems.

---

Phase 3: Dashboard (MUST)

Present dashboard sorted by score ascending (worst first):

## Data Effectiveness Dashboard

| System | Capture | Storage | Recall | Action | Total | Grade |
|--------|---------|---------|--------|--------|-------|-------|
| ...    | 0-3     | 0-3     | 0-3    | 0-3    | 0-12  | A-F   |

Composite Score: X/12 (Grade)
Systems below threshold (< 6): N
Orphaned systems (Recall = 0): N
Unbounded growth (Storage < 2): N

Zero findings below threshold: Skip walkthrough, go to Phase 6. With findings: Begin walkthrough.

---

Phase 4: Finding-by-Finding Walkthrough (MUST)

Sort systems by total score ascending (worst first). For each system below threshold (< 6/12):

1. Show current scores with dimension breakdown.
2. Identify weakest dimension(s).
3. Recommend specific remediation action.
4. Get user decision.

Decision Options

• Fix Now: Implement remediation in this session.
• Defer: Create TDMS entry via /add-debt.
• Skip: Acknowledge with justification.

Batch Shortcuts

After 3+ similar low-severity findings: offer batch acknowledgment.

Scope Explosion Guard

More than 15 systems below threshold: offer filtered review (F-grade only first, then D-grade).

---

Phase 5: Route Gaps (MUST)

1. Run node scripts/route-lifecycle-gaps.js — routes Action<2 gaps through learning-to-automation pipeline.
2. Run node scripts/route-enforcement-gaps.js — routes CLAUDE.md enforcement gaps.
3. Run node scripts/verify-enforcement.js — verify enforcement effectiveness.
4. Present routing results to user.

Done when: All gap routing scripts executed and results shown.

---

Phase 6: Summary & Actions (MUST)

## Audit Summary

- Systems audited: N
- Composite score: X/12 (Grade)
- Systems improved: N
- Systems deferred: N
- Systems skipped: N
- Gaps routed: N

### Remediation Plan
[Ordered list of deferred items with priority]

### Key Insights
[2-3 data-driven observations about system health trends]

Write report to .claude/tmp/data-effectiveness-audit-report-{YYYY-MM-DD}.md.

---

Phase 7: Verification (SHOULD)

Re-run scoring to confirm improvements:

1. Regenerate lifecycle scores.
2. Compare before/after composite score.
3. Verify fixed systems now score >= 6/12.

---

Phase 8: Closure (MUST)

1. Auto-learnings (MUST): 2-3 data-driven insights about system health.
2. Invocation tracking (MUST):

   cd scripts/reviews && npx tsx write-invocation.ts --data '{"skill":"data-effectiveness-audit","type":"skill","success":true,"schema_version":1,"completeness":"stub","origin":{"type":"manual"},"context":{"score":SCORE,"grade":"GRADE"}}'
   

3. Closure signal (MUST): List all artifacts created/modified.
4. Cleanup (SHOULD): Remove any temporary files.

---

Key Scripts

Script	Purpose
scripts/generate-lifecycle-scores-md.js	Generate dashboard from JSONL
scripts/route-lifecycle-gaps.js	Route Action<2 gaps
scripts/route-enforcement-gaps.js	Route CLAUDE.md enforcement gaps
scripts/lib/learning-router.js	Core routing library
scripts/verify-enforcement.js	Verify enforcement effectiveness
scripts/ratchet-baselines.js	Tighten violation baselines
scripts/rotate-jsonl.js	Apply rotation policies

Output Artifacts

Artifact	Purpose
.claude/state/lifecycle-scores.jsonl	Updated scores
.planning/system-wide-standardization/learnings-effectiveness-audit/LIFECYCLE_SCORES.md	Generated dashboard
.claude/state/learning-routes.jsonl	Routing decisions
.claude/tmp/data-effectiveness-audit-report-{date}.md	Session report

Integration

• Health Check: scripts/health/checkers/data-effectiveness.js (15% weight)
• Alerts: /alerts checks velocity-regression, stale-planning, deferred-items
• Session: session-end includes planning data + commit analytics

Guard Rails

• Zero findings below threshold: Skip walkthrough, go to Summary.
• Scope explosion: >15 systems below threshold -> offer filtered review.
• Script failure: Report error, suggest manual scoring.
• Disengagement: Save progress, show resume instructions.

---

Version History

Version	Date	Description
1.0	2026-03-13	Initial creation from Data Effectiveness Audit plan