| name | data-effectiveness-audit |
| description | Comprehensive audit of data system effectiveness — lifecycle scoring across Capture, Storage, Recall, and Action dimensions for all data systems in the project. |
Document Version: 1.0
Last Updated: 2026-03-13
Status: ACTIVE
Data Effectiveness Audit
Systematic audit of all data systems to identify write-only data, broken
feedback loops, and enforcement gaps. Uses lifecycle scoring
(Capture/Storage/Recall/Action, 0-3 each, 0-12 total) to quantify effectiveness
and route gaps through the learning-to-automation pipeline.
Invocation: /data-effectiveness-audit
Critical Rules (MUST follow)
- READ lifecycle-scores.jsonl first (MUST) — never generate findings
without current data.
- Present findings interactively (MUST) — one system at a time, worst
first, with user decisions.
- Use conversational Q&A for decisions (MUST) — NEVER use AskUserQuestion.
- Update scores after decisions (MUST) — write to lifecycle-scores.jsonl
immediately.
- Regenerate dashboard after updates (MUST) — run
node scripts/generate-lifecycle-scores-md.js.
- Route gaps through pipeline (MUST) — Action<2 gaps routed via
scripts/route-lifecycle-gaps.js.
- Create TDMS entries (MUST) — for deferred findings via
/add-debt.
When to Use
- User explicitly invokes
/data-effectiveness-audit
- Periodic health check on data pipeline effectiveness
- After adding new JSONL files or data systems
- When health score Data Effectiveness dimension drops
- When
/alerts surfaces data staleness warnings
When NOT to Use
- For individual file rotation — use
scripts/rotate-jsonl.js
- For learning routing — use
scripts/lib/learning-router.js directly
- For general health monitoring — use
/health-ecosystem-audit
- For code quality auditing — use
/audit-code
- Running all ecosystem audits — use
/comprehensive-ecosystem-audit
Routing Guide
| Concern | Use |
|---|
| Data system lifecycle scoring | /data-effectiveness-audit |
| Health monitoring system audit | /health-ecosystem-audit |
| Hook internals, pre-commit | /hook-ecosystem-audit |
| PR review workflow | /pr-ecosystem-audit |
| Session lifecycle | /session-ecosystem-audit |
| Technical debt management | /tdms-ecosystem-audit |
| All of the above | /comprehensive-ecosystem-audit |
Process Overview
WARM-UP Orientation -> Effort estimate, process overview
PHASE 1 Load & Score -> Read lifecycle-scores.jsonl, check staleness
PHASE 2 Re-Score -> Score all systems across 4 dimensions (if stale)
PHASE 3 Dashboard -> Composite grades, domain breakdown (worst first)
PHASE 4 Walkthrough -> Interactive finding-by-finding decisions
PHASE 5 Route Gaps -> Run gap routing scripts
PHASE 6 Summary -> Decision aggregate + remediation plan
PHASE 7 Verification -> Re-run scoring to confirm improvements
PHASE 8 Closure -> Retro, invocation tracking, artifact list
Warm-Up (MUST)
Data Effectiveness Audit
Phases: Load scores -> Re-score -> Dashboard -> Walkthrough -> Route gaps -> Summary
Estimated time: 15-25 min (depends on system count + finding severity)
Done when: User confirms proceed.
Audit Domains (8)
D1. Capture Completeness
Score systems on data capture quality (0-3):
| Score | Meaning |
|---|
| 0 | No structured capture |
| 1 | Ad-hoc/manual capture |
| 2 | Systematic capture with schema |
| 3 | Automated capture with validation |
D2. Storage Health
Score systems on data storage (0-3):
| Score | Meaning |
|---|
| 0 | No rotation or management |
| 1 | Exists but unbounded |
| 2 | Rotation policy applied |
| 3 | Rotation + backup + versioned schema |
D3. Consumer Coverage (Recall)
Score systems on data consumption (0-3):
| Score | Meaning |
|---|
| 0 | Write-only (no consumer) |
| 1 | One informational consumer |
| 2 | Multiple active consumers |
| 3 | Consumers trigger automated actions |
D4. Enforcement (Action)
Score systems on automated enforcement (0-3):
| Score | Meaning |
|---|
| 0 | No enforcement |
| 1 | Alerting/informational |
| 2 | Semi-automated (gates + manual) |
| 3 | Fully automated (gates + auto-fix) |
D5. Feedback Loop Closure
Check that data flows form closed loops: capture -> store -> recall -> act ->
improve capture.
D6. Orphan Detection
Identify files with zero consumers (Recall = 0).
D7. Growth Management
Verify all JSONL files have rotation policies (Storage >= 2).
D8. Cross-System Integration
Count cross-skill data flows and identify isolated systems.
Scoring Rubric
| Score | Grade | Meaning |
|---|
| 10-12 | A | Excellent — fully wired and automated |
| 8-9 | B | Good — most loops closed, minor gaps |
| 6-7 | C | Adequate — functional but gaps exist |
| 4-5 | D | Poor — significant gaps, action needed |
| 0-3 | F | Critical — write-only or broken |
Threshold: Systems below 6/12 are flagged for immediate remediation.
Phase 1: Load & Score (MUST)
- Read
.claude/state/lifecycle-scores.jsonl for current scores.
- Check staleness: if newest entry is > 7 days old, mark as stale.
- List all known data systems (JSONL files, state files, metrics).
- Identify any new systems not yet scored.
Done when: Current scores loaded and staleness assessed.
Phase 2: Re-Score (CONDITIONAL)
If scores are stale or new systems found:
- For each data system, score across all 4 dimensions (C/S/R/A).
- Compute total (0-12) and letter grade.
- Write updated scores to
.claude/state/lifecycle-scores.jsonl.
- Run
node scripts/generate-lifecycle-scores-md.js to regenerate dashboard.
Done when: All systems scored, dashboard regenerated.
Skip if: All scores are fresh (< 7 days) and no new systems.
Phase 3: Dashboard (MUST)
Present dashboard sorted by score ascending (worst first):
## Data Effectiveness Dashboard
| System | Capture | Storage | Recall | Action | Total | Grade |
|--------|---------|---------|--------|--------|-------|-------|
| ... | 0-3 | 0-3 | 0-3 | 0-3 | 0-12 | A-F |
Composite Score: X/12 (Grade)
Systems below threshold (< 6): N
Orphaned systems (Recall = 0): N
Unbounded growth (Storage < 2): N
Zero findings below threshold: Skip walkthrough, go to Phase 6. With
findings: Begin walkthrough.
Phase 4: Finding-by-Finding Walkthrough (MUST)
Sort systems by total score ascending (worst first). For each system below
threshold (< 6/12):
- Show current scores with dimension breakdown.
- Identify weakest dimension(s).
- Recommend specific remediation action.
- Get user decision.
Decision Options
- Fix Now: Implement remediation in this session.
- Defer: Create TDMS entry via
/add-debt.
- Skip: Acknowledge with justification.
Batch Shortcuts
After 3+ similar low-severity findings: offer batch acknowledgment.
Scope Explosion Guard
More than 15 systems below threshold: offer filtered review (F-grade only first,
then D-grade).
Phase 5: Route Gaps (MUST)
- Run
node scripts/route-lifecycle-gaps.js — routes Action<2 gaps through
learning-to-automation pipeline.
- Run
node scripts/route-enforcement-gaps.js — routes CLAUDE.md enforcement
gaps.
- Run
node scripts/verify-enforcement.js — verify enforcement effectiveness.
- Present routing results to user.
Done when: All gap routing scripts executed and results shown.
Phase 6: Summary & Actions (MUST)
## Audit Summary
- Systems audited: N
- Composite score: X/12 (Grade)
- Systems improved: N
- Systems deferred: N
- Systems skipped: N
- Gaps routed: N
### Remediation Plan
[Ordered list of deferred items with priority]
### Key Insights
[2-3 data-driven observations about system health trends]
Write report to .claude/tmp/data-effectiveness-audit-report-{YYYY-MM-DD}.md.
Phase 7: Verification (SHOULD)
Re-run scoring to confirm improvements:
- Regenerate lifecycle scores.
- Compare before/after composite score.
- Verify fixed systems now score >= 6/12.
Phase 8: Closure (MUST)
- Auto-learnings (MUST): 2-3 data-driven insights about system health.
- Invocation tracking (MUST):
cd scripts/reviews && npx tsx write-invocation.ts --data '{"skill":"data-effectiveness-audit","type":"skill","success":true,"schema_version":1,"completeness":"stub","origin":{"type":"manual"},"context":{"score":SCORE,"grade":"GRADE"}}'
- Closure signal (MUST): List all artifacts created/modified.
- Cleanup (SHOULD): Remove any temporary files.
Key Scripts
| Script | Purpose |
|---|
scripts/generate-lifecycle-scores-md.js | Generate dashboard from JSONL |
scripts/route-lifecycle-gaps.js | Route Action<2 gaps |
scripts/route-enforcement-gaps.js | Route CLAUDE.md enforcement gaps |
scripts/lib/learning-router.js | Core routing library |
scripts/verify-enforcement.js | Verify enforcement effectiveness |
scripts/ratchet-baselines.js | Tighten violation baselines |
scripts/rotate-jsonl.js | Apply rotation policies |
Output Artifacts
| Artifact | Purpose |
|---|
.claude/state/lifecycle-scores.jsonl | Updated scores |
.planning/system-wide-standardization/learnings-effectiveness-audit/LIFECYCLE_SCORES.md | Generated dashboard |
.claude/state/learning-routes.jsonl | Routing decisions |
.claude/tmp/data-effectiveness-audit-report-{date}.md | Session report |
Integration
- Health Check:
scripts/health/checkers/data-effectiveness.js (15% weight)
- Alerts:
/alerts checks velocity-regression, stale-planning,
deferred-items
- Session: session-end includes planning data + commit analytics
Guard Rails
- Zero findings below threshold: Skip walkthrough, go to Summary.
- Scope explosion: >15 systems below threshold -> offer filtered review.
- Script failure: Report error, suggest manual scoring.
- Disengagement: Save progress, show resume instructions.
Version History
| Version | Date | Description |
|---|
| 1.0 | 2026-03-13 | Initial creation from Data Effectiveness Audit plan |
