vanta-patterns

name	vanta-patterns
description	Weekly governance retrospective on Vanta itself. Analyzes routing telemetry, repeat invariants, decision reversals, and stale memory to surface patterns that would otherwise stay invisible. Outputs vanta-health.md and proposes self-improvements.
argument-hint	[--weekly | --monthly | --since=YYYY-MM-DD]
user-invocable	true
model	opus

Vanta-Patterns — Self-Governance Loop

Vanta logs everything: route hits, route misses, decisions, episodes, sync events. This skill turns that telemetry into actionable insight about how Vanta itself is performing.

When to Run

• Weekly (every Monday) — review the week's routing health
• Before any v-bump — confirm features are being used
• After a frustrating session — find the pattern in the friction
• User says "vanta health", "how is vanta doing", "patterns this week"

Process

Step 1 — Window selection

Default: last 7 days (rolling). Override via --monthly, --since=YYYY-MM-DD, or --all.

WINDOW_START=$(date -u -v-7d +%Y-%m-%dT 2>/dev/null || date -u -d '7 days ago' +%Y-%m-%dT)
echo "Analyzing window: $WINDOW_START → now"

Step 2 — Pull telemetry

Read four log files, all jsonl, all in ~/.vanta/:

# R12 P1 / R8 P1 — read across rotated `.bak.<ts>` siblings. Earlier impl
# read only the live file, so weekly health metrics ignored everything past
# the most recent 5MB rotation. Counts collapse falsely on long-running
# installs. Use a small Node helper that walks bak siblings.
_LOGS=~/.vanta
node - "$WINDOW_START" "$_LOGS" << 'COUNT' | while IFS=':' read -r name n; do
  case "$name" in
    routes)   _ROUTES="$n";;
    misses)   _MISSES="$n";;
    queue)    _QUEUE="$n";;
    episodes) _EPISODES="$n";;
  esac
done
const fs = require('fs');
const path = require('path');
const since = process.argv[2];
const dir = process.argv[3];
function* siblings(file) {
  const base = path.basename(file);
  let entries = [];
  try { entries = fs.readdirSync(dir).filter(n => n.startsWith(base + '.bak.')); } catch {}
  for (const e of entries.sort()) yield path.join(dir, e);
  if (fs.existsSync(file)) yield file;
}
function count(file) {
  let n = 0;
  for (const f of siblings(file)) {
    try {
      for (const l of fs.readFileSync(f, 'utf8').split('\n')) {
         (!l) ;
         (l >= since) n++;  // crude TS prefix compare; matches awk usage
      }
    } catch {}
  }
   n;
}
console.log(   + count(path.join(, )));
console.log(   + count(path.join(, )));
console.log(    + count(path.join(, )));
console.log( + count(path.join(, )));
COUNT
 

Step 3 — Compute four metrics

Use Node to compute aggregates (no jq dependency). The script outputs the report.

node - "$WINDOW_START" << 'JS'
const fs = require('fs');
const path = require('path');
const os = require('os');
const since = process.argv[2];
const dir = path.join(os.homedir(), '.vanta');
// R12 P1 / R8 P1 — read across rotated `.bak.<ts>` siblings + live file.
// Producer no longer compacts on rotate, so older entries live in baks.
const read = f => {
  try {
    const base = path.basename(f);
    const parts = [];
    let baks = [];
    try { baks = fs.readdirSync(dir).filter(n => n.startsWith(base + '.bak.')).sort(); } catch {}
    for (const b of baks) {
      try { parts.push(fs.readFileSync(path.join(dir, b), 'utf8')); } catch {}
    }
    try { parts.push(fs.readFileSync(path.join(dir, f), 'utf8')); } catch {}
    const merged = parts.map(p => p.endsWith('\n') ? p : p + '\n').join('');
    return merged.trim().split('\n').filter(Boolean).map(l => {
      try { return JSON.parse(l); } catch { return null; }
    }).filter(Boolean);
  } catch { return []; }
};

const routes = read('routing-events.jsonl').filter(e => (e.ts || '') >= since);
const misses = read('missed-intents.jsonl').filter(e => (e.ts || '') >= since);
const episodes = read('episodes.jsonl').filter(e => (e.ts || '') >= since);
const queue = read().filter(e => (e.ts || ) >= since);

// Top routes
const routeCounts = {};
routes.forEach(r => { routeCounts[r.route || ] = (routeCounts[r.route || ] || 0) + 1; });
const topRoutes = Object.entries(routeCounts).((a,b)=>b[]-a[]).slice(,);

// Top missed phrases (cluster similar ones)
const missCounts = {};
misses.forEach(m => {
  const key = (m.phrase || '').toLowerCase().slice(, );
  missCounts[key] = (missCounts[key] || ) + ;
});
const topMisses = Object.entries(missCounts).filter(([_,c])=>c>=).sort((a,b)=>b[]-a[]).slice(,);

// Repeat episode topics (recurring problems)
const topicCounts = {};
episodes.forEach(e => (e.topics || []).forEach(t => { topicCounts[t] = (topicCounts[t] || ) + ; }));
const repeatTopics = Object.entries(topicCounts).filter(([_,c])=>c>=).sort((a,b)=>b[]-a[]).slice(,);

// Sync coverage: % of queued sessions actually synced
const synced = queue.filter(q => q.synced).length;
const syncCoverage = queue.length >  ? Math.round((synced / queue.length) * ) : ;

// Outcome distribution
const outcomes = { resolved: , blocked: , decided: , 'in-progress':  };
episodes.forEach(e => { if (outcomes[e.outcome] !== undefined) outcomes[e.outcome]++; });

// Generate report
const today = new Date().toISOString().slice(, );
const report = `# Vanta Health Report — 

Window:  → 

## Activity

- Sessions captured: 
- Routes invoked: 
- Routing misses: 
- Episodes recorded: 
- Sync coverage: % (/ synced)

## Top Routes



## Top Missed Phrases (worth adding routes)



## Repeat Topics (recurring problems)



## Outcome Distribution

- Resolved: 
- Blocked: 
- Decided: 
- In-progress: 

## Recommendations






`;

const reportPath = path.join(os.homedir(), '.vanta', 'vanta-health.md');
fs.writeFileSync(reportPath, report);
console.log(report);
console.log(`\n✓ Written to `);
JS

Step 4 — Run /council on the report (governance loop)

If the report flags ≥2 warnings, propose a council review of Vanta itself:

The health report flagged N issues. Run /council on ~/.vanta/vanta-health.md to design fixes? [y/n]

This closes the loop: Vanta uses its own adversarial review process to redesign itself based on observed failures.

Step 5 — Auto-propose route additions

For each missed phrase appearing 3+ times, propose adding it to vanta-run's routing table. Format:

Suggested new route for "<phrase>":
| "<phrase>", "<variants>" | <framework> | `Skill("<target>")` |

Add to ~/.claude/skills/vanta-run/SKILL.md? [y/n]

Apply confirmed additions to the SKILL.md routing table directly.

What Good Reports Look Like

A healthy Vanta produces reports with:

• High sync coverage (>80%): learnings are being captured
• Few repeated misses: the routing table covers common intents
• Topic distribution matches active work: what you're doing is what's logged
• Outcomes skew resolved over blocked: progress is being made

A report that's mostly empty means vanta isn't being used heavily yet — that's fine, just informational.

v3.10 — Self-Improving Loop Surfaces

After v3.10, vanta-patterns has three additional telemetry streams to consult. These are read-only signals from the self-improving loop; the skill does NOT mutate any of them. They feed tools/vanta-soak-report.js — run that for the rendered view.

Stream	Path	Producer	Surface
Rule effectiveness	~/.vanta/rule-effectiveness.jsonl	vanta-rewriter.js decisions joined to actions/cancellations	Soak §7. Wilson CI lower bound + rolling-window rate per rule. Quarantined rules are skipped at runtime.
Invariant evidence	~/.vanta/invariant-evidence.jsonl	vanta-resolve.js (origin=user-prompt) + vanta-council-feedback.js (council_tp mirror)	Soak §8. Top-cited and cold (30d unused) invariants. C-7 default-deny: only user-prompt origin counts.
Recent failures	~/.vanta/recent-failures.jsonl	auto-sync.js Stop hook → vanta-failure-extract.js parser	Brief surfacing (24h window, top 3 distinct). C-2 hardened: structured allowlisted fields ONLY — never freeform output.

If any of these files don't exist yet on the operator's box, tools/vanta-soak-report.js shows a "not deployed" stub for the section. Run setup.sh from ~/Projects/vanta to install the bins.

Operator escape hatch

For rule-effectiveness specifically, bin/vanta-rule-tune.js is the human override surface. Operator only — never expose to the user:

• vanta-rule-tune list — every rule with current scores
• vanta-rule-tune status <rule> — JSON detail
• vanta-rule-tune compute — recompute scores, snapshot
• vanta-rule-tune quarantine <rule> — manual skip
• vanta-rule-tune rehabilitate <rule> — flip back active, open new scoring epoch
• vanta-rule-tune auto-quarantine [--dry-run] — quarantine every eligible rule

Manual setStatus calls always have higher status_seq than concurrent snapshots, so operator decisions are never stomped (R1 council fix). Auto-rehab fires automatically when a rule's source content hash changes.