Run any Skill in Manus with one click

vaudeville-tier-advisor

Stars1

Forks0

UpdatedApril 14, 2026 at 09:51

Analyze vaudeville warn logs and eval data to recommend rule tier promotions and demotions. Use when the user says "promote rule", "tier advisor", "warn log analysis", "should we enforce", "demote rule", "rule metrics", "tier recommendation", "warn to block", "shadow to warn", "rule performance", "check rule health", "evaluate rule tiers", "promotion analysis", or invokes /tier-advisor. Do NOT use for editing rule files — use /rule-admin for that.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

paulnsorensen

paulnsorensen/vaudeville

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software Quality Assurance Analysts and TestersComputer and Mathematical Occupations·SOC 15-1253

File Explorer

4 files

SKILL.md

readonly

name	vaudeville:tier-advisor
description	Analyze vaudeville warn logs and eval data to recommend rule tier promotions and demotions. Use when the user says "promote rule", "tier advisor", "warn log analysis", "should we enforce", "demote rule", "rule metrics", "tier recommendation", "warn to block", "shadow to warn", "rule performance", "check rule health", "evaluate rule tiers", "promotion analysis", or invokes /tier-advisor. Do NOT use for editing rule files — use /rule-admin for that.
model	sonnet
context	fork
allowed-tools	Bash(uv:), Bash(duckdb:), Bash(python3:*), Read, Glob

tier-advisor

Reads ~/.vaudeville/logs/ JSONL files, ingests into DuckDB, computes per-rule metrics with user-agreement proxy, and outputs promotion/demotion recommendations.

Steps

1. Ingest

Run the ingest script to load fresh log data:

uv run python3 ${CLAUDE_PLUGIN_ROOT}/skills/tier-advisor/scripts/ingest.py

This reads ~/.vaudeville/logs/events.jsonl and violations.jsonl, deduplicates, and replaces table contents in ~/.claude/analytics/sessions.duckdb as table vaudeville_verdicts.

2. Analyze

Run the analysis script to compute per-rule metrics:

uv run python3 ${CLAUDE_PLUGIN_ROOT}/skills/tier-advisor/scripts/analyze.py

This JOINs vaudeville_verdicts against raw_entries to compute the agreement proxy (next user message after a violation indicates whether the user agreed or disagreed with the verdict).

3. Report

Run the report script to produce grouped recommendations:

uv run python3 ${CLAUDE_PLUGIN_ROOT}/skills/tier-advisor/scripts/report.py

Output is markdown grouped by recommendation: promote-to-block, promote-to-warn, demote, insufficient-data.

4. Present

Show the report to the user. If the user wants to act on a recommendation, tell them to use /rule-admin to promote, demote, or edit rule tiers.

Gotchas

Agreement proxy is approximate — the next-message heuristic misclassifies multi-turn corrections where the user responds several messages later
DuckDB raw_entries table may not exist if session-analytics hasn't been ingested yet — analyze.py returns empty agreement data, not an error
Empty ~/.vaudeville/logs/ produces zero records from ingest.py (exit 1), which is expected — tell the user to run some sessions first
Short keywords in agreement matching use word boundaries, but multi-word phrases still use substring matching — "it's fine" matches inside longer text

Thresholds

Transition	Min evals	Agreement	Violation rate	Confidence p50
shadow → warn	≥50	≥70%	2%–40%	—
warn → block	≥200	≥85%	5%–30%	≥0.7
warn → shadow	any	<50%	>60%	—
delete candidate	shadow ≥14d	no improvement	—	—

More from this repository

same repository

add-hook

paulnsorensen/vaudeville

Unified hook creation entry point that routes natural language descriptions to the right specialist agent. Use when the user wants to enforce behavior in Claude Code sessions. Trigger when the user says "add a hook", "create a hook", "new hook", "enforce X", "guard against X", "block X", "prevent X", "detect when Claude does X", "catch X", "stop Claude from X", "add enforcement", "quality gate for X", or describes any behavior to enforce. This skill routes to vaudeville:hard-hook-writer (JS/Python/Bash) or vaudeville:slm-rule-writer (SLM/YAML) based on whether the enforcement is structural or semantic. Also trigger when the user says "add a rule", "new rule", "create a detector", "SLM rule for X", or describes behavior requiring semantic classification. Do NOT use for suggesting hooks from usage data (use vaudeville:hook-suggester), querying session analytics (use vaudeville:session-analytics), checking daemon status (use `vaudeville:status` skill), or debugging existing hooks.

2026-04-261

vaudeville-hook-suggester

paulnsorensen/vaudeville

Analyze Claude Code session history to suggest hooks tailored to the user's actual usage patterns. Mines vaudeville:session-analytics data for dangerous commands, tool misuse, high error rates, permission friction, missing quality gates, broken hooks, and repeated automatable patterns — then generates concrete hook implementations via vaudeville:add-hook. Use when the user asks "what hooks should I add", "suggest hooks", "analyze my usage for hooks", "what should I enforce", "improve my hooks", "hook suggestions", or wants to discover enforcement opportunities they haven't thought of. Also trigger for "vaudeville suggestions", "hook recommendations", "what am I doing wrong", "what hooks do I need", "hook audit", "analyze my hooks", "optimize my workflow", "what's failing in my sessions", "guard against", or any question about optimizing their Claude Code workflow through hooks. Do NOT use for creating a specific known hook (use vaudeville:add-hook), querying raw session data (use vaudeville:session-analytics)

2026-04-261

vaudeville-rule-admin

paulnsorensen/vaudeville

Promote, demote, or edit vaudeville rule tiers. Use when the user says "promote rule", "demote rule", "move rule to warn", "move rule to block", "rule-admin", "change tier", "promote to warn", "promote to block", "demote to shadow", "delete rule", "enable rule", "disable rule", or invokes /rule-admin. Typically used after /tier-advisor produces recommendations. Do NOT use for analysis — use /tier-advisor for that.

2026-04-261

vaudeville-rule-audit

paulnsorensen/vaudeville

Audit existing vaudeville rules for design-level usefulness. Distinct from /tier-advisor (which audits eval data for promotion). This skill asks: does the rule's event × tier combination produce useful behavior change *at any reachable tier*? Surfaces rules that are useless by design — rules whose violations cannot be acted on in a future turn, rules stuck at SLM model ceilings, rules whose max-useful tier is below their current tier, rules duplicated by other rules, and rules that should be hard hooks instead of SLM rules. Use when the user says "audit my rules", "find useless rules", "are my rules useful", "review rule design", "rule audit", "which rules should I delete", "do my rules make sense", "design audit", or invokes /rule-audit. Do NOT use for promotion analysis (use /tier-advisor) or modifying tiers (use /rule-admin).

2026-04-261

vaudeville-session-analytics

paulnsorensen/vaudeville

Query Claude Code's own JSONL session logs via DuckDB for usage analytics, tool patterns, error forensics, hook impact analysis, and routing decisions. Use when the user asks about their Claude usage patterns, tool frequencies, error rates, permission denials, agent routing, skill invocations, MCP server usage, session timelines, hook effectiveness, or any question about "how has Claude been working". Also trigger when the user says "session analytics", "query my logs", "tool usage", "how often do I use", "check my sessions", "analyze my usage", "hook stats", "what hooks fired", or asks about specific tool/agent/skill/hook behavior across sessions. Also trigger for "vaudeville hook analytics", "hook effectiveness", "hook firing rate", "enforcement stats", or any question about how vaudeville hooks are performing. This skill turns raw JSONL into a queryable DuckDB database — use it instead of writing ad-hoc Python scripts to parse logs. Do NOT use for debugging current code issues, reading individual session t

2026-04-251

vaudeville-tune

paulnsorensen/vaudeville

Tune a single vaudeville SLM rule to meet precision/recall/f1 targets. Runs `vaudeville tune <rule>` which launches a ralphify autonomous agent loop to iteratively improve the rule's prompt until it meets accuracy thresholds. The loop exits early when thresholds are met via completion promise. Use when the user says "tune rule", "fix rule accuracy", "improve rule precision", "reduce false positives", "recall is too low", "tune violation-detector", or wants to iterate on an SLM rule prompt until it meets accuracy targets.

2026-04-191

name	vaudeville:tier-advisor
description	Analyze vaudeville warn logs and eval data to recommend rule tier promotions and demotions. Use when the user says "promote rule", "tier advisor", "warn log analysis", "should we enforce", "demote rule", "rule metrics", "tier recommendation", "warn to block", "shadow to warn", "rule performance", "check rule health", "evaluate rule tiers", "promotion analysis", or invokes /tier-advisor. Do NOT use for editing rule files — use /rule-admin for that.
model	sonnet
context	fork
allowed-tools	Bash(uv:), Bash(duckdb:), Bash(python3:*), Read, Glob

tier-advisor

Reads ~/.vaudeville/logs/ JSONL files, ingests into DuckDB, computes per-rule metrics with user-agreement proxy, and outputs promotion/demotion recommendations.

Steps

1. Ingest

Run the ingest script to load fresh log data:

uv run python3 ${CLAUDE_PLUGIN_ROOT}/skills/tier-advisor/scripts/ingest.py

This reads ~/.vaudeville/logs/events.jsonl and violations.jsonl, deduplicates, and replaces table contents in ~/.claude/analytics/sessions.duckdb as table vaudeville_verdicts.

2. Analyze

Run the analysis script to compute per-rule metrics:

uv run python3 ${CLAUDE_PLUGIN_ROOT}/skills/tier-advisor/scripts/analyze.py

This JOINs vaudeville_verdicts against raw_entries to compute the agreement proxy (next user message after a violation indicates whether the user agreed or disagreed with the verdict).

3. Report

Run the report script to produce grouped recommendations:

uv run python3 ${CLAUDE_PLUGIN_ROOT}/skills/tier-advisor/scripts/report.py

Output is markdown grouped by recommendation: promote-to-block, promote-to-warn, demote, insufficient-data.

4. Present

Show the report to the user. If the user wants to act on a recommendation, tell them to use /rule-admin to promote, demote, or edit rule tiers.

Gotchas

Agreement proxy is approximate — the next-message heuristic misclassifies multi-turn corrections where the user responds several messages later
DuckDB raw_entries table may not exist if session-analytics hasn't been ingested yet — analyze.py returns empty agreement data, not an error
Empty ~/.vaudeville/logs/ produces zero records from ingest.py (exit 1), which is expected — tell the user to run some sessions first
Short keywords in agreement matching use word boundaries, but multi-word phrases still use substring matching — "it's fine" matches inside longer text

Thresholds

Transition	Min evals	Agreement	Violation rate	Confidence p50
shadow → warn	≥50	≥70%	2%–40%	—
warn → block	≥200	≥85%	5%–30%	≥0.7
warn → shadow	any	<50%	>60%	—
delete candidate	shadow ≥14d	no improvement	—	—