| name | session-analysis |
| description | Run and interpret session transcript analysis — skill activation rates, steering compliance, tool usage across kiro sessions. Use when measuring skill field usage, checking recall-check compliance, reviewing what skills actually activate, or judging skill retirement. Trigger: session review, skill usage, field compliance, steering compliance, never activated, session:skills, session:parse. |
| metadata | {"type":"reference","invocation":"both","practice":null} |
Session Analysis
Parses ~/.kiro/sessions/cli transcripts. Both tasks REQUIRE --days:
mise run session:skills -- --days 30
mise run session:parse -- --days 30
mise run session:review 7
Direct: python tools/session-analyzer/skill_usage.py --days N (JSON to stdout).
Reading the skill_usage report
| Key | Meaning / how to act |
|---|
sessions / crew_research_dev_sessions | window size; dev sessions are excluded from field-usage judgments (we trigger our own skills) |
skill_activations_sessions | sessions where each skill's content entered context — the field-usage number |
never_activated | deployed skills with zero activations in the window — retirement CANDIDATES, not verdicts (check the skill was actually deployed and correct for the whole window first; see multi-agent-validation precedent, t09 rec #5) |
activation_signal_kinds | how activation was detected — weight explicit loads over incidental mentions |
steering_compliance.history_questions vs .history_q_with_recall_search | recall-check compliance ratio — the ticket 23 metric. Baseline 21% (2026-07-17, 60/284) |
recall_check_compliance | convenience block (added ticket 23): same counters plus precomputed rate and the baseline string — cite this in compliance reports |
Measurement soundness: session JSONLs do NOT embed injected steering text (verified 2026-07-19: zero matches for steering markers in stored transcripts) — compliance regexes match real conversation content, not the steering that teaches the phrases.
Transcript format (verified 2026-07-25, ticket 34 spike): user prompts are kind: Prompt JSONL lines (data.content[].data); there are NO "USER MESSAGE BEGIN" wrappers in stored transcripts (317/319 sessions — only interactive sessions with injected context embed them). Slash invocations arrive as the EXPANDED skill body — skill_usage.py matches prompt H1s against deployed skills' H1s (fixed 2026-07-25; slash_invoke counts before this date are ~0 and undercount user-invoked skills — don't compare across the boundary).
| steering_compliance.eval_run_sessions vs .eval_run_with_nohup | eval-execution steering compliance |
| sessions_per_project | denominator context — low-session projects give noisy rates |
Interpretation rules
- Windows matter: ≥30 days for retirement judgments; ≥7 days for compliance deltas; a 2-day window is a smoke test, not evidence.
- Compare like windows: compliance measured over N days compares against a baseline of similar N (weekday mix shifts usage).
- Skill fixed mid-window? Restart the clock — activations before the fix don't count for the fixed version (frontmatter/body breakage precedent: multi-agent-validation).
- Report rates WITH raw counts (42/98, not "43%") — small denominators must be visible.
- Retirement route: a confirmed retirement verdict lands as a
compositions/deprecated.yaml entry (delete skill + registry entry in the same commit, per AGENTS.md). Activation data is the evidence; deprecated.yaml is the mechanism.
When to run
- Ticket 23 measurement (recall-check compliance vs 21% baseline)
- Monthly skill portfolio review (retirement candidates, trigger-space overlaps like planning-cycles vs sdd — t09 rec #2)
- After any steering gate change (did compliance move?)
