copilot-session-telemetry

# Copilot Session Telemetry Audit one skill-execution episode from a Copilot session's local telemetry — what the run actually burned (tokens, time, scripted vs ad-hoc steps). It measures; it does not judge skill quality. Goals, metric definitions, and classification rules live in `references/` — load them only as needed. > **Terminology**: an **episode** is one execution run of the target skill — a > time-window *slice* of the session's `events.jsonl` transcript, from the skill's > invocation to the next external skill invocation (or session end). One session > can contain many episodes. `--mode session` measures everything in the session > before this audit's own activation instead. ## Relationship to waza (hand-off) Vocabulary: **"benchmark" is reserved for waza's authored evals** — this skill produces **telemetry**. The two are complementary halves of one loop: 1. **Audit** a real run here → find the hotspot (a repeated ad-hoc shell block, a discovery-heavy subagent window, a SKILL.md loaded every turn). 2. **Hand off** the evidence to the [`waza-cli`](../waza-cli/SKILL.md) skill: scaffold an eval case (`waza new eval <skill>`), assert the corrected behavior with graders (`action_sequence`, `tool_constraint`, `text`, …), and gate regressions with `waza gate`. 3. **Re-audit** after the fix and compare the new dated report against the previous one to confirm the improvement held in production. Do not load this skill to judge skill output quality — that is waza's domain. ## Workflow 1. **Identify target skill and session.** Determine the skill name that executed (repo-level in `skills/<name>/` or `.github/skills/<name>/`, or user-level/published). Locate the session dir: the current session (`$env:COPILOT_HOME` or `~/.copilot` + `session-state/<uuid>`), or scan `session-state/*/workspace.yaml` for `repository`/`git_root` matching the current repo (most recently modified first). Do NOT use the `sessions` SQLite catalog for discovery — it can be days stale. 2. **Pre-flight telemetry check.** Confirm `events.jsonl` exists in the session dir and contains a `skill.invoked` **event type** whose `data.name` equals the target skill. Filter on the event `type` field exactly — a substring match on `skill.invoked` also catches `skill.invoked_ref` events, whose `data.name` is an empty string; those are noise, not evidence of a missing skill. If either check is missing, stop and report the specific gap — never silently fall back to the stale SQLite catalog. 3. **Run the audit.** ```powershell pwsh -NoProfile -File skills/copilot-session-telemetry/scripts/Invoke-SessionTelemetry.ps1 ` -SkillName <target-skill> -SessionDir <session-dir> ``` The orchestrator resolves the telemetry location (target skill's `telemetry/` folder when repo-level — `skills/<skill>/` or `.github/skills/<skill>/` —, `telemetry/<skill>/` at repo root when user-level; an existing legacy `benchmarks/` folder is reused with a warning; override with `-OutDir`), runs `scripts/compute_metrics.py`, and writes: - `YYYYMMDD-HHMMSS-episodeN-report.md` — dated report with the full metric tables (the only persisted artifact; the engine also echoes the full metrics as JSON on stdout) Options: `-Episode N` (earlier invocation of the skill), `-Mode episode|session`, `-EventsPath` (direct file), `-OutDir`, `-RepoRoot`. **Modes**: `-Mode episode` (default) measures one execution run of the skill. `-Mode session` measures everything before this audit's own activation — the whole-workflow number that survives ambiguous attribution; per-episode detail appears as a secondary breakdown table. Session mode writes `…-session-report.md` (keep episode and session reports separate — they answer different questions and must never cross-compare). 4. **Present results.** Show the four-goal scorecard. When a metric looks off versus the previous dated report, name one concrete improvement: raise the deterministic ratio by scripting a repeated ad-hoc shell command; cut tokens by trimming context loaded per step; cut semantic steps by batching edits. ## Reading the output - Tokens are primary; cost is informational only (BYOK sessions report 0). - Active time excludes `ask_user` waits — human idle never penalizes the workflow. - A high `uncertain` count (usually ad-hoc `powershell` one-liners) is a signal, not noise: each one is a candidate for promotion to a repo script, or for a waza eval case that locks the fix in (see Relationship above). - `query_read` steps count as decision points — grep/view spam is visible, not free. - **Degenerate-episode guard**: an episode with ≤ 2 tool steps (or a < 2 min window while the session holds far more work) is flagged in the report's Caveats — a skill-load measurement must never be mistaken for the workflow. Use `-Mode session` for a whole-workflow number. - **Skill-chain & subagent attribution**: loads of driver/helper skills named by the target's `SKILL.md` (or its `machina:` contract) and skill loads inside subagent windows do not terminate the episode — they appear in the report's Boundaries table with their own sub-window totals. - Classification rules and the tool-name alias map: `references/classification.md`. - Metric formulas and episode attribution (self-exclusion, timestamp windows): `references/metrics.md`. Event shapes: `references/event-schema.md`. ## Edge cases - **Multiple invocations**: default is the most recent episode; use `-Episode N`. - **Meta/driver skills belong to their owner**: when a driver skill (e.g. `machina-driving`) is chain-merged into a declaring orchestrator's episode, its cost is already owned by that orchestrator's scorecard (shown as a boundary row). Do NOT audit a shared driver as its own episode — it would double-count and mix callers (several orchestrators can declare the same driver). Audit a driver directly only when the session drove the workflow with no orchestrator involved (e.g. a direct human-conductor run). - **Sub-agent (`task`) work**: included in totals and broken out in `sub_agent_tokens`. - **Audit self-contamination**: excluded automatically — the episode ends at the audit's own activation (hard stop, even across skill chains). - **No/old telemetry**: pre-flight fails with a named gap; the engine still writes a `*-cannot-audit.md` stub (exit code 3) so the failure itself is a durable record.