copilot-usage-telemetry

name

copilot-usage-telemetry

description

Pull GitHub Copilot usage statistics without UI or screenshots. Two Python CLIs: one reads live credit/quota numbers from GitHub's private Copilot API, the other runs fast, context-free queries over the plugin's per-session `main.jsonl` debug log (tokens, requests, tool calls). Use when the user wants to track AI credits, premium-request usage, token counts, or inspect what a chat session actually did.

Copilot Usage Telemetry

Two standalone tools that turn the Copilot status-bar hover numbers and the hidden session logs into scriptable data.

Script	Source	What it gives
scripts/copilot_stats.py	live GitHub API copilot_internal/user	AI credits / quota, plan, feature flags
scripts/session_log.py	local debug-logs/*/main.jsonl	tokens, context window, requests, tool calls — by line number
scripts/usage_track.py	orchestrates the two above	persists ONE telemetry row per UPD run into a SQLite DB

Dependencies: requests, python-dotenv (only copilot_stats.py needs them; session_log.py and usage_track.py are stdlib-only — usage_track.py imports the other two lazily and degrades to NULL columns if they fail).

---

1. copilot_stats.py — credits & quota (needs a token)

Reads COPILOT_GITHUB_TOKEN from a .env (searched upward from the current directory AND the script's directory). The token is never printed. See .env.example for the template. Requires a token from a Copilot-enabled account (a classic GitHub PAT works; fine-grained PATs are rejected).

# Quick summary — the status-bar number (used %, remaining/entitlement, overage)
python ./modules/084-copilot-usage-telemetry/tools/scripts/copilot_stats.py credits

# Same data as JSON (text and json carry identical fields)
python ./modules/084-copilot-usage-telemetry/tools/scripts/copilot_stats.py credits --format json

# Everything the endpoint exposes (plan, flags, orgs, regional endpoints, full quotas)
python ./modules/084-copilot-usage-telemetry/tools/scripts/copilot_stats.py info

# `info --format json` prints the RAW endpoint response
python ./modules/084-copilot-usage-telemetry/tools/scripts/copilot_stats.py info --format json

Key field for tracking: AI credits = premium_interactions.remaining (out of entitlement, e.g. 30000). The value is fractional and personal — it is your own quota, not a number shared across your org. The status-bar "N% used" = 100 - percent_remaining. Measure remaining before and after a run; the delta is what the run cost.

Two gotchas this endpoint has — both handled by usage_track.py:

1. Eventually consistent. The endpoint serves a cached snapshot whose quota_snapshots.premium_interactions.timestamp_utc only advances every few minutes. So immediately after a run, remaining may be unchanged — the spend shows up a few minutes later. usage_track.py end therefore marks the run stale when the snapshot did not move, and you re-settle it later with refresh.
2. Cumulative, not per-run. remaining is a running total, so re-reading it "now" includes every run since. The only crisp per-run boundary is the next run's begin reading: spent(run N) = credits_start(N) − credits_start(N+1). refresh uses that boundary automatically once a later run exists; until then it falls back to a fresh read.

---

2. session_log.py — query the session log (no token, stdlib only)

The Copilot Chat extension writes one main.jsonl per session. These get large (several MB), so load nothing into context — query it. Every view returns line numbers (coordinates) so you can jump straight to a block.

This tool is intentionally generic: it knows nothing about UPD blocks or the iterative-prompt pattern. It only provides primitives.

S=./modules/084-copilot-usage-telemetry/tools/scripts/session_log.py

# List recent logs, grouped by workspace (with the resolved folder path).
# Default shows the 5 most-recently-modified sessions; --limit N changes that;
# --all shows every log. Copy the path you want and pass it to other commands.
python $S locate
python $S locate --limit 10
python $S locate --all

# IMPORTANT: every other command works on ONE log. Pass the explicit path you
# copied from `locate` as the last argument. If you omit it, the MOST RECENT
# log across ALL workspaces is used — which is often NOT the one you're in,
# because any keystroke in any chat updates a different log's mtime.
LOG="C:/Users/<you>/AppData/Roaming/Code - Insiders/User/workspaceStorage/<wsId>/GitHub.copilot-chat/debug-logs/<sid>/main.jsonl"

# Count events by type (tool_call, llm_request, user_message, …)
python $S types "$LOG"

# All user requests, with line numbers (great for finding UPD boundaries)
python $S requests "$LOG"

# Every raw line containing a substring (case-insensitive)
python $S grep "UPD2" "$LOG"

# Every tool_call for a tool (exact or substring match)
python $S tool read_file "$LOG"

# Tiny JSONPath-ish query (dot path, [index], * wildcard) over each event
python $S jsonpath attrs.model "$LOG"
python $S jsonpath attrs.inputTokens "$LOG"

# Pretty-print one event by line number, with optional context
python $S view 209 --context 2 "$LOG"

Any command accepts an explicit log path as the last positional arg (default: the most recent log across all workspaces), and --json for structured output.

Event schema (from the troubleshoot skill)

Each line is one JSON event with ts, dur, type, name, spanId, parentSpanId, status, attrs. Useful types:

• user_message — attrs.content (the raw user turn)
• llm_request — attrs.model, inputTokens, outputTokens, maxTokens, userRequest
• tool_call — name, attrs.args, attrs.result, status, dur
• turn_start / turn_end — tool-loop iteration boundaries

Typical workflow

1. locate → find your workspace by its folder path, copy the session log path.
2. requests <log> → get the line numbers of each user turn (e.g. where UPD2 starts).
3. jsonpath attrs.inputTokens <log> / attrs.outputTokens <log> → token counts per request.
4. tool <name> <log> → see which tools ran and how long they took.
5. view <line> --context N <log> → drill into one event when you need detail.

debug-logs vs chatSessions — which file to use

A workspace folder (workspaceStorage/<wsId>/) can hold the same session id in two places. They are different artifacts:

	chatSessions/<sid>.jsonl	GitHub.copilot-chat/debug-logs/<sid>/main.jsonl
Purpose	UI conversation persistence (what the chat panel renders)	Tracing / telemetry of the agent loop
Always present?	Yes — every session	Only when debug logging is on → rarer, fewer sessions
Format	delta/CRDT: kind:0 full state + kind:1/2 patches	one JSON event per line
Has user + assistant text	Yes (requests[].message, requests[].response)	Yes (user_message, agent_response)
Has model id	Yes (requests[].modelId)	Yes (llm_request.attrs.model)
Has token counts	No	Yes (inputTokens, outputTokens, maxTokens)
Has tool calls + timings	No (only rendered references)	Yes (tool_call.attrs.args/result, dur)
Has turn boundaries	No	Yes (turn_start / turn_end)

For this module (token & credit telemetry) always use debug-logs/main.jsonl — it is the only source with per-request token counts and tool timings. chatSessions is the right source if you only need the rendered conversation (it covers more history), which is exactly what module 250's export tool reads. session_log.py targets debug-logs exclusively.

---

3. usage_track.py — persist one telemetry row per UPD run

The orchestrator. It calls copilot_stats.py (credits) and session_log.py (tokens) on the agent's behalf and writes the result into a SQLite database at ~/.copilot-telemetry/telemetry.db (override with COPILOT_TELEMETRY_DIR). One row = one iterative-prompt UPD run. The AI agent never parses the log or talks to GitHub itself — it just runs two commands.

Two-phase workflow

T=./modules/084-copilot-usage-telemetry/tools/scripts/usage_track.py

# 1) START of a UPD — captures start time + start credits, prints a MARKER.
python $T begin "UPD7"
#   -> run_id: 12 ; marker: FG8FBJ7EV547HBJEH

# 2) The agent ECHOES that marker into the chat (one line) so it lands in the
#    current session's debug log. This is what lets `end` find the right log.

# 3) END of the UPD — after RESULT is written and committed. Captures end
#    credits, resolves the log BY MARKER, extracts tokens/model/context.
python $T end 12 --write-min 6 --read-min 20

--write-min / --read-min are the human minutes the user reports (time spent writing the prompt and reading the result) — the only fields the logs cannot supply. The agent asks the user for these during the report.

Why the marker

UPD5 showed "most recent log" is unreliable (any keystroke in any chat bumps a different log's mtime). begin prints a unique random string; the agent echoes it; end scans the most-recent logs for that exact string — the log that contains it is provably THIS session's log. No guessing.

Other commands

python $T refresh 12              # re-settle a run once later runs/log have flushed
python $T list                    # recent runs (id, status, label, spent, stale, reqs, peak ctx, out, model)
python $T list --workspace <path> # only runs from one workspace
python $T show 12                 # full JSON record of one run
python $T export --format md      # the UPD1 tracking table, ready to paste
python $T export --format csv     # same, as CSV

What the token columns mean

Per-request inputTokens in the log is the cumulative context sent that turn, so summing it across a run is meaningless (it N-counts the context). The row therefore stores:

• input_tokens = peak context — the high-water mark of inputTokens across the run (how big the conversation got). context_start / context_finish are the first/last values.
• output_tokens = sum of outputTokens (genuinely new tokens the model produced — this one IS additive).
• llm_requests = number of llm_request events in the run.
• work_started_at / work_finished_at = first/last llm_request timestamps (real agent work span), as opposed to started_at / ended_at which are wall-clock when you ran begin / end (these include your think/read time and Q&A lag).

Each run is bounded by the next run's marker, so a closed run never scans past the next begin and swallows a later run's events.

Settling credits — run refresh after the next run starts

Because the credit endpoint is eventually consistent and cumulative (see §1), the cleanest spend number arrives only after the next run's begin. Workflow: end records a best-effort (often stale=yes) number now; once you start the next UPD, run refresh <previous_run_id> and it back-fills the exact spend from the next run's start reading.

Known limitation

When end runs inside the same turn, the final agent_response may not be flushed to the log yet, so response_text can be empty and output_tokens may be slightly short. Run refresh <run_id> afterwards (next turn) to top it up. Credits are likewise often stale at end time — refresh settles them. context_max is the per-request response budget (maxTokens), not the model's full context window — the log does not expose the latter.

---

Agent integration block

This is the block an agent should follow to record telemetry on every run. Other agents (e.g. iterative-prompt) reference this section instead of copying the steps inline.

If modules/084-copilot-usage-telemetry/tools/scripts/usage_track.py exists, track each UPD run:

1. At the start of a UPD, before doing the work, run usage_track.py begin "UPD<N>". It prints a unique marker. Echo that marker on its own line into the chat so it lands in the session log (this is how the run is later matched to the right log).
2. At the end of the UPD, after the ### RESULT is written and committed, ask the user two quick numbers — minutes spent writing the prompt and minutes spent reading the result — then run usage_track.py end <run_id> --write-min <W> --read-min <R>.

The two marker echoes (begin/end) bracket the run inside the log. Never block the UPD on telemetry — if a telemetry command fails, note it and continue.

---

Installing this telemetry into another agent

To bolt usage tracking onto the iterative-prompt agent — or any other agent that has a recurring run boundary — add a small optional block that points at the Agent integration block above. Keep it optional so the agent still works when this module is absent.

Add to the agent's instruction file (e.g. .github/agents/<name>.agent.md):

## Usage telemetry (optional)

If `modules/084-copilot-usage-telemetry/tools/scripts/usage_track.py` exists,
follow the **Agent integration block** in
`modules/084-copilot-usage-telemetry/tools/SKILL.md` to record one telemetry
row per run (`begin` + echo marker at the start, `end` after the commit).

Why a reference and not the full steps: the agent file stays short, and the canonical procedure lives in one place (this SKILL) so updates apply everywhere.

---

Security

• The real .env is gitignored; only .env.example is committed.
• copilot_stats.py never prints the token (only a non-secret fingerprint on auth errors).
• Session logs may contain file contents, paths, and pasted secrets — treat exports like server logs.