copilot-metrics-tracker

name	copilot-metrics-tracker
description	Use when tracking token usage and graphify impact during a Copilot + Obsidian pilot

Copilot Metrics Tracker

Overview

Manual tracking template for measuring Graphify + Obsidian impact on GitHub Copilot token usage and response quality during a pilot.

When to Use

• Running a pilot of Graphify + Obsidian + Copilot
• Need before/after metrics for token usage
• Want to verify Copilot is using vault context

Quick Reference

Metric	Where to Find It
Token count	VS Code status bar (bottom right, Copilot icon) — shows "X tokens used"
Context source	Copilot Chat response → "Used X sources" link at bottom — click to see what it read
Response quality	Manual 1-5 rating per response

Copilot Usage Display Forms

Copilot shows usage differently depending on context:

• Status bar: Token count updates in real-time during generation
• Chat responses: "Used X sources" link at bottom of each response — click to expand
• Inline completions: Token count shown after completion appears
• /memory command: Shows what Copilot has stored for the current repo

To track before/after:

1. Note token count in status bar before graphify --update
2. Run graphify . --obsidian ~/obsidian-code-vault --update
3. Ask Copilot a question
4. Check "Used X sources" — did it include vault notes?
5. Note new token count

Tracking Template

Copy this to a note in your vault or a spreadsheet:

## Pilot Metrics - [Date]

### Session 1
- Tokens before graphify update: ___
- Tokens after graphify update: ___
- Copilot used vault context: yes/no
- Response quality (1-5): ___
- Notes: ___

### Session 2
- Tokens before graphify update: ___
- Tokens after graphify update: ___
- Copilot used vault context: yes/no
- Response quality (1-5): ___
- Notes: ___

What to Measure

1. Token usage — Check Copilot status bar before and after graphify --update
2. Context hits — Does Copilot reference vault notes? (Check context panel in Chat)
3. Response accuracy — Rate responses on whether they understand project-specific context
4. Onboarding time — How long to get useful answers about an unfamiliar codebase

Common Pitfalls

• Token count fluctuates per session — compare similar tasks
• Copilot may use both vault AND raw files — check context panel
• First graphify run takes longer — incremental updates are faster

After the Pilot

Summarize findings in docs/pilot-results.md:

• Total tokens saved (estimate)
• Responses where vault context helped
• Responses where vault context was missing
• Recommendations for team adoption