mk-observe

# mk-observe — agent harness observability Reads the native session stores each harness already writes, normalises them into one run model, scores them, **keeps the processed runs in its own store** (`~/.mk-observe/store/`) and shows the result in the terminal or a local web UI. Plain Node, no dependencies, no LLM call anywhere, nothing sent off the machine, harness stores are opened read-only. ```bash node .claude/skills/observe/scripts/observe.cjs daemon install # machine-level: copies the skill to ~/.mk-observe/app, always-on UI for ALL projects at http://127.0.0.1:3467 node .claude/skills/observe/scripts/observe.cjs daemon status # installed? running? pid, app copy version, log path node .claude/skills/observe/scripts/observe.cjs store status # what the persistent store holds (runs, archived, size) node .claude/skills/observe/scripts/observe.cjs serve # one-off UI for the current project node .claude/skills/observe/scripts/observe.cjs stats --days 14 # terminal twin of the UI node .claude/skills/observe/scripts/observe.cjs runs # one line per run node .claude/skills/observe/scripts/observe.cjs run <id-prefix> # event timeline of one run node .claude/skills/observe/scripts/observe.cjs export > obs.json ``` Options: `--cwd <project>` (default: current dir), `--all` (every project any harness knows), `--days N` (default 30), `--harness claude-code,opencode,kiro`, `--json`, `--port`, `--interval <s>`, `--no-open`; `daemon install --in-place` (run from this checkout instead of the copy), `--skill` (symlink `~/.claude/skills/observe` so `/mk-observe` exists in every project). ## How it runs | Mode | Started by | What happens | |---|---|---| | `stats` / `runs` / `run` / `export` | the developer (or the agent, when asked) | one pass over the stores, prints, exits | | `serve` | the developer | UI for one project; collects when a request arrives, 15 s cache; stops with the terminal | | `daemon install` | the developer, once per machine | copies this skill to `~/.mk-observe/app/observe` (so no project checkout has to stay put) and registers a user-level service (launchd on macOS, `systemd --user` on Linux) that runs `serve --all` from there; starts at login, restarts if it dies, re-collects every 60 s in the background so the UI answers instantly. Log: `~/.mk-observe/daemon.log`. Re-run after upgrading the kit (`daemon status` says when the copy is older). `daemon uninstall` removes service + copy, keeps the store. | Nothing is triggered by the agent's own session: no hook, no prompt, no LLM. The daemon only reads files, parses a transcript again only when its size or mtime changed (a 450 MB session parses in ~1.5 s; an idle refresh is ~0.1 s), keeps only file metadata in RAM for unchanged sessions (the runs themselves live in the store), and runs at `nice 10`, so it never competes with the harness you are working in. ## Machine-level, not per project The rest of the kit is installed per project (hooks, rules and agents are project-scoped). Observe is different: it reads *every* project's store, so it lives once per machine — the daemon copy under `~/.mk-observe/app`, the store under `~/.mk-observe/store`. A project only needs the skill folder if you want `/mk-observe` inside that project; `daemon install --skill` links it into `~/.claude/skills` instead, once for all projects. ## Persistent store Harnesses clean up after themselves (Claude Code deletes transcripts after `cleanupPeriodDays`, 30 by default). Every pass — CLI, `serve` or the daemon — writes what it processed into the store: | Path | Holds | |---|---| | `~/.mk-observe/store/index.json` | one summary row per run (scores, counts, hook stats) — what `stats` and the UI aggregate | | `~/.mk-observe/store/runs/<id>.json` | the full run with its event timeline — what `run <id>` and the UI detail view show | Analysis is always built **from the store**; live collection only adds or rewrites runs whose source changed (size/mtime for transcripts, `time_updated`/`updated_at` for the sqlite stores). A run whose source vanished is flagged **archived** and keeps counting. `store status` shows size and age; `store prune --days N` deletes runs older than N days (nothing is pruned automatically). `MK_OBSERVE_STORE=/path` relocates it. A CLI run and the daemon can share the store: the index is merged on write, never clobbered. ## Where the data comes from | Harness | Store | Tokens | Harness-reported | Interventions | Kit hooks | Extras | |---|---|---|---|---|---|---| | Claude Code | `~/.claude/projects/<cwd-slug>/*.jsonl` | input / output / cache read / cache write **per model**, per API call | none | rejected tool calls, interrupts | `.claude/hooks/.logs/hook-log.jsonl` (+ hook errors the harness itself reports) | slash commands, compactions, API errors, refusal fallbacks, PR links, effort, thinking | | opencode | `~/.local/share/opencode/opencode.db` | input / output / reasoning / cache per model | USD computed by opencode | interrupted tool calls only | none (kit ships no hooks for opencode) | lines added / removed | | Kiro CLI | `~/Library/Application Support/kiro-cli/data.sqlite3` | schema has per-turn token fields but kiro-cli ≤ 2.21 leaves them null; read when present | credits | cancelled tool uses | `hook-log.jsonl` via the kit adapter | context %, time to first chunk | **Observe never turns tokens into money.** It stores the raw counts per model, per run (and whatever number the harness itself reports — opencode's USD, Kiro's credits — as `reported`, clearly labelled). Cost is the reader's job: `Σ over models (input × p_in + output × p_out + cacheRead × p_cache_read + cacheWrite × p_cache_write)` with the price table valid for *their* contract at the time. This keeps the numbers true across harnesses, providers and price changes; a hard-coded price list would be wrong within a month. Missing data is shown as `n/a` or a neutral score with the reason in the formula. It is never faked as 0. ## How scoring works Four dimensions, 0–100, equal weight, composite = plain mean. Every number carries its formula and the run ids behind it. | Dimension | Question | Formula | |---|---|---| | Acceptance | Is the code kept? | `100 × (1 − rejected edits / edits)` (rejected = failed or hook-blocked Write/Edit) | | Success | Did the task finish? | `100 × done / ended runs` (done = no dangling tool call, checklist complete) | | Autonomy | Self-driving or rescued? | `100 × (1 − interventions / user turns)` (permission ask, rejection, interrupt) | | Reliability | Stable or flaky? | `100 × (1 − mean(tool error rate, hook block rate, abandoned rate))` | Grades are percentiles inside the fleet you actually have (harnesses, agents, models or runs): A ≥ p80, B ≥ p60, C ≥ p40, D ≥ p20. Fewer than 5 members → static bands (A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60) and the grade is marked uncalibrated (`*`). Fewer than 5 runs → low confidence. Behaviour fingerprint per run: prompt length band (short ≤ 20 words, good ≤ 250, long), task size from input tokens, retry loops (3 consecutive errors on the same tool), subagents spawned, skills used (`/mk-x` in the prompt, `SKILL.md` reads), reports written under `plans/`, checklist completion. ## Kit health, adoption, workflow, insights - **Kit health** — hook runs, blocks, crashes and durations per hook (from `hook-log.jsonl`; entries carry `sid` + `harness` since kit v2.7.0 and are otherwise matched to runs by time window). Hook errors reported by the harness itself are attributed to the hook named in the error; a `Cannot find module …/.claude/hooks/x.cjs` is flagged as *unresolved path* — the session was started from a subdirectory, so nothing was protecting it. - **Adoption** — a run "uses the kit" when it typed a kit slash command (`/mk:…` or any command matching a skill in the project's `.claude/skills/`), read a `SKILL.md`, called the Skill tool, or spawned a subagent. Observe reads the installed catalogue (`.claude/metadata.json` name + version, `skills/*/SKILL.md`, `agents/*.md`, hooks in `settings.json`) and reports: kit share of runs, commands/skills/agents/hooks by frequency, kit vs harness agents, **never-used** skills/agents/hooks, adoption per project with kit version, per week. - **Workflow & outcomes** — prompt bands, steering per run, median duration and tool calls, marathons (> 4 h or > 250 tool calls), compactions, API errors and refusal fallbacks, retry loops, abandoned runs, model switches, subagent use, weekday × hour heatmap, hot files, tool errors by tool; outcomes: `git commit` / `git push` / PR links, test commands (npm/pnpm/yarn/bun test, pytest, go test, cargo test, jest, vitest, …) and how many failed, edit-heavy runs that ran tests, files edited, reports under `plans/`. - **Insights** — threshold rules over those numbers, each with level (`bad` / `warn` / `info` / `good`), evidence, a suggestion and the run ids: kit adoption < 30 %, projects without the kit, older kit versions (per kit name), unused skills/agents, non-kit slash commands, short prompts > 40 %, steering avg > 5, compactions in > 25 % of runs, marathons, few subagents, tests in < 30 % of edit-heavy runs, test failure rate, tool error rate > 20 % on ≥ 20 calls, hook crashes / slow hooks, retry loops ≥ 10 %, abandoned > 20 %, API errors, one model ≥ 80 % of tokens, cache hit rate < 50 %, model switches, off-hours share, weakest dimension < 60. Rate rules wait for ≥ 5 runs. Rules live in `scripts/lib/metrics.cjs` (`buildInsights`) — thresholds are plain numbers, adjust them to the team. ## When the user asks - "How is the agent doing / which harness is better here" → run `stats` (or `serve`) and explain the numbers **with their formulas**. Quote the coverage limits above when relevant. - "How much did this week cost" → `stats` (Usage by model) or `export --json`; give the **token table per model** and the formula above, and let the user apply their own price list. Do not invent prices. - "Is the team using the kit / what should we improve" → `stats --all` (Adoption, Workflow & outcomes, Insights) or the Insights tab; read the findings top-down (bad → warn → info → good) and quote the evidence and run ids. - "Why is autonomy low" → `runs`, then `run <id>` on the worst ones; interventions are listed as `permission_ask` events. - "Are the hooks working" → the Kit health section; a hook with crashes > 0 or no runs at all is the finding. - "Is the old data still there / the session got cleaned up" → `store status`; archived runs are listed like any other, with an `archived` pill in the UI. - Do not edit the harness stores. Only `~/.mk-observe/store` is written, and only by observe itself.