| name | kelly-agent-observability |
| description | Agent Fleet Observability Desk App-in-Skill — local mock dashboard visualizing a fleet of LLM agents running behind a shared AI gateway for your organization. Use when the user invokes $kelly-agent-observability or /kelly-agent-observability, wants to review agent health, call volume, latency, error rates, cost, or trace-level chain breaks across an agent fleet. This is a demo/reference dashboard over generated mock telemetry, not a live monitoring integration. |
Agent Fleet Observability Desk
Overview
Use this skill as a local operator surface over a MOCK fleet of LLM agents
running behind a shared AI gateway for a generic organization. It aggregates
per-agent call volume, latency (p50/p95), error/failure rate, and cost into one
file-backed App-in-Skill dashboard with a Fleet Overview, a sortable Agent
Health table, and a Trace Detail view that highlights exactly where a chain of
tool calls broke.
Default interaction mode: App UI. Unless the user explicitly asks for chat-only
handling, seed mock telemetry if missing, start/reuse the local app with
app/start.sh, and give the actual local URL. Use chat-only mode only when the
user says "纯聊天", "chat only", "no UI", or similar.
This is a demo/reference dashboard: all data is generated locally by
scripts/generate_fleet_data.ts. There is no real gateway, no real agents, and
no external network calls anywhere in this skill.
App UI Screenshots
 |  |
Fleet Overview
Total calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid. | Agent Health
Sortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent. |
 |  |
Trace Detail
Ordered step timeline for one trace; the step where the chain broke is visually flagged. | Chinese UI
Full zh-CN chrome via the language toggle or lang=zh. |
Boundary
- Local, read-only mock dashboard. The app reads and writes local files only.
- The only mutating endpoint is
POST /api/handoffs, which appends a
human-in-the-loop acknowledgement or investigation note to a local
app/.data/handoffs.jsonl file. It never calls any remote system, agent
gateway, or third-party API.
- Never reference any real company, brand, or production system. Agent names
are generic archetypes (Booking Assistant, Support Triage, etc.) for a
generic organization ("your organization", "the fleet").
Local App
Start the dashboard with:
skills/kelly-agent-observability/app/start.sh
First run installs hono and @hono/node-server, then seeds mock telemetry
into app/.data/fleet.json if it doesn't already exist. The frontend is
zero-build vanilla JS/HTML/CSS. The app uses local HTTP on 127.0.0.1,
preferring port 3100 through 4100, or KELLY_AGENT_OBS_UI_PORT when set.
To regenerate mock telemetry at any time:
node skills/kelly-agent-observability/scripts/generate_fleet_data.ts
Demo Mode
?demo=1 opens a deterministic, fully offline mock fleet (8 agents, 48h of
hourly buckets, ~16 traces per agent) for documentation and screenshots,
without touching app/.data/fleet.json.?demo=agents and ?demo=trace select named mock scenes (agent health table,
a broken trace).lang=en or lang=zh forces UI chrome language for screenshots.- Demo API responses never read or write local files.
UI language: support English and Chinese chrome with Auto default.
Data Model
Read references/fleet-schema.md before editing the app, scripts, or the
generator. Primary local files:
app/.data/fleet.json: generated mock telemetry (agents, per-agent metrics,
hourly buckets, traces). Regenerated by scripts/generate_fleet_data.ts.app/.data/handoffs.jsonl: append-only human-in-the-loop handoff log written
by POST /api/handoffs.
Views
#/overview: total calls (24h), total cost (today), degraded/critical/
healthy agent counts, and a per-agent card grid with a call-volume sparkline.#/agents: sortable Agent Health table (call volume, p50/p95 latency, error
rate, cost, status badge).#/agents/<agent_id>: agent detail — metrics, sparkline, recent traces, and
a handoff form (acknowledge / needs investigation).#/traces/<trace_id>: trace detail — ordered step timeline with the
chain-break step visually flagged, and a handoff form.#/handoffs: history of recorded handoffs.#/settings: data provider and status-threshold summary.
Safety
- Never invent a real incident or real company data; this is illustrative mock
telemetry only.
- Keep all writes local and file-based (
app/.data/handoffs.jsonl). No
external network calls anywhere in this skill.
- Redact nothing sensitive is collected here — there are no credentials in
this skill.
Execution reports
Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.
