skill-health

> **${var}** — View selector. > - **empty** → health check across all scheduled skills (default). > - a **skill slug** (e.g. `token-movers`) → health check for that one skill. > - `analytics` or `metrics` (optionally `analytics:HOURS`, e.g. `metrics:72`) → fleet metrics view over the last HOURS (default 168 = 7d, cap 720). > - a bare **integer** (e.g. `168`) → metrics view with that window in hours (legacy shorthand). <!-- autoresearch: variation C — more robust: memory/issues integration per CLAUDE.md health-skill contract, state-change-gated notifications, graceful missing-data; folds in B's TL;DR+action-directives+top-5 and A's skill-runs fallback. Analytics view absorbed from skill-analytics: ranked fleet view, exit-taxonomy distribution, significance-gated notify + article + dashboard JSON. --> ## Overview This skill provides two views over the same GitHub-Actions skill-run data. They share a preamble but branch into distinct logic: - **health** (default): per-skill classification, issue filing/resolution against `memory/issues/`, and a state-change-gated notification. This is the load-bearing self-healing view — its issue contract, `memory/skill-health/` scoring, and `### skill-health` log shape are depended on by the health loop and other skills. Do not weaken it. - **analytics** (metrics): a fleet-wide ranked view — top runners, failure rates, exit-taxonomy distribution, silent-scheduled detection, and anomaly flags — with a significance-gated notification plus an article and a dashboard JSON spec. `heartbeat` gives binary ok/not-ok per run and the health view audits skills one degradation-band at a time; the analytics view is the only place the operator sees the entire fleet ranked side-by-side. ## Shared preamble (run for either view) 1. Read `memory/MEMORY.md` for high-level context and scan the last ~3 days of `memory/logs/` for recent activity — drop anything already reported so you don't re-report the same signal. 2. Compute `${today}` (UTC date, `YYYY-MM-DD`). 3. **Parse `${var}` → selector** (trim whitespace first): - **empty** → `VIEW=health`, `TARGET=all` (all scheduled skills). - lowercase first token is `analytics` or `metrics` → `VIEW=analytics`. Parse an optional window argument after a `:` or a space (`analytics:72`, `metrics 336`): if it is a positive integer, `WINDOW_HOURS = min(that, 720)`; otherwise `WINDOW_HOURS = 168`. - a bare positive integer (e.g. `168`) → `VIEW=analytics`, `WINDOW_HOURS = min(that, 720)` (legacy skill-analytics shorthand). - anything else (a non-keyword, non-integer slug) → `VIEW=health`, `TARGET=<that slug>` (single-skill health check). 4. Dispatch: if `VIEW=health`, run **Health view**; if `VIEW=analytics`, run **Analytics view**. --- # Health view `VIEW=health`. Audit skill quality metrics, detect API degradation, **file issues for new failures and resolve them when skills recover**, and notify only when fleet health state actually changes. If `TARGET` is a single skill slug, only check that skill. ## Data sources 1. **`memory/cron-state.json`** — Per-skill quality metrics (as before). 2. **`memory/skill-health/*.json`** — Per-skill quality analysis (Haiku post-run). 3. **`memory/skill-health/last-report.json`** — Last run's classification snapshot (this skill writes it). Used to dedup notifications and detect flapping. 4. **`aeon.yml`** — Enabled skills and schedules. 5. **`memory/issues/INDEX.md`** and `memory/issues/ISS-*.md` — Open issues tracker. Check before filing, update on recovery. 6. **`./scripts/skill-runs --hours 168 --failures --json`** — Fallback source for failures that never wrote to cron-state (runs that crashed before writing, etc.). Run once, parse JSON. 7. **`memory/logs/YYYY-MM-DD.md`** (last 3 days) — Grep for `SKILL_*_ERROR` or `EMPTY` signatures keyed to skills missing from skill-health/*.json. ## Steps ### 1. Gather state - Parse `aeon.yml` → list of enabled skills with schedules. If `TARGET` is a single skill, filter to just that skill. - Load `memory/cron-state.json` (if missing or unparseable, treat as empty — first run, not failure). - Load every `memory/skill-health/*.json` (except `last-report.json`). - Load `memory/skill-health/last-report.json` if present → `prev_report`. If missing, `prev_report = {}`. - Run `./scripts/skill-runs --hours 168 --failures --json 2>/dev/null || echo '{}'` → extract any skill with failures in the last 7d that isn't in cron-state (runs that failed before writing state). - Parse `memory/issues/INDEX.md` → extract open issues with `detected_by: skill-health` and their affected skills. If missing, treat as empty. ### 2. Classify each enabled skill For each enabled skill, assign one status using the **first matching rule**: | Status | Trigger | |---|---| | **CRITICAL** | `consecutive_failures >= 3` OR (status==failed AND days_since_last_success >= 3) | | **DEGRADED** | `success_rate < 0.6` OR (latest `skill-health/*.json` avg_score < 2.5 over ≥3 runs) | | **FLAPPING** | 3+ status transitions (success↔failed) in last 7 days per cron-state history *or* `skill-runs` output | | **WARNING** | `success_rate < 0.8` OR `consecutive_failures >= 1` | | **HEALTHY** | `success_rate >= 0.8` AND `consecutive_failures == 0` AND (no skill-health data OR avg_score >= 3) | | **NO DATA** | no entry in cron-state AND never seen in skill-runs | Compute **severity score** for sorting: `consecutive_failures × (1 + days_since_last_success/7)`. Ties broken by days_since_last_success desc. For each CRITICAL/DEGRADED/FLAPPING skill, record: - `last_error` (from cron-state or nearest log signature) - `api_host` if the error clearly names one (e.g. `api.coingecko.com`, `api.github.com`) - `suggested_action` — one of: `FIX CONFIG` (missing secret, bad arg), `WAIT-API` (rate limit, 5xx, timeout on third-party host), `INVESTIGATE` (unrecognised error), `DISPATCH-SKILL` (NO DATA but scheduled — scheduler gap) ### 3. Detect systemic patterns Group non-HEALTHY skills by shared `api_host` OR shared `last_error` signature. If ≥2 skills share one: - Emit a single `SYSTEMIC:` callout (e.g. `SYSTEMIC: 3 skills failing on api.coingecko.com (rate_limit)`). - Do **not** duplicate the same error across per-skill rows — reference the systemic line. ### 4. Reconcile with memory/issues/ **Precondition guard:** only perform issue filing/resolution if `memory/issues/INDEX.md` already exists. If it is missing, the operator has not opted into the issue-tracker contract yet — log `SKILL_HEALTH_ISSUE_TRACKER_MISSING` to `memory/logs/${today}.md`, skip this entire step (and the reconciliation side of step 5), and continue with classification + notification only. Do **not** auto-create `INDEX.md`. For each CRITICAL or FLAPPING skill, check if an issue with `status: open` or `status: fix-pending` already has this skill in `affected_skills` AND a matching `root_cause` signature: - **Matching issue exists, same root cause** → do nothing (no new file, no notification for this skill). - **Matching issue exists, different root cause** → append a note to the existing ISS file's body: `Update YYYY-MM-DD: new signature: <error>`. Do not file a new issue. - **No matching issue** → file a new one (see below). **Reconcile `fix-pending` issues first.** `skill-repair` sets `status: fix-pending` with a `fix_pr` when it opens a repair PR - it cannot know whether that PR merges, so it deliberately does not claim `resolved`. This step closes that loop. For each issue with `status: fix-pending` and a non-null `fix_pr`, check the PR's real state (`gh pr view <fix_pr> --json state,mergedAt`) and reconcile: - **Merged** (`mergedAt` non-null) → the fix shipped. Set `status: resolved`, `resolved_at: <mergedAt>`, move the row from Open to Resolved in INDEX.md. - **Closed without merging** (`state: CLOSED`, `mergedAt: null`) → the fix did not ship. Set `status: open`, set `fix_pr: null`, and append `Update <YYYY-MM-DD>: fix PR <url> was closed unmerged; issue reopened.` to the body. The skill is still broken; leaving it `fix-pending` would hide that. - **Still open** → leave untouched. The repair is in flight. If `gh` is unavailable or the lookup errors, leave the issue untouched and log it - never resolve on an unverified assumption. For each skill now HEALTHY whose name appears in any `status: open` issue's `affected_skills`: - Skip `status: fix-pending` issues. Those wait for the reconcile step above; a lucky HEALTHY classification must not close a repair that has not merged. - Remove the skill from that issue's `affected_skills`. If the list becomes empty, set `status: resolved`, set `resolved_at: <now ISO>`, and move the row from Open to Resolved in INDEX.md. For each skill in a `status: open` **critical** issue that is still DEGRADED or WARNING only because of historical metrics, pipe its cron-state JSON object to `python3 scripts/skill_health_recovery.py '<detected_at>'`. If it prints `recovered`, remove the skill from `affected_skills` and resolve an empty issue exactly as above. A successful run after detection proves that specific failure incident recovered even when lifetime `success_rate` remains low. Do not apply this shortcut to FLAPPING/high issues (one successful run does not prove flapping stopped) or to `status: fix-pending` issues (those wait for the reconcile step above). Invalid or missing state/timestamps print `active` and fail closed. **Filing a new issue:** 1. Find next ID: scan `memory/issues/ISS-*.md`, take max `NNN`, add 1. Format as zero-padded 3 digits (`ISS-042`). 2. Write `memory/issues/ISS-NNN.md` with YAML frontmatter: ```yaml --- id: ISS-NNN title: <skill> <concise failure> status: open # open | fix-pending | resolved. fix-pending = repair PR open, unmerged. Health resolves fix-pending only after merge (or reopens if closed unmerged). HEALTHY recovery still resolves status: open issues with no pending PR. severity: critical | high | medium | low # critical=CRITICAL status, high=FLAPPING, medium=DEGRADED category: rate-limit | timeout | missing-secret | config | api-change | sandbox-limitation | unknown detected_by: skill-health detected_at: <ISO-8601 UTC, e.g. 2026-09-03T12:00:00Z> # must carry Z or a +00:00 offset; a naive stamp fails closed to active affected_skills: [<skill>, ...] # may grow later root_cause: <error signature, 1 line> fix_pr: null --- ## What happened <2-3 line summary> ## Signal - consecutive_failures: N - days_since_last_success: N - last_error: "<error>" - related skills: <list or "none"> ``` 3. Append a row to `memory/issues/INDEX.md` under **Open**: `| ISS-NNN | title | severity | category | YYYY-MM-DD | skill-a, skill-b |`. All issue writes must be atomic per file — never partial updates mid-run. ### 5. Decide whether to notify Build a stable signature from the current classification: sorted list of `CRITICAL+FLAPPING+DEGRADED skill names + SYSTEMIC callouts`. SHA-256 it → `current_hash`. - If `current_hash == prev_report.hash` AND `now - prev_report.last_notified_at < 24h` → **do not notify**. State unchanged. - Otherwise → **notify** (there's new signal or the daily reminder cadence elapsed). Always write `memory/skill-health/last-report.json`: ```json { "hash": "<current_hash>", "last_notified_at": "<ISO if notified this run, else previous value>", "last_run_at": "<ISO now>", "classification": { "critical": [...], "degraded": [...], "flapping": [...], "warning": [...], "healthy_count": N, "no_data": [...] } } ``` ### 6. Format the report **Top line:** `HEALTH: OK` | `HEALTH: WARNING(W)` | `HEALTH: DEGRADED(D)` | `HEALTH: CRITICAL(C)` — most severe wins. **Body (notify-channel format, max 1 message):** ``` *Skill Health — ${today}* HEALTH: CRITICAL(2) [systemic: api.coingecko.com rate_limit — 3 skills] 🔴 CRITICAL - token-movers — 5 fails, 3d down — WAIT-API (rate_limit) → ISS-042 - defi-monitor — 4 fails, 2d down — WAIT-API (rate_limit) → ISS-042 🟡 DEGRADED / FLAPPING - digest — 52% success (14d), avg quality 2.1 — INVESTIGATE → ISS-043 ⚪ NO DATA (2): skill-x, skill-y — DISPATCH-SKILL 🟢 HEALTHY: 34 Open issues: 2 · Resolved this run: 1 (rss-digest) ``` Rules for formatting: - Cap per-section rows at 5; collapse the rest as `+N more — see memory/issues/INDEX.md`. - Omit HEALTHY list (count only). Omit any empty section. - Always end with `Open issues: X · Resolved this run: Y`. - If NO CRITICAL/DEGRADED/FLAPPING and no new/resolved issues → body is just `HEALTH: OK — N skills healthy`. ### 7. Notify and log - If the gate in step 5 said notify → `./notify "<report body>"`. Update `last_notified_at` in last-report.json to now. - If gate said skip → do not call `./notify`. Log to memory/logs/${today}.md: ``` ### skill-health - view: health - SKILL_HEALTH_NOOP — state unchanged since <prev_run_at>, hash=<short> ``` On notify, log to memory/logs/${today}.md: ``` ### skill-health - view: health - HEALTH: <OK|WARNING|DEGRADED|CRITICAL> - filed: [ISS-NNN, ...] - resolved: [ISS-NNN, ...] - open: N - systemic: <pattern or none> ``` If all skills healthy, the body-only shortcut from step 6 still fires (once per 24h, per gate) so the operator gets confirmation the audit actually ran — but suppress if last-report.json shows a notify <24h ago with the same OK hash. ## Health-view constraints - Never file two open issues for the same `(skill, root_cause)` pair — always check INDEX.md first. - Never edit a Resolved issue. If a previously-resolved issue re-fires, file a new ISS with a pointer (`related: ISS-NNN`) in the body. - Do not notify on pure HEALTHY runs more than once per 24h. - If in single-skill mode (selector was a skill slug), skip INDEX.md updates only if the single skill is HEALTHY — otherwise file/resolve as normal. - Never touch `memory/issues/INDEX.md` Resolved section except to move rows into it; never delete rows. --- # Analytics view `VIEW=analytics`. Generate a fleet-level performance view of every Aeon skill that has run in the window. **The point of this view is to answer four questions in one report:** which skills run most, which fail most, which are silently skipping (new exit taxonomy from the autoresearch-evolution rewrites), and which scheduled skills haven't fired at all. ## Why this exists