skill-health

name	skill-health
description	Fleet skill observability with two views - health audits per-skill metrics and files/resolves issues in memory/issues/; analytics ranks the fleet by 7d runs, success rates, and anomaly flags.
scorable	false
metadata	{"title":"Skill Health","category":"evolution","var":"","tags":["meta"]}

${var} — View selector.

• empty → health check across all scheduled skills (default).
• a skill slug (e.g. token-movers) → health check for that one skill.
• analytics or metrics (optionally analytics:HOURS, e.g. metrics:72) → fleet metrics view over the last HOURS (default 168 = 7d, cap 720).
• a bare integer (e.g. 168) → metrics view with that window in hours (legacy shorthand).

Overview

This skill provides two views over the same GitHub-Actions skill-run data. They share a preamble but branch into distinct logic:

• health (default): per-skill classification, issue filing/resolution against memory/issues/, and a state-change-gated notification. This is the load-bearing self-healing view — its issue contract, memory/skill-health/ scoring, and ### skill-health log shape are depended on by the health loop and other skills. Do not weaken it.
• analytics (metrics): a fleet-wide ranked view — top runners, failure rates, exit-taxonomy distribution, silent-scheduled detection, and anomaly flags — with a significance-gated notification plus an article and a dashboard JSON spec. heartbeat gives binary ok/not-ok per run and the health view audits skills one degradation-band at a time; the analytics view is the only place the operator sees the entire fleet ranked side-by-side.

Shared preamble (run for either view)

1. Read memory/MEMORY.md for high-level context and scan the last ~3 days of memory/logs/ for recent activity — drop anything already reported so you don't re-report the same signal.
2. Compute ${today} (UTC date, YYYY-MM-DD).
3. Parse ${var} → selector (trim whitespace first):
   • empty → VIEW=health, TARGET=all (all scheduled skills).
   • lowercase first token is analytics or metrics → VIEW=analytics. Parse an optional window argument after a : or a space (analytics:72, metrics 336): if it is a positive integer, WINDOW_HOURS = min(that, 720); otherwise WINDOW_HOURS = 168.
   • a bare positive integer (e.g. 168) → VIEW=analytics, WINDOW_HOURS = min(that, 720) (legacy skill-analytics shorthand).
   • anything else (a non-keyword, non-integer slug) → VIEW=health, TARGET=<that slug> (single-skill health check).
4. Dispatch: if VIEW=health, run Health view; if VIEW=analytics, run Analytics view.

---

Health view

VIEW=health. Audit skill quality metrics, detect API degradation, file issues for new failures and resolve them when skills recover, and notify only when fleet health state actually changes. If TARGET is a single skill slug, only check that skill.

Data sources

1. memory/cron-state.json — Per-skill quality metrics (as before).
2. memory/skill-health/*.json — Per-skill quality analysis (Haiku post-run).
3. memory/skill-health/last-report.json — Last run's classification snapshot (this skill writes it). Used to dedup notifications and detect flapping.
4. aeon.yml — Enabled skills and schedules.
5. memory/issues/INDEX.md and memory/issues/ISS-*.md — Open issues tracker. Check before filing, update on recovery.
6. ./scripts/skill-runs --hours 168 --failures --json — Fallback source for failures that never wrote to cron-state (runs that crashed before writing, etc.). Run once, parse JSON.
7. memory/logs/YYYY-MM-DD.md (last 3 days) — Grep for SKILL_*_ERROR or EMPTY signatures keyed to skills missing from skill-health/*.json.

Steps

1. Gather state

• Parse aeon.yml → list of enabled skills with schedules. If TARGET is a single skill, filter to just that skill.
• Load memory/cron-state.json (if missing or unparseable, treat as empty — first run, not failure).
• Load every memory/skill-health/*.json (except last-report.json).
• Load memory/skill-health/last-report.json if present → prev_report. If missing, prev_report = {}.
• Run ./scripts/skill-runs --hours 168 --failures --json 2>/dev/null || echo '{}' → extract any skill with failures in the last 7d that isn't in cron-state (runs that failed before writing state).
• Parse memory/issues/INDEX.md → extract open issues with detected_by: skill-health and their affected skills. If missing, treat as empty.

2. Classify each enabled skill

For each enabled skill, assign one status using the first matching rule:

Status	Trigger
CRITICAL	consecutive_failures >= 3 OR (status==failed AND days_since_last_success >= 3)
DEGRADED	success_rate < 0.6 OR (latest skill-health/*.json avg_score < 2.5 over ≥3 runs)
FLAPPING	3+ status transitions (success↔failed) in last 7 days per cron-state history or skill-runs output
WARNING	success_rate < 0.8 OR consecutive_failures >= 1
HEALTHY	success_rate >= 0.8 AND consecutive_failures == 0 AND (no skill-health data OR avg_score >= 3)
NO DATA	no entry in cron-state AND never seen in skill-runs

Compute severity score for sorting: consecutive_failures × (1 + days_since_last_success/7). Ties broken by days_since_last_success desc.

For each CRITICAL/DEGRADED/FLAPPING skill, record:

• last_error (from cron-state or nearest log signature)
• api_host if the error clearly names one (e.g. api.coingecko.com, api.github.com)
• suggested_action — one of: FIX CONFIG (missing secret, bad arg), WAIT-API (rate limit, 5xx, timeout on third-party host), INVESTIGATE (unrecognised error), DISPATCH-SKILL (NO DATA but scheduled — scheduler gap)

3. Detect systemic patterns

Group non-HEALTHY skills by shared api_host OR shared last_error signature. If ≥2 skills share one:

• Emit a single SYSTEMIC: callout (e.g. SYSTEMIC: 3 skills failing on api.coingecko.com (rate_limit)).
• Do not duplicate the same error across per-skill rows — reference the systemic line.

4. Reconcile with memory/issues/

Precondition guard: only perform issue filing/resolution if memory/issues/INDEX.md already exists. If it is missing, the operator has not opted into the issue-tracker contract yet — log SKILL_HEALTH_ISSUE_TRACKER_MISSING to memory/logs/${today}.md, skip this entire step (and the reconciliation side of step 5), and continue with classification + notification only. Do not auto-create INDEX.md.

For each CRITICAL or FLAPPING skill, check if an issue with status: open or status: fix-pending already has this skill in affected_skills AND a matching root_cause signature:

• Matching issue exists, same root cause → do nothing (no new file, no notification for this skill).
• Matching issue exists, different root cause → append a note to the existing ISS file's body: Update YYYY-MM-DD: new signature: <error>. Do not file a new issue.
• No matching issue → file a new one (see below).

Reconcile fix-pending issues first. skill-repair sets status: fix-pending with a fix_pr when it opens a repair PR - it cannot know whether that PR merges, so it deliberately does not claim resolved. This step closes that loop. For each issue with status: fix-pending and a non-null fix_pr, check the PR's real state (gh pr view <fix_pr> --json state,mergedAt) and reconcile:

• Merged (mergedAt non-null) → the fix shipped. Set status: resolved, resolved_at: <mergedAt>, move the row from Open to Resolved in INDEX.md.
• Closed without merging (state: CLOSED, mergedAt: null) → the fix did not ship. Set status: open, set fix_pr: null, and append Update <YYYY-MM-DD>: fix PR <url> was closed unmerged; issue reopened. to the body. The skill is still broken; leaving it fix-pending would hide that.
• Still open → leave untouched. The repair is in flight.

If gh is unavailable or the lookup errors, leave the issue untouched and log it - never resolve on an unverified assumption.

For each skill now HEALTHY whose name appears in any status: open issue's affected_skills:

• Skip status: fix-pending issues. Those wait for the reconcile step above; a lucky HEALTHY classification must not close a repair that has not merged.
• Remove the skill from that issue's affected_skills. If the list becomes empty, set status: resolved, set resolved_at: <now ISO>, and move the row from Open to Resolved in INDEX.md.

For each skill in a status: open critical issue that is still DEGRADED or WARNING only because of historical metrics, pipe its cron-state JSON object to python3 scripts/skill_health_recovery.py '<detected_at>'. If it prints recovered, remove the skill from affected_skills and resolve an empty issue exactly as above. A successful run after detection proves that specific failure incident recovered even when lifetime success_rate remains low. Do not apply this shortcut to FLAPPING/high issues (one successful run does not prove flapping stopped) or to status: fix-pending issues (those wait for the reconcile step above). Invalid or missing state/timestamps print active and fail closed.

Filing a new issue:

1. Find next ID: scan memory/issues/ISS-*.md, take max NNN, add 1. Format as zero-padded 3 digits (ISS-042).
2. Write memory/issues/ISS-NNN.md with YAML frontmatter:

   ---
   id: ISS-NNN
   title: <skill> <concise failure>
   status: open   # open | fix-pending | resolved. fix-pending = repair PR open, unmerged. Health resolves fix-pending only after merge (or reopens if closed unmerged). HEALTHY recovery still resolves status: open issues with no pending PR.
   severity: critical | high | medium | low   # critical=CRITICAL status, high=FLAPPING, medium=DEGRADED
   category: rate-limit | timeout | missing-secret | config | api-change | sandbox-limitation | unknown
   detected_by: skill-health
   detected_at: <ISO-8601 UTC, e.g. 2026-09-03T12:00:00Z>   # must carry Z or a +00:00 offset; a naive stamp fails closed to active
   affected_skills: [<skill>, ...]    # may grow later
   root_cause: <error signature, 1 line>
   fix_pr: null
   ---
   
   ## What happened
   <2-3 line summary>
   
   ## Signal
   - consecutive_failures: N
   - days_since_last_success: N
   - last_error: "<error>"
   - related skills: <list or "none">
   

3. Append a row to memory/issues/INDEX.md under Open: | ISS-NNN | title | severity | category | YYYY-MM-DD | skill-a, skill-b |.

All issue writes must be atomic per file — never partial updates mid-run.

5. Decide whether to notify

Build a stable signature from the current classification: sorted list of CRITICAL+FLAPPING+DEGRADED skill names + SYSTEMIC callouts. SHA-256 it → current_hash.

• If current_hash == prev_report.hash AND now - prev_report.last_notified_at < 24h → do not notify. State unchanged.
• Otherwise → notify (there's new signal or the daily reminder cadence elapsed).

Always write memory/skill-health/last-report.json:

{
  "hash": "<current_hash>",
  "last_notified_at": "<ISO if notified this run, else previous value>",
  "last_run_at": "<ISO now>",
  "classification": { "critical": [...], "degraded": [...], "flapping": [...], "warning": [...], "healthy_count": N, "no_data": [...] }
}

6. Format the report

Top line: HEALTH: OK | HEALTH: WARNING(W) | HEALTH: DEGRADED(D) | HEALTH: CRITICAL(C) — most severe wins.

Body (notify-channel format, max 1 message):

*Skill Health — ${today}*
HEALTH: CRITICAL(2)  [systemic: api.coingecko.com rate_limit — 3 skills]

🔴 CRITICAL
- token-movers — 5 fails, 3d down — WAIT-API (rate_limit) → ISS-042
- defi-monitor — 4 fails, 2d down — WAIT-API (rate_limit) → ISS-042

🟡 DEGRADED / FLAPPING
- digest — 52% success (14d), avg quality 2.1 — INVESTIGATE → ISS-043

⚪ NO DATA (2): skill-x, skill-y — DISPATCH-SKILL
🟢 HEALTHY: 34

Open issues: 2 · Resolved this run: 1 (rss-digest)

Rules for formatting:

• Cap per-section rows at 5; collapse the rest as +N more — see memory/issues/INDEX.md.
• Omit HEALTHY list (count only). Omit any empty section.
• Always end with Open issues: X · Resolved this run: Y.
• If NO CRITICAL/DEGRADED/FLAPPING and no new/resolved issues → body is just HEALTH: OK — N skills healthy.

7. Notify and log

• If the gate in step 5 said notify → ./notify "<report body>". Update last_notified_at in last-report.json to now.
• If gate said skip → do not call ./notify. Log to memory/logs/${today}.md:

  ### skill-health
  - view: health
  - SKILL_HEALTH_NOOP — state unchanged since <prev_run_at>, hash=<short>
  

On notify, log to memory/logs/${today}.md:

### skill-health
- view: health
- HEALTH: <OK|WARNING|DEGRADED|CRITICAL>
- filed: [ISS-NNN, ...]
- resolved: [ISS-NNN, ...]
- open: N
- systemic: <pattern or none>

If all skills healthy, the body-only shortcut from step 6 still fires (once per 24h, per gate) so the operator gets confirmation the audit actually ran — but suppress if last-report.json shows a notify <24h ago with the same OK hash.

Health-view constraints

• Never file two open issues for the same (skill, root_cause) pair — always check INDEX.md first.
• Never edit a Resolved issue. If a previously-resolved issue re-fires, file a new ISS with a pointer (related: ISS-NNN) in the body.
• Do not notify on pure HEALTHY runs more than once per 24h.
• If in single-skill mode (selector was a skill slug), skip INDEX.md updates only if the single skill is HEALTHY — otherwise file/resolve as normal.
• Never touch memory/issues/INDEX.md Resolved section except to move rows into it; never delete rows.

---

Analytics view

VIEW=analytics. Generate a fleet-level performance view of every Aeon skill that has run in the window. The point of this view is to answer four questions in one report: which skills run most, which fail most, which are silently skipping (new exit taxonomy from the autoresearch-evolution rewrites), and which scheduled skills haven't fired at all.

Why this exists