llm-provider-resilience

name	llm-provider-resilience
description	Use when LLM provider flakiness breaks cron jobs or probes.
version	1.0.0
author	Hermes Agent
license	MIT
platforms	["macos","linux"]
metadata	{"hermes":{"tags":["llm","provider","cron","fallback","reliability","opencode-go"],"related_skills":["cron-job-provider-migration","we-mp-rss-troubleshooting"]}}

LLM Provider Resilience

Use when agent cron jobs fail with provider-related errors, when a model provider is intermittently flaky, or when deciding between provider migration vs adding a fallback. Covers the failure taxonomy, the fallback_model config mechanism, and the verified re-run/probe patterns.

Failure taxonomy — read the cron error, classify FIRST

Symptom in cron output	Class	Meaning
TimeoutError: Cron job '<name>' idle for Ns (limit 600s) — last activity: waiting for non-streaming API response	HANG	Request sent, response never arrived. Client has no (or too-long) request timeout. The 600s cron watchdog kills the job.
RuntimeError: HTTP 429 / 401 / 5xx or Connection error	ERROR / RATE-LIMIT	Provider responded with a failure.
Job runs instantly, exit 0, but did nothing useful	FALSE-HEALTH	Script logic doesn't check its own output quantity (e.g. a collection script reporting "30/30 ok" while fetching 0 items). Not a provider issue — a script bug.

Superseded-path trap (2026-08-21, wechat-article-discover): a cron job can be broken not because its code is wrong but because its collection path was replaced. Retrying it confirms it's still broken (e.g. Done: 30/30 ok, 0 err, new_articles=0 → exit 1, all feeds return 0 in Playwright mode) — and re-enabling it produces a false daily error. Before re-enabling a previously-paused errored job, confirm its path still exists: (a) cookie-renew success (token write, container HTTP 200) does NOT imply discovery success (article-list fetch auth) — the two are separate; (b) check whether the pipeline now ingests through a mirror/other job (e.g. wechat2rss mirror via rss-feed-scan + wechat-inbox-pipeline). If superseded, keep it paused and say why.

Verify the underlying WeChat ban directly (200013 probe, 2026-08-26): to confirm WHETHER the account is still banned before deciding to re-enable wechat-article-discover, don't trust the unblock-probe's silence alone — it exits silently on BOTH ret==200013 (still banned) and probe failure. Run the probe yourself. The container's driver.token module supplies token+cookie; heredoc via docker exec ... <<'PY' is unreliable (empty output), so write the probe to a local file, docker cp it in, then run it:

# /tmp/probe_ret.py — docker cp into we-mp-rss, run with container python
import sys, requests, urllib3; urllib3.disable_warnings()
sys.path.insert(0, '/app')
from driver.token import get
token, cookie = get('token',''), get('cookie','')
url = "https://mp.weixin.qq.com/cgi-bin/appmsgpublish"
params = {"sub":"list","sub_action":"list_ex","begin":0,"count":1,
          "fakeid":"MzA3MjgyODI4MzMw","type":"101_1","free_publish_type":1,
          "token":token,"lang":"zh_CN","f":"json","ajax":1}
headers = {"User-Agent":"Mozilla/5.0","Referer":"https://mp.weixin.qq.com/"}
cd = {}
for item in cookie.split(";"):
    item = item.strip()
    if "=" in item:
        k,v = item.split("=",1); cd[k.strip()] = v.strip()
try:
    r = requests.get(url, params=params, headers=headers, cookies=cd, timeout=15, verify=False)
    print("RET=" + (r.json().get(,{}).get()))
 Exception  e:
    ( + (e))

docker cp /tmp/probe_ret.py we-mp-rss:/tmp/probe_ret.py
docker exec we-mp-rss /app/env_x86_64/bin/python3 /tmp/probe_ret.py 2>&1 | grep -E "RET=|ERROR="
# RET=200013 → still banned (keep job paused); RET=0 → unblocked (unblock-probe will auto-restore); RET=200003 → session expired

Hang vs error matters for the fix: fallback chains trigger on errors/timeouts/rate-limits, NOT on pure hangs (see below). If the class is HANG, a fallback alone may not help — you also need a client-side request timeout so the hang becomes a timeout the fallback can catch, or accept the 600s watchdog kill + next-tick retry.

fallback_model — automatic backup provider (no migration needed)

Config key in ~/.hermes/config.yaml (verified in agent/agent_init.py + agent/agent_runtime_helpers.py, 2026-08-12):

fallback_model:            # single dict OR ordered chain
  - provider: minimax-cn
    model: <model>
  - provider: deepseek
    model: deepseek-chat

• Each entry REQUIRES provider + model (config validation rejects entries missing either field).
• Chain entries tried in order; a failed primary call activates the chain for the rest of that turn.
• Rate-limit cooldown (~60s) and provider reset-time tracking gate when the primary is retried; chain index resets per turn.
• Cron agent jobs inherit the global model config → ONE global fallback_model covers all jobs. Per-job model/provider overrides in cronjob records take precedence.
• Hang caveat: fallback engages on client-visible failures. A pure hang with no client timeout never surfaces an error, so the fallback never fires — the cron watchdog kills the job first. Pair fallback config with a request timeout if hangs are the observed class.

Re-running errored agent jobs

• cronjob(action='run', job_id=...) executes the job immediately and reliably (verified 2026-08-12: 8/8 errored agent jobs re-ran to last_status: ok in one pass, plus a 9th that had re-errored on its schedule).
• last_status: error on the job list is a HISTORICAL ARTIFACT of the last execution — after a provider outage day, it does not mean the job is broken now. Re-run, then verify the OUTPUT FILE content (~/.hermes/cron/output/<job_id>/<latest>.md) shows real work, not just exit 0.
• After a mass provider-outage day, batch cronjob run 4-6 jobs per tool-call block; jobs re-fire sequentially, so rate-limit collisions are unlikely, but stagger anyway if the provider has frequency limits.

Migrating cron jobs to a new provider — cronjob update has NO model override

To point all agent cron jobs at a different provider/model, the cronjob tool's update action does NOT expose a model/provider parameter — calling it with just job_id/name is a no-op that leaves provider unchanged (verified 2026-08-19). The dependable path is a direct batch edit of ~/.hermes/cron/jobs.json, then revalidate JSON, then restart the gateway (the scheduler caches job config in memory):

import json
p = '/Users/jinguo/.hermes/cron/jobs.json'
data = json.load(open(p))
for j in data['jobs']:
    if j.get('model'):            # agent jobs only; no_agent/script jobs have model=None
        j['provider'] = 'ark'
        j['model'] = 'deepseek-v4-flash'
        j['base_url'] = 'https://ark.cn-beijing.volces.com/api/coding/v3'
json.dump(data, open(p, 'w'), ensure_ascii=False, indent=2)

Then python3 -c "import json; json.load(open(p))" to revalidate, back up jobs.json first, and verify the result via cronjob(action='list'). Full workflow detail lives in the cron-job-provider-migration skill (user-owned — recommend hermes curator adopt cron-job-provider-migration to make it curatable).

Background probe hardening — never trust signal.alarm alone

Real case (2026-08-10): a background verification docker exec we-mp-rss ... do_job(mp=feed) hung 10 HOURS past its signal.alarm(50) — SIGALRM does not reliably interrupt requests stuck in C-level sockets, and the process stayed alive holding the pipe. The scheduled result was lost.

• Wrap any scheduled/background probe in an OUTER hard timeout: gtimeout 90 docker exec ... (macOS: brew install coreutils for gtimeout), or run it with terminal(background=true) + process(action='kill') fallback.
• Never chain sleep N && probe in a background process and assume the probe terminates on its own.
• For long cooldown-window probes (hours), prefer a one-shot cron job or a foreground poll with bounded waits over a single huge sleep.

Large-context probe: a SMALL curl can fake provider health (2026-08-21, ark)

When a batch of agent-driven crons fails with RuntimeError: Request timed out but no_agent script jobs stay green, suspect provider-wide large-context degradation, not per-job bugs. Verified against ark (deepseek-v4-flash): a small probe (max_tokens=10) returned HTTP 200 in ~2s WHILE the same provider was timing out on the cron's real workload — because agent-driven cron prompts embed the full skills catalog + memory (~130K tokens), and large-context requests are the FIRST thing to degrade during a provider outage window. A green small-probe is NOT evidence of health.

• Diagnostic tell: ALL agent-driven jobs on one provider fail simultaneously with the same timeout signature; no_agent script jobs (no LLM call) stay green. That split is the signal — it points at the provider, not the jobs.
• Probe LARGE, not small: reproduce the real workload with an ~80K-char prompt + stream:true. If that returns 200 fast, the outage window has passed and jobs will recover on re-run.
• Bulk re-run: hermes cron run <job_id> per failed job. Heavy agent jobs exceed 180s foreground — trigger them with terminal(background=true). Output Job is already being fired by the scheduler; not run again means it's mid-run (my earlier trigger queued it), NOT a failed trigger.
• Verify via agent.log, not the stale list right after triggering (last_status doesn't refresh until the run completes): grep "Job '<name>' completed successfully" ~/.hermes/logs/agent.log. Heavy ingest jobs (wechat-inbox-pipeline) take 40–60 min across 70+ tool turns with ~99% prompt-cache hit — steady API-call progression in agent.log means healthy, not stuck.

Host provider inventory (macOS, ~/.hermes/.env keys)

Primary (2026-08-19, after migration): ark (base_url: https://ark.cn-beijing.volces.com/api/coding/v3, model deepseek-v4-flash, ~1M context). This is a custom_providers entry, not a built-in. Migrated here 2026-08-19 (main session, all 17 agent cron jobs, and delegation) from opencode-go.

Earlier primary (2026-08): opencode-go / deepseek-v4-flash (base_url: https://opencode.ai/zen/go/v1). Intermittent hangs observed 08-10 (mass), 08-12, 08-13 — transient, self-healing on retry, but noisy. opencode-go is a REAL built-in provider (key OPENCODE_GO_API_KEY), NOT a display alias — don't assume a cron provider label you don't recognize is a normalization artifact; check hermes-agent skill's provider table first.

Fallback candidates with keys already present: minimax-cn (already in config.providers — the zero-config choice), deepseek, anthropic, xiaomi. Check key NAMES only: grep -oE "^[A-Z_]+_(API_KEY|KEY|TOKEN)" ~/.hermes/.env | sort -u.

Recurring wave pattern — evening/morning clusters, not random

Observed 08-13 → 08-17: LLM-provider failures recur in WAVES clustered in evening (20:30–23:00) and morning (09:00–10:30) windows, not uniformly. A single wave takes out 3–8 unrelated agent crons simultaneously with the same signature (TimeoutError: Cron job 'X' idle for NNNs or RuntimeError: Connection error). Diagnosis shortcut: count failures by hour in executions.db — if 3+ jobs failed within a 30-min window, it is a provider wave, NOT N independent job bugs:

SELECT substr(started_at,12,5) hm, count(*) FROM executions
WHERE started_at > datetime('now','-24 hours') AND status='failed'
GROUP BY hm ORDER BY hm;

Response: wait for the wave to pass (provider self-heals in 1–2h), then batch re-run all errored jobs (cronjob run, 4–6 per tool-call block). Do NOT chase each job individually during the wave — they will fail again until the provider recovers.

Zombie cron round — executions.db says running but the agent is dead

Distinct from the HANG class: the execution row stays at running forever (never transitions to completed/failed) because the agent process died mid-run without the executor writing a terminal state. Recurred 3× on wiki/wechat inbox crons (08-13, 08-14).

Three-signal zombie check (all must hold):

1. ps aux | grep -E '<job_id>' | grep -v grep → empty (no agent subprocess)
2. Heartbeat stale: cat ~/wiki/heartbeat/<job>.last-run vs date (hours old)
3. Work product frozen: find <inbox-dir> -name '*.md' -newermt '<round start>' | wc -l → 0

Fix: verify dead, then cronjob action=run — do NOT wait for the next scheduled round (a zombie never completes, so the next tick just starts another round on the same unprocessed files). The unprocessed files were never consumed — the re-run picks them up.

Verification checklist

• Classified the failure: hang vs error vs rate-limit vs false-health (read the output file, not just status)
• For error/rate-limit classes: fallback_model configured with entries that each have provider+model
• For hang class: client request timeout present, or accept watchdog kill + next-tick retry
• Re-ran errored jobs via cronjob run and verified real work in the output files
• Background probes wrapped in an outer hard timeout