| name | llm-provider-resilience |
| description | Use when LLM provider flakiness breaks cron jobs or probes. |
| version | 1.0.0 |
| author | Hermes Agent |
| license | MIT |
| platforms | ["macos","linux"] |
| metadata | {"hermes":{"tags":["llm","provider","cron","fallback","reliability","opencode-go"],"related_skills":["cron-job-provider-migration","we-mp-rss-troubleshooting"]}} |
LLM Provider Resilience
Use when agent cron jobs fail with provider-related errors, when a model provider is intermittently flaky, or when deciding between provider migration vs adding a fallback. Covers the failure taxonomy, the fallback_model config mechanism, and the verified re-run/probe patterns.
Failure taxonomy — read the cron error, classify FIRST
| Symptom in cron output | Class | Meaning |
|---|
TimeoutError: Cron job '<name>' idle for Ns (limit 600s) — last activity: waiting for non-streaming API response | HANG | Request sent, response never arrived. Client has no (or too-long) request timeout. The 600s cron watchdog kills the job. |
RuntimeError: HTTP 429 / 401 / 5xx or Connection error | ERROR / RATE-LIMIT | Provider responded with a failure. |
| Job runs instantly, exit 0, but did nothing useful | FALSE-HEALTH | Script logic doesn't check its own output quantity (e.g. a collection script reporting "30/30 ok" while fetching 0 items). Not a provider issue — a script bug. |
Superseded-path trap (2026-08-21, wechat-article-discover): a cron job can be broken not because its code is wrong but because its collection path was replaced. Retrying it confirms it's still broken (e.g. Done: 30/30 ok, 0 err, new_articles=0 → exit 1, all feeds return 0 in Playwright mode) — and re-enabling it produces a false daily error. Before re-enabling a previously-paused errored job, confirm its path still exists: (a) cookie-renew success (token write, container HTTP 200) does NOT imply discovery success (article-list fetch auth) — the two are separate; (b) check whether the pipeline now ingests through a mirror/other job (e.g. wechat2rss mirror via rss-feed-scan + wechat-inbox-pipeline). If superseded, keep it paused and say why.
Verify the underlying WeChat ban directly (200013 probe, 2026-08-26): to confirm WHETHER the account is still banned before deciding to re-enable wechat-article-discover, don't trust the unblock-probe's silence alone — it exits silently on BOTH ret==200013 (still banned) and probe failure. Run the probe yourself. The container's driver.token module supplies token+cookie; heredoc via docker exec ... <<'PY' is unreliable (empty output), so write the probe to a local file, docker cp it in, then run it:
import sys, requests, urllib3; urllib3.disable_warnings()
sys.path.insert(0, '/app')
from driver.token import get
token, cookie = get('token',''), get('cookie','')
url = "https://mp.weixin.qq.com/cgi-bin/appmsgpublish"
params = {"sub":"list","sub_action":"list_ex","begin":0,"count":1,
"fakeid":"MzA3MjgyODI4MzMw","type":"101_1","free_publish_type":1,
"token":token,"lang":"zh_CN","f":"json","ajax":1}
headers = {"User-Agent":"Mozilla/5.0","Referer":"https://mp.weixin.qq.com/"}
cd = {}
for item in cookie.split(";"):
item = item.strip()
if "=" in item:
k,v = item.split("=",1); cd[k.strip()] = v.strip()
try:
r = requests.get(url, params=params, headers=headers, cookies=cd, timeout=15, verify=False)
print("RET=" + (r.json().get(,{}).get()))
Exception e:
( + (e))
docker cp /tmp/probe_ret.py we-mp-rss:/tmp/probe_ret.py
docker exec we-mp-rss /app/env_x86_64/bin/python3 /tmp/probe_ret.py 2>&1 | grep -E "RET=|ERROR="
Hang vs error matters for the fix: fallback chains trigger on errors/timeouts/rate-limits, NOT on pure hangs (see below). If the class is HANG, a fallback alone may not help — you also need a client-side request timeout so the hang becomes a timeout the fallback can catch, or accept the 600s watchdog kill + next-tick retry.
fallback_model — automatic backup provider (no migration needed)
Config key in ~/.hermes/config.yaml (verified in agent/agent_init.py + agent/agent_runtime_helpers.py, 2026-08-12):
fallback_model:
- provider: minimax-cn
model: <model>
- provider: deepseek
model: deepseek-chat
- Each entry REQUIRES
provider + model (config validation rejects entries missing either field).
- Chain entries tried in order; a failed primary call activates the chain for the rest of that turn.
- Rate-limit cooldown (~60s) and provider reset-time tracking gate when the primary is retried; chain index resets per turn.
- Cron agent jobs inherit the global model config → ONE global
fallback_model covers all jobs. Per-job model/provider overrides in cronjob records take precedence.
- Hang caveat: fallback engages on client-visible failures. A pure hang with no client timeout never surfaces an error, so the fallback never fires — the cron watchdog kills the job first. Pair fallback config with a request timeout if hangs are the observed class.
Re-running errored agent jobs
cronjob(action='run', job_id=...) executes the job immediately and reliably (verified 2026-08-12: 8/8 errored agent jobs re-ran to last_status: ok in one pass, plus a 9th that had re-errored on its schedule).last_status: error on the job list is a HISTORICAL ARTIFACT of the last execution — after a provider outage day, it does not mean the job is broken now. Re-run, then verify the OUTPUT FILE content (~/.hermes/cron/output/<job_id>/<latest>.md) shows real work, not just exit 0.- After a mass provider-outage day, batch
cronjob run 4-6 jobs per tool-call block; jobs re-fire sequentially, so rate-limit collisions are unlikely, but stagger anyway if the provider has frequency limits.
Migrating cron jobs to a new provider — cronjob update has NO model override
To point all agent cron jobs at a different provider/model, the cronjob tool's update action does NOT expose a model/provider parameter — calling it with just job_id/name is a no-op that leaves provider unchanged (verified 2026-08-19). The dependable path is a direct batch edit of ~/.hermes/cron/jobs.json, then revalidate JSON, then restart the gateway (the scheduler caches job config in memory):
import json
p = '/Users/jinguo/.hermes/cron/jobs.json'
data = json.load(open(p))
for j in data['jobs']:
if j.get('model'):
j['provider'] = 'ark'
j['model'] = 'deepseek-v4-flash'
j['base_url'] = 'https://ark.cn-beijing.volces.com/api/coding/v3'
json.dump(data, open(p, 'w'), ensure_ascii=False, indent=2)
Then python3 -c "import json; json.load(open(p))" to revalidate, back up jobs.json first, and verify the result via cronjob(action='list'). Full workflow detail lives in the cron-job-provider-migration skill (user-owned — recommend hermes curator adopt cron-job-provider-migration to make it curatable).
Background probe hardening — never trust signal.alarm alone
Real case (2026-08-10): a background verification docker exec we-mp-rss ... do_job(mp=feed) hung 10 HOURS past its signal.alarm(50) — SIGALRM does not reliably interrupt requests stuck in C-level sockets, and the process stayed alive holding the pipe. The scheduled result was lost.
- Wrap any scheduled/background probe in an OUTER hard timeout:
gtimeout 90 docker exec ... (macOS: brew install coreutils for gtimeout), or run it with terminal(background=true) + process(action='kill') fallback.
- Never chain
sleep N && probe in a background process and assume the probe terminates on its own.
- For long cooldown-window probes (hours), prefer a one-shot cron job or a foreground poll with bounded waits over a single huge
sleep.
Large-context probe: a SMALL curl can fake provider health (2026-08-21, ark)
When a batch of agent-driven crons fails with RuntimeError: Request timed out but no_agent script jobs stay green, suspect provider-wide large-context degradation, not per-job bugs. Verified against ark (deepseek-v4-flash): a small probe (max_tokens=10) returned HTTP 200 in ~2s WHILE the same provider was timing out on the cron's real workload — because agent-driven cron prompts embed the full skills catalog + memory (~130K tokens), and large-context requests are the FIRST thing to degrade during a provider outage window. A green small-probe is NOT evidence of health.
- Diagnostic tell: ALL agent-driven jobs on one provider fail simultaneously with the same timeout signature; no_agent script jobs (no LLM call) stay green. That split is the signal — it points at the provider, not the jobs.
- Probe LARGE, not small: reproduce the real workload with an ~80K-char prompt +
stream:true. If that returns 200 fast, the outage window has passed and jobs will recover on re-run.
- Bulk re-run:
hermes cron run <job_id> per failed job. Heavy agent jobs exceed 180s foreground — trigger them with terminal(background=true). Output Job is already being fired by the scheduler; not run again means it's mid-run (my earlier trigger queued it), NOT a failed trigger.
- Verify via agent.log, not the stale list right after triggering (
last_status doesn't refresh until the run completes): grep "Job '<name>' completed successfully" ~/.hermes/logs/agent.log. Heavy ingest jobs (wechat-inbox-pipeline) take 40–60 min across 70+ tool turns with ~99% prompt-cache hit — steady API-call progression in agent.log means healthy, not stuck.
Host provider inventory (macOS, ~/.hermes/.env keys)
Primary (2026-08-19, after migration): ark (base_url: https://ark.cn-beijing.volces.com/api/coding/v3, model deepseek-v4-flash, ~1M context). This is a custom_providers entry, not a built-in. Migrated here 2026-08-19 (main session, all 17 agent cron jobs, and delegation) from opencode-go.
Earlier primary (2026-08): opencode-go / deepseek-v4-flash (base_url: https://opencode.ai/zen/go/v1). Intermittent hangs observed 08-10 (mass), 08-12, 08-13 — transient, self-healing on retry, but noisy. opencode-go is a REAL built-in provider (key OPENCODE_GO_API_KEY), NOT a display alias — don't assume a cron provider label you don't recognize is a normalization artifact; check hermes-agent skill's provider table first.
Fallback candidates with keys already present: minimax-cn (already in config.providers — the zero-config choice), deepseek, anthropic, xiaomi. Check key NAMES only: grep -oE "^[A-Z_]+_(API_KEY|KEY|TOKEN)" ~/.hermes/.env | sort -u.
Recurring wave pattern — evening/morning clusters, not random
Observed 08-13 → 08-17: LLM-provider failures recur in WAVES clustered in evening (20:30–23:00) and morning (09:00–10:30) windows, not uniformly. A single wave takes out 3–8 unrelated agent crons simultaneously with the same signature (TimeoutError: Cron job 'X' idle for NNNs or RuntimeError: Connection error). Diagnosis shortcut: count failures by hour in executions.db — if 3+ jobs failed within a 30-min window, it is a provider wave, NOT N independent job bugs:
SELECT substr(started_at,12,5) hm, count(*) FROM executions
WHERE started_at > datetime('now','-24 hours') AND status='failed'
GROUP BY hm ORDER BY hm;
Response: wait for the wave to pass (provider self-heals in 1–2h), then batch re-run all errored jobs (cronjob run, 4–6 per tool-call block). Do NOT chase each job individually during the wave — they will fail again until the provider recovers.
Zombie cron round — executions.db says running but the agent is dead
Distinct from the HANG class: the execution row stays at running forever (never transitions to completed/failed) because the agent process died mid-run without the executor writing a terminal state. Recurred 3× on wiki/wechat inbox crons (08-13, 08-14).
Three-signal zombie check (all must hold):
ps aux | grep -E '<job_id>' | grep -v grep → empty (no agent subprocess)- Heartbeat stale:
cat ~/wiki/heartbeat/<job>.last-run vs date (hours old)
- Work product frozen:
find <inbox-dir> -name '*.md' -newermt '<round start>' | wc -l → 0
Fix: verify dead, then cronjob action=run — do NOT wait for the next scheduled round (a zombie never completes, so the next tick just starts another round on the same unprocessed files). The unprocessed files were never consumed — the re-run picks them up.
Verification checklist
