cron-management

name	cron-management
description	Comprehensive Hermes cron job management — heartbeat monitoring (detect stale jobs), prompt recovery (extract + modify + recreate), bottleneck diagnosis (why is inbox piling up), ground-truth fire verification, and error debugging (why did this cron error). Four first-class concerns unified under one entry point so an agent searching 'cron' finds the right playbook immediately.
tags	["cron","heartbeat","monitoring","prompt-recovery","bottleneck","stale-detection","provider-migration","schedule","error-debugging"]
category	devops
related_skills	["wiki-auto-pipeline-recipes","wiki-maintenance","hermes-tool-corruption-pitfalls","devops/ci-ubuntu-docker-test"]

Cron Management

Hermes cron jobs require four distinct operational skills. This umbrella is the single entry point — load it first, then read the relevant reference for the specific concern.

Five First-Class Concerns + Report Generation

Concern	When to load	Reference
Heartbeat monitoring	"Is my cron still running?" "How do I detect a stale job?"	references/heartbeat-monitoring.md
Monitor prompt template	"Canonical cron-heartbeat-monitor prompt"	references/cron-monitor-prompt-template.md
Prompt recovery	"I need to fix the prompt for cron X" "Did this job actually fire?"	references/prompt-recovery.md
Bottleneck diagnosis	"Inbox is piling up, why?" "How do I clear the backlog?"	references/bottleneck-diagnosis.md
Error debugging	"Why is this cron erroring?" "last_status=error"	references/cron-error-debugging.md
Status reporting	"cron status" — delta report with error timestamps, stale-vs-current-vs-delivery classification	references/cron-status-reporting.md
Report generation	"Generate daily wiki report" "Send status email"	references/wiki-daily-report-html-pipeline.md
no_agent migration	"Report cron keeps timing out" "Broken pipe / request timeout"	references/cron-report-no-agent-migration.md

Quick Decision Tree

Cron job concern?
│
├── Stale / not running / unknown status
│   → heartbeat-monitoring.md (touch + monitor + diagnose-stale)
│
├── Need to fix the prompt / change model / change schedule
│   → prompt-recovery.md (extract from output file, modify, recreate)
│
├── Inbox accumulating faster than cron can clear
│   → bottleneck-diagnosis.md (4-step diagnosis + 4 fix options)
│
└── last_status=error / job is failing
    → cron-error-debugging.md (systematic 5-step debug methodology)

User asks "cron status"?
    → cron-status-reporting.md (list → delta vs last check → trace evidence chain →
      classify stale/current/delivery → report with precise timestamps)

### Report cron (LLM-agent) consistently times out?

Run the diagnostic from `references/cron-report-no-agent-migration.md`:

1. Is the report pure data-collection + template + send? → **Migrate to no_agent=True + Python script** (`references/cron-report-no-agent-migration.md`)
2. Does it need LLM reasoning (summarization, commentary)? → Fix timeout via skill-content reduction or `dialog_timeout_s` increase (SKILL.md rule #11)

Script-level proxy trap: send-mail.py hardcodes a proxy regardless of env vars

Separate from Rule #3 above (which covers HTTP_PROXY env vars). scripts/send-mail.py has a hardcoded proxy at PROXY_HOST='127.0.0.1', PROXY_PORT=10808 — it routes SMTP traffic through the local xray/v2rayN proxy regardless of what HTTP_PROXY / NO_PROXY say. The SSL handshake through this proxy can time out (socket.timeout: _ssl.c:1112: The handshake operation timed out), even when the proxy itself responds to HTTP CONNECT.

Fix: Toggle PROXY_ENABLED = False at the top of scripts/send-mail.py (the connect_via_proxy() function has a gate). Gmail SMTP ports 587/465 are reachable directly on this network, so no proxy is needed. See references/wiki-daily-report-html-pipeline.md → section 6 pitfall for the full diagnostic procedure.

Universal Rules (apply to all five)

1. Scheduler runs LLM-driven jobs sequentially. Only one agent-powered cron runs at a time. If a job takes 30-60 minutes, all other jobs due during that window are skipped — their next_run_at passes without being dispatched. This is the single most common cause of 'missed' cron runs. Long jobs (rss-feed-scan, newsletter-link-extract) can cascade and delay even daily jobs (e.g., daily-status-report at 23:00). Mitigations:

   • Keep high-frequency jobs short (target < 10 minutes per run)
   • Schedule heavy jobs at times when few other jobs are due
   • After a heavy job completes, check if other jobs are past-due and manually cron run them. Caution: cron run does NOT preempt the currently running job — it only queues the job for the next tick. If another job is still running, the queued job sits in a FIFO queue and won't execute until the current one finishes. Also: cron run does NOT refresh last_run_at/last_status until the run completes — a list right after triggering still shows the previous (often error) status; that is expected, not a failed trigger. Verify via agent.log completion markers (Job '<name>' completed successfully) — full recipe in references/cron-status-reporting.md → "Manual re-run".
   • no_agent scripts run independently and don't block the queue

   Pitfall — Provider connection errors inflate job runtime. When the LLM provider has intermittent connection failures (e.g., opencode-go APIConnectionError, Broken pipe [Errno 32], stream stalls of 300–1000s), a job that normally completes in 5–10 minutes can stretch to 60–90 minutes due to retry loops and stream timeouts. This blocks the scheduler queue the same way a genuinely heavy job does: all subsequent jobs due during that window are skipped. After the slow job finally finishes, the scheduler defers its next run by hours to prevent overlap (e.g., a 30m-scheduled job that ran 90m may have its next run pushed 2+ hours out). Diagnosis: Cross-reference the cron's runtime in hermes cron list with grep -E "(Connection error|Stream stale|Broken pipe)" ~/.hermes/logs/gateway.error.log near the same timestamp. Fix: After the provider recovers, manually re-run affected jobs with hermes cron run job-id=<id>. The scheduler resumes normal cadence once the backlog clears.

2. git log with Chinese/non-ASCII filenames produces octal escape sequences. When stdout is not a terminal (piped, redirected), git defaults to quoting non-ASCII filenames with octal escapes (). This breaks grep-based file counting and downstream text processing. use to emit raw UTF-8:

10b. A no_agent cron's script resolves under ~/.hermes/scripts/, NOT the project dir — same-name copies can drift, and you WILL patch the wrong one. When a no_agent job has "script": "sync-wiki-book.sh" (or similar), the scheduler runs ~/.hermes/scripts/sync-wiki-book.sh. If the same filename also exists in the project (e.g. ~/wiki/scripts/sync-wiki-book.sh), the two are SEPARATE files that drift. Confirmed 2026-08-02: patching the project copy did nothing for the cron (the hermes copy was actually the newer, complete version). Before editing any no_agent script: (1) confirm which file the cron runs — grep -o '"script": "[^"]*"' ~/.hermes/cron/jobs.json for the job, then ls -la ~/.hermes/scripts/<name>; (2) diff the hermes copy against any project same-name copy to see which is newer; (3) edit the cron runner, then cp it over the project copy so they stay in sync. Never assume which copy is canonical — verify first.

11. Skill content size directly causes RuntimeError: request timeout in agent-driven crons. The agent must load all skill content into context before it can start executing the prompt. If the combined skill content exceeds ~100KB (roughly), the agent exhausts its time budget (default dialog_timeout_s: 300 = 5 min) just reading documentation, never reaching actual tool execution.

    Verified failure mode (wechat-inbox-pipeline, 2026-07-24):

    • Old version (1 skill, 82KB): ran successfully, ingested 12 articles
    • Recreated version (4 skills, 356KB total): 109 consecutive timeout failures, 0 successful runs
    • Every output file ended with ## Error followed by RuntimeError: request timeout — agent never made a single tool call

    Rule of thumb: keep total loaded skill content under 100KB for agent-driven crons with dialog_timeout_s: 300. For larger content needs:

    • Use no_agent scripts instead (no LLM overhead, no timeout limit)
    • Load only the single most relevant skill in the cron's skills list; list other skills as related_skills in the cron prompt for the agent to decide
    • Split a wide pipeline into multiple crons, each loading only its phase's skill
    • Increase provider request_timeout_seconds (affects httpx/OpenAI client per-call timeout):

      hermes config set providers.<id>.request_timeout_seconds 600
      

      Example: hermes config set providers.opencode-go.request_timeout_seconds 600
    • Increase agent.max_turns (tool-calling iteration limit):

      hermes config set agent.max_turns 150
      

    This lets a 356KB pipeline complete without reducing skills, at the cost of slower timeout detection for all jobs using that provider. request_timeout_seconds is set under providers.<id>.request_timeout_seconds in config.yaml. max_turns is the tool-calling iteration limit.

    • Trim verbose documentation from SKILL.md files (edge cases, changelogs, old pitfall catalogues) — put detail in references/ files the agent can decide to read

When This Skill Should NOT Be Loaded

• If the cron has never been set up, see hermes-tool-corruption-pitfalls for the broader Hermes system.
• If the cron is part of a specific pipeline (e.g., wiki-inbox-scan), defer to the pipeline-specific skill first (e.g., wiki-auto-pipeline-recipes, rss-to-wiki-pipeline).

See Also

• hermes-tool-corruption-pitfalls — execute_code cron-mode block, read_file caching pitfalls
• wiki-auto-pipeline-recipes — wiki pipeline cron prompt best practices, MiniMax model compatibility
• devops/cron-job-provider-migration — yidong / MiniMax provider switching workflow
• references/heartbeat-monitoring.md — focused skill content for prompt extraction + modification, including P31 (wechat-inbox-pipeline missing heartbeat touch, 2026-07-22)
• references/cron-monitor-prompt-template.md — canonical cron-heartbeat-monitor prompt (added 2026-07-22)
• references/heartbeat-sessions/ — 12 daily monitor/pipeline session transcripts
• references/prompt-recovery-details/ — 5 detailed reference docs (cron-rebuild-and-providers, cron-fire-evidence, cron-fire-vs-trigger, yidong-provider-quirks, yidong-fact-forcing-gate)
• references/tirith-pitfalls.md — tirith security-scanner blocks in cron mode: pipe-to-interpreter, schemeless-URL, execute_code; diagnosis recipe + workarounds
• references/cron-error-debugging.md — systematic 5-step methodology for debugging erroring cron jobs
• references/cron-status-reporting.md — the "cron status" report workflow: delta vs last check, evidence-chain tracing, stale-error vs delivery-error classification, FOUR provider-failure signatures (A: DNS outage / B: mid-stream stall / C: explicit Router.Unavailable 500 / D: slow-but-functional degradation), the recurring ~09:00 DNS window (7/31 + 8/5, absent 8/6; manual-trigger does NOT dodge it), and jobs.json last_error as the fastest evidence source for no_agent script failures
• references/wiki-daily-report-html-pipeline.md — generate and email a comprehensive HTML wiki status report (cron report generation pattern)
• scripts/cron-daily-report.py — no_agent report generation script (data collection + HTML email via direct SMTP; see references/cron-report-no-agent-migration.md for the pattern)
• scripts/diagnose-stale.py — automated stale-classification script