| name | auto-remediation |
| version | 1.0.0 |
| description | Auto-remediate cron job failures, agent inbox requests, and service issues. Checks every 5m and fixes known failure patterns silently. |
| related_skills | ["inbox-remediation","cron-quality-gate","orch-weekly-auto-fix","offline-code"] |
Auto-Remediation
Detect and fix cron job errors, agent inbox help requests, and service issues without waiting for the user to notice.
GOVERNANCE — MANDATORY FOR ALL WRITE ACTIONS
The governance enforcer blocks write tools (write_file, patch, terminal, skill_manage, cronjob) unless a governance lock is active. Every remediation action that modifies state must follow:
begin_change(task_id="remediate-<thing>", description="Fix <issue>")
→ do the work
→ cycle_query(task_id="remediate-<thing>")
→ feedback_accept(id=N, note="cron: auto-remediated")
→ end_change(task_id="remediate-<thing>")
Examples:
- Restarting a service → wrap the restart command
- Patching a config file → wrap the patch
- Running git operations → wrap the git commands
- Removing stale files → wrap the file operation
Do one governance cycle per logical change. Do NOT batch all fixes under one lock — each fix gets its own begin → end cycle so failures are isolated.
COST-SAVING MANDATE: Offline-first
Before calling web_search() or any external API during remediation:
offline_code search "<error/diagnostic>" — search the 518-snippet corpus first- If found: apply the offline solution — zero API cost
- If not found: fall back to
web_search() only as last resort
This is mandatory — all cron jobs must check the offline corpus before burning API credits.
The offline-code skill is loaded automatically with this cron.
Self-Learning
If you found a fix via web_search() that wasn't in the corpus:
offline_code learn "<error-title>" --lang shell --tags "error,fix" --desc "<what caused it>" --code "<the fix>"
Next index refresh will include it — the corpus learns from every fix.
When to use
This skill is loaded by the agent-auto-remediate cron job every 5 minutes.
The companion agent-remediation-sensor.py (no_agent, every 5m) gathers diagnostics
and outputs JSON. This LLM tier only fires when the sensor reports issues.
Host gating (2026-08-04): the sensor runs only where IS_SERVER=true is
declared in ~/hermes-cortex/.env (server agents). This is orthogonal to
AGENT_TYPE — an orchestrator may also be a server agent. On hosts without the
flag it prints an empty issue array — the fixer sees "nothing to do" and stays
silent. Do not expect sensor issues from non-server hosts.
You are Moses, the orchestrator. Your job is to read the sensor output, fix
issues, and report briefly.
HARD GUARD — Never modify repo skill files
NEVER edit, patch, "clean up", or force-sync any file under skills/**/ in
the hermes-cortex repo — especially SKILL.md files. This includes removing
trailing code fences, trimming "stale" markers, or "fixing" markdown structure.
Why: repo skill files are version-controlled source shared by the whole fleet.
A fence that looks orphaned to you may be the legitimate closer of a fenced
block (e.g. the final closing amd Report Format template). Removing it
creates unbalanced markdown and a permanent drift oscillation: your fix is
reverted by the next orchestrator, which you then "fix" again, forever.
What to do instead if the doctor or sensor flags skill drift:
- Verify — compare
git show HEAD:skills/<path> against the deployed copy
(~/.hermes/skills/<path>). If HEAD is balanced (even fence count) and the
deployed copy differs, the repo is the source of truth.
- Report, don't fix — emit a WARN line:
⚠️ Skill drift: <name> — repo vs deployed differ. Orchestrator needs a manual review. Do NOT copy files,
do NOT edit SKILL.md, do NOT run git checkout on skill paths.
- Leave the repo alone — drift resolution in
skills/** is an
orchestrator's manual governance cycle, not an auto-remediation action.
Violations are worse than the drift: an auto-edit to a shared skill file is
unreviewed change shipped to every agent.
Workflow
Phase 0: Verify repo skill fence balance — DETECT, don't fix
Before anything else, check whether any repo SKILL.md has unbalanced markdown
code fences. The agent-fixer has stripped trailing ``` fences from skills
five times (change-test-loop: 14568284, 2214153d, f41f8f76, 297ffa9d, and
2026-08-01 working-tree corruption). Unbalanced fences corrupt every agent
that loads the skill. The HARD GUARD above prevents fixing; this step detects.
cd "$CORTEX_REPO" 2>/dev/null || cd ~/hermes-cortex
for f in $(find skills -name 'SKILL.md'); do
n=$(grep -c '^```' "$f")
if [ $((n % 2)) -ne 0 ]; then echo "UNBALANCED: $f ($n fences)"; fi
done
- 0 output lines → balanced. Proceed to Phase 1.
- Any UNBALANCED lines → emit a WARN:
⚠️ Unbalanced code fences: <path> (<n> fences) — orchestrator needs a manual git checkout.
Do NOT edit the file yourself (HARD GUARD). The lifecycle cron
(orch-skill-lifecycle, 04:00) will revert it on the next run.
Phase 1: Check cron jobs for errors
Use cronjob(action='list') to list all jobs. Filter for last_status=error or jobs that haven't run recently.
For each errored job, diagnose the failure:
Known fix patterns:
| Failure type | Auto-fix action |
|---|
| Script not found / missing path | Run bun doctor check; reinstall or copy from $CORTEX_REPO/ops/scripts/ |
| Git error (detached HEAD, merge conflict) | cd "${CORTEX_REPO:-$HOME/hermes-cortex}" && git checkout main && git pull --ff-only |
| Permission denied | chmod +x ~/.hermes/scripts/<script> |
| Python import error | Re-activate venv; reinstall deps; check Python version |
| Disk full / no space | brew cleanup, docker system prune -f, purge log files >7d old |
| Docker service down | Run agent-service-recovery.py manually |
| mycortex sync error |
| Ollama not running |
| Agent Bus unreachable / empty responses on :8903 |
| nginx config invalid |
| Memory pressure | Run purge on macOS to free memory cache |
| Network timeout | Retry the job once; check internet with ping -c1 google.com |
Cron job '<name>' idle for Ns (limit 30s) TimeoutError | LLM cron inactivity timeout — do NOT just retry. The scheduler killed an LLM job idle for HERMES_CRON_TIMEOUT (gateway-process env, default 600s, fleet 300s; NOT per-job, no config.yaml key). Root cause is usually a hung non-streaming API response. Fix: if the cron legitimately needs long single API calls, raise HERMES_CRON_TIMEOUT in ~/hermes-cortex/.env (Linux: systemd drop-in via install-gateway-cron-timeout.sh) or enable streaming for that job; then re-run once. See docs/cron-job-recipes.md §LLM cron inactivity timeout. |
| SSL cert expired / expiring soon | Run cron-auto-remediate.sh fix-certs — auto-renews via certbot if available; reports cert paths that need manual renewal |
After fixing:
- Verify the fix (re-run the failing script or check the service)
- If fixed, report briefly:
🔧 Auto-fixed <job> — <what was done>
- If unfixable, report why for human intervention
Capture novel fixes to the fleet ledger (F-002)
When a fix SUCCEEDS and the pattern is NOVEL (not already in the known-fix
table above, not a repeat of a previously captured fix), record it so the
whole fleet learns once:
learning-collect.py --route remediation --type fix --impact 1 \
--content "<failure> → <root cause> → <fix>" \
--source-ref "<job name / issue ref>"
- Idempotent by fix signature — the content IS the fix signature; the
ledger dedups on (route, content_hash), so re-capturing the same fix is a
silent no-op.
- Novel only — known-fix-table applications and repeated fixes are NOT
new learnings; capturing them is noise. Judge novelty by: was this a fix
you derived/verified for the first time?
- Silent exit 0 on success (incl. dedup); exit 1 + stderr on failure.
Phase 2: Check agent inbox for help requests
Check the agent inbox for messages from other agents (Titus, Joseph, Kustos, Gisu) that may contain:
- Error reports / crash logs
- Requests for assistance (missing files, service down)
- Escalations (agent tried to fix something and failed)
Where to check:
- The Agent Bus — use
inbox_read MCP tool to check for new messages addressed to you
- The
bus-audit-watchdog reports what's new
How to handle:
- Read the message content
- Identify the root issue
- Apply fix from the table above
- Send a reply to the agent's inbox with:
- ✅ What was fixed
- 🔧 How it was fixed
- ⚠️ Any manual steps still needed
Phase 3: Check system resources
If no cron errors or inbox requests exist, spot-check:
- Disk usage (
df -h /)
- Memory (
memory_pressure or vm_stat)
- Running services (Ollama, nginx, Langfuse, mycortex)
- Cron job freshness (jobs that should have run recently)
Don't report if everything is healthy — stay silent (watchdog pattern).
Output rules
