Patterns for creating, testing, debugging, and maintaining cron-driven automation in workspace-hub, including log strategy, failure analysis, and safe git-aware job design.
Patterns for creating, testing, debugging, and maintaining cron-driven automation in workspace-hub, including log strategy, failure analysis, and safe git-aware job design.
works manually but fails in cron due to PATH/env differences
command too complex inline in YAML
missing log file path prevents health monitoring
git operations race or fail under cron
task declared in YAML but not installed on machine
cron entry exists but script path changed
MANDATORY: Cron Script Prologue
Every cron wrapper script MUST start with PATH injection. Cron's environment is minimal and does not include $HOME/.local/bin where uv, node, etc. live. A script that runs fine in your shell but breaks silently in cron is almost always a PATH issue.
The cron-health-check script had uv: command not found for 5 consecutive days because of this exact issue. The schedule-tasks.yaml command lines include PATH overrides, but wrapper scripts called from those commands do not inherit them in subshells.
Health Monitoring Realities
False-Positive Awareness
The cron-health-check reports STALE for weekly jobs (Sunday/Monday) when viewed on Tuesday. Day-of-week schedules (0 3 * * 0, 0 4 * * 1) naturally exceed the 25h default staleness threshold. When doing a health review:
Weekly jobs scheduled for yesterday or today = EXPECTED, not stale
Weekly jobs scheduled for 3+ days ago = ACTUALLY stale and needs attention
Daily jobs > 25h old = genuinely stale
Error scans must be bounded to recent log output, not the full append-only log. Use tail -n 100 (or another explicit recent window) before matching ERROR/Traceback, otherwise an old transient failure in logs/daily/cron.log can keep a healthy job red indefinitely.
Do not let cron-health self-poison: its own log legitimately contains [ERROR] rows when reporting other broken jobs. Either skip error-pattern scanning for the cron-health task itself or restrict self-checking to the current run's process exit/result artifact.
Make ERROR: matching case-sensitive and anchored, e.g. (^|[[:space:]])ERROR:, so warning prose such as 403 Client Error: Forbidden is not counted as an operational failure.
Log Path Gotchas
log: null in YAML means the health checker cannot monitor the task -- this produces a MISS on glob matching. Every task should have a log: field with a valid glob pattern.
The glob pattern is expanded relative to $WORKSPACE_HUB, so logs/quality/thing-*.log resolves to /mnt/local-analysis/workspace-hub/logs/quality/thing-*.log.
Pre-Create GitHub Labels
Any script that creates issues with gh issue create --label "X" must pre-create the label:
Without this, gh issue create fails with could not add label: 'X' not found and the issue is silently not created.
Legal Scan vs Learning Pipeline
The comprehensive-learning pipeline can be blocked by legal-sanity-scan when session JSONL logs contain client names that match deny-list patterns. Session logs are append-only operational data, not source code. Watch for this pattern:
RESULT: FAIL — N block violation(s) found in learning logs
WARNING: learning artifact commit failed — changes remain local
Changes accumulate uncommitted across multiple runs
Either exclude the log directory from the legal scan, or use --diff-only for the learning pipeline's commit path.
Health Review Workflow
To perform a comprehensive cron health review:
Run cron-health-check.sh manually -- this is your first pass
Check the cron log directory: tail -10 logs/research/2026-04-*.log for research, etc.
Cross-reference crontab -l vs config/scheduled-tasks/schedule-tasks.yaml
Also compare against bash scripts/cron/setup-cron.sh --dry-run -- this is the most reliable rendered view for the current host
Verify the /today daily report at logs/daily/YYYY-MM-DD.md is fresh
If reports exist but health-check is broken, fix the script first, then rerun
For full closeout of a "fix all cron jobs / daily report is current" request, collect the evidence bundle in references/cron-daily-report-closeout.md before reporting success.
Canonical vs Drift-Prone Sources
When auditing cron drift in workspace-hub, treat these as canonical:
config/scheduled-tasks/schedule-tasks.yaml
scripts/cron/setup-cron.sh
scripts/cron/validate-schedule.py
config/workstations/registry.yaml
Treat these as legacy or drift-prone unless they are explicitly refreshed to match YAML:
scripts/coordination/context/setup_cron.sh
scripts/coordination/productivity/crontab.example
docs/ops/scheduled-tasks.md
docs/WORKSPACE_HUB_CAPABILITIES_SUMMARY.md
scripts/cron/crontab-template.sh
If live crontab and one of the legacy sources disagree, prefer YAML + setup-cron.sh --dry-run, not the legacy file.
Important operational lesson:
cron-health-check can report a low or even zero issue count while the installed crontab is still drifted from YAML.
Never use the health report alone as proof that cron reconciliation succeeded.
Primary proof of success is:
fresh crontab header from setup-cron.sh --replace
exact parity between crontab -l and bash scripts/cron/setup-cron.sh --dry-run
at least one near-term canary job (for example cron-health at 05:45 or daily-today at 06:00) producing a fresh log after reconciliation.
Troubleshooting Guidance
Missing job on machine
compare crontab -l with setup-cron.sh --dry-run
if YAML is canonical, reconcile with bash scripts/cron/setup-cron.sh --replace
do NOT use the default additive install mode for drift repair
Converting an LLM-driven cron wrapper to a deterministic script
When a weekly/daily task used to shell out to Claude/Codex/Gemini and you replace it with a deterministic script, update the scheduler metadata and test contract at the same time — not just the script body.
Required checks:
update requires: in config/scheduled-tasks/schedule-tasks.yaml so it matches the new runtime (python3/uv/bash instead of claude, etc.)
update is_claude_task: (or equivalent task metadata) so the scheduler no longer advertises the job as provider-driven when it is now deterministic
update the task description: to remove mutating/agent-only behavior that is no longer true
preserve the canonical task id when the cadence/path should stay stable; replace the implementation behind it instead of creating a second overlapping task
add/refresh wrapper tests so --dry-run / print-mode output reflects the new deterministic command line
support redirectable output roots (CLI flag or env var) so tests and manual runs can write to temp directories instead of dirtying the repo
Common failure mode:
the script becomes deterministic, but YAML still says requires: [claude, ...] and is_claude_task: true, while tests/log paths still assume the old wrapper behavior. This creates governance drift and confusing operator docs even if the new script itself works.
additive/default mode can leave stale commands installed
additive/default mode can preserve duplicate inline jobs
one real example was a duplicated notification-purge cron line because append-mode dedupe did not recognize a non-script inline find ... -delete command
drift reviews should compare live crontab against dry-run output both before and after replacement
Silent failure
add or inspect log redirection
run the exact cron command manually
check for non-interactive shell assumptions
Git contention
replace raw git pipelines with git-safe wrappers
Health monitoring blind spots
make sure the task has a stable log: glob in YAML
verify monitoring scripts can parse YAML and locate latest log files
Key Workspace-Hub References
config/scheduled-tasks/schedule-tasks.yaml
scripts/cron/setup-cron.sh
scripts/cron/lib/git-safe.sh
scripts/cron/comprehensive-learning-nightly.sh
scripts/monitoring/cron-health-check.sh
YAML Date Escaping Gotcha
The % character in crontab date format strings must be escaped as \%. In schedule-tasks.yaml, the YAML >- block scalar strips trailing newlines but also adds an extra layer of backslash handling through the setup-cron.sh Python rendering pipeline. This creates a quadruple-escape trap:
Correct in schedule-tasks.yaml: $(date +\\\\%Y\\\\%m\\\\%d) (4 backslashes in YAML → \%Y\%m\%d in crontab → %Y%m%d for date)
Broken: $(date +\\\\\\\\%Y\\\\\\\\%m\\\\\\\\%d) (too many escapes)
Also broken: $(date +\\\\%Y\\\\%m\\\\%d) with only 2 (becomes %Y%m%d with single backslash which also fails)
If a cron log shows date format errors or the log file has literal backslash names, check the YAML escaping level. Use sed -n '/id: my-job/,/^[^ ]/p' config/scheduled-tasks/schedule-tasks.yaml to inspect.
Mandatory: schedule Field
Every task in schedule-tasks.yaml MUST have either a schedule: field or a schedule_by_machine: field. The cron-health entry (#1512) was declared in YAML without either, so setup-cron.sh silently skipped it -- the Python parser got an empty schedule string and continued. The cron-health-check.py could not find the crontab entry because it was never installed.
The requires: list in schedule-tasks.yaml is validated against the flattened capabilities in config/workstations/registry.yaml. The validator (scripts/cron/validate-schedule.py) merges all values from agent_clis, languages, and tools lists for the host machine.
Rule: Any tool name in requires: must appear in the machine's capabilities. Adding requires: [gh] or requires: [npm] will FAIL validation unless gh / npm are also added to the tools: list in registry.yaml for the target machine.
# WRONG — validation fails with "unknown capability 'gh'":-id:my-new-jobrequires: [python3, uv, gh]
# CORRECT — first add gh to registry.yaml:# dev-primary capabilities:# tools: [uv, git, gh] # ← add here# Then in schedule-tasks.yaml:-id:my-new-jobrequires: [python3, uv, gh]
After adding new capabilities to registry.yaml, verify:
uv run --no-project python scripts/cron/validate-schedule.py
Hermes Gateway Cron Scheduler
Hermes has its OWN cron scheduler inside the Gateway process (separate from system crontab). Jobs managed via the hermes cron CLI (gmail-daily-digest, memory-bridge-daily, etc.) require the Gateway to be running.
Starting the Gateway:
# Use the systemd service directly — the hermes CLI wrapper# requires sudo which may not resolve hermes in root PATH
systemctl --hermes-gateway
# Equivalent: sudo systemctl start hermes-gateway
Diagnosing a dead Hermes cron job:
sudo systemctl status hermes-gateway # if not active, no Hermes cron fires
hermes cron list # check job next_run_at dates — stale dates mean gateway has been dead
Warning "No messaging platforms enabled" is non-fatal — the cron ticker still runs. 'local' delivery works fine. Only 'origin'/platform deliveries may be affected.
uv PEP 723 Inline Metadata in Cron
Scripts with # /// script\n# dependencies = ["pyyaml"]\n# /// blocks are PEP 723 inline-metadata scripts. uv treats them as self-contained scripts and auto-installs dependencies.
CRITICAL: Do NOT use python between uv run and the script path:
# WRONG — bypasses PEP 723 metadata, dependencies NOT installed:
uv run --no-project python scripts/ai/my-script.py
# RIGHT — uv recognizes the script's inline metadata block:
uv run --no-project scripts/ai/my-script.py
This bug caused agent-radar to fail for days with "Install PyYAML: uv add pyyaml" even though the script's metadata block declares pyyaml as a dependency.
False-Positive Counting in Daily Reports
When investigating alerts like "13 CVE references", "42 ERROR/FAIL", etc., always verify the raw count against unique root causes. Common traps:
Error counts across rolling windows of multi-day logs inflate perceived severity
Benchmark regressions on sub-millisecond tests are often run-to-run noise, not code issues
Absolute Paths in Cron Shell Context
Cron's minimal environment may not have $HOME set when PATH is expanded at parse time. Use hardcoded absolute paths rather than $HOME expansions in shell scripts:
# UNRELIABLE in cron:export PATH="$HOME/.local/bin:$PATH"# RELIABLE:export PATH="/home/vamsee/.local/bin:$PATH"# BEST: use absolute path for specific commands:
/home/vamsee/.local/bin/uv run --no-project ...
Weekly Today Reports Pattern
The daily productivity report is only healthy if you verify both the daily artifact and whether a weekly variant is actually scheduled.
scripts/productivity/daily_today.sh already supports --week and writes logs/weekly/YYYY-Www.md
if weekly reports are absent, first check whether a weekly-today task exists in config/scheduled-tasks/schedule-tasks.yaml
do not assume weekly reporting is broken just because scripts/coordination/productivity/crontab.example mentions it; that file may be stale or intentionally deprecated
preferred scheduling is 5 minutes after the daily run to avoid same-minute overlap with daily-today
No script changes are needed for this pattern; only the YAML task and cron reinstall.
Documentation cleanup rule:
if scripts/coordination/productivity/crontab.example still contains installable cron lines, consider converting it to deprecation-only guidance that points operators back to config/scheduled-tasks/schedule-tasks.yaml and scripts/cron/setup-cron.sh.
Rule of Thumb
If a cron task is important enough to debug twice, it is important enough to have:
