self-monitor

name	self-monitor
version	1.1.0
author	Polycat
tags	["monitoring","health","infrastructure"]
license	MIT
platform	universal
description	Proactive self-monitoring of infrastructure, services, and health. Tracks disk/memory/load, service health, cron job status, recent errors. Auto-fixes safe issues. Triggers on: health check, heartbeat, monitor status, service status, infrastructure check.

Compatible with Claude Code, Codex CLI, Cursor, Windsurf, and any SKILL.md-compatible agent.

Self Monitor

Proactive self-monitoring: infrastructure, services, and health.

Usage

Run during heartbeats or scheduled checks.

1. Infrastructure Health

# Disk usage
df -h / | awk 'NR==2 {print $5}' | tr -d '%'

# Memory usage  
free -m | awk 'NR==2 {printf "%.0f", $3/$2*100}'

# Load average
uptime | awk -F'load average:' '{print $2}' | awk -F',' '{print $1}'

# Top processes by memory
ps aux --sort=-%mem | head -10

# Top processes by CPU
ps aux --sort=-%cpu | head -10

Thresholds:

Metric	Warning	Critical
Disk	> 80%	> 90%
Memory	> 85%	> 95%
Load	> 2.0	> 4.0

2. Service Health

# Check if a process is running
pgrep -f "your_process_name" >/dev/null && echo "OK" || echo "FAIL"

# Check HTTP endpoint
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health

# Check systemd service
systemctl is-active --quiet nginx && echo "OK" || echo "FAIL"

# Check Docker container
docker ps --filter "name=mycontainer" --filter "status=running" -q | grep -q . && echo "OK" || echo "FAIL"

# Tailscale (if using)
tailscale status --json 2>/dev/null | jq -r '.Self.Online' || echo "FAIL"

3. Cron Job Health

# Check recent cron executions
grep CRON /var/log/syslog | tail -20

# Count failures in last 24h
grep -c "CRON.*error\|CRON.*fail" /var/log/syslog

# List scheduled jobs
crontab -l

4. Recent Errors

# Check system logs for errors
journalctl -p err --since "1 hour ago" 2>/dev/null | tail -20

# Check application logs
tail -50 ~/workspace/projects/*/logs/*.log 2>/dev/null | grep -i "error"

# Check dmesg for hardware/kernel issues
dmesg | tail -20 | grep -i "error\|fail\|warn"

Quick Health Check (for heartbeat)

#!/bin/bash
# Quick health snapshot

DISK=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
MEM=$(free -m | awk 'NR==2 {printf "%.0f", $3/$2*100}')
LOAD=$(uptime | awk -F'load average:' '{print $2}' | awk -F',' '{print $1}' | xargs)

echo "Disk: ${DISK}% | Mem: ${MEM}% | Load: ${LOAD}"

# Alert if thresholds exceeded
[ "$DISK" -gt 90 ] && echo "⚠️ Disk critical!"
[ "$MEM" -gt 95 ] && echo "⚠️ Memory critical!"

Root Cause Iron Law (inspired by gstack /investigate)

Never apply a fix without identifying the root cause first.

When an issue is detected:

1. Investigate -- collect evidence (logs, metrics, timeline)
2. Analyze -- identify the root cause, not just the symptom
3. Hypothesize -- state the cause in one sentence before acting
4. Fix -- apply the smallest targeted fix
5. Verify -- confirm the fix resolved the root cause, not just the symptom

If root cause is unclear after 2 attempts:

• Log what you tried and what you observed
• Escalate to owner with evidence, not guesses
• Do NOT apply speculative fixes that could mask the real issue

Anti-patterns:

• Restarting a service without checking why it died
• Clearing disk without checking what filled it
• Killing a process without checking what caused high usage

Destructive Command Guard (inspired by gstack /guard)

Before executing any of these commands, stop and verify:

Pattern	Risk	Required Check
rm -rf	Data loss	Verify path is correct, not /, not home
DROP TABLE / DROP DATABASE	Data loss	Confirm backup exists
git push --force	History loss	Confirm branch and remote
kill -9	Corruption	Try graceful stop first
chmod 777	Security	Use minimal permissions instead
truncate / > file	Data loss	Confirm file is a log, not config
docker system prune -a	Image loss	List what will be removed first

Subagents MUST NOT execute destructive commands. Return the command to main session for approval.

Proactive Actions

When issues detected:

Issue	Auto-Action	Alert?
Disk > 90%	Investigate what grew, then clean	Yes
Key process down	Check logs first, then restart	Yes
Cron 3+ failures	Root cause analysis, then report	Yes
Memory > 95%	Identify leak/offender, then act	Yes

Auto-fixable (safe, only after root cause check):

# Clean old logs (> 7 days)
find /var/log -name "*.log" -mtime +7 -delete 2>/dev/null
find ~/.cache -type f -mtime +7 -delete 2>/dev/null

# Clean temp files
rm -f /tmp/agent-temp-* 2>/dev/null
rm -rf ~/.cache/pip 2>/dev/null

Report Format

## 🔍 Self-Monitor Report - [TIME]

### Health Summary
| Metric | Value | Status |
|--------|-------|--------|
| Disk | XX% | ✅/⚠️/🔴 |
| Memory | XX% | ✅/⚠️/🔴 |
| Load | X.X | ✅/⚠️/🔴 |
| Services | X/Y up | ✅/⚠️ |

### Issues Found
- [Issue 1]: [Action taken or recommended]

### Top Resource Consumers
| Process | CPU% | MEM% |
|---------|------|------|
| ... | ... | ... |

Integration with Scheduled Tasks

Add to your crontab or task scheduler:

# Run health check every 30 minutes
*/30 * * * * /path/to/health-check.sh >> /var/log/health-check.log 2>&1

Or run manually as part of your workflow:

./health-check.sh

---

Security Checks

Run daily (or alongside any health check). Uses only sha256sum/shasum, grep, find, jq. Designed to complete in under 5 seconds.

Critical Files to Monitor

/path/to/workspace/SOUL.md
/path/to/workspace/IDENTITY.md
/path/to/workspace/MEMORY.md
/path/to/workspace/AGENTS.md
/path/to/workspace/TOOLS.md

1. File Integrity (SHA256 Baseline)

See ../security-guardian/scripts/check_integrity.sh for the full script.

Logic:

• On first run: generate data/security-baseline.json with SHA256 of each critical file
• On subsequent runs: compare current hashes against baseline; report any drift
• Manual reset: "reset security baseline" → regenerate baseline (do this after intentional edits to SOUL.md/MEMORY.md)

Baseline format:

{
  "generated_at": "2026-04-02T07:00:00Z",
  "files": {
    "/path/to/workspace/SOUL.md": "sha256:<hash>"
  }
}

2. Prompt Injection Scan

See ../security-guardian/scripts/scan_injections.sh for the full script.

Scans last 3 days of memory/*.md files for patterns like:

• "ignore previous instructions"
• "you are now" / "pretend you are"
• "forget everything" / "override your"
• "new system prompt"

3. Credential Leak Scan

See ../security-guardian/scripts/scan_credentials.sh for the full script.

Scans workspace *.md, *.json, *.txt, *.yaml (excluding .git/, node_modules/) for:

• api_key = 'xxx' / token = 'xxx'
• sk-... (OpenAI/Anthropic keys)
• ghp_... (GitHub tokens)
• Hardcoded passwords

Security Report Format

🛡️ Security Report — YYYY-MM-DD HH:MM UTC

✅ File Integrity: OK (5/5 files match baseline)
⚠️  Injection Scan: 1 suspicious pattern in memory/2026-04-01.md
✅ Credential Scan: Clean

 Summary: 1 warning — review flagged file

Severity & Alerting

Level	Meaning	Action
✅ OK	All clear	Log only
⚠️ WARNING	Suspicious but not confirmed	Note in report
🚨 CRITICAL	Confirmed drift or credential exposed	Alert owner immediately

CRITICAL triggers:

• SOUL.md or IDENTITY.md hash changed without "reset baseline" command
• Credential pattern found in a committed/shared file
• Injection pattern found in a file that was then acted upon

On CRITICAL: alert owner via WhatsApp.

Cron Schedule

Run daily at 06:00 UTC (before morning briefing). Silent unless CRITICAL found.

{
  "id": "security-daily",
  "schedule": "0 6 * * *",
  "timezone": "UTC",
  "task": "Run self-monitor security checks: file integrity, injection scan, credential scan. Alert owner via WhatsApp if CRITICAL.",
  "delivery": { "mode": "silent" }
}

---

📊 Skill Analytics Section

Include this in the daily self-monitor report (after security checks).

LOG="/path/to/workspace/data/skill-analytics.jsonl"
TODAY=$(date -u +%Y-%m-%d)

if [ ! -f "$LOG" ] || [ $(wc -l < "$LOG") -eq 0 ]; then
  echo "### 📊 Skill Analytics\n_No data yet._"
else
  echo "### 📊 Skill Analytics (last 24h)"
  echo ""
  
  # Total today
  TOTAL=$(grep "$TODAY" "$LOG" | wc -l)
  echo "**Invocations:** $TOTAL"
  echo ""
  
  # Top 5 skills today
  echo "**Top skills:**"
  grep "$TODAY" "$LOG" \
    | jq -r '.skill' \
    | sort | uniq -c | sort -rn \
    | head -5 \
    | awk '{printf "- %s (%d)\n", $2, $1}'
  echo ""
  
  # Unused skills (never logged)
  KNOWN="ai-pa billing-monitor calendar-setup eval hebrew-nikud maintenance meetings memory-tiering monday-for-agents owner-briefing pa-onboarding self-learning self-monitor skill-master skill-scout supervisor whatsapp youtube-watcher skill-analytics"
  UNUSED=""
  for s in $KNOWN; do
    COUNT=$(jq -r '.skill' "$LOG" | grep -c "^${s}$" 2>/dev/null || echo 0)
    [ "$COUNT" -eq 0 ] && UNUSED="$UNUSED $s"
  done
  [ -n "$UNUSED" ] && echo "**Never used:**$UNUSED" || echo "**Never used:** none"
fi

Output Example

### 📊 Skill Analytics (last 24h)

**Invocations:** 11

**Top skills:**
- supervisor (4)
- owner-briefing (2)
- meetings (2)
- whatsapp (2)
- self-monitor (1)

**Never used:** hebrew-nikud youtube-watcher