| name | lms-health-check |
| description | Health, liveness & performance check of the production LMS-TMS site, with known auto-fixes. Use when asked to check if the site/prod is up, healthy, slow, or "loading forever", or to verify after a deploy/firewall change. |
LMS-TMS Health Check
Verify the production LMS-TMS is healthy, alive, and fast — and auto-fix the known DB-connectivity failure mode.
Infra specifics (SSH target, app container name) come from environment variables set in the gitignored .claude/settings.local.json (LMS_DOMAIN, LMS_SSH_TARGET, LMS_APP_CONTAINER). Never hardcode host IPs, container IDs, or credentials in this file.
Step 1 — External checks (no credentials needed; work from anywhere)
Let DOMAIN="${LMS_DOMAIN:-lms-tms.tertiaryinfotech.com}".
- Homepage:
curl -sS -o /dev/null -w "%{http_code} %{time_total}s" --max-time 12 "https://$DOMAIN/" → expect 200 in < 3s.
- DB-backed readiness (the key canary):
curl -sS --max-time 12 "https://$DOMAIN/api/health" → returns JSON.
- Parse the body, not just the status. The endpoint currently returns HTTP 200 even when the DB is down (known bug — see CLAUDE.md). Healthy =
"database":"connected".
- A ~15s hang on this call means the app→DB path is broken (the classic outage).
If both pass → report healthy and stop (when running on a schedule, stay silent unless state changed).
Step 2 — Diagnose (only if the canary fails/hangs or database != connected)
Requires LMS_SSH_TARGET + LMS_APP_CONTAINER. SSH in (ssh "$LMS_SSH_TARGET"):
- App container up?
docker ps --filter name=$LMS_APP_CONTAINER --format '{{.Status}}'
- App → DB reachable?
docker exec $LMS_APP_CONTAINER node -e "const n=require('net');const s=n.connect({host:'<db-host>',port:6433});s.setTimeout(5000);s.on('connect',()=>{console.log('OK');s.end()});s.on('timeout',()=>console.log('TIMEOUT'));s.on('error',e=>console.log('ERR '+e.message))"
- Firewall intact?
ufw status | grep -E '5432|5439|6433' and iptables -S DOCKER-USER
Step 3 — Known auto-fix (DB connectivity)
Root cause of the 2026-07-01 outage: a UFW DENY on the DB ports blocked the app container's hairpin path to the DB (UFW does not filter Docker-published ports for the internet, but does break the container's INPUT-chain hairpin). Fix:
ufw insert 1 allow from 10.0.0.0/8 to any port 6433 proto tcp
If the internet-lockdown (DOCKER-USER) rules are missing after a Docker restart, re-apply: /usr/local/sbin/lms-db-firewall.sh.
Re-run Step 1 to confirm recovery.
Step 4 — Performance snapshot (optional)
- API/page response times (
time_total above) should be < 1s warm.
- On the server:
docker stats --no-stream $LMS_APP_CONTAINER (CPU/mem), uptime (load), df -h / (disk).
Reporting
- Scheduled/loop runs: report only on a state change (healthy→broken, broken→fixed, fix-failed). Stay quiet when healthy.
- Manual runs: give a short table (homepage, DB canary, response times) + any action taken.
