Write content into Blot's production test blogs (gittest, dbtest, drivetest, icloudtest - one per sync client) the way a real user would, wait for Blot to sync and build it, then verify it end to end - the published page, CDN and /_assets responses, the…
Look after Blot's production Redis host (its own EC2 instance, built and operated with config/redis/): triage a "Redis host:" alert email from app/scheduler/check-redis-host.js (failed background save, Redis refused writes - OOM/MISCONF/NOREPLICAS, other…
Work out why the blot-blue and blot-green unit containers from the most recent production deployment have restarted — distinguishing a normal deploy-triggered restart from a crash (V8 heap OOM, Linux cgroup OOM kill, a lost sync folder lock - `[LOCK…
Grow or shrink the EBS volume that holds the app host's data directory (/var/www/blot/data), or move its contents to another host, with config/host/data-volume/resize.sh (grow in place; shrink by live rsync passes, a brief read-only freeze and a mount swap…
Daily cheap check that no runtime or OS Blot depends on (Node.js, Alpine, OpenResty/nginx, Pandoc, Amazon Linux 2 on the app/proxy host, Amazon Linux 2023 and redis6 on the Redis host, the macserver's Node) is behind on a security release or past end-of-life.…
Analyze the production openresty access log for IP ranges (IPv4 /24, IPv6 /48) that cost a lot of egress and are broadly abusive (scraping with rotating user agents, repeat scanning, hammering the CDN hostname), set aside CDN edges and legitimate crawlers,…
Investigate an "iCloud resync found changes" admin email (the ICLOUD_RESYNC_ISSUE email a macserver-requested resync sends when it found changes, Fix() repairs or errors, with the macserver's reason), an "iCloud sync issue" hourly digest, or any other iCloud…
Analyze production Node.js app container response times to find slow-rendering sites, cross-checking against nginx queuing delay to rule out false positives (a site only looks slow because the event loop was blocked by a different, pathological site). Use…