data-volume

# Data volume The production host's data directory (`/var/www/blot/data`: blog folders; git repositories and generated assets live only in S3) is one XFS-formatted EBS volume. Blog folders are planned to move to S3 (#2258), after which the volume is removed. Until `data/git` is deleted from the host (the cleanup PR of #2259 has it as a deployment step) the old bare repositories are still on it, unused. It is mounted by `mount-data-volume.service` by its volume ID (`/etc/blot/data-volume`, checked against the `.blot-data-volume` marker at the volume's root), shared, and the app containers bind it with `bind-propagation=rslave` (config/host/README.md). `config/host/data-volume/README.md` explains the scripts and every step. This skill does not repeat it: it holds the rules, what to expect, and the log. Related: `scripts/read-only.js` (the app's read-only freeze, also usable on its own), `redis-host` (Redis, which shares the host), and the host cycle in `config/host/README.md`, which runs this script's `move` flow. ## Rules 1. **Ask the operator before every production command**: `resize.sh` (even `--dry-run`: it copies host.sh to the host, and to the new host for `move prepare`), `ssh blot`, `docker exec`, `read-only.js on/off`. State the exact command. Read-only AWS `describe-*` and CloudWatch calls (`--profile blot --region us-west-2`) are fine without asking; say what you ran. 2. **Let the operator run `resize.sh` in their own terminal.** It asks yes/no at each step and the "Freeze now?" moment is theirs to choose; `--yes` would skip it. Watch from the side (CloudWatch, plain HTTP requests) and read what they paste. 3. **Drill before changing anything on the freeze, copy or swap path.** `drill.sh launch/setup` on a throwaway from production's AMI (and `launch-target/setup-target` for a move), then `resize.sh --host drill …`. Keep a subagent away from production mechanically: an ssh config passed with `-F` (so the `blot` alias does not resolve) and `--host drill` on every call. 4. **Tear down only by explicit ID**, re-checking `BlotDrill=true` on each resource immediately before deleting it, and never anything that is the app host or its volumes. Show the operator the list and get a yes first. 5. **Never delete the old volume or its pre-resize snapshot without the operator.** They are the rollback for about a week after a shrink, and after a move the old host with its volume is the rollback until the cycle's finish. **`move freeze` leaves the app frozen and the source read-only on purpose.** Never walk away from it: the next step is the switch, or `move thaw`, inside the freeze TTL. 6. **No freeze within 10 minutes of 01:00 and 05:00 UTC** (host cron and the scheduler delete files then; the script refuses). 7. **After `finish` released an old volume on the host, deploy the app before the next resize.** The release breaks the running containers' link to host mounts until they are recreated (README, "After it"). 8. Public repo: no production instance, volume or snapshot IDs, account IDs, key names or ssh details in commits, PRs or issues (drill IDs are fine), and sizes only as percentages here. ## What to expect at production scale - Pass 1 (everything) runs for hours at roughly 100-120 MB/s, the site fully live; the instance's EBS byte burst balance drains a few percent per five minutes during it and refills afterwards. Watch `EBSByteBalance%` on the instance and read latency on the old volume; stopping the copy (`ssh blot sudo bash /tmp/blot-data-volume/host.sh stop-copy`) and re-running later loses nothing. - Pass 2 (nothing changed) is a metadata walk of both trees: minutes with a cold cache. - The frozen pass runs in parallel shards and takes about a minute; the whole read-only window (grace, lock drain, remount, copy, snapshot call, swap) is about two minutes. Blogs and the dashboard keep serving; writes get 503s, syncs wait and catch up within seconds of the freeze lifting, iCloud's retry within about 20s. - Afterwards: an app deploy; then `resize.sh finish` releases the old volume on the host without a reboot and detaches it. ## Incident log Newest last. Times in UTC. ``` ### <date> — <operation> - What / why: … - Timings: … - Problems: … - Follow-up: … ``` ### 2026-10-09 — first shrink (gp2 to a smaller gp3 volume) - What / why: the data volume was about half used and on gp2; shrunk to a gp3 volume at about 70% used after the copy (logs/ and tmp/ contents not copied), roughly 40% cheaper. PRs #2099 (read-only freeze), #2100 (mount by ID, rslave), #2101 (Sync paused), #2102 (scripts), #2103 (finish releases on the host). - Drill first (AL2, kernel 5.10): it found that systemd 219 rejects the copy unit's properties (every live pass would have failed to start) and that `docker exec` fails on paused containers; both fixed before production. - Timings: pass 1 1h49m, pass 2 4m46s, frozen copy 1m11s over ~3,100 shards, read-only window 2m00s, ending 18:12:50 (swap 18:12:48). EBS byte burst balance bottomed out around 45%; old-volume read latency about 1 ms during the copy. - Afterwards: proxy and app deploys; the host still held the old volume under the new one, released by hand without a reboot (umount, umount, mount-data-volume.sh; containers unaffected), then `finish` detached it. That sequence is now `host.sh release-old`, offered by `finish`. - Follow-up: delete the old volume and its snapshot from 16 Oct (TODO). ### 2026-10-15 — second shrink, after generated assets moved to S3 - What / why: #2111 made the storage bucket the only home of generated assets, so data/static was deleted (about 60% of what the volume held; a transient `systemd-run ... rm -rf`, minutes). Then shrunk to a gp3 volume at about 85% used, with gp3's included IOPS and throughput (no `--throughput`: the app never needed more). New volume vol-0d6b1e4c0ff79d852, old vol-07c767ac8b6aeab4f, snapshot snap-02793374b4e56ea7b. No drill: #2111 only changed printed text in the scripts. - Timings: pass 1 about 40 min at ~118 MB/s (EBS byte balance bottomed out around 75%), frozen copy 16s over ~1,900 shards, read-only window 59s. - Problems: the final copy printed `Terminated sleep "$budget"`: teardown killed the watchdog's sleep before the watchdog, which could have flagged an in-time copy as over budget (#2116). `finish` then refused to release the old volume because chronyd (PrivateTmp, up since boot) had it in its own namespace, and asked for another deploy, which can't help; its copy is a slave of the host's mount and goes with the host's umount, so it is now a `follower=` that doesn't block (#2117). Release and detach then went cleanly. - Afterwards: the operator deleted the old volume the same day (the snapshot is the rollback). Follow-up: delete that snapshot from 22 Oct (TODO). New data-disk emails at 90/95/99% and a daily-email line (#2115) watch the smaller margin; grow with `resize.sh grow`.