vps-ops

name	vps-ops
description	Operating the Hetzner VPS — container inventory, database access, deploys, backups, cleanup. Load before any work that touches the server, including deploys, docker commands, database queries, disk cleanup, or diagnosing a production incident.

VPS operations

The server hosts naŭ Platform and several unrelated systems. Assume anything you did not personally deploy belongs to something else, and check before touching it.

Access

ssh nau — connects as root. Applications live in ~/apps/<service>/, each with its own docker-compose.yml and .env.

Hetzner, Ubuntu 24.04 LTS. As of 2026-08-21: 7.6 GB RAM, 75 GB disk (49% used), 4 GB swap. (An older protocol document described a 4 GB / 40 GB CX23 with a warning not to run all services at once. That is obsolete — 24 containers run comfortably. If you find resource limits derived from 4 GB, they are stale.)

What runs here

naŭ Platform: api, app, accounts, flownau, flownau-renderer, nauthenticity, zazu, plus nau-gateway (the only container binding 80/443) and nau-backup.

Not naŭ — do not touch without asking:

System	Containers	What it is
Karen Explora	karenexplora-freescout, karenexplora-freescout-db	Self-hosted helpdesk
Violeta Cuesta	nau-violetacuesta-web, violeta-listmonk, violeta-listmonk-postgres	Client site + newsletter
Connect	connect, connect-postgres	—
Monitoring	netdata	Host metrics

Databases

Seven Postgres instances, each with its own role. There is no postgres superuser role — trying to use one fails with role "postgres" does not exist. Read the credentials from the container:

U=$(docker exec <svc>-postgres printenv POSTGRES_USER)
D=$(docker exec <svc>-postgres printenv POSTGRES_DB)
docker exec <svc>-postgres psql -U "$U" -d "$D" -c "SELECT ..."

Container	Role / DB	Size (2026-08-21)
api-postgres	nau_api	13 MB
flownau-postgres	flownau	29 MB
nauthenticity-postgres	nauthenticity (pgvector)	33 MB
zazu-postgres	zazu	9 MB
connect-postgres	connect	8 MB
karenexplora-freescout-db	freescout	13 MB
violeta-listmonk-postgres	listmonk	11 MB

Three Redis instances (api, flownau, nauthenticity) hold BullMQ queue state — transient, not backed up.

Deploys

Push to main → GitHub Actions → build image → push to GHCR → SSH → docker compose pull → prisma migrate deploy → docker compose up -d. No manual step, and no manual step should be added.

Migration gotcha. prisma migrate deploy runs from the container's default working directory (/app), but only flownau keeps its schema there. The deploy scripts pass -w explicitly:

Service	Schema path
api	/app/apps/api
nauthenticity	/app/apps/nauthenticity
zazu	/app/packages/zazu-db
flownau	/app (default)

Without -w the command fails with Could not find Prisma Schema — and it failed that way on every deploy for months without anyone noticing, because the script did not stop on error. The deploy scripts now start with set -e for exactly this reason. If you ever see a deploy step "succeed" while printing an error, treat that as a bug in the script, not noise.

Rollback: every push produces a sha-<git-sha> tag in GHCR. Pin it in .env and docker compose up -d.

Before deploying a worker service

nauthenticity runs BullMQ workers (ingestion, download, optimization, compute) and flownau runs Remotion renders and a post scheduler. Restarting mid-job loses that job — recoverStuckRuns re-enqueues stuck runs on startup but cannot recover work that was mid-execution.

Check queues first, and prefer 03:00–06:00 UTC for worker-touching deploys. Note that nauthenticity has no curl inside the container, so query the queue endpoint from the host.

The API runs journal summary crons at 18:00, 20:00 and 23:00 UTC; avoid deploying across those.

Backups

State as of 2026-08-21 — read this before trusting anything: the nightly nau-backup job (0 2 * * *) dumps correctly but its rclone upload to R2 has been returning 401 Unauthorized since at least 2026-07-31, with zero successful uploads in the entire log. The credential (R2_ACCESS_KEY_ID=56daa1ab…) is dead. Zazu uses a different R2 key and is unaffected.

Two structural faults in that script, worth not repeating anywhere:

1. It uses set -e around a per-service loop, so the first failed upload aborts the whole run. Only api was ever dumped; nauthenticity, zazu and flownau never were.
2. It fails silently — nothing alerts, so it looked configured for weeks while protecting nothing.

The correct shape for a backup job: dump everything → upload everything → verify → only then delete local copies, with a per-service failure isolated from the rest, and a loud alert on any failure. Verify by checksum after transfer, not by exit code alone.

Manual backup (this pattern is proven):

# dump all seven, plus globals for roles/passwords, then checksum
docker exec <c> pg_dump -U "$U" -d "$D" --no-owner --no-privileges | gzip > <c>.sql.gz
docker exec <c> pg_dumpall -U "$U" --globals-only | gzip > <c>.globals.sql.gz
sha256sum *.gz > SHA256SUMS

Then scp off the server and run sha256sum -c on the copy. A backup that only exists on the machine it protects is not a backup.

Existing snapshots: ~/backups/manual-<timestamp>/ on the server, mirrored to C:\Users\Sam\backups\nau-vps\ locally.

Cleanup

• docker image prune -f after docker compose up -d — removes the previous untagged image.
• Never docker system prune --volumes in production. Volumes hold the databases.
• docker system df to check reclaimable space; report it, do not prune automatically.
• Before deleting any file or directory: look inside it. A deploy12.tar.gz in ~/apps/ turned out to be a build artifact of a client site, deleted on the strength of its name alone. It was harmless, but that was luck, not method. Archive to ~/backups/ first when unsure.