dmo-infra

name

dmo-infra

description

Entry point for ALL infrastructure changes on the dungeon_minus_one project's DigitalOcean resources — inspecting, creating, updating, scaling, or destroying the DOKS cluster, managed PostgreSQL, Spaces bucket, CDN endpoint, container registry, Doppler config, and DNSimple records. Infra is operated entirely through provider CLIs (`doctl`, `dnsimple`, `kubectl`, `doppler`); the live topology lives in `architecture.md` next to this file. Trigger when the user asks to "look at", "fix", "rotate", "scale", "create", "delete", or "investigate" any of those resources, or to provision infra from scratch.

Dungeon Minus One — Infrastructure (CLI-driven)

Production is operated imperatively through the providers' own CLIs. There is no OpenTofu/Terraform and no separate state file — architecture.md (next to this file) is the source of truth for what exists. Read it first; keep it accurate after structural changes.

There is no staging environment. Local development is the test loop, then releases go straight to prod.

Tooling

Domain	CLI	Auth
DigitalOcean (k8s, DB, CDN, Spaces keys, registry, certs, LB)	doctl	doctl auth init
DNS / domain (dungeonminusone.com)	dnsimple	dnsimple auth login (or DNSIMPLE_TOKEN)
Cluster internals (namespaces, deployments, rollouts)	kubectl	make k8s-kubeconfig
Runtime secrets	doppler	doppler login
Spaces bucket-level ops (CORS, objects)	boto3 / S3 API	SPACES_ACCESS_KEY + SPACES_SECRET_KEY

Resource inventory

See architecture.md for the topology diagram and the full inventory table (names, sizes, regions, relationships). IDs are intentionally not committed — resolve them live (the repo is public):

doctl kubernetes cluster list   --format ID,Name,Region,Version
doctl databases list            --format ID,Name,Engine,Region,Status
doctl compute cdn list          --format ID,Origin,CustomDomain
doctl compute load-balancer list --format ID,Name,IP,Status
doctl compute certificate list  --format ID,Name,Type,State,NotAfter
dnsimple zones records list dungeonminusone.com

Prerequisites

doctl auth init         # one-time, paste token from cloud.digitalocean.com
doctl account get       # sanity check
kubectl config current-context   # expect do-nyc3-dungeon-k8s
doppler me              # expect dungeon-minus-one workplace
dnsimple auth status    # expect the dungeonminusone.com account

If kubectl is pointed elsewhere: make k8s-kubeconfig (wraps doctl kubernetes cluster kubeconfig save dungeon-k8s).

Workflow rules

1. Read before write. Run a list/get to confirm the ID and the surrounding state before any delete, update, or recreate. Cloud state drifts and stale memory is a footgun.
2. No destructive ops without explicit user authorization (the global "operate carefully" rule applies — match scope, don't expand it).
3. Keep architecture.md accurate. After any structural change (new/removed resource, resized pool, changed topology), update architecture.md so the diagram stays the source of truth. Ephemeral ops (a rotation, a temporary scale-up) don't need an edit.
4. Never commit secrets or resource IDs to tracked files. This repo is public. Resolve UUIDs/connection strings/cert IDs live; never paste them into architecture.md, this skill, or other committed docs.

Bootstrap from zero (greenfield)

This is the doctl/dnsimple runbook that replaces the old OpenTofu config. Run it only to rebuild the environment from scratch; values mirror architecture.md.

REGION=nyc3

# 1. DOKS cluster — single app-pool node, autoscale off
doctl kubernetes cluster create dungeon-k8s \
  --region $REGION --version 1.34.1-do.2 \
  --node-pool "name=app-pool;size=s-2vcpu-2gb;count=1;auto-scale=false;tag=env:k8s"
doctl kubernetes cluster kubeconfig save dungeon-k8s

# 2. Managed PostgreSQL + database
doctl databases create dungeon-db --engine pg --version 18 \
  --size db-s-1vcpu-1gb --num-nodes 1 --region $REGION --wait
DB=$(doctl databases list --format ID,Name --no-header | awk '/dungeon-db/{print $1}')
doctl databases db create $DB dungeon_prod
# Allow the cluster to reach the DB (firewall rule type=k8s)
CLUSTER=$(doctl kubernetes cluster list --format ID,Name --no-header | awk '/dungeon-k8s/{print $1}')
doctl databases firewalls append $DB --rule k8s:$CLUSTER

# 3. Spaces bucket (then apply CORS via the boto3 snippet in "Spaces" below)
doctl spaces bucket create dungeon-minus-one-assets-prod --region $REGION --acl public-read
#   CORS: GET,HEAD · origin https://dungeonminusone.com · headers * · max-age 3600

# 4. CDN over the bucket (needs a custom cert id — see "TLS Certificates")
CERT=$(doctl compute certificate list --format ID,Name --no-header | awk '/dungeon-cert/{print $1; exit}')
doctl compute cdn create \
  --origin dungeon-minus-one-assets-prod.$REGION.digitaloceanspaces.com \
  --domain assets.dungeonminusone.com --certificate-id $CERT --ttl 3600

# 5. Doppler operator + secrets, then deploy
make k8s-setup-prod
make k8s-deploy TAG=<tag>

# 6. DNS — point apex/www at the LB IP, assets at the CDN (see "DNS" below)

doctl spaces bucket create requires doctl ≥ recent; if unavailable, create the bucket with the boto3 s3.create_bucket call instead.

Kubernetes

The cluster runs a single node pool (app-pool, s-2vcpu-2gb, autoscaling off, count 1) to keep cost down. Steady state is one node; the deploy pipeline temporarily scales to two and then retires the oldest node (see "Deploy-time node rotation" below). There is no in-cluster monitoring stack — use DO's built-in node/LB graphs + kubectl logs for observability.

# Cluster
doctl kubernetes cluster get dungeon-k8s
doctl kubernetes cluster node-pool list dungeon-k8s
doctl kubernetes cluster node-pool get dungeon-k8s <pool-id> -o json   # includes node ids + names

# Scale node pool count (autoscale must be off; it is)
doctl kubernetes cluster node-pool update dungeon-k8s <pool-id> --count N

# Upgrade
doctl kubernetes cluster upgrade dungeon-k8s --version <version>

# Refresh kubeconfig
doctl kubernetes cluster kubeconfig save dungeon-k8s

Deploy-time node rotation (1 → 2 → 1, oldest out)

make k8s-deploy already does this automatically: scale the pool to 2, apply + roll out, migrate, then delete-node the oldest node so each release ends on a single fresh node. The dungeon-app PodDisruptionBudget (minAvailable: 1) keeps the app serving while the old node drains. To run the rotation by hand:

POOL=$(doctl kubernetes cluster node-pool list dungeon-k8s --format ID --no-header | head -1)
doctl kubernetes cluster node-pool update dungeon-k8s $POOL --count 2   # wait for 2 Ready nodes
# ... apply manifests / roll out / migrate ...
OLDEST=$(kubectl get nodes --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[0].metadata.name}')
NODE_ID=$(doctl kubernetes cluster node-pool get dungeon-k8s $POOL -o json | jq -r --arg n "$OLDEST" '.[0].nodes[] | select(.name==$n) | .id')
doctl kubernetes cluster node-pool delete-node dungeon-k8s $POOL $NODE_ID --force   # drains by default

delete-node drains gracefully unless you pass --skip-drain; it removes that specific node and decrements the pool count (unlike --count 1, which lets DO pick which node to drop).

Resizing the node pool (sizes are immutable → migrate)

DO node pool sizes cannot be changed in place. To move to a different size, create a new pool, drain onto it, and delete the old one:

doctl kubernetes options sizes | grep <slug>                       # confirm size exists
doctl kubernetes cluster node-pool create dungeon-k8s \
  --name app-pool --size <slug> --count 1                          # autoscale off by default
kubectl get nodes -w                                               # wait for the new node Ready
for n in $(kubectl get nodes -l doks.digitalocean.com/node-pool=<old-pool> -o name); do \
  kubectl cordon $n && kubectl drain $n --ignore-daemonsets --delete-emptydir-data; done
doctl kubernetes cluster node-pool delete dungeon-k8s <old-pool-id>

Then update architecture.md to reflect the new node size.

Namespace ops use kubectl directly (cluster-internal):

kubectl get ns
kubectl get all -n prod-dungeon
kubectl rollout status deployment/dungeon-app -n prod-dungeon
kubectl logs -f -l app=dungeon-app -n prod-dungeon --all-containers

Managed PostgreSQL

Resolve the cluster id once, then reuse it:

DB=$(doctl databases list --format ID,Name --no-header | awk '/dungeon-db/{print $1}')

# Cluster
doctl databases list --format ID,Name,Status,Engine,Region
doctl databases get $DB

# DBs inside the cluster
doctl databases db list $DB
doctl databases db create $DB <name>
doctl databases db delete $DB <name> --force

# Connection string (sensitive — do not paste into tracked files)
doctl databases connection $DB --format URI

# Firewall (k8s/IP rules)
doctl databases firewalls list $DB
doctl databases firewalls append $DB --rule k8s:<cluster-id>

# Backups
doctl databases backups $DB

For schema migrations run from inside a pod:

make k8s-db-migrate

CDN

# List / get
doctl compute cdn list --format ID,Origin,CustomDomain
doctl compute cdn get <id>

# Create (custom domain requires a matching DO-managed certificate)
doctl compute cdn create \
  --origin dungeon-minus-one-assets-prod.nyc3.digitaloceanspaces.com \
  --domain assets.dungeonminusone.com \
  --certificate-id <cert-id> \
  --ttl 3600

# Flush cache after a release (force re-fetch from origin)
doctl compute cdn flush <id> --files "*"

# Repoint CDN at a different cert (used by cert rotation)
doctl compute cdn update <cdn-id> --certificate-id <cert-id>

# Delete
doctl compute cdn delete <id> --force

TLS Certificates

Certs are custom (manually issued; DNS is on DNSimple so DO can't auto-renew managed certs). Both the LB (k8s/prod/service.yaml do-loadbalancer-certificate-id) and the CDN bind a cert by id. Renewal is automated — prefer it over manual steps:

# Inspect
doctl compute certificate list --format ID,Name,Type,State,NotAfter,DNSNames
doctl compute certificate get <id> --format ID,Name,NotAfter,Type,State

# Automated renewal + rotation (issues via lego/DNSimple, repoints LB + CDN):
#   GitHub Actions > "Renew TLS Certificate" > Run workflow
#     staging: true   # first, validates DNS-01 safely
#     staging: false  # then performs the real rotation
# or locally (needs DNSIMPLE_OAUTH_TOKEN, ACME_EMAIL, doctl + kubectl auth):
DNSIMPLE_OAUTH_TOKEN=... ACME_EMAIL=[e-mail] FORCE=true ./scripts/renew_certs.sh

# Manual upload of an externally-issued cert (fallback)
doctl compute certificate create --type custom --name dungeon-cert-$(date -u +%Y%m%d) \
  --leaf-certificate-path leaf.pem --certificate-chain-path fullchain.pem --private-key-path privkey.pem

After creating a cert, repoint both the LB (edit the service annotation + re-apply, or kubectl patch svc dungeon-app-lb) and the CDN (doctl compute cdn update), then delete the superseded cert (doctl compute certificate delete <old-id> --force; DO refuses while it's still referenced).

Spaces (S3-compatible)

doctl spaces only manages access keys (and, on recent versions, bucket create/delete). For bucket-level ops (CORS, objects) use the S3-compatible API via boto3 (already in the venv) or aws/s3cmd. Required env: SPACES_ACCESS_KEY + SPACES_SECRET_KEY (the GH Actions secret names — same keys work locally).

# Keys
doctl spaces keys list
doctl spaces keys create --name <key-name> --grants "permission=read;bucket=<bucket>"

# Bucket ops via boto3 (paste into venv/bin/python)
venv/bin/python - <<'PY'
import boto3, os
s3 = boto3.session.Session().client(
    "s3",
    region_name="nyc3",
    endpoint_url="https://nyc3.digitaloceanspaces.com",
    aws_access_key_id=os.environ["SPACES_ACCESS_KEY"],
    aws_secret_access_key=os.environ["SPACES_SECRET_KEY"],
)
print([b["Name"] for b in s3.list_buckets()["Buckets"]])
PY

Common Spaces ops the bucket-level helper handles:

• List buckets → s3.list_buckets()
• List objects → s3.list_objects_v2(Bucket="<name>", Prefix="<p>")
• Get/Put CORS → s3.get_bucket_cors(Bucket=...), s3.put_bucket_cors(...) (prod rule: methods GET,HEAD, origin https://dungeonminusone.com, headers *, max-age 3600)
• Delete bucket → must be empty first; s3.delete_objects(Bucket=..., Delete={"Objects": [...]}) then s3.delete_bucket(Bucket=...)

When permanently adding/removing a bucket, update architecture.md.

Container registry

doctl registry get
doctl registry repository list dungeon-minus-one
doctl registry repository list-tags dungeon-minus-one/dungeon-minus-one
doctl registry repository delete-tag dungeon-minus-one/dungeon-minus-one <tag>

# Garbage-collect untagged blobs (frees storage)
doctl registry garbage-collection start
doctl registry garbage-collection get-active

Doppler (runtime secrets)

doppler projects
doppler environments --project dungeon-minus-one
doppler configs --project dungeon-minus-one
doppler secrets --project dungeon-minus-one --config prd --only-names
doppler secrets get <NAME> --project dungeon-minus-one --config prd --plain

# Add/update a secret
doppler secrets set <NAME>=<value> --project dungeon-minus-one --config prd

# Service token for the K8s operator (one-time per env)
doppler configs tokens create --project dungeon-minus-one --config prd \
  --name k8s-operator-prod --max-age 0 --plain

DNS (DNSimple — not DigitalOcean DNS)

dungeonminusone.com is registered at DNSimple. There is no doctl integration; use the dnsimple CLI (it has a built-in dnsimple ai helper that prints agent-oriented usage). Auth once with dnsimple auth login (stores a context), or pass DNSIMPLE_TOKEN / --token per call. records is an alias for zones records; the apex is the empty name (--name "").

ZONE=dungeonminusone.com

# List records (filter by name/type)
dnsimple records list $ZONE
dnsimple records list $ZONE --type A --name www

# Create
dnsimple records create $ZONE --type A --name www --content 1.2.3.4 --ttl 300

# Update (needs the record id from `list`)
dnsimple records update $ZONE <record-id> --content 5.6.7.8

# Delete
dnsimple records delete $ZONE <record-id>

The production site is served off the apex (@): an A record → 129.212.144.182 (the DO LB IP) plus an AAAA record. There is no www record. The assets CNAME points at the CDN. These are created by hand and shouldn't normally change. If the LB is recreated, look up the new IP and repoint the apex A record (and the AAAA if the LB's IPv6 changed):

LB_IP=$(kubectl get svc dungeon-app-lb -n prod-dungeon -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
APEX_A=$(dnsimple records list $ZONE --name "" --type A --json | jq -r '.data[0].id')
dnsimple records update $ZONE "$APEX_A" --content "$LB_IP"

Load balancer

The LB is created and owned by the k8s Service of type LoadBalancer (k8s/prod/service.yaml); never doctl compute load-balancer delete it directly — that orphans the Service. To replace it:

1. Edit k8s/prod/service.yaml (annotations, IP, etc.).
2. make k8s-deploy re-applies; DO reconciles the LB to match.

For inspecting only:

doctl compute load-balancer list
doctl compute load-balancer get <id>

Verification recipes

After any non-trivial change, verify end-to-end:

# App health
curl -fsS https://dungeonminusone.com/health

# Asset CDN
curl -fsS -I https://assets.dungeonminusone.com/<TAG>/index.html

# Pod state
make k8s-status
make k8s-logs

# DB connectivity (from a pod)
make k8s-shell
# then: alembic current && python -c "from app.db import session_factory; ..."