setup-cpu-proxy-server

name

setup-cpu-proxy-server

description

Set up and verify a new Draw Things CPU proxy and Envoy server using the scripts in Scripts/ServerManagement/CPUScript. Use when Codex needs to bootstrap a root-access Linux CPU server over SSH/Tailscale, upload model-list and envoy-config.yaml, install Docker/Tailscale images, install or copy compute.drawthings.ai TLS certificates, launch proxy_service and envoy_grpc_web_proxy, and debug port 443, Envoy 8443, TLS, or gRPC Echo connectivity.

Draw Things CPU Proxy Server Setup

Use this workflow from Scripts/ServerManagement/CPUScript.

Files

• InitCPUServer.sh: install/update Docker, install Tailscale, run tailscale up --ssh, lock the root password, and pull drawthingsai/draw-things-proxy-server-cli:latest plus envoyproxy/envoy:v1.28-latest.
• InitCertificate.sh: request a Let's Encrypt cert for compute.drawthings.ai with standalone certbot on port 80, set the permissions required by the non-root proxy container, and install the renewal deploy hook.
• restart-drawthings-tls.sh: Certbot deploy hook that repairs renewed certificate permissions and restarts proxy_service and envoy_grpc_web_proxy so both processes load the renewed certificate.
• LaunchCPUServer.sh: start proxy_service on public 443 and Tailscale control port TS_IP:50002.
• LaunchEnvoyServer.sh: start envoy_grpc_web_proxy with host networking and the provided Envoy config.
• model-list: mounted through its containing directory as /app/Documents/<filename> so host-side edits and control-panel writes target the same file.
• envoy-config.yaml: mounted read-only as /etc/envoy/envoy.yaml.

Inputs

Have these ready before launch:

• New CPU server SSH target, usually root@<tailscale-ip>.
• Datadog API key. Treat it as a secret; do not commit or echo it in summaries.
• Let's Encrypt notification email if issuing a new cert.
• Optional existing cert source server, if DNS is not routed to the new VM yet.
• Updated model-list.
• A known-good envoy-config.yaml; the admin socket must include a port, typically 127.0.0.1:9901.

Initial State Check

Check SSH, OS, Docker, Tailscale, certs, containers, and ports:

ssh root@<new-ts-ip> 'hostname; uname -a; id; command -v apt || true; command -v docker || true; command -v tailscale || true; tailscale ip -4 2>/dev/null || true'
ssh root@<new-ts-ip> 'docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || true; ss -ltnp | grep -E ":(80|443|8443|50002)" || true; ls -la /etc/letsencrypt/live/compute.drawthings.ai 2>/dev/null || true'

Upload the Script Bundle

Install the CPUScript files on the new server:

ssh root@<new-ts-ip> 'mkdir -p /root/CPUScript'
scp InitCertificate.sh InitCPUServer.sh LaunchCPUServer.sh LaunchEnvoyServer.sh restart-drawthings-tls.sh envoy-config.yaml model-list root@<new-ts-ip>:/root/CPUScript/
ssh root@<new-ts-ip> 'chmod +x /root/CPUScript/*.sh; wc -l /root/CPUScript/model-list; ls -la /root/CPUScript'

If the model list changes, recopy only that file and verify checksum:

scp model-list root@<new-ts-ip>:/root/CPUScript/model-list
shasum -a 256 model-list
ssh root@<new-ts-ip> 'shasum -a 256 /root/CPUScript/model-list; wc -l /root/CPUScript/model-list'

If the local Envoy config is stale, copy the known-good config from the current CPU server, then upload it:

scp root@<current-cpu-ts-ip>:/root/envoy/envoy-config.yaml ./envoy-config.yaml
rg -n "port_value|address:|filename:|sni:|admin:" envoy-config.yaml
scp envoy-config.yaml root@<new-ts-ip>:/root/CPUScript/envoy-config.yaml

Run CPU Server Init

Run the dependency/image setup:

ssh root@<new-ts-ip> 'cd /root/CPUScript && ./InitCPUServer.sh'

Expect apt output, Docker install/update, Tailscale install/update, root password lock, and Docker image pulls. A pending kernel upgrade warning is not a launch blocker.

Certificate Strategy

The scripts expect:

/etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem
/etc/letsencrypt/live/compute.drawthings.ai/privkey.pem

If compute.drawthings.ai already routes to the new VM and public port 80 reaches it, run:

ssh root@<new-ts-ip> 'cd /root/CPUScript && printf "%s\n" "<email>" | ./InitCertificate.sh'

If DNS is not routed to the new VM yet, certbot standalone will fail. In that case, copy only the active cert and key from the current CPU server. Do not copy the whole /etc/letsencrypt unless deliberately migrating Certbot renewal ownership; that also copies ACME account keys, renewal configs, and archived cert material.

ssh root@<current-cpu-ts-ip> 'ls -l /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem; openssl x509 -in /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem -noout -subject -issuer -dates'

mkdir -p /private/tmp/cpu-cert-transfer
chmod 700 /private/tmp/cpu-cert-transfer
scp root@<current-cpu-ts-ip>:/etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem root@<current-cpu-ts-ip>:/etc/letsencrypt/live/compute.drawthings.ai/privkey.pem /private/tmp/cpu-cert-transfer/
chmod 644 /private/tmp/cpu-cert-transfer/fullchain.pem
chmod 600 /private/tmp/cpu-cert-transfer/privkey.pem

ssh root@<new-ts-ip> 'mkdir -p /etc/letsencrypt/live/compute.drawthings.ai && chmod 755 /etc/letsencrypt /etc/letsencrypt/live /etc/letsencrypt/live/compute.drawthings.ai'
scp /private/tmp/cpu-cert-transfer/fullchain.pem /private/tmp/cpu-cert-transfer/privkey.pem root@<new-ts-ip>:/etc/letsencrypt/live/compute.drawthings.ai/
ssh root@<new-ts-ip> 'chmod 644 /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem; chmod 644 /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem; openssl x509 -in /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem -noout -subject -issuer -dates'
rm -f /private/tmp/cpu-cert-transfer/fullchain.pem /private/tmp/cpu-cert-transfer/privkey.pem

Use 0644 for privkey.pem with the current Docker launch because the proxy process inside the container may not run as root. If proxy_service logs Permission denied opening privkey.pem, fix permissions and recreate the container.

Automatic Renewal and TLS Reload

Copying only fullchain.pem and privkey.pem creates usable TLS files but does not migrate Certbot renewal ownership. A managed certificate must appear in certbot certificates and have all of:

/etc/letsencrypt/renewal/compute.drawthings.ai.conf
/etc/letsencrypt/archive/compute.drawthings.ai/
/etc/letsencrypt/live/compute.drawthings.ai/ -> ../../archive/compute.drawthings.ai/

InitCertificate.sh installs this executable deploy hook:

/etc/letsencrypt/renewal-hooks/deploy/restart-drawthings-tls.sh

Certbot normally creates /etc/letsencrypt/archive as root-only and creates each renewed private key as mode 0600. The proxy runs as appuser, so following the live symlink fails unless the archive directories are searchable and the private key is readable. The hook reapplies:

chmod 0711 /etc/letsencrypt/archive
chmod 0755 /etc/letsencrypt/archive/compute.drawthings.ai
chmod 0644 /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem

The proxy and the statically configured Envoy listener load the certificate into memory at process startup. After a successful renewal, the deploy hook restarts both containers so they load the new certificate:

/usr/bin/docker restart proxy_service
/usr/bin/docker restart envoy_grpc_web_proxy

Verify the snap timer and the complete renewal-to-reload path:

systemctl is-enabled snap.certbot.renew.timer
systemctl is-active snap.certbot.renew.timer
certbot renew --dry-run --run-deploy-hooks
docker ps --filter name=proxy_service
docker ps --filter name=envoy_grpc_web_proxy

Expected results are enabled, active, Congratulations, all simulated renewals succeeded, and both containers returning to Up. --run-deploy-hooks is needed only to make a dry run execute deploy hooks; a real successful renewal automatically executes every executable in /etc/letsencrypt/renewal-hooks/deploy.

Confirm that ports 443 and 8443 serve the same renewed certificate:

openssl s_client -connect 127.0.0.1:443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -serial
openssl s_client -connect 127.0.0.1:8443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -serial

Launch Services

Launch the CPU proxy first. Any extra arguments after the model-list path are passed through to ProxyServiceCLI, including -g GPU worker ranges:

ssh root@<new-ts-ip> 'cd /root/CPUScript && ./LaunchCPUServer.sh "<DATADOG_API_KEY>" /root/CPUScript/model-list -g 100.70.225.70:40001-40008:1 -g 100.115.89.107:40001-40008:1 -g 100.83.253.70:40001-40008:1 -g 100.64.66.84:40001-40008:1 -g 100.121.197.14:40001-40008:1 -g 100.110.198.32:40001-40008:1'

LaunchCPUServer.sh mounts the directory containing the model list as /app/Documents and passes --model-list-path /app/Documents/<filename>. It also makes that directory and file writable by the container's non-root appuser; this is required because update-model-list writes atomically by creating a temporary file in the same directory.

Do not use the old single-file read-only mount:

-v /root/CPUScript/model-list:/app/model-list:ro

That mount lets the server read the file, but it prevents the control-panel update-model-list RPC from writing back. The server currently uses try? around the file write, so the CLI can appear to succeed even when the file was not updated.

Then launch Envoy:

ssh root@<new-ts-ip> 'cd /root/CPUScript && ./LaunchEnvoyServer.sh /root/CPUScript/envoy-config.yaml'

Expected ports:

• proxy_service: 0.0.0.0:443->8080/tcp and <tailscale-ip>:50002->50000/tcp.
• envoy_grpc_web_proxy: host network, listening on 0.0.0.0:8443.
• Envoy admin: 127.0.0.1:9901.

Verification

Verify containers, listeners, logs, TLS, and Envoy readiness:

ssh root@<new-ts-ip> 'docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"'
ssh root@<new-ts-ip> 'docker inspect --format "{{range .Mounts}}{{println .Source \"->\" .Destination \"RW=\" .RW}}{{end}}" proxy_service'
ssh root@<new-ts-ip> 'docker inspect --format "{{range .Args}}{{println .}}{{end}}" proxy_service | grep -E "^100\\..*:40001-40008:1$" || true'
ssh root@<new-ts-ip> 'ss -ltnp | grep -E ":(443|8443|50002|9901)" || true'
ssh root@<new-ts-ip> 'docker logs --tail 50 proxy_service 2>&1; docker logs --tail 80 envoy_grpc_web_proxy 2>&1'
ssh root@<new-ts-ip> 'openssl s_client -connect 127.0.0.1:443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -subject -dates'
ssh root@<new-ts-ip> 'openssl s_client -connect 127.0.0.1:8443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -subject -dates'
ssh root@<new-ts-ip> 'curl -fsS --max-time 5 http://127.0.0.1:9901/ready'

Verify external GCP access:

curl -vk --connect-timeout 10 https://<external-ip>:443/ImageGenerationService/Echo
openssl s_client -connect <external-ip>:443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -subject -dates -ext subjectAltName
curl -vk --resolve compute.drawthings.ai:443:<external-ip> --connect-timeout 10 https://compute.drawthings.ai:443/ImageGenerationService/Echo

An HTTP 415 from a raw GET to a gRPC endpoint is acceptable; it means the request reached the service but was not a valid gRPC call.

Model List Updates

With the updated launch script, both of these update the same model-list file:

• Editing /root/CPUScript/model-list on the CPU host.
• Running control-panel update from a local checkout:

bazel-bin/Apps/ProxyServerControlPanelCLI -h <new-ts-ip> -p 50002 update-model-list /path/to/model-list

The container should show:

/root/CPUScript -> /app/Documents RW= true

If this mount is not writable, update-model-list will not reliably update the host file. After a successful update, Echo and FilesExist read modelListPath on demand, so they should reflect the new file without restarting. A restart is still useful after changing launch arguments such as -g GPU workers.

DNS and App Client Notes

The copied Let's Encrypt cert is for compute.drawthings.ai, not an IP address. If the app connects with:

static let host = "<external-ip>"
static let port = 443

and uses full hostname verification, TLS fails because the certificate SAN has DNS:compute.drawthings.ai and no IP SAN. Symptoms may surface as GRPC.ConnectionPoolError.shutdown, and the server logs may show no Echo call because the client fails before the RPC reaches application code.

Before DNS is updated, either:

• connect to <external-ip> but set gRPC TLS hostnameOverride: "compute.drawthings.ai" / SNI to compute.drawthings.ai, or
• temporarily disable hostname verification only for a controlled test path.

After DNS is updated, prefer:

static let host = "compute.drawthings.ai"
static let port = 443

Then full TLS verification should pass without special handling.

Troubleshooting

• InitCertificate.sh fails with Let's Encrypt unauthorized or 503: compute.drawthings.ai is not routed to this VM on public port 80. Copy the existing cert/key to launch now, then fix DNS/load-balancer routing for renewal.
• proxy_service restarts with fopen(... privkey.pem ... Permission denied): make /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem readable by the container, then rerun LaunchCPUServer.sh.
• update-model-list prints success but the host file does not change: the model-list path is probably mounted read-only or the containing directory is not writable by the container user. Relaunch with the updated LaunchCPUServer.sh directory mount.
• Envoy restarts with validation error on AdminValidationError.Address ... port_specifier is required: update envoy-config.yaml so admin.address.socket_address includes port_value: 9901, or copy the known-good config from the current CPU server.
• grpcurl <ip>:443 ... fails certificate verification: use -servername compute.drawthings.ai or update DNS and use the hostname.
• No Echo log appears on the server: suspect client-side TLS/channel setup before debugging server Echo handling.