wabi-deploy-debug

name	wabi-deploy-debug
description	Wabi production deployment and runtime debugging. Covers the Rust wabi-server binary (rust_embed static frontend), the SvelteKit build-mode mismatch (adapter-static SPA vs adapter-node SSR and why the Rust server needs index.html), Cloudflare/caddy/cloudflared tunnel WebSockets, WabiDB data-dir locks, and the selfhostability check. Use when a Wabi deploy will not boot, the login page is blank or stuck on "Starting Wabi"/"Work Offline", socket.io WS fails through Cloudflare, wabi-server restart-loops with "engine already running", or the user asks whether Wabi is still selfhostable or hardwired to wabi.chat.
version	1.0.2
author	Hermes Agent
license	MIT
metadata	{"hermes":{"tags":["wabi","deploy","debug","sveltekit","cloudflare","cloudflared","wabidb","rust-embed","spa"],"related_skills":["wabi-frontend-polish"]}}

Wabi Deploy & Runtime Debug

Class-level workflow for the recurring "the deployed Wabi will not boot / login is broken / socket will not connect / is this still selfhostable" task. This is the deploy + runtime layer, NOT visual polish (that is wabi-frontend-polish).

The deploy stack (Tim dev server, root@100.96.11.45): a single Rust wabi-server binary (binds :3000) built with cargo build --release -p wabi-server, served behind a caddy reverse proxy (:8088) and cloudflared tunnels (wabi-cloudflared-named etc.) that expose wabi.chat. The binary embeds the frontend via rust_embed (StaticAssets = frontend/build), so the release build MUST run bun run build (adapter-static, see below) BEFORE cargo build --release.

Build mode is the #1 footgun

frontend/svelte.config.js selects the adapter:

• STATIC_BUILD=1 → @sveltejs/adapter-static with fallback: 'index.html' → emits a top-level index.html + _app/. The Rust serve_static SPA-fallback needs this index.html. This is what the Rust binary requires.
• default (no env) → @sveltejs/adapter-node → emits handler.js/server//client/ but NO top-level index.html. The Rust serve_static cannot serve it (404 on every route). Only use this if you also run the SSR handler.js somewhere — the Rust binary does NOT.

So: the deploy pipeline is STATIC_BUILD=1 bun run build then cargo build --release -p wabi-server. adapter-node alone 404s.

Addon feature flag is the #2 footgun

The Lore addon is optional in Cargo.toml:

[features]
default = []
addons = ["wabi-webhooks", "wabi-lore"]

Building with cargo build --release -p wabi-server (no --features addons) produces a binary with zero Lore API routes — GET /api/addons/lore/health returns not_found. The addon code compiles but the routes are never registered.

Fix: cargo build --release -p wabi-server --features addons

Without this flag, the LoreChannelShell will show "Lore service unavailable" even with a healthy Lore server running, because the API endpoints don't exist.

Symptom → cause map

Symptom	Likely cause	Fix
localhost:3000 not responding, localhost:3001 healthy	Server on wrong port; serverUrl.ts rewrites :3000 → :3001	Fix all four rewrite paths in serverUrl.ts to return :3000 instead of :3001; rebuild frontend; restart wabi-server; hard-refresh browser
Every route 404, /health 200	adapter-node build (no index.html) embedded	rebuild with STATIC_BUILD=1
Page stuck on "Starting Wabi" / blank, no API calls	boot shell never hidden OR SPA boot crash (see below)	see SPA boot crash
Tab crashes / "can't establish connection to wss://wabi.chat/socket.io"	cloudflared strips WS Upgrade (esp. quic tunnels)	polling fallback + tunnel protocol (see WS section)
Public https://wabi.chat → Cloudflare 502 on ALL routes, but Tim origin healthy	dead tunnel edge / stale QUIC connectors — NOT origin, NOT locks	see "Dead tunnel edge" below + references/cloudflared-ws-and-wabidb-lock.md
wabi-server restart-loops "engine already running"	stale WabiDB lock (deeper path)	remove BOTH lock files (see Locks)
Login flash then bounce to login; /api/user/me → 401 token revoked	Permanent ban in data/wabi-server/revocations.json users: [id] (legacy revoke_user)	Clear users: [], restart wabi-server. See references/login-bounce-token-revocation.md
Logged in but messages vanish / can't create channels; console Join as: null after force-reset	Socket double-connect with empty username; createChannel ignored REST; and/or nav calls joinChannel only (never sets currentChannel)	references/post-login-socket-and-channels.md. Probe REST first. Nav must use switchChannel (no registry gate).

SPA boot crash (the big one) — ROOT CAUSE: terser minification

Symptom: STATIC_BUILD=1 build deploys, but on load the tab either (a) sits forever on the "Starting Wabi" boot shell (boot shell never hides — wabi:boot-hide event never fires, no JS error), or (b) the renderer crashes (Playwright page.on('crash'), document empty, no catchable pageerror). Dev (bun run dev, SSR) and bun run check are clean. In a REAL browser it usually just sits stuck (the headless OOM is a Playwright artifact, not a true crash).

ACTUAL ROOT CAUSE (found 2026-07-19): frontend/vite.config.ts used minify: 'terser' with terserOptions.compress.drop_console. Terser's aggressive compress/drop_console pass breaks Svelte's store runtime / circular re-export init order in the client bundle, so a store used as $store is undefined → n.subscribe is not a function → uncaught → boot IIFE dies before dismissDocumentBootShell() hides the shell. This is a BUILD/MINIFIER bug, not an app-code regression — the pre-overnight anchor also "crashes" only in the flaky headless harness, and the app worked for the user on Jul-17.

THE FIX: frontend/vite.config.ts → minify: !process.env.TAURI_DEBUG (esbuild by default; false only under TAURI_DEBUG). Drop the terserOptions: { compress: { drop_console } } block — it is inert once terser is no longer the minifier and was the thing breaking the app. After the fix, STATIC_BUILD=1 bun run build produces a working SPA. Rebuild the Rust binary to embed it, redeploy.

CORS / Frontend Port Detection Issue

Symptom: Frontend makes API calls to http://100.x.x.x:3001 but server is on port 3000. Console shows:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://100.87.255.66:3001/api/public/frontend-app-metadata

Root Cause: frontend/src/lib/serverUrl.ts has a port rewrite bug:

// 4. Direct container access: frontend on :3000, backend on :8080 on the same host.
if (port === '3000') {
    return { url: `${protocol}//${hostname}:3001`, source: 'docker_port_rewrite' };
}

When the embedded SPA is served on port 3000 (via rust_embed), the frontend incorrectly rewrites the backend URL to port 3001. This is backwards logic for the embedded-serve pattern where frontend and API share the same origin.

Fix (confirmed 2026-07-28, Ronin's machine):

frontend/src/lib/serverUrl.ts has four places that incorrectly rewrite port 3000 → 3001. All must be changed to :3000 (same port, embedded-serve pattern):

1. SSR default (line 141): http://localhost:3001 → http://localhost:3000
2. env_override_dev_rewrite (line 177): :3001 → :3000
3. env_override_dev_rewrite_invalid (line 181): :3001 → :3000
4. dev_tauri (line 193): :3001 → :3000
5. dev_vite (line 200): :3001 → :3000
6. docker_port_rewrite (line 206): :3001 → :3000

After editing serverUrl.ts, rebuild frontend and restart wabi-server:

cd /home/Ronin/wabi/frontend && npm run build
cp -r .svelte-kit/output/* build/
pkill -f wabi-server; sleep 2
WABIDB_ROOT_KEY=... WABI_CORS_ORIGINS="..." ./target/release/wabi-server --port 3000 --host 0.0.0.0 --data-dir ./data &

Hard-refresh browser (Ctrl+Shift+R) to get the new JS bundle.

CORS Headers Stripped by Cloudflare Tunnel

Symptom: After fixing the port rewrite issue, localhost:3000 CORS works correctly with Access-Control-Allow-Origin: https://wabi.chat, but wabi.chat responses are missing this header. The API is accessible but browser CORS blocks the response.

Root Cause: Cloudflare's reverse proxy for named tunnels does not automatically pass through dynamic CORS headers. Even though wabi-server correctly sets Access-Control-Allow-Origin, Access-Control-Allow-Credentials, etc., Cloudflare strips them from the response.

Fix - WABI_CORS_ORIGINS + Caddy header_up:

1. Set WABI_CORS_ORIGINS="https://wabi.chat,http://localhost:3000,http://localhost:5173" when starting wabi-server
2. Update Caddyfile.tunnel to forward CORS headers:

reverse_proxy wabi-server:3000 {
    header_up X-Forwarded-Proto https
    header_up X-Forwarded-Host {host}
    header_up X-Forwarded-For {http.request.remote.host}
    # Pass through CORS headers
    header_up Access-Control-Allow-Origin {upstream_response.header.Access-Control-Allow-Origin}
    header_up Access-Control-Allow-Credentials {upstream_response.header.Access-Control-Allow-Credentials}
    header_up Access-Control-Allow-Methods {upstream_response.header.Access-Control-Allow-Methods}
    header_up Access-Control-Allow-Headers {upstream_response.header.Access-Control-Allow-Headers}
}

3. IMPORTANT: For named tunnels, the Caddyfile changes require either:
   • Reconfiguring via Cloudflare dashboard, OR
   • Using a quick tunnel (cloudflared tunnel --url http://caddy-tunnel:8088) which reads local Caddyfile

Verification:

# Test localhost CORS
curl -s -H "Origin: https://wabi.chat" -X OPTIONS http://localhost:3000/api/auth/login -D - | grep "access-control"

# Test wabi.chat CORS (should show access-control-allow-origin after fix)
curl -s -H "Origin: https://wabi.chat" -X OPTIONS https://wabi.chat/api/auth/login -D - | grep "access-control"

Orphan STDB-era containers (noise, safe to remove)

After the WabiDB cutover, docker compose up on Tim warns Found orphan containers (wabi-stdb-proxy, wabi-stdb-publisher, wabi-spacetimedb) for this project. These are stopped STDB-era leftovers (Exited 0/137, weeks idle) — noise, NOT a cutover signal when wabi-server is healthy. Safe to remove with docker rm (containers only; does NOT touch data/ or uploads/, and do NOT delete STDB-era data/spacetimedb/**/db.lock unless explicitly cleaning orphans):

ssh tim@100.96.11.45 'docker rm wabi-stdb-proxy wabi-stdb-publisher wabi-spacetimedb'

Verified 2026-08-05: after docker rm, Tim runs only wabi-server (healthy) + 3 cloudflared connectors + caddy-tunnel. Orphan removal is optional housekeeping; the compose warning can be left alone safely.

WabiDB Locks (restart-loop trap)

wabi-server owns a data dir (e.g. data/wabi-server/). There are TWO lock files:

• data/wabi-server/.lock (top-level)
• data/wabi-server/wabidb/.lock (DEEPER — the engine lock)

A stale deeper lock causes Error: engine already running and the container restart-loops even after docker stop + docker rm + up. The deploy script MUST remove BOTH: rm -f data/wabi-server/.lock data/wabi-server/wabidb/.lock. If you only clear the top lock, the new container loops until the deeper lock is gone.

• json! macro resolution in adapter code. Files using json!() without use serde_json; (or use serde_json::json) fail with cannot find macro json in this scope. When adding new code that uses json!, either add use serde_json::json; at the top or use the fully qualified serde_json::json!(). Recipe: grep -n 'json!(.*)' core/crates/wabi-server/src/adapter/mod.rs | grep -v use | grep -v serde_json. When adding new code that uses json!, either add use serde_json::json; at the top or use the fully qualified serde_json::json!(). Recipe: grep -n 'json!(.*)' core/crates/wabi-server/src/adapter/mod.rs | grep -v use | grep -v serde_json.

• json! macro resolution in adapter code. Files that use json!() without use serde_json; (or use serde_json::json) fail with cannot find macro json in this scope. The wabi-server/src/adapter/mod.rs and wiring_handlers.rs both use json!() extensively but rely on re-exports. When adding new code that uses json!, either add use serde_json::json; at the top or use the fully qualified serde_json::json!(). Fix recipe: grep -n 'json!(.*)' core/crates/wabi-server/src/adapter/mod.rs | grep -v use | grep -v serde_json to find unqualified usages in files missing the import.

• ChannelKind match exhaustiveness (E0004). The ChannelKind enum (defined in wabidb) has variants Text, Voice, Dm, GroupDm, Announcement, Whiteboard, Wiki, Forum, Incident, Gallery, Category. Any match on channel_kind (in channels.rs, adapter/mod.rs, socket-types.ts, etc.) MUST cover all 11 variants or fail to compile. gallery (media) and (channel grouping header) are the two that are most often missed. Check ALL match arms after adding a new channel type: .

Related references

• references/port-mismatch-debug.md
• references/frontend-serverurl-port-rewrite.md
• references/cloudflared-ws-and-wabidb-lock.md
• references/post-login-store-crash.md
• references/cloudflare-cors-header-stripping.md
• references/wabi-cors-header-stripping.md (NEW: Cloudflare CORS header forwarding setup)
• references/csp-unsafe-eval-and-beacon-block.md (NEW: CSP script-src missing 'unsafe-eval' blocks SvelteKit runtime eval)
• references/avatar-upload-cross-account.md — profile picture cross-account visibility, /uploads serving, client merge race
• references/message-identity-new-eats-old.md — UUID message ids, merge/key/dedupe, don't delete dedupeByIdKey, SW stale chunk proof
• references/sw-truth-and-css-cascade.md — (2026-08-08 audit) SW never caches chunks; "deploys don't change anything" = CSS cascade / capability gates / z-index / stubs, NOT caching
• references/lore-runtime-and-create-repo-fix.md — container-traversal permission fix (host home dir o+rx), create-repo URL contract (POST /repos vs /repos/{id}/link), feature-flag vs runtime-disabled distinction
• references/tim-restart-loop-and-login-ux.md — Tim restart-loop diagnosis, lock cleanup, and login/boot UX hardening when backend 502 causes "stuck signed in"
• scripts/css-cascade-audit.py — list every class defined in 2+ sheets and which @import wins; run first when UI looks stale despite verified deploy