local-testing

# Local testing **After a change to the network architecture (netcode, connecting, identity and accounts, the server's auth, the masterserver, the site's match reports and broker): run `scripts/regression.sh`** (about 15 minutes; not for gameplay, UI or rendering changes) - every check below in a fresh scratch directory and a PASS/FAIL table at the end; `SOAK=1` adds the netcode soak (~35 min), `REMOTE=1` the remote part (`remote_e2e/regression_remote.sh`: the MMR rules on the remote test site, one real ranked match through real NAT, a web client over TURN only), `ONLY="mmr keys"` runs some checks only. Such a fix is tested with it before it is committed; a new kind of such change gets its check added there. The site's MMR rules are tested by signed simulated reports (`run_mmr_tests.py` in the site), real ranked matches only for one or two end-to-end cases. Everything runs from scripts in `scripts/`, each isolated: own appdata dirs, own scratch dir, and they check their ports are free (the scenarios share 18000/18410/18420/18430/10002 - run one at a time) (`SCRATCH`, default `/tmp/hypersomnia-nettest` - point it at the session scratchpad). Build into your own build folder first and pass it as `BIN` (default `build/Debug-x64-clang-claude/Hypersomnia`). ## Isolation rules - read first, all of them were learned the hard way 1. **Heartbeats go to the production masterserver by default.** `notified_server_list` lives in the `"server"` section; a top-level key is silently ignored. Every test server needs `"server": { "notified_server_list": "127.0.0.1:18430", "suppress_new_community_server_webhook": true }`. Check `Sending heartbeat through UDP to` in the server log before anything long. A miss lists the test server publicly and posts "new server" notifications to the community channels. 2. **Never use the `build` MCP or `cmake/build.sh`.** Both work on `build/current`, which the user's builds use; `build.sh` also re-points the symlink. Configure with plain `cmake` into `build/<Config>-x64-clang-claude[-suffix]` and build there with `nice -n 19 ninja -j10`. Generated sources go into the build dir, so parallel builds don't clash. **While implementing and testing, build and test only `build/Debug-x64-clang-claude`** (the fast Debug native one, no debug info - like the user's `cmkdf`). The Steam, headless, web and CrazyGames variants only when the change touches their code (`#if` paths, headless-only sources, web or Steam APIs) - and even then only once, literally at the end of the batch, after the user has tested; never in the middle of iterating. Rebuilding all five for every small fix made the iterations too slow. **A commit needs only the Linux build** - a variant that stops compiling is fixed later, at the very end; build the other variants before a commit only for a big architectural change across platforms. 3. **Never `pkill -f <pattern>` (or `pgrep -f`) in a command whose own text contains the pattern** - it kills your own shell (exit 144). Put the kill in a script file and run the script, or kill by exact PID. 4. **Port 3000 is the user's local hypersomnia.io** (the site repo, `node app.js`, its `private/mmr.db`). Never post to it. 5. **Check ports are free first** (`ss -tulpn`, `lsof -ti tcp:<port>`) and abort if busy - a server that fails to bind silently leaves the requests going to whatever else listens there. 6. **Silence, nothing on the user's screen.** Every script sources `scripts/test_env.sh` (the one place for it: `ALSOFT_DRIVERS=null`, `VISUAL=true`, and `$GAME_TEST_FLAGS` = `--no-update-on-launch --mute --no-open-logs`) and passes `$GAME_TEST_FLAGS` to every game it starts - a new script does the same, a game started by hand too (`. .claude/skills/local-testing/scripts/test_env.sh` first). Firefox profiles get `user_pref("media.volume_scale", "0.0");`. 7. On remote hosts, production runs as `Hypersomnia-Headless.AppImage --as-service` (masterserver, official server) - find it with `ps`. Work only in `~/yojimbo-test`, restart only by your own PIDs, and verify afterwards that the production processes still have their old uptime (`ps -eo etime,args | grep Hypersomnia-Headless`). ## Scenarios (scripts/) | Script | What it proves | |---|---| | `broker.sh start\|stop\|sql` | A scratch copy of the site on 13500 as the identity broker and report endpoint, with its own broker key; prints `BROKER_URL` and `BROKER_PUBLIC_KEY` to export for the scenarios | | `scenario_keys.sh` | Master with `official_servers.d` (servers official by their `--print-server-key`), official casual (de) + ranked (pl) + community servers, 3 CLI clients: signed heartbeats, every client self-signed (unlinked keys never ask the broker) and taken by ranked too, `.keys/account.key` 0600. `COMMUNITY_PASSWORD`, `KEEP_RUNNING`, `*_SECS` | | `scenario_webrtc_ranked.sh` | Native client over WebRTC (`--connect rtc:ranked-pl:1` - a ranked instance's alias has its number) through the local signalling server to the official ranked server by its alias, with a broker cert | | `scenario_web_firefox.sh [connect] [wait]` | The real web build in Firefox under Xvfb joining `/game/<connect>` (default `pl`) over WebRTC with a cert from the local broker; screenshot in `$SCRATCH/web_test/screen.png` | | `scenario_outage.sh` | No masterserver, broker down vs up, joining by address: self-signed plays casual and is told ranked needs an account; a broker cert plays both | | `scenario_crazygames.sh` | The CrazyGames build in Chromium with a stubbed SDK (`crazygames_sdk_stub.py`, token signed by `CG_KEY`) joins an official ranked server over WebRTC as `crazygames_...` | | `scenario_steam.sh` | The Steam build with the user's running Steam: a real Web API ticket -> broker cert -> ranked (broker started with `STEAM_API_KEY_FILE`) | | `scenario_ranked_report.sh` | A whole ranked match to its natural end in about a minute: the test map `de_e2e_short` (de_cyberaqua, max team score 2, 5 s rounds, no warmup) on a ranked server, two key clients clicking Auto-assign, the signed report at the local broker, the site's match and MMR rows | | `scenario_tournament.sh` (`CASES=...`) | Tournaments end to end, a PASS/FAIL table: fixed teams with a bye, an open first round (auto close, leaving before the start, 2v2), crashes (SIGKILL) mid-match, in a freeze after someone left, between rounds and in an open round - resumed by the same flags, nothing replayed or lost; the live results page's JSON on the way; the page in the masterserver's list. Run it after any tournament change (~25 min; the cases are listed at its top). Clients run small (480x270, `LP_NUM_THREADS=2`) - four full-size ones starve each other into timeouts | | `check_environment_variables.sh` (`PAGE=<site repo>`) | The `HYPERSOMNIA_*` names in the game (`environment_variables.{h,cpp}`), `README_SERVER.md`'s table and the config generator are the same - after adding or renaming one | | `scenario_ranked_corners.sh` (`CASE=unlink\|broker_down`) | A linked account (a key linked to a fake Steam id straight in the site's DB, its broker behind a stoppable proxy) in a ranked match: mid-match its key is unlinked on the site, or its broker goes down; it is killed and started again - it must rejoin as the same `steam_` account (its saved steam_cert.json) and the report must name that account. Note: a key put in by hand is linked only after the game's link check - started with `--connect` right away, it joins as `key_`; the script lets alice save her steam_cert.json first | | `scenario_disconnects.sh` (`CASES=...`) | Filmstrips of each disconnect as the player sees it, with timings: server SIGTERM/SIGKILL/SIGSTOP, full, the same key joining again, wrong password, AFK, client SIGSTOP/SIGKILL/SIGTERM. Expected: every kick reason shows at once; a dead server or a dropped network shows "Nothing came from the server for 10 seconds" (native only); a 6 s server freeze is survived; a killed client frees its slot in about 5 s, a quitting one at once | | `scenario_map_download.sh <out>` (`MAP`, `EXT`, `LATENCY`, `LOSS`, `KILL_AT`) | A client without the server's map downloads it - over UDP or from an HTTP provider (`slow_http.py` serves one at a set rate, with CORS for the browser); a long download never disconnects, a server killed meanwhile shows "Nothing came from the server for 10 seconds". In the browser: `scenario_web_firefox.sh` with `MAP`, `EXT` (and `KILL_SERVER=1`, `TAB_AWAY=<s>`) | | `scenario_admin.sh` | A key linked to a Steam account on a server with a password: with only the `steam_` account in `admin_profiles` it is asked for the password (outside the Steam build a linked key is no admin), with its `key_` too it gets in without one | | `scenario_password_limit.sh` | 7 joins without the password: 5x "Wrong password", then "Too many wrong passwords" | | `link_flow_test.py`, `crazygames_cert_test.py` | The site's link/unlink/start endpoints and the broker's CrazyGames token checks, at the HTTP level | | netcode's own tests (handshake, replays, forged challenges) | Not in any build target: `printf 'void netcode_test(void);int netcode_init(void);void netcode_term(void);int main(void){netcode_init();netcode_test();netcode_term();return 0;}' > main.c` in a scratch dir, then `clang -O1 -DNETCODE_ENABLE_TESTS=1 -DNETCODE_DEBUG=1 -I$Y/netcode -I$Y/include main.c $Y/netcode/netcode.c -lsodium -lm -o nctest` (`Y=src/3rdparty/yojimbo`) and `stdbuf -oL ./nctest` - a failed check aborts (exit 132) after naming it | | `pw_driver.js`, `fake_steam_session.py` | Chromium (Playwright in `~/.cache/hs-playwright`) driven over HTTP for clicking the web build; a Steam session cookie for the local site | | `[VARIANT=crazygames] build_web.sh` | Web Release build (`-DUSE_O3=1`) in `build/Release-Web-clang-claude-web[-crazygames]`, the same steps as `build.sh Release Web` | | `[USER_FILES=dir] prepare_web.sh <web build> <out>` | Copies a web build and points its packaged config at the local master and the broker on 13500 (see below); USER_FILES also writes files into its `/user` before start | | `webserve.py <dir> <port>` | Serves a web build with COOP/COEP; any path serves the game, so `/game/<id>` joins right away | | `steam_tickets.py IDENTITY...` | Steam Web API tickets from the user's running Steam client, to probe `AuthenticateUserTicket` without the game | | `xinput.py click x y` / `rclick x y` / `drag x1 y1 x2 y2 [x3 y3 ...]` (rests 1.5 s at the points in between) / `type TEXT` / `key NAME` / `scroll x y n` (0 notches = only move the mouse) | Clicks, drags (e.g. a window edge), types, presses keys and scrolls on `$DISPLAY` through XTest (no xdotool needed) | | `focus_bounce.py <window name> [secs]` | Takes the X focus from the game's window and gives it back (alt-tab to a browser and back - Xvfb has no window manager); `xclip -selection CLIPBOARD -in` sets the clipboard | | `local_stack.sh [web build] [hours]` | Leaves running, for the user to test by hand: master, official ranked [PL] and casual [DE] servers, the web build on http://localhost:8765 pointed at them; `local_stack.sh stop` | | `build_headless_ubuntu2204.sh` | Headless Release build inside an unprivileged Ubuntu 22.04 rootfs (bwrap) - glibc 2.35, runs on both arenaus and arena; prints where the libc++ to ship lies | | `build_headless_ubuntu_container.sh` | The same in Ubuntu 24.04 (glibc 2.39) - arena only | Pass/fail is read from the logs - the scripts grep the meaningful lines: `Received an identity cert for X`, `Authenticated "nick". ID: steam_...|key_...`, `Ranked matches need`, `Received initial public vars`. ## The web build headless - how it works - **Config without rebuilding:** the web build packages `default_config.json` into `Hypersomnia.data` uncompressed. `prepare_web.sh` replaces the provider lines in place with text of the same length padded with spaces, so the file packager offsets stay valid. (In the game UI the same can be set in Settings - Advanced - Network.) - **Joining from the URL:** the page passes `window.location.pathname` as a CLI argument; `/game/<id>` connects to that WebRTC id. - **Firefox:** `firefox --no-remote --profile <fresh dir> --new-instance URL` under `Xvfb`, with `user.js` setting `devtools.console.stdout.content=true` - the game's console output then lands in Firefox's stdout. WebGL works through llvmpipe. Allow ~60 s for the 100 MB data to load. - The web account is a key in its own IDBFS database (`/hypersomnia_keys/account.key`, apart from `/user`), made on the first start. Link with Steam (Settings - Account) opens `<website>/link_key/start?state=..&key=..&sig=..&device=..&os=..` (the state signed with the key) in a new tab (native: in the browser); signed in through Steam anew (every link), the player confirms the Steam account on that page and the key is linked at once - the game asks for its cert every 3 s and sees the Steam account there (no redirect back, no local listener, no code; the same natively, in a tab or an iframe). The game then asks "Link to this Steam account?" (the profile link, [Link] [Cancel] - Cancel unlinks; steam_cert.json is saved only after Link); linking moves no progress. A refusal (another account's key, 32 keys) shows on the page only; the page is good for 3 min, then the game stops waiting. Unlink from Steam is one signed GET `/link_key/unlink?t&key&sig`, answered at once. ## UI checks Every game started for a test gets `$GAME_TEST_FLAGS` (see point 6): `--mute` - master volume 0 for that run only (under Xvfb it still plays sound through the user's speakers); `--no-open-logs` - a failure log is saved but never opened in the user's `$VISUAL`, which pops up on their screen and blocks the process until closed. Read the logs yourself from `<appdata>/logs/` (`exit_failure_debug_log.txt`, `ensure_failed_debug_log.txt`). Run the game under `Xvfb :77 -screen 0 1280x720x24` with a 1280x720 non-fullscreen window (conf.d `"window"`), `import -window root shot.png` to screenshot, read the image, and `python3 xinput.py click x y` to click. A fresh appdata starts in the tutorial: `xinput.py key Escape` opens the menu (Settings at ~156,555). The appdata remembers the last connect string - the first screen may be a reconnect dialog with Abort. ### Identity UI end to end (scripts/identity_ui) Helpers, all under `SCRATCH_ROOT` (default `/tmp/hypersomnia-ui`; the broker is `broker.sh` with `SCRATCH=$SCRATCH_ROOT/nettest`, started with `STEAM_API_KEY_FILE` and a `SESSION_SECRET` kept in `$SCRATCH_ROOT/nettest/session_secret`): - `ui.sh conf <name> [broker url]` (a client in `$SCRATCH_ROOT/ui/<name>`, main menu, the scratch site's leaderboards; `NICK=`), `ui.sh start <name> <display> [game args]`, `stop`, `shot <display> <png>`, `click`, `type`, `key`. `BIN=` for the Steam build. The game never opens a real browser there: `fake_bin/xdg-open` writes the URLs to `<client>/opened_urls`. - `browser_sign_in.sh [steam id64]` (with `OPENED_URLS=<client>/opened_urls`) plays the player's browser for Link with Steam: opens the last `/link_key/start` URL signed in to Steam on the scratch site and clicks Link - the game then learns it by itself. - account.key is read only at the start and after Restore / Start a new account - a key swapped on disk counts from the next start (restart the client to test it). Every start (not of a key made just now) asks the broker quietly whether the key is linked ("Asking quietly whether the key is linked") - unlinked on the site, it drops its steam_cert.json; linked elsewhere (`link_key.py`), Settings open with "This key is linked to a Steam account - is it yours?" until answered. Not in the Steam build or CrazyGames. - `cp imgui_big.ini <client>/user/imgui.ini` first: Settings at 20,20 1240x680, so whole panes fit a screenshot - then Settings 155,555, Account 126,112 with Info 102,144 / Link with Steam 245,144 / Backup 385,144 - the panes' own fields move with their content, so screenshot first. Without it (the default 1280x720 layout): Info 307,278 / Link 440,278 / Backup 590,278, Link with Steam... 397,337. Every confirmation and result is a centered modal. Backup of a key linked to Steam says only "Your progress is on your Steam account - nothing to back up...". - `srv.sh start|stop <name> <port> [TEAMS_EQUAL]` - a server trusting the scratch broker. - `link_key.py <account.key> [steam id64]` - links that key to a fake Steam session's account, as confirming the game's link page would (prints the confirm POST's status) (a Steam account takes at most 32 keys - a test account full of them answers 409). - `../link_flow_test.py <site dir> <site url>` (with `SESSION_SECRET`) - the whole link protocol against the site, bans and replays included. - `slow_broker.py <port> <delay>` - the broker answering after `delay` s (a closed port is a dead broker). - `ui.sh conf <name> ""` - no identity broker at all. `srv.sh` wipes the server's dir, so every start is a **new server key**: a client joining with the old key (`--connect <address>,<key>`, its last join) gets "another key than the one you chose".