benchmark-rendering

# Benchmarking a rendering change (A/B in one run) The fastest reliable method: temporarily add a global toggle for the measured thing, flip it automatically every few seconds inside ONE game run, and log running averages of GPU frame time (GL timer query) and FPS for both variants. Then remove the temporary code and rebuild. One run with alternating phases cancels out scene, thermal and clock differences between separate runs. ## Facts that save time - `src/view/frame_profiler.h` measures only CPU time and has no long averages - useless for GPU passes. The rendering script only enqueues commands; the GPU cost of an FBO clear or fullscreen quad never shows there. - FPS alone hides GPU costs: on the dev machine (RX 5700, ~1080p) the game is CPU-bound - ~1110 FPS with a ~0.5 ms GPU frame. Always report the GPU time, not just FPS. - **Always pass `/range`** to launch straight into the shooting range, no input needed: `../build/current/Hypersomnia /range`. It's a URL-style location argument (`src/cmd_line_params.h`, also `/menu`, `/editor`, `/tutorial`). Confirm in the log: `Activity SHOOTING_RANGE was specified from the command line.` and `Launched mode: SHOOTING_RANGE`. Without it, the game launches `last_activity` from `hypersomnia/user/runtime_prefs.json` - which is whatever the user did last (editor, menu, a server...) - and only if `skip_tutorial` is true, else the tutorial. `launch_at_startup: MAIN_MENU` in the config would send it to the menu too. - A different scene (editor, a map on a server) needs a different measuring setup - ask the user if the range isn't representative. - **A specific map, with a fixed camera:** open it in the editor with `--edit <project dir>`, e.g. `../build/current/Hypersomnia --edit user/projects/de_cyberaqua_bench` (cwd = `hypersomnia/`). Confirm `Launched mode: EDITOR` and that no `DTOR finished: ~editor_setup` follows right after loading - builds before 2026-10-01 fell through to the project selector with `--edit` (missing `break` in `work.cpp`). - Never benchmark on the official `content/arenas/*` or the user's own `user/projects/*`: the editor writes `navmesh.nav`, `editor_view.json` and autosaves there. `user/projects` is a symlink to `../content/arenas`, so a copy there lands in the repo too. Copy the arena to the scratchpad instead - `--edit` takes an absolute path - rename its json to match the folder, drop `last_saved.json`, and set `meta.name`. - Builds before 2026-10-05: `--edit` opened the editor only when the last activity was the editor, and overwrote `user/editor_last_project.txt` with the scratch path - the user then edited the scratch copy unknowingly. Since then `--edit` always opens the editor and leaves both untouched; `--playtest` starts playtesting right away. Still never name the copy like the original. - The editor camera comes from `editor_view.json` (`panned_camera.transform.pos`, `zoom`) - set it to frame what is measured, e.g. zoom 0.45 at the average position of the lights to catch many of them. - A frame "saved" by the user reaches `editor_view.json` only when their editor quits - ask them to quit it first, and check that the camera lies within the map (the min/max of the nodes' `pos` in the map json). `Num visible entities: 0` and ~1200 FPS in the profile mean an empty camera, not a fast scene. - Downloaded maps: `hypersomnia/user/downloads/maps` may be a symlink (to `~/poprawionemapy` in Oct 2026) - resolve it with `readlink -f` before deciding which copy the user edits. `~/.config/Hypersomnia/...` holds older copies. - Editor playtests run the map's `playtesting.mode`. `quick_test` spawns no bots even with `spawn_bots` - set `"mode": "bomb_defusal"` in the scratch copy to measure bots. - **Screenshots of the game:** `import -window $(xwininfo -name <title> | awk '/Window id/{print $4}') out.png`. Never `import -window root` - it captures whatever the user has on screen. A window on another i3 workspace is unmapped and comes out black; on X11 without a compositor it still renders (the timings stay valid). - **Never use `--apply-config` for a benchmark variant:** the patch lands in `user/runtime_prefs.json` when the game quits (even on `timeout`'s SIGTERM), silently changing the user's settings. Back up `runtime_prefs.json` and diff it after the run, or toggle the setting in code with a `/* BENCH TEMP */` flag. The editor also saves its camera to `editor_view.json` on quit - reset it before every run so the scene stays the same. - **CPU-side light work:** the profiler has `light_ray_casts`, `light_aabb_queries`, `lights_recalculated`, `lights_reused` and `light_jobs_cpu_us` per frame (`frame_profiler.h`, summed by the light jobs in `launch_visibility_jobs.cpp` from the per-thread `thread_physics_query_counters`). Log them in the A/B harness instead of the GPU query when the measured thing is CPU work. The light jobs run in parallel on the thread pool, so their summed CPU time barely moves the FPS on a many-core machine - report the CPU sum, not just FPS. - Uncapped already: `vsync_mode` is `OFF` in `default_config.json`, and `max_fps` is disabled (`OFF_max_fps` in runtime prefs). Verify if the numbers look capped (e.g. exactly 60/144/400 FPS). - **On the user's screen, deliberately:** the one exception to running test games under Xvfb - the real GPU and window are what's measured. Still silent and never opening a failure log in the user's editor: source `.claude/skills/local-testing/scripts/test_env.sh` first and pass `$GAME_TEST_FLAGS`. - Launch with cwd = `hypersomnia/`: `cd hypersomnia && timeout 76 ../build/<config>/Hypersomnia /range $GAME_TEST_FLAGS > <scratchpad>/bench.txt 2>&1` (the folder built for the benchmark). From the repo root the game fails to read its config and litters the root with empty `cache/`, `user/`, `logs/`. - LOG output goes to stdout, so it lands in the redirected file even when `timeout` kills the game. - The user allows launching and killing the game by yourself for benchmarks. - **Check that no other game instance runs before measuring:** `ps -eo pid,pcpu,lstart,args | grep -i "[h]ypersomnia"`. The user often keeps one open (launched via `ninja run -C build/current`), or plays the release AppImage between runs - it runs as `/tmp/.mount_*/usr/bin/Hypersomnia`, which a `build/` grep misses, and took 320% CPU in one case. It skews the absolute CPU numbers and FPS - ask them to close it rather than killing it, then rerun. An A/B in one run keeps the ratio roughly right even then. - **Detect disturbances during every run, not only before it:** the user compiles, plays and runs other agents meanwhile. Sample every second with `top -b -n2 -d1` (the second iteration - `ps` pcpu is a lifetime average and misleads) for `ninja|clang++|cc1plus|ld.lld|Hypersomnia|Xvfb` above ~3% CPU, and drop and rerun a run that had any. Also drop runs whose `BENCHPROF` shows `Debug details` (~0.4-0.5 ms) - the user opened the performance overlay in the window. A game started from Claude's shell inherits nice 5 (only root can lower it) - fine when the machine is idle. - **Name the benchmark binary:** a symlink in `build/current` (e.g. `HypersomniaBenchJedrula -> Hypersomnia`) makes the process distinct from the user's game. Kill it with `pkill -x HypersomniaBenc` (comm is cut to 15 chars) - `pkill -f <pattern>` kills Claude's own shell, whose command line contains the pattern. - **A separate worktree for the instrumentation** (`git worktree add --detach`, submodules copied as described below): the user and other agents keep building the main checkout. `git stash` in a worktree shares the stash list with the main checkout - save the instrumentation with `git diff > patch` instead, and `git apply --3way` it onto a newer master. Port the final fix to the main checkout without the `/* BENCH TEMP */` hunks and commit only those files (`git commit -o`). - **Ready instrumentation** in `instrumentation/` next to this file (header-only, include them where marked): - `bench_gpu_segments.h` - the per pass GPU timing below. Hooks: `bench_gpu::set_fbo(id)` in `fbo::set_as_current_impl` (0 in `set_current_to_none_impl`), `set_prog(id)` in `shader_program::set_as_current_impl`, `set_tag(int)` for the `light_pass` uniform in `shader_program::perform(set_uniform_command)`, `on_draw(tris)` after `glDrawArrays`, `begin_frame()` / `end_frame(frame_s)` around the `renderer_backend.perform` loop in `work.cpp`. `BENCH_GPU=0` switches the blocking readback off and keeps only `BENCHFRAME` (FPS, p50/p90/p99/max frame time). - `bench_jobs.h` - CPU time of the thread pool jobs per frame: `bench_jobs::scope(bench_jobs::FOW)` at the top of `fow_job`, `LIGHT` in `light_job` (`launch_visibility_jobs.cpp`), `ILLUM` / `GAME_GUI` / `POST_GUI` in the jobs of `work.cpp`, `HELP_WALL` around `help_until_no_tasks` + `wait_for_all_tasks_to_complete`, `end_frame()` after it. The other slots served as plain counters (`ns[slot] += n`, printed /1e6 per frame). - `bench_cpu_timers.h` - per simulation step: `report_and_reset()` + `scope(STEP)` at the top of `standard_solve`, `scope(...)` in the AI and pathfinding functions. When `report_and_reset()` returns true, log the cosmos profiler too (`performance.prepare_summary_info(); performance.summary(s);` - needs `augs/templates/introspect.h` in that file). - **"Help tasks" in the game thread profile is the wait for the thread pool** - the frame waits for its longest job. Time each job (`bench_jobs.h`): the parallel light jobs sum up high but rarely matter; a single long job does. - **Which pass eats the GPU:** time segments of the command stream - `glQueryCounter(GL_TIMESTAMP)` in `renderer_backend::perform` at every change of the bound FBO (`fbo::set_as_current_impl`) or program (`shader_program::set_as_current_impl`), read back blocking at the frame end, summed per (fbo id, program id). Log the ids at creation (`fbo::fbo`, the path ctor of `shader_program`) to name them; `all_necessary_fbos::apply` creates them in a fixed order. Split further by a tag: the `light_pass` uniform (hook `shader_program::perform(set_uniform_command)`) or the `dedicated_buffer` of a `drawcall_dedicated_command`. Trust the total and big segments only - the cost of an FBO switch can land in a neighbouring segment. The blocking readback lowers the FPS: measure FPS in separate runs without it. - **GLSL: return early instead of skipping work in an `else`:** on radeonsi a uniform `if (mode == 2) { cheap } else { expensive }` ran at the cost of the expensive branch (0.90 ms); returning right after the cheap path got 0.29 ms. ## Frame pacing / smoothness (judder), and comparing two builds FPS and GPU averages say nothing about judder. Use `smoothness/` next to this file: - `smoothness.patch` (`git apply` it; applies to HEAD and to tag 3.0.0) adds `src/bench_smoothness.h` and hooks in `work.cpp`. With `HYPERSOMNIA_SMOOTHNESS_BENCH=<out.csv>` in the editor it starts playtesting after 3 s (like Space), holds A+W from 6 to 10 s, dumps one CSV row per frame and quits by SIGTERM. Per frame: game frame start and delta, GPU timestamps after the frame's commands (`GL_TIMESTAMP`, so fill rate counts), swap times, the drawn camera position (after pixel snapping), the raw camera, the viewed character, logic steps. Not for committing: `GL_TIMESTAMP` does not exist in WebGL. - `analyze_smoothness.py name=run.csv name=run2.csv ...` groups runs by name and prints p50/p99/p99.9/max of: display intervals, GPU and swap times, latency, position judder (deviation from a moving average, in screen px), the step per simulated 144 Hz refresh, backward camera steps per second, and hitches. The display time of a frame = max(GPU end, swap end) - right for X11 without a compositor, vsync off. - Run 3-5 times per build, alternating builds. The bench script needs a window in view; a run that logs `Launched mode: SHOOTING_RANGE` or `WM_DELETE_WINDOW` was disturbed by the user - drop it and rerun. - Use the same scratchpad copy of the map for both builds (3.0.0 loads HEAD's map json fine) and restore its `editor_view.json` before every run. **Another build (e.g. a release tag):** `git worktree add "$(git rev-parse --show-toplevel)-<tag>" <tag>` (next to the repo, as `~/Hypersomnia-<tag>`), then copy the submodule directories from the main checkout instead of cloning (`git ls-tree -r <tag> | awk '$2=="commit"{print $4}'`; compare with HEAD first) with `rsync -a --exclude=.git` - a copied `.git` file would make every git command in the worktree fail. Build with `BUILD_FOLDER_SUFFIX=fast cmake/build.sh RelWithDebInfo x64 -DGENERATE_DEBUG_INFORMATION=0`, then `ninja` in `build/current` (~15 min, run it in the background) - after `. "$(git rev-parse --path-format=absolute --git-common-dir)/../.claude/skills/local-testing/scripts/lib.sh" && wait_build_slot` (the main checkout's - a release tag's worktree may not have it) in the same shell. Copy `hypersomnia/user/runtime_prefs.json` into the worktree. Before commit a5c684fd5 `--edit` falls through to the project selector: add the `break;` after `launch_editor(params.editor_target);`. Remove the worktree when done: `git worktree remove --force <path>` (the build inside is the only thing lost). **What the smoothness numbers showed (Oct 2026):** the judder came from the camera logic, not from frame times - backward 1 px camera steps from pixel snapping and 1 px "snap to target" jumps of the smoothing (fixed in 39f1b07fd). Frame times were steady on both HEAD and 3.0.0; rare 4-5 ms stalls inside `renderer_backend.perform` (0-1 per 4 s, random moments, both builds) are driver noise. Deterministic stalls happen only at loading: `texImage2D` uploads (8-20 ms) when the editor opens, and the first `LIGHT_SHADOW_MASKS` upload (7 ms) when playtesting starts. ## Gotchas: FPS caps and vsync Check these before trusting the numbers. A capped frame rate makes both variants look identical. Suspicious FPS values are the round ones: 60, 144, 165, 240, 400, or anything exactly flat across phases. - **In-game vsync:** `window.vsync_mode` (`OFF` / `ON` / `ADAPTIVE`, `src/augs/window_framework/window_settings.h`). Default `OFF` in `hypersomnia/default_config.json`, but `hypersomnia/user/runtime_prefs.json` and any file in `hypersomnia/user/conf.d/` override it. Grep all three for `vsync_mode`. - **In-game FPS limit:** `window.max_fps` is a `maybe<int>`. In JSON, `"max_fps": 400` means ENABLED at 400, `"OFF_max_fps": 400` means disabled (value kept). Values below 10 are ignored (`work.cpp`, `max_fps.value >= 10`). The limiter method `max_fps_method` (`SLEEP` / `SLEEP_ZERO` / `YIELD` / `BUSY`) changes the frame pacing, but not the GPU time. - **Driver vsync (Mesa, AMD):** the driver can force vsync regardless of the game - `vblank_mode` env var or `~/.drirc`. Launch with `vblank_mode=0` to be sure: `vblank_mode=0 timeout 76 ../build/current/Hypersomnia /range ...`. - **Compositor / hidden window:** a minimized or fully covered window may get throttled by the compositor or the driver. The game itself does not throttle unfocused windows. Keep the game window visible during the run. - **Swap moment:** `performance.swap_window_buffers_when` (`AFTER_HELPING_LOGIC_THREAD` by default) moves where the frame blocks. Keep it the same for both variants. Since the toggle lives in one run, this is automatic. - **Settings changed in the game menu** land in `runtime_prefs.json` on a graceful quit. `timeout` kills with SIGTERM, so a benchmark run shouldn't change them - but don't open the settings mid-benchmark. - **The GPU time is immune to all of the above.** `GL_TIME_ELAPSED` measures the executed commands only, so even with a cap it gives the right diff. Only the FPS column gets flattened by caps. ## Steps 1. **Toggle.** Add a global flag next to the measured code, e.g. in the rendering script's header: ```cpp /* BENCH TEMP */ inline std::atomic<bool> bench_skip_<thing> = false; ``` Read it once per frame (`const bool bench_skip = bench_skip_<thing>.load();`) and skip ALL of the measured work with it: FBO binds and clears, draw calls, and the composite fullscreen quad. Mark every temporary edit with `/* BENCH TEMP */`. 2. **Measure.** Paste `bench_snippet.cpp` (next to this file) into `src/work.cpp`, around the `renderer_backend.perform(...)` loop in the render thread (search for `rendering_result.clear();`),