chat-perf

# Chat Performance Testing ## When to use - Before/after modifying chat rendering code (`chatListRenderer.ts`, `chatInputPart.ts`, markdown rendering) - When changing the streaming response pipeline or SSE processing - When modifying disposable/lifecycle patterns in chat components - To compare performance between two VS Code releases - In CI to gate PRs that touch chat UI code ## Quick start ```bash # Run perf regression test (compares local dev build vs VS Code 1.115.0): npm run perf:chat -- --scenario text-only --runs 3 # Run all scenarios with no baseline (just measure): npm run perf:chat -- --no-baseline --runs 3 # Compare two local builds (apples-to-apples): npm run perf:chat -- --build /path/to/build-A --baseline-build /path/to/build-B --runs 5 # Build a local production package and compare against a release: npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5 # Run memory leak check (10 messages in one session): npm run perf:chat-leak # Run leak check with more messages for accuracy: npm run perf:chat-leak -- --messages 20 --verbose ``` ## Perf regression test **Script:** `scripts/chat-simulation/test-chat-perf-regression.js` **npm:** `npm run perf:chat` Launches VS Code via Playwright Electron, opens the chat panel, sends a message with a mock LLM response, and measures timing, layout, and rendering metrics. By default, downloads VS Code 1.115.0 as a baseline, benchmarks it, then benchmarks the local dev build and compares. > **You don't always need a baseline.** A baseline exists only for *comparison* (regression detection). If you just want the current build's numbers — profiling a single change, capturing traces/heap snapshots, or iterating on a scenario — pass `--no-baseline` to skip downloading and benchmarking the baseline entirely (roughly halves runtime). Baseline comparison is what turns raw measurements into a pass/fail verdict; without it you still get all the metrics, just no verdict. ### Key flags | Flag | Default | Description | |---|---|---| | `--runs <n>` | `5` | Runs per scenario. More = more stable. Use 5+ for CI. | | `--scenario <id>` / `-s` | all | Scenario to test (repeatable). See `common/perf-scenarios.js`. | | `--build <path\|ver>` / `-b` | local dev | Build to test. Accepts path or version (`1.110.0`, `insiders`, commit hash). | | `--baseline <path>` | — | Compare against a previously saved baseline JSON file. | | `--baseline-build <path\|ver>` | `1.115.0` | Version or local path to benchmark as baseline. | | `--no-baseline` | — | Skip the baseline entirely — just measure the test build (no download, no comparison, ~2× faster). Use when you only need raw numbers, not a regression verdict. | | `--save-baseline` | — | Save results as the new baseline (requires `--baseline <path>`). | | `--resume <path>` | — | Resume a previous run, adding more iterations to increase confidence. | | `--threshold <frac>` | `0.2` | Regression threshold (0.2 = flag if 20% slower). | | `--production-build` | — | Build a local bundled package via `gulp vscode` for comparison against a release baseline. | | `--no-cache` | — | Ignore cached baseline data, always run fresh. | | `--force` | — | Skip build mode mismatch confirmation prompt. | | `--ci` | — | CI mode: write Markdown summary to `ci-summary.md` (implies `--no-cache`, `--heap-snapshots`, `--cleanup-diagnostics`). | | `--heap-snapshots` | — | Take heap snapshots after each run (slow; auto-enabled in `--ci` mode). | | `--gc-object-stats` | — | **GC deep-dives only.** Enables V8 `gc_stats` tracing (per-type heap object dump on every GC). ⚠️ Corrupts all timing metrics — a major GC landing mid-request adds ~550ms — so never use it for benchmarking. Off by default; prefer heap snapshots for memory analysis. | | `--cleanup-diagnostics` | — | Delete heap snapshots, CPU profiles, and traces to save disk. During runs, only the latest run's files are kept; after comparison, files for non-regressed scenarios are deleted. Auto-enabled in `--ci` mode. | | `--setting <k=v>` | — | Set a VS Code setting override for all builds (repeatable). | | `--test-setting <k=v>` | — | Set a VS Code setting override for the test build only. | | `--baseline-setting <k=v>` | — | Set a VS Code setting override for the baseline build only. | | `--verbose` | — | Print per-run details including response content. | ### Comparing two remote builds ```bash # Compare 1.110.0 against 1.115.0 (no local build needed): npm run perf:chat -- --build 1.110.0 --baseline-build 1.115.0 --runs 5 ``` ### Comparing two local builds Both `--build` and `--baseline-build` accept local paths to VS Code executables. This enables apples-to-apples comparisons between any two builds: ```bash # Compare two dev builds (e.g. feature branch vs main): npm run perf:chat -- \ --build .build/electron/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \ --baseline-build /path/to/other/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \ --runs 5 # Compare two production builds: npm run perf:chat -- \ --build ../VSCode-darwin-arm64-feature/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \ --baseline-build ../VSCode-darwin-arm64-main/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \ --runs 5 ``` Local path baselines are never cached (the build may change between runs). Version string baselines are cached for reuse. ### Build modes and mismatch detection The tool classifies builds into three modes based on the executable path: | Mode | Source | Characteristics | |---|---|---| | `dev` | `.build/electron/` (local dev) | Unbundled sources, `VSCODE_DEV=1`, `NODE_ENV=development`. Higher memory and startup overhead. | | `production` | `../VSCode-<platform>-<arch>/` (from `gulp vscode`) | Bundled JS, no dev flags. Matches release characteristics but uses local source. | | `release` | `.vscode-test/` (downloaded via `@vscode/test-electron`) | Official published build. | When test and baseline builds have different modes (e.g. dev vs release), the tool shows a warning and prompts for confirmation. Use `--force` or `--ci` to skip the prompt. Using `--production-build` builds a local bundled package via `gulp vscode` for fair comparison against a release baseline. This eliminates dev-mode overhead while still testing your local changes. ```bash # Production build vs release baseline (fair comparison): npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5 ``` ### Settings overrides Use `--setting`, `--test-setting`, and `--baseline-setting` to inject VS Code settings into the launched instance. This is useful for A/B testing experimental features: ```bash # Enable a feature for the test build only: npm run perf:chat -- --test-setting chat.experimental.incrementalRendering.enabled=true --runs 3 # Compare two builds with different settings: npm run perf:chat -- \ --baseline-build "../vscode2/.build/electron/Code - OSS.app/Contents/MacOS/Code - OSS" \ --baseline-setting chat.experimental.incrementalRendering.enabled=true \ --test-setting chat.experimental.incrementalRendering.enabled=false \ --runs 3 # Set a value for both builds: npm run perf:chat -- --setting chat.mcp.enabled=false --runs 3 ``` Precedence: `--test-setting` / `--baseline-setting` override `--setting` for the same key. Values are auto-parsed: `true`/`false` become booleans, numbers become numbers, everything else stays a string. ### Resuming a run for more confidence When results exceed the threshold but aren't statistically significant, the tool prints a `--resume` hint. Use it to add more iterations to an existing run: ```bash # Initial run with 3 iterations — may be inconclusive: npm run perf:chat -- --scenario text-only --runs 3 # Add 3 more runs to the same results file (both test + baseline): npm run perf:chat -- --resume .chat-simulation-data/2026-04-14T02-15-14/results.json --runs 3 # Keep adding until confidence is reached: npm run perf:chat -- --resume .chat-simulation-data/2026-04-14T02-15-14/results.json --runs 5 ``` `--resume` loads the previous `results.json` and its associated `baseline-*.json`, runs N more iterations for both builds, merges rawRuns, recomputes stats, and re-runs the comparison. The updated files are written back in-place. You can resume multiple times — samples accumulate. ### Statistical significance Regression detection uses **Welch's t-test** to avoid false positives from noisy measurements. A metric is only flagged as `REGRESSION` when it both exceeds the threshold AND is statistically significant (p < 0.05). Otherwise it's reported as `(likely noise — p=X, not significant)`. With typical variance (cv ≈ 20%), you need: - **n ≥ 5** per build to detect a 35% regression at 95% confidence - **n ≥ 10** per build to detect a 20% regression reliably Confidence levels reported: `high` (p < 0.01), `medium` (p < 0.05), `low` (p < 0.1), `none`. ### Exit codes - `0` — all metrics within threshold, or exceeding threshold but not statistically significant - `1` — statistically significant regression detected, or all runs failed ### Scenarios Scenarios are defined in `scripts/chat-simulation/common/perf-scenarios.js` and registered via `registerPerfScenarios()`. There are three categories: - **Content-only** — plain streaming responses (e.g. `text-only`, `large-codeblock`, `rapid-stream`) - **Tool-call** — multi-turn scenarios with tool invocations (e.g. `tool-read-file`, `tool-edit-file`) - **Multi-turn user** — multi-turn conversations with user follow-ups, thinking blocks (e.g. `thinking-response`, `multi-turn-user`, `long-conversation`) Run `npm run perf:chat -- --help` to see the full list of registered scenario IDs. ### Metrics collected - **Timing:** time to first token, time to complete, time to render complete (includes typewriter animation) - **Rendering:** layout count, layout duration (ms), style recalculation count, forced reflows, long tasks (>50ms), long animation frame count and duration - **Memory:** heap before/after, heap delta post-GC (informational, noisy for single requests) - **Extension host:** heap before/after/delta via CDP inspector ### Regression triggers vs informational metrics Only these metrics trigger a regression failure (when they exceed the threshold with statistical significance): - `timeToFirstToken`, `timeToComplete` — user-perceived latency - `layoutDurationMs` — total layout time from the trace (the *real* layout cost) - `forcedReflowCount` — forced synchronous layouts are always bad - `longTaskCount` — main thread jank These are reported but **informational only** (won't fail CI): - `longAnimationFrameCount` — number of long animation frames; noisy and compositor-driven, so tracked for signal but does not gate. Real jank is gated via `longTaskCount` / `layoutDurationMs`. - `layoutCount` — number of layout ops; inflated by CSS animations (compositor-driven, cheap). A build can do *more but cheaper* layouts, so gate on `layoutDurationMs`, not this count. - `recalcStyleCount` — number of style recalcs; inflated by CSS animations (compositor-driven, cheap) - `timeToRenderComplete` — includes typewriter animation tail - Memory/heap metrics — too noisy for single-request benchmarks ### Statistics Results use **IQR-based outlier removal** and **median** (not mean) to handle startup jitter. The **coefficient of variation (cv)** is reported — under 15% is stable, over 15% gets a ⚠ warning. Baseline comparison uses **Welch's t-test** on raw run values to determine statistical significance before flagging regressions. Use 5+ runs to get stable results. ## Memory leak check **Script:** `scripts/chat-simulation/test-chat-mem-leaks.js` **npm:** `npm run perf:chat-leak` Launches one VS Code session, sends N messages sequentially, forces GC between each, and measures renderer heap and DOM node count. Uses **linear regression** on the samples to compute per-message growth rate, which is compared against a threshold. ### Key flags | Flag | Default | Description | |---|---|---| | `--messages <n>` / `-n` | `10` | Number of messages to send. More = more accurate slope. | | `--build <path\|ver>` / `-b` | local dev | Build to test. | | `--threshold <MB>` | `2` | Max per-message heap growth in MB. | | `--setting <k=v>` | — | Set a VS Code setting override (repeatable). | | `--verbose` | — | Print per-message heap/DOM counts. | ### What it measures - **Heap growth slope** (MB/message) — linear regression over forced-GC heap samples. A leak shows as sustained positive slope. - **DOM node growth** (nodes/message) — catches rendering leaks where elements aren't cleaned up. Healthy chat virtualizes old messages so node count plateaus. ### Interpreting results - `0.3–1.0 MB/msg` — normal (V8 internal overhead, string interning) - `>2.0 MB/msg` — likely leak, investigate retained objects - DOM nodes stable after first message — normal (chat list virtualization working) - DOM nodes growing linearly — rendering leak, check disposable cleanup ## CI runs & pinpointing regressions The perf + leak checks run in CI via the **`.github/workflows/chat-perf.yml`** workflow (a scheduled daily `workflow_dispatch`, plus manual dispatch). Each run benchmarks the current `main` as the **test** build against a fixed release **baseline** (from `config.jsonc`, e.g. `1.122.0`). Because the baseline is fixed, the **test** median for a metric across successive daily runs traces `main`'s trajectory over time — that's what lets you bisect *when* something regressed or went flaky. ### Finding and reading historic runs ```bash # List recent runs (most recent first) — note the run IDs and dates gh run list --workflow chat-perf.yml -R microsoft/vscode --limit 30 \ --json databaseId,status,conclusion,createdAt,headBranch \ --jq '.[] | [.databaseId, (.conclusion//.status), .createdAt, .headBranch] | @tsv' # See a run's per-job results gh run view <run-id> -R microsoft/vscode --json jobs \ --jq '.jobs[] | [.name, (.conclusion//.status)] | @tsv' # Trigger a run manually against any ref/version: gh workflow run chat-perf.yml -R microsoft/vscode --ref main \ -f test_build=<branch|sha|version> -f baseline_build=<version> ``` ### Artifacts per run (and retention — this matters for old runs) | Artifact | Contents | Retention | |---|---|---|