run-load-test

# Run the Omnigent load test Drives `dev/loadtest/` end to end: collect inputs → run → read `summary.md` → explain the latencies. **Each Locust user is a real `omnigent host`** that registers over the host tunnel, creates host-bound sessions, and drives **real multi-turn conversations** — every turn is a genuine post→idle loop through the host's runner, with the **LLM mocked** (zero latency) so the numbers are Omnigent's own overhead. `-u N` scales the number of hosts. It **boots its own local stack** (server + mock LLM), so there is no server to point at, and it runs **from a repo checkout** only. For single-request latency micro-benchmarks (not concurrency), that is a different tool: `dev/benchmarks/`. ## 1. Ensure deps (repo checkout) ```bash uv sync --extra loadtest --extra agents-sdk ``` Run with that same interpreter (e.g. `.venv/bin/python`), from the repo root. ## 2. Gather inputs Ask the user (AskUserQuestion when several are unknown); all have defaults. | Input | Flag | Default | Notes | |---|---|---|---| | Hosts | `--users` | 4 | Concurrent hosts (N) — the main scale knob. | | Spawn rate | `--spawn-rate` | 1 | Hosts started per second. | | Run time | `--run-time` | 120s | `40s` / `5m` / `1h`. | | Sessions/host | `--sessions-per-user` | 2 | Host-bound sessions each host drives. | | Turns/session | `--turns-per-session` | 4 | Turns per session — history grows across them. | | Reply length | `--reply-words` | 60 | Words in the mocked (streamed) reply per turn. | **Capacity caveat — say this to the user if they ask for large N:** turns run on real host + runner subprocesses, so N hosts × M sessions = N×M runner processes on *this* box. It is capacity-limited by design (real turns, not faked). Start at `--users 2 --sessions-per-user 1 --turns-per-session 2 --run-time 40s` to confirm the stack boots (~10-30s), then ramp to a few dozen hosts at most. At high N the load box saturates before the server (Locust warns about CPU). ## 3. Run ```bash python dev/loadtest/run.py \ --users <N> --spawn-rate <R> --run-time <T> \ --sessions-per-user <S> --turns-per-session <TU> ``` It boots the stack, prints the server URL + registered agent, runs Locust, and writes `dev/loadtest/results/omnigent_load_test-<timestamp>/`. ## 4. Read and explain `Read` the `summary.md` and relay it. Focus on: - **Outcome / failures** first. Exit 0 + 0 failures = PASS. Non-zero failures are the headline — check `console.log` and, for a host that failed to register, the per-host `results/.../host-workspaces/<name>/host.log`. At high N, failures usually mean the *load box* saturated, not the server. - **turn** — the headline latency: one full post→idle agent turn on a host's runner (mocked LLM), so it is Omnigent's per-turn overhead. It **grows across a conversation** as history accumulates, so a rising p95/p99 with larger `--turns-per-session` is expected and is the interesting signal. - **host online** — host tunnel registration cost; **session create** — the host-bound create; **Ops/s** — aggregate throughput at this concurrency. If failures appeared or the tail looks high, suggest a concrete next step (lower N if the load box is saturated, raise `--turns-per-session` to study history growth, lengthen `--run-time` for steady state, or check server logs/metrics). ## Notes - Scenario file: `dev/loadtest/omnigent_load_test.py`; driver + report: `dev/loadtest/run.py`. Full reference: `dev/loadtest/README.md`.