kev-deploy

# Deploy Kev on Modal `scripts/kev_serve.py` is a single self-contained Modal app. `modal deploy` on it builds an image with the kev package at a pinned commit, loads a Kev checkpoint from the Hugging Face Hub on a GPU sized for it, and serves TypeSafe's System One protocol (`POST /v1/systemone`, `GET /v1/models`) at `https://<workspace>--kev-api.modal.run`. It scales to zero when idle. The file is short on purpose: if the user's case does not fit, read it and edit it. ## 1. Check the prerequisites ```bash python3 -c "import modal" 2>/dev/null || pip install modal modal profile current 2>/dev/null || modal setup # opens a browser to sign in; the user must do this step modal skills install -y # optional: Modal's own agent skill + docs (into .agents/, or -g for home) ``` If `modal setup` is needed, tell the user and wait; do not try to authenticate for them. `modal skills install` gives you Modal's official skill and documentation, which helps with anything beyond this file: GPUs, secrets, volumes, logs, billing. ## 2. Pick the model | `KEV_MODEL` | GPU (automatic; fallbacks in parentheses) | $/h while up | Model time, 6 questions (new / repeated state) | Cold start (cached weights) | When | | --- | --- | --- | --- | --- | --- | | `jaredpalmer/kev-0.8b` | L4 (L40S) | 0.80 | 23 / 16 ms | ~40 s | cheapest, prototyping | | `jaredpalmer/kev-4b` (default) | L40S (H100) | 1.95 | 42 / 28 ms (H100: 18 / 13 ms) | ~35 s | the default: best quality per dollar | | `jaredpalmer/kev-9b` | H100 (H200, L40S) | 3.95 | 24 / 17 ms | ~55 s | accuracy on smaller GPUs | | `jaredpalmer/kev-27b` | B200 (H200, H100) | 6.25 | 47 / 32 ms (H200: 67 / 50 ms) | ~50 s | best released accuracy; 51 GB of full weights | Model time is the `latency_ms` the API returns (median of 20 requests, measured in the Kev repo: `runs/serve-*`, `runs/grouping-4b-h100`, `runs/fused-27b-*` and `runs/serving-27b-r23`; Kev-27B's B200 and H100 figures were measured on its previous version, the same architecture in bf16). A new state is the normal call, since every ticket is a new state; a repeated state is served from a prefix cache. The very first cold start of an account also downloads the weights and compiles kernels (1-2 minutes); both are cached on the `kev-hf-cache` volume afterwards. Other GPUs work with `KEV_GPU` but are worse picks: an L4 runs out of compute on Kev-4B, and an A100 is slower than an L40S here and costs more. Kev-27B is compute-bound under load: a B200 serves ~57 mixed requests/s at 64 concurrent clients (H200 ~40, H100 ~36) for about the same cost per request, with the lowest latency. Its first cold start downloads 51 GB of weights (several minutes). A warm container costs the GPU's hourly rate only while it is up; after five idle minutes it scales to zero. `KEV_MIN_CONTAINERS=1` keeps one warm (no cold starts, pays the hourly rate all the time). `@revision` pins a checkpoint revision (`jaredpalmer/kev-4b@v7-base`). ## 3. Deploy Always set an API key unless the user explicitly wants a public URL; without one, anyone with the URL can spend their GPU time. ```bash curl -LO https://raw.githubusercontent.com/jaredpalmer/kev/main/skills/kev-deploy/scripts/kev_serve.py # or use the skill's copy export KEV_API_KEY=$(openssl rand -hex 24) # save it: it is the endpoint's bearer token KEV_MODEL=jaredpalmer/kev-4b modal deploy kev_serve.py # prints the URL ``` Settings are read at deploy time; redeploying with other values replaces the model behind the same URL. `KEV_APP_NAME=kev-9b` gives a second, independent endpoint (`https://<workspace>--kev-9b-api.modal.run`). `KEV_GPU=H100` overrides the GPU list (comma-separated). `KEV_REGION=us` (or `us-east`, `eu`, ...) pins where the container runs: without it Modal takes the first region with a free GPU, which can be another continent (an unpinned Kev-4B landed in Frankfurt and added ~150 ms to every round trip from the US). A pinned region costs 1.15-1.75x on Modal; pin it near the callers for latency-sensitive use. `KEV_TRUNCATE_STATES=1` reads the first 65,536 tokens of a longer document instead of refusing it (see Long documents under Troubleshooting). ### Throughput A container answers concurrent requests in batches: its model thread takes everything waiting and runs it through shared passes. In-process, Kev-4B on an H100 serves about 95 six-question requests/s (120 on mixed short records); an L40S about 45. Over HTTP the front door matters more than the GPU (Kev-4B, H100, a client in the same region, measured 2026-09-24): | Front door | One request, round trip | 8 / 32 concurrent clients | Scaling | | --- | --- | --- | --- | | web endpoint (default) | ~77 ms | 77 / 103 req/s | Modal adds containers past 32 concurrent requests each | | `KEV_FLASH=1` (experimental) | ~46 ms | 70 / 112 req/s per container | one container stays up; past ~32 concurrent per container requests queue at the proxy (p99 ~4 s at 64) | For steady high traffic, keep containers warm (`KEV_MIN_CONTAINERS=2` or more) so bursts do not wait for a cold start, and size it at about 32 concurrent requests per container. `KEV_FLASH=1` needs `KEV_REGION` (its proxy is regional) and its URL is printed as `https://<workspace>--<app>-kev.<region>.modal.direct`. ## 4. Verify The first request after a deploy or an idle period waits for the cold start. Modal answers a request that waits longer than 150 s with an HTTP 303 to a result URL, so warm the endpoint with a redirect-following call first: ```bash curl -sL --max-time 900 $KEV_URL/v1/models -H "authorization: Bearer $KEV_API_KEY" # returns once the model is loaded ``` ```bash curl -s $KEV_URL/v1/systemone -H "authorization: Bearer $KEV_API_KEY" -H 'content-type: application/json' -d '{ "state": "Order 4411 arrived late and the box was crushed. Two charges appear on my card.", "model": "kev-latest", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"returns": "Exchanges, refunds", "shipping": "Delivery, delays", "billing": "Charges, payments"}}, "urgent": {"type": "noul", "instructions": "Does this need urgent human attention?"}}}' curl -s $KEV_URL/v1/models -H "authorization: Bearer $KEV_API_KEY" # served checkpoint, base, temperature ``` Expect per-question `probabilities` (calibrated by the checkpoint's own temperature), `choice` / `noul` / `score`, and `latency_ms`, the model time: tens of milliseconds warm (table above). The round trip adds the network and Modal's proxy, about 80-100 ms from a client in the US to a us-east container over a kept-alive connection, more with a new TLS connection per request, so reuse one HTTP client. The first request of a new shape (question set, state length) runs without a CUDA graph, about 2-4x slower; the server captures one in the background and later requests use it. `/v1/models` reports the served checkpoint, its temperature and the number of captured graphs. A request without the key must return 401. ## 5. Wire it in The protocol is TypeSafe's, so only the base URL and key change: ```python client = TypeSafeClient(api_key=KEV_API_KEY, base_url=KEV_URL, model="kev-latest") # was: TypeSafeClient(api_key=TYPESAFE_KEY) ``` ```ts const r = await fetch(`${KEV_URL}/v1/systemone`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${KEV_API_KEY}` }, body: JSON.stringify({ state, model: "kev-latest", questions }) }); // question type "noul", not "boolean" ``` Kev answers typed questions about a state (Choice over named options, Noul yes/no, Score over ordered levels) without generating text. It was trained on public classification, policy and rule data; on the user's own domain, measure it on a few hundred labelled examples before relying on it, and if it falls short, fine-tune it with the `kev-finetune` skill. ## 6. Take it down ```bash modal app stop kev # or the KEV_APP_NAME used; the URL stops working immediately modal volume delete kev-hf-cache # optional: the cached weights (shared with kev-finetune; next deploy re-downloads) ``` ## Troubleshooting - **First request returns nothing or a 303**: the cold start is still running (weights download on the very first start, or Modal is still finding a GPU; `modal app logs kev` says "waiting to be scheduled"). Follow redirects (`curl -L`), use a longer client timeout, or deploy with `KEV_MIN_CONTAINERS=1`. - **CUDA out of memory on start**: the GPU is too small for that checkpoint (Kev-9B needs ~18 GB, Kev-4B ~10 GB); use the table above or `KEV_GPU=H100`. - **Slow round trips with fast `latency_ms`**: the container is far from the caller or every request opens a new connection; set `KEV_REGION` and reuse the HTTP client. - **401 with the right key**: the key is fixed at deploy time; redeploy with the same `KEV_API_KEY` exported. - **Long documents**: a state may have up to 65,536 tokens and a question with its options 8,192 more (more when the state is shorter). A longer state gets a 422 that gives its token count and the limit (the TypeSafe SDK raises `TypeSafeUnprocessableEntityError` with that message), as a longer question does: shorten or split the document. Deploying with `KEV_TRUNCATE_STATES=1` reads only the first 65,536 tokens instead, and then every response carries `truncated` and `usage.state_tokens` / `state_tokens_used`. Kev-27B trained on states of up to 32,768 tokens and the smaller models on 384, so all four accept long documents but Kev-27B answers them best. A new long state is slow, a repeated one is served from the prefix cache: Kev-27B on an H200 took 8-10 s for a 60k-token state and 0.28 s for the same request again; a 70k-token state read to 65,536 tokens with `KEV_TRUNCATE_STATES=1` took 12 s, then 0.29 s. The cache holds 65,536 state tokens in all, so a new state that long replaces the cached one, which is dropped before the new state's pass rather than after it (no second copy resident) (`runs/kev-deploy-71d4829` and `runs/kev-deploy-2ea5660` in the repo). - **Logs**: `modal app logs kev` shows the load line (`serving <model> on <GPU> ... ready in Ns`) and every request.