managed-model-endpoints

# Managed model endpoints A **managed model endpoint** is a model service the **daemon** owns: you register it **once**, then every `compute_provider` cell against it just works — the daemon swaps the resident model off the device (one model at a time, via the resident's own approved stop), runs your approved start script, waits for the readiness route, then runs your cell, streaming its lifecycle progress into the cell as it goes. You never run the container runtime yourself, never poll readiness in cells, and never see the credential value. Two verbs: `register()` (asks the user once) and ordinary inference cells. Container specifics — image, registry login, internal port, cache mount target, readiness route — come from the **model's own runbook skill**; this skill is the translation contract. ## Calling a registered endpoint — inference cells Calling a registered endpoint — use the `using-model-endpoint` skill (this skill is the REGISTRATION contract; that one documents the call side in full). The ONLY dispatch form is the `compute_provider` tool with the endpoint's registered name (`list_compute` shows them): ``` compute_provider(provider="boltz2-service", code=""" import requests r = requests.post(BASE_URL + "/v1/infer", json=payload) """) ``` The daemon brings the model up on demand (a first cold start downloads image + weights — minutes; let it run) and preloads `BASE_URL` into the cell — both as a Python variable (use it directly, as above) and as `os.environ["BASE_URL"]` (plus `INFER_API_KEY` for remote endpoints). Endpoints are **not kernel environments**: `environment="boltz2-service"` on a plain python cell fails — plain cells get no `BASE_URL`. ## Enablement — once per machine The user connects the family under **Customize → Compute → Model endpoints** (the setup flow saves the family credential first — connect-without-key is not a state) and picks ONE mode: **Local** (container registrations) or **URL/remote** (https against the configured host). Until connected, `free_port()`/`register()` raise a precise error — relay it; in the wrong mode they refuse with a teaching error naming the setting (existing endpoints of the unarmed leg keep dispatching — only NEW registrations refuse). **Disconnecting is a full teardown**: every active local service is stopped via its approved stop script and every registration (local AND hosted) is removed; caches stay on disk; a failing stop keeps that one row, FAILED. The "Local machine GPU" toggle never gates registration — it governs cell GPU access only; the approval card is the per-registration gate. **Credential contract (platform rule):** every registration passes `credential="NVIDIA_API_KEY"` — the daemon rejects any other name. Locally the value feeds the start script's registry login and never enters your kernel env; for remote endpoints it authenticates the upstream and is delivered only into the inference cell's env (as `INFER_API_KEY`), never the repl kernel. ## Register (repl kernel) ```python port = host.model_endpoints.free_port() # local only; random 20000-29999 host.model_endpoints.register( name="boltz2-service", # <model>-service -- descriptive, never the # bare model name (collides/ambiguous) url=f"http://127.0.0.1:{port}", # LITERAL 127.0.0.1 -- `localhost` rejected credential="NVIDIA_API_KEY", # the family credential NAME, never a value skill="<model-runbook-skill>", start=START_SCRIPT, # composed below stop="docker stop boltz2-service",# exit 0 ONLY once actually stopped live="/v1/health/ready", # readiness ROUTE (200 = model answers; # "up but loading" must read not-ready) ) ``` Name endpoints `<model>-service` (e.g. `diffdock-service`) — unambiguous in provider lists; never just the bare model name. Name the CONTAINER after the endpoint too (the template above does): the UI then follows the service's own logs live while it starts. `register()` **always cards the user** (scripts verbatim, port, service dir, credential name). One exception: a **byte-identical** re-registration is silent — same bytes are approved forever; any byte change re-cards. The registration stays inspectable under Customize → Compute. Re-registering to fix scripts: **reuse the existing url** — never call `free_port()` again (the port is the endpoint's stable mutex). ## Remote endpoints — upstream APIs (no lifecycle) Pass `url="https://<upstream>"` and **omit `start`/`stop`/`live`** — no port, no scripts, no readiness. Requires URL/remote mode (the setup radio; in Local mode https registrations refuse). The url's HOST must equal the configured upstream host exactly — you pick the path leaf, never the authority. After approval, cells are plain HTTP clients of `BASE_URL` authenticating with `$INFER_API_KEY`. `list_compute` labels every row `location: "local" | "remote"`. ## Composing the start script The daemon hands scripts three things in their **process environment** (never argv, never sudo): `HOST_PORT` (the registered port), `SERVICE_DIR` (this endpoint's persistent directory — put the model cache here), and the credential value under its own name. Nothing else is inherited — ambient tokens are not visible; the ONLY secret a script sees is its registered credential. The start script must be **idempotent** (cold create / warm start / crash re-entry), with the **port-mismatch guard** — the runtime freezes port mappings at container creation, so a container created under an OLD port must be recreated or readiness can never pass: ```bash mkdir -p "$SERVICE_DIR/cache" # docker login persists auth in $DOCKER_CONFIG/config.json; scope it to the # service dir so the credential dies with the service (never ~/.docker). export DOCKER_CONFIG="$SERVICE_DIR/.docker" create_service() { docker run -d --name boltz2-service \ --restart unless-stopped \ -p 127.0.0.1:${HOST_PORT}:8000 --gpus all \ -e NVIDIA_API_KEY \ -v "$SERVICE_DIR/cache:<cache target from the runbook>" \ <image from the runbook> } if docker inspect boltz2-service >/dev/null 2>&1 && \ [ "$(docker inspect -f '{{(index (index .HostConfig.PortBindings "8000/tcp") 0).HostPort}}' boltz2-service)" != "$HOST_PORT" ]; then docker rm -f boltz2-service # stale port mapping -- recreate below fi if docker inspect boltz2-service >/dev/null 2>&1; then docker start boltz2-service # warm wake -- no credential, no chown needed else echo "$NVIDIA_API_KEY" | docker login <registry> --username '<user>' --password-stdin docker pull <image from the runbook> # Cache must be writable by the CONTAINER's user, whose uid the image # defines (container uid != host uid). chown needs root the script doesn't # have; a throwaway root container does it -- and the chmod, which the # host user can no longer do once the dir is chowned away -- without sudo. CUID="$(docker inspect --format '{{.Config.User}}' <image from the runbook> 2>/dev/null | cut -d: -f1)" case "$CUID" in ''|root) CUID=0;; *[!0-9]*) CUID=1000;; esac # named user -> default 1000; runbook may override if [ "$CUID" != "0" ]; then docker run --rm -v "$SERVICE_DIR/cache:/c" alpine sh -c "chown -R $CUID:$CUID /c && chmod 700 /c" else chmod 700 "$SERVICE_DIR/cache" 2>/dev/null || true fi create_service fi # RUNTIME-binding guard (one retry): after a port-conflict crash the engine # can start the container yet silently skip port programming -- the CONFIG # still matches $HOST_PORT (so the guard above cannot catch it) but # `docker port` prints nothing and the model serves to nobody. Recreate. if [ -z "$(docker port boltz2-service 2>/dev/null)" ]; then docker rm -f boltz2-service create_service fi ``` Translation rules: - **Keep scripts ASCII** — non-ASCII (em dashes, arrows, curly quotes) triggers the approval card's spoofing warning; use `--` and `->` in comments. - **`export DOCKER_CONFIG="$SERVICE_DIR/.docker"`** before any `docker login` — login persists the credential in `config.json`, and scoping it to the service dir means Remove honestly reclaims it (never `~/.docker`, which outlives stop/Remove/Disable). - **`-p 127.0.0.1:${HOST_PORT}:<internal>`** — loopback-only publish; the internal port comes from the runbook. - If the image reads a different env name, bridge env→env at the top: `export OTHER_NAME="$NVIDIA_API_KEY"` (never argv, never a file). - **`-e NAME` bare** (argv is world-readable); the key rides the login stdin pipe only. - **`-d`, no `--rm`** — managed containers are **stopped, never removed**: stop parks them with weights loaded; `--rm` throws the cache away. - Cache under `$SERVICE_DIR`, owned by the **container's** uid: the runbook states it when it matters; otherwise derive it post-pull with `docker inspect --format '{{.Config.User}}' <image>` (empty or `root` ⇒ runs as root, no chown needed; a NAMED user can't be resolved without running the image — default `1000`). Getting it wrong is the cache-empty symptom: the container can't write the mount, weights leak into the writable layer and die on recreate (or the image crash-loops on Permission denied). Never `777` — world-writable cache on a multi-user host. The mount TARGET comes from the runbook. - Cells need **no auth header** against local endpoints — the credential is a pull key that never enters your kernel. ## Failures A failed start/stop flips the endpoint **FAILED** (transcript on the endpoint panel — never echoed into cell errors; ask the user to read it there) and your cell errors with the daemon's one-line cause. FAILED is **sticky**: further cells fail fast until the user presses Stop or you re-register (byte-identical re-register also clears it). If a stop is stuck (exit 0 but the port never frees), removal is refused while the port is bound — recover out-of-band; the daemon absorbs the freed port on its next probe. A first-ever cold start downloads image + weights — minutes, once; the cell streams the phase lines live and the endpoint detail view streams the full script output, so let it run.