openapi-spec-conventions

# OpenAPI Spec Conventions Treadstone's API surface is published through three distinct OpenAPI artifacts — each serving a different consumer. Understanding which artifact does what prevents accidentally breaking the SDK or the docs. ## The Three-Tier Spec ``` FastAPI routes │ ▼ build_full_openapi_spec(app) ← treadstone/openapi_spec.py │ ├─► openapi.json ← Full spec (admin + audit + control plane) │ │ Used by: make gen-web-types │ │ Contains: ALL routes │ │ │ └─► filter_public_openapi() │ │ │ └─► openapi-public.json ← SDK spec (control plane only) │ │ Used by: make gen-sdk-python │ │ Excludes: /v1/admin, /v1/audit │ │ Excludes: sandbox runtime paths │ └─► merge_sandbox_paths() ← treadstone/openapi_spec.py │ └─► Runtime GET /openapi.json ← Swagger UI docs │ Used by: /docs page │ = public spec + sandbox proxy paths │ Excludes: /v1/admin, /v1/audit └─► Displayed as "Sandbox: shell", "Sandbox: file" tag groups ``` ### Tier 1 — `openapi.json` (full, exported) Generated by `make gen-openapi` → `scripts/export_openapi.py`. Contains every route including admin and audit. Used **only** to generate `web/src/api/schema.d.ts` via `make gen-web-types`. Never served at runtime. Gitignored. ### Tier 2 — `openapi-public.json` (SDK source) Derived from Tier 1 by `filter_public_openapi()`, which drops all paths under `/v1/admin` and `/v1/audit`. Sandbox runtime proxy paths are **not** merged here — the SDK stays focused on the control plane. Source for `make gen-sdk-python` → `sdk/python/`. Gitignored. ### Tier 3 — Runtime `/openapi.json` (Swagger UI) Served by `_public_openapi()` in `treadstone/main.py`. Starts from Tier 2, then merges sandbox runtime paths via `merge_sandbox_paths()`. This is what users see at `/docs`. --- ## Sandbox Runtime Paths ### Why they exist in docs but not in the SDK The real proxy implementation (`treadstone/api/sandbox_proxy.py`) uses `include_in_schema=False` — it is a transparent HTTP forwarder, not a typed API endpoint, so it intentionally produces no OpenAPI schema. To help developers understand the sandbox's capabilities, `merge_sandbox_paths()` reads a static snapshot of the sandbox's own OpenAPI spec and injects those paths into the runtime docs at the correct proxy prefix. The Python SDK does **not** include these paths. Data-plane operations are meant to be called via `agent_sandbox.Sandbox(base_url=sandbox.urls.proxy)`, not through generated Treadstone SDK methods. ### How merge_sandbox_paths() works Source file: `treadstone/openapi_spec.py` ``` scripts/sandbox_openapi_base.json ← static snapshot of sandbox internal OpenAPI │ ├─ rename conflicting schemas: │ Response → SandboxApiResponse │ ValidationError → SandboxValidationError │ HTTPValidationError → SandboxHTTPValidationError │ ├─ prefix every path: │ /v1/shell/exec → /v1/sandboxes/{sandbox_id}/proxy/v1/shell/exec │ ├─ inject sandbox_id path parameter into every operation │ └─ retag operations: ["shell"] → ["Sandbox: shell"] ["file"] → ["Sandbox: file"] ... ``` Called from: `main.py` → `_public_openapi()` only. **Not** called from `export_openapi.py` (SDK generation must stay clean). ### Updating the sandbox spec snapshot When the sandbox runtime adds new routes or changes its schema, update the snapshot: ```bash # Fetch the updated spec from a running sandbox instance, then: cp /path/to/new/sandbox_openapi.json scripts/sandbox_openapi_base.json # Remove the /terminal route (HTML page, not REST API): python3 -c " import json with open('scripts/sandbox_openapi_base.json') as f: s = json.load(f) s['paths'].pop('/terminal', None) with open('scripts/sandbox_openapi_base.json', 'w') as f: json.dump(s, f, indent=2) " ``` Then restart the API server — `_public_openapi()` caches on first call, so a restart is required for the updated spec to appear in `/docs`. --- ## Adding New API Routes ### Adding a Treadstone control-plane route 1. Add the FastAPI route in `treadstone/api/<module>.py` with a `tags=["tag-name"]`. 2. Assign a clear `operationId` via the function name (FastAPI generates it as `{tag}-{function_name}` using `custom_generate_unique_id` in `main.py`). 3. Run `make gen-openapi` to regenerate both `openapi.json` and `openapi-public.json`. 4. If the web app needs the new types, run `make gen-web-types`. 5. If the SDK needs new methods, run `make gen-sdk-python`. 6. Check for the `@audit_log` decorator — changes to auth, admin, sandbox lifecycle, or API key management require audit log coverage (see `AGENTS.md`). **Two-commit pattern for large SDK diffs** (see AGENTS.md → OpenAPI / SDK Generation): - Commit 1: source changes (`treadstone/`, `tests/`, `alembic/`, `web/src/api/schema.d.ts`) - Commit 2: `chore: regenerate Python SDK from OpenAPI` — `sdk/python/` only ### Hiding a route from the public SDK (admin / audit style) Add the path prefix to `HIDDEN_FROM_PUBLIC_PATH_PREFIXES` in `treadstone/openapi_spec.py`: ```python HIDDEN_FROM_PUBLIC_PATH_PREFIXES: tuple[str, ...] = ("/v1/admin", "/v1/audit") ``` Routes matching these prefixes appear in `openapi.json` (web types) but are stripped from `openapi-public.json` (SDK) and from the runtime `/openapi.json` (Swagger UI). ### Hiding a route from ALL OpenAPI output (proxy style) Use `include_in_schema=False` on the FastAPI route decorator. This is appropriate for transparent proxies or internal endpoints that have no meaningful schema. --- ## SDK Generation The Python SDK lives in `sdk/python/` and is generated by `openapi-python-client` from `openapi-public.json`. ```bash make gen-openapi # Regenerate openapi.json + openapi-public.json make gen-sdk-python # Regenerate sdk/python/ from openapi-public.json make gen-clients # Both gen-web-types + gen-sdk-python ``` **SDK method naming** derives from `tags[0]` and the FastAPI function name. Example: `POST /v1/sandboxes` with `tags=["sandboxes"]` and function `create_sandbox` → SDK method `sandboxes_create_sandbox.sync(...)`. Keeping tags consistent and function names descriptive is important for the SDK's usability. --- ## Control Plane vs Data Plane (Examples Pattern) The `examples/` directory illustrates the two-plane architecture: ``` Control plane → treadstone_sdk.AuthenticatedClient Manages: sandbox lifecycle, templates, API keys Data plane → agent_sandbox.Sandbox(base_url=sandbox.urls.proxy) Operates inside: shell, file, browser, jupyter, mcp ``` The Treadstone Python SDK intentionally does **not** include data-plane methods. Users connect the two planes by extracting `sandbox_detail.urls.proxy` from the control plane and passing it as `base_url` to `agent_sandbox.Sandbox`. See `examples/data_plane/01_agent_sandbox_runtime.py` for the authoritative connection pattern. --- ## Key Files | File | Purpose | |------|---------| | `treadstone/openapi_spec.py` | `build_full_openapi_spec`, `filter_public_openapi`, `merge_sandbox_paths` | | `treadstone/main.py` | `_public_openapi()` — runtime spec served at `/openapi.json` | | `scripts/export_openapi.py` | Exports `openapi.json` + `openapi-public.json` for SDK/web generation | | `scripts/sandbox_openapi_base.json` | Static snapshot of sandbox internal OpenAPI (no `/terminal`) | | `openapi-client-config.yaml` | SDK generator config (package name overrides, post-gen hooks) | | `sdk/python/` | Generated Python SDK — do not hand-edit | ## Checklist: When touching API or OpenAPI - [ ] Route has correct `tags=[...]` (drives SDK method names) - [ ] Function name is descriptive (drives SDK method names) - [ ] Admin/audit/internal routes use `HIDDEN_FROM_PUBLIC_PATH_PREFIXES` or `include_in_schema=False` - [ ] Audit log coverage added for control-plane changes (per AGENTS.md) - [ ] `make gen-openapi` run after any route changes - [ ] `make gen-sdk-python` run if SDK consumers need new methods - [ ] `_public_openapi()` cache cleared (API restart) if sandbox spec snapshot updated - [ ] Two-commit pattern used if SDK diff is large