gateway-debug

# Gateway Debug Local-process debugger workflow for the gateway, based on `gateway/DEBUG_GUIDE.md` **Option 2A** (controller + policy-engine locally, Envoy in Docker) and **Option 2B** (also runs Python Executor locally — only when a Python policy is under test). Envoy itself is **never** the debug target — we always run it in Docker. We use `dlv` against the gateway-controller and policy-engine Go binaries directly. > **Path convention.** All paths in this skill are written as > `<REPO_ROOT>/...` — substitute your api-platform checkout (e.g. > `~/git/api-platform`) for `<REPO_ROOT>`. To paste shell blocks as-is, each > block first sets a matching variable: > ```bash > REPO_ROOT="$(git rev-parse --show-toplevel)" > ``` > and then references `$REPO_ROOT/gateway/...`. Run the commands from anywhere > inside the checkout — no hard-coded paths. ## When to use - "Debug why the controller / policy engine is doing X" - "Step through controller code when I POST to /rest-apis" - "Find and fix a bug in gateway-controller / policy-engine" - A specific request returns the wrong response and source-level inspection is needed (when [[gateway-integration-tests]] step 3 — logs / config dumps — wasn't enough) - An Integration Test (IT) reproducer exists but you need to step through the gateway side, not the test side ## Workflow Work through phases 1-6 + 8 every time. Phase 7 (the fix loop) is **optional** — only run it when the user explicitly asked you to fix the issue, not just to investigate / explain it. 1. **Decide which component to debug** (Step 1) — usually controller XOR policy-engine; pick wrong and you'll watch the wrong process while the bug fires elsewhere. 2. **Prepare the env** — picking **Path A** (deployment-only) or **Path B** (request-runtime through Envoy) drives which substeps you run: edit a compose file, run the builder, start Envoy (Step 2). 3. **Launch the chosen component under `dlv`** in headless mode (Step 3). 4. **Reproduce the issue** with `curl` against the management REST API + the example YAMLs in `gateway/examples/` (Step 4). 5. **Triage from logs + config dumps** before touching the debugger (Step 5). 6. **Attach `dlv`, set function-name breakpoints, find root cause** (Step 6). 7. **(Optional) Apply a fix and re-verify through the same loop** (Step 7) — skip if the request was investigate-only. 8. **Clean up**: revert the compose / config.toml edits, stop processes, `docker compose down` (Step 8). ## Layout - Gateway tree: `<REPO_ROOT>/gateway/`. Most commands `cd` into it. - Controller entry: `<REPO_ROOT>/gateway/gateway-controller/cmd/controller` - Policy-engine entry: `<REPO_ROOT>/gateway/gateway-runtime/policy-engine/cmd/policy-engine` - Builder entry: `<REPO_ROOT>/gateway/gateway-builder/cmd/builder` - Python executor: `<REPO_ROOT>/gateway/gateway-runtime/python-executor/main.py` - Builder output (required by PE): `<REPO_ROOT>/gateway/gateway-builder/target/output/` - VS Code launch envs (use these for local processes too): `<REPO_ROOT>/.vscode/launch.json` - Example API/secret/key YAMLs: `<REPO_ROOT>/gateway/examples/` — including `sample-echo-api.yaml` (debug-friendly echo upstream wired to the in-compose `sample-backend`) - Management REST API spec: `<REPO_ROOT>/gateway/gateway-controller/api/management-openapi.yaml` - Admin REST API spec (config_dump etc.): `<REPO_ROOT>/gateway/gateway-controller/api/admin-openapi.yaml` - Controller config: `<REPO_ROOT>/gateway/configs/config.toml` ## Step 1: Decide which component to debug Before touching anything, work out where the bug lives. Wrong pick = you stare at the wrong process. Use config dumps to localise — config flows **controller → (xDS) → policy-engine + Envoy**, so the first dump that looks wrong is the culprit. Bring the stack up briefly in normal Docker mode if needed: ```bash REPO_ROOT="$(git rev-parse --show-toplevel)" cd "$REPO_ROOT/gateway" docker compose up -d gateway-controller gateway-runtime sample-backend ``` Then: ```bash # Controller — what it decided to deploy (APIs, routes, clusters, policy chains) curl -s http://localhost:9094/api/admin/v0.9/config_dump | jq . # 9094 in docker-compose; 9092 when running locally curl -s http://localhost:9094/api/admin/v0.9/xds_sync_status | jq . # Policy-engine — what it received from controller via xDS curl -s http://localhost:9002/config_dump | jq . # Envoy — live listeners/routes/clusters + per-endpoint health curl -s http://localhost:9901/config_dump | jq . curl -s http://localhost:9901/clusters ``` > **Admin port** for the controller is `9092` everywhere except the *dev* > docker-compose (`<REPO_ROOT>/gateway/docker-compose.yaml`), which remaps it > to `9094`. The IT compose (`<REPO_ROOT>/gateway/it/docker-compose.test.yaml`) > and the local dlv-driven controller (Step 3a) both bind the native `9092`. > Pick the port that matches the mode you're in. Rule of thumb: | Symptom | Debug this | |---|---| | `POST /rest-apis` returns wrong status / wrong response body | **controller** | | Controller `config_dump` is missing the API or has the wrong route/cluster/policy chain | **controller** | | Controller dump looks correct but PE `config_dump` doesn't have it | **controller** (xDS push) — but break in PE's xDS handler to confirm | | PE dump has it but request behaviour at the router is wrong (response transform, auth, rate limit, header munging…) | **policy-engine** | | Routing/upstream selection looks wrong and PE metadata is correct | **policy-engine** (`upstream` mapping) — Envoy itself we don't debug | | Python policy misbehaves | **python-executor** (Option 2B) | | Anything else | start with the **controller** — it owns the deployment record | Stop the temporary Docker stack before launching local processes: ```bash cd "$REPO_ROOT/gateway" && docker compose down ``` ## Step 2: Prepare the environment > **One-time provisioning (required before the first `docker compose up`).** Run setup once from > `<REPO_ROOT>/gateway`. It generates `api-platform.env` (required runtime defaults), the router's HTTPS > listener certificate, the AES-256 at-rest encryption key, and the gateway-controller **admin > credentials**. Startup fails if basic auth is enabled with no credential, so run it non-interactively > with fixed credentials to keep the `-u admin:admin` examples in this skill valid: > > ```bash > ADMIN_USERNAME=admin ADMIN_PASSWORD=admin ./scripts/setup.sh > ``` > > The gateway never auto-generates keys/certs and has no demo mode: the compose `env_file:` is now > `required: true` (a missing `api-platform.env` fails `docker compose up`), and the controller exits at > startup if the encryption key is missing. Full reference: > [Gateway Quick Start](../../../docs/gateway/quick-start-guide.md). > **Compose target — pick one before editing anything.** Step 2 (and Step 8 > cleanup) operates on **one** compose file. Subsequent substeps reference > "the compose file" generically: > > | Mode | Compose file (`<COMPOSE>`) | Run `docker compose` from (`<COMPOSE_CWD>`) | Use when | > |---|---|---|---| > | **Standalone** *(default)* | `<REPO_ROOT>/gateway/docker-compose.yaml` | `<REPO_ROOT>/gateway` (default filename, no `-f` needed) | Investigating from scratch; no IT scenario in play | > | **IT handoff** | `<REPO_ROOT>/gateway/it/docker-compose.test.yaml` | `<REPO_ROOT>/gateway/it` (use `-f docker-compose.test.yaml` — non-default filename) | Reproducing a failing integration test (handoff from [[gateway-integration-tests]]) — keeps the IT mocks (`mock-jwks`, `mock-platform-api`, …) that the scenario depends on | > **Pick a path before doing any 2x substep.** Two flows in the gateway > need different setups; choose based on what the bug actually involves. > Once you've picked, the rest of Step 2 (and Step 8 cleanup) applies only > to the substeps your path lists. > > | Path | What the bug involves | Setup to run | Then go to | > |---|---|---|---| > | **A. Deployment path** | The controller's REST API + its internal state. Examples: deploy / update / delete a REST API, secret CRUD, api-key generate / list / regenerate / revoke, MCP / LLM-resource CRUD. **No traffic flows through Envoy or the policy engine** in the failing flow. | **2b only** (and only if you'll also launch the policy-engine, which Path A normally doesn't) | **3a** — launch controller under dlv | > | **B. Request-runtime path** | An actual HTTP request through the gateway misbehaves: wrong upstream, wrong status, header munging, auth/rate-limit verdict, response transform, etc. Envoy receives the request and ext_proc-calls the policy engine. | **2a + 2b + 2c** (compose edit + builder + Envoy in Docker) | **3a + 3b** — launch both controller and policy-engine under dlv | > > Path B is the broader one — anything that exercises the runtime data > plane. Path A is faster and dockerless: the controller alone, in Step 3a, > is self-sufficient for the management-API flows. ### 2a. Edit the compose file (so the in-Docker Envoy talks to the host process) Make **two** changes to the `gateway-runtime` service block: 1. Point the runtime at the host. Replace whatever the file currently has for `GATEWAY_CONTROLLER_HOST` (in the dev compose it's `gateway-controller`; in the IT compose it's `it-gateway-controller`) with `host.docker.internal`: ```yaml gateway-runtime: environment: - GATEWAY_CONTROLLER_HOST=host.docker.internal ``` 2. Comment out the **Policy Engine** port mappings — those ports belong to the local process now, leaving them mapped collides on bind: ```yaml ports: - "8080:8080" # HTTP ingress — keep - "8443:8443" # HTTPS ingress — keep - "8081:8081" # xDS-managed listener — keep - "9901:9901" # Envoy admin — keep # - "9002:9002" # PE Admin — COMMENTED OUT # - "9003:9003" # PE Metrics — COMMENTED OUT ``` **Remember to revert these in Step 8** (`git checkout -- <COMPOSE>`). A leftover `host.docker.internal` will silently break plain `docker compose up` (or `make test`, for the IT compose) for other people on the repo. ### 2b. Run the gateway-builder (required before launching policy-engine) The policy-engine source includes generated files (`plugin_registry.go`, `build_info.go`) that the builder emits — without them, `dlv debug` against the PE entry point won't compile. Run the builder once before the first PE debug session, and again whenever `build.yaml` or any compiled policy changes: ```bash REPO_ROOT="$(git rev-parse --show-toplevel)" cd "$REPO_ROOT/gateway/gateway-builder" go run ./cmd/builder \ -build-file ../build.yaml \ -system-build-lock ../system-policies/system-build-lock.yaml \ -policy-engine-src ../gateway-runtime/policy-engine \ -out-dir ./target/output \ -log-level info ``` Confirm: - `ls "$REPO_ROOT/gateway/gateway-builder/target/output/gateway-controller/policies/" | wc -l` — should be 40+ policy YAMLs. - `ls "$REPO_ROOT/gateway/gateway-runtime/policy-engine/cmd/policy-engine/" | grep -E 'plugin_registry|build_info'` — both files should exist (gitignored). > Builder output is also where the controller reads policy definitions from > (`controller.policies.definitions_path`, wired to `APIP_GW_CONTROLLER_POLICIES_DEFINITIONS_PATH` > via a `{{ env }}` token in the config — the prefix override no longer applies on its own). > If the controller starts and complains it can't load policy definitions, > you skipped the builder. ### 2c. Start the Envoy router (and any backends) in Docker Standalone mode: ```bash REPO_ROOT="$(git rev-parse --show-toplevel)" cd "$REPO_ROOT/gateway" docker compose up -d gateway-runtime sample-backend docker compose logs -ft gateway-runtime # tail [rtr] / [pol] streams ``` IT-handoff mode (bring up only the services your scenario needs — minimally `gateway-runtime` + the backends/mocks it touches, e.g. `sample-backend`, `mock-jwks`, `mock-platform-api`): ```bash REPO_ROOT="$(git rev-parse --show-toplevel)" cd "$REPO_ROOT/gateway/it" docker compose -f docker-compose.test.yaml up -d gateway-runtime sample-backend mock-platform-api # add others as the scenario requires docker compose -f docker-compose.test.yaml logs -ft gateway-runtime ``` Envoy will sit waiting for xDS from `host.docker.internal:18000` (controller) and ext_proc on `host.docker.internal:9001` (policy engine). Until you launch those in Step 3, `curl http://localhost:9901/ready` returns 503 — that's expected and clears once Step 3 brings both up. ## Step 3: Launch the chosen component under `dlv` `dlv debug` builds from source and starts the binary. Use `--headless --listen=127.0.0.1:<port> --accept-multiclient --continue` so the process runs immediately and you can attach / detach freely. **Pass the same env vars that `.vscode/launch.json` does for that configuration.** The blocks below extract those env vars and run `dlv` from a single Bash invocation. > ⚠️ **Config change — the `APIP_GW_` prefix override was removed.** The gateway loaders now read > **only** the `-config` file layered over defaults; an environment value reaches a setting solely > through a `{{ env "NAME" }}` interpolation token in that file. The `APIP_GW_*` env vars below > therefore take effect **only** for keys whose config value is a matching token. `configs/config.toml` > currently tokenizes storage, control-plane, logging, metrics and the policies path — but **not** the > machine-specific dev paths (LLM-template dir, downstream TLS cert/key, lua script) or the local > tcp policy-engine/analytics split used here. For from-source `dlv` runs, point `-config` at a > **local, git-ignored** `config.toml` (copied from `configs/config-template.toml`) with those values > filled in — or add `{{ env }}` tokens for them to that local config so the variables below apply. > _Follow-up: ship a ready-made dev `config.toml` template for this recipe._ ### 3a. Gateway Controller (dlv on `127.0.0.1:2345`) ```bash REPO_ROOT="$(git rev-parse --show-toplevel)" cd "$REPO_ROOT/gateway/gateway-controller" # Env vars mirror .vscode/launch.json → "Gateway Controller" APIP_GW_CONTROLPLANE_HOST="" \ APIP_GW_GATEWAY_REGISTRATION_TOKEN="" \