dev-mitm-proxy

# MITM Proxy The MITM proxy is the network engine's HTTPS interception boundary. It intercepts traffic from the air-gapped guest VM, normalizes it into typed facts, hands a `SecurityEvent` to the security engine, and preserves allowed runtime bytes for upstream. Treat it as a system, not a collection of hacks -- every capability must be general-purpose. ## Security boundary Network code parses transport bytes, routes traffic, and emits typed `SecurityEvent` facts. It must not broker credentials, create credential refs, run CEL/security decisions, or sanitize ledger projections. Those belong to the security engine plugin rail. Every security plugin has the same data contract: it receives a `SecurityEvent` and returns a `SecurityEvent`; the plugin stage only controls ordering (`preprocess`, `postprocess`, or `logging`). There are two materialization paths and they must never be collapsed: - **Runtime materialization** prepares bytes for the real upstream. It may resolve a broker ref back to a real credential because the protocol needs it. - **Ledger materialization** prepares the event stored in `session.db`, structured logs, route JSON, and UI stats. It must contain only broker refs, hashes, bounded previews, typed detections, and plugin execution evidence. No credential logic belongs in HTTP header formatters, DB readers, frontend transforms, debug harnesses, or route adapters. If a future change needs capture, injection, redaction, threat intel, or PII handling, implement it as a security plugin stage over `SecurityEvent -> SecurityEvent`. ## Pipeline ``` Guest curl -> iptables REDIRECT -> capsem-net-proxy (guest, port 10443) -> vsock port 5002 -> Host MITM proxy -> SNI parse -> network metadata capture -> TLS terminate (rustls, per-domain cert minted from Capsem CA) -> HTTP request parse (hyper) -> SecurityEvent -> SecurityRuleSet + plugin rail -> Forward to real upstream over TLS -> Record telemetry to session DB -> Stream response back to guest ``` ## Key source files | File | What | |------|------| | `crates/capsem-core/src/net/mitm_proxy.rs` | Async MITM proxy (rustls + hyper): TLS termination, HTTP inspection, upstream bridging | | `crates/capsem-core/src/net/cert_authority.rs` | CA loader + bounded single-flight domain certificate cache | | `crates/capsem-core/src/security_engine/` | Rule/plugin/decision rail over `SecurityEvent` | | `crates/capsem-core/src/net/sni.rs` | SNI parser for TLS ClientHello | | `crates/capsem-agent/src/net_proxy.rs` | Guest-side TCP-to-vsock relay | ## Cross-process network limits `config/data/runtime-limits.toml` owns the supported concurrent network-client ceiling and its required multiples. `capsem-proto/build.rs` validates that file and generates the shared constants. DNS session capacity equals the client ceiling. HTTP descriptor-grant capacity multiplies it by `traffic_grants_per_client` because a pool can hold active and idle connections while a retiring connection is still closing. The process descriptor floor multiplies it by `file_descriptors_per_client` and must cover both ends of every overlapping grant. The ledger producer queue multiplies the client ceiling by `producer_events_per_network_client`; its admission batch must divide that queue exactly. `admission_coalesce_wait_microseconds` gives a lone audit event a bounded chance to join its immediate followers before the logger performs the commitment and admission IPC exchanges. Never add a numeric connection, ledger queue, batch or coalescing ceiling to the proxy, service broker, VM owner, logger, core DNS session, or guest DNS forwarder. A limit duplicated across those processes can make one stage admit work that the next stage sheds, which previously produced intermittent HTTP 502 and DNS SERVFAIL responses under load. The guest relay, VM owner, service broker and confined proxy must call `capsem_foundation::unix::process::ensure_open_file_limit` with the generated floor before accepting traffic; raising only one process leaves the same cliff at the next hop. Change the config-owned values, exercise every consumer through the generated constants, and run the official HTTPS and DNS load benchmarks twice before and twice after the change. Any traffic error or capacity refusal fails the run. User overrides belong in the typed settings contract tracked by issue #349. Extend and run `tests/citadel/test_runtime_limit_authority.py` whenever a new consumer or derived limit is added. The same file's `[cache]` table owns DNS answer count and TTLs plus the TLS leaf count. Use its generated `capsem_proto::CACHE_*` constants; do not place a second cache bound in the proxy or DNS modules. Use `just bench mitm-scale` when a one-VM curve plateaus while the host still looks idle. The lane uses fresh 1/2/4/8-VM cohorts, keeps per-VM work fixed and records the mock upstream outside the Capsem process tree. Compare the curve to physical cores, not only logical CPUs: after the VM owners, confined proxies and ledgers consume the physical-core budget, SMT contention is expected to raise per-VM latency. A busy fixture, request error or hidden descriptor refusal invalidates the run; do not explain it as product saturation. ## Content-Encoding: the systemic rule The proxy MUST handle response decompression as a general capability. This is not optional, not per-feature. 1. Normalize `Accept-Encoding` in outgoing requests to only allow encodings we can decompress (gzip at minimum) 2. Transparently decompress response bodies before any parsing (SSE, body preview, telemetry) 3. Never strip encoding headers as a workaround -- that breaks upstream behavior **Why this matters**: Failing to handle gzip on Anthropic SSE responses caused all model/token/cost metadata to be NULL. The SSE parser received compressed garbage. This went undetected because Google's API happened to not compress SSE in testing. The fix was general-purpose decompression, not an Anthropic-specific hack. ## Serde optimization for ai_traffic parsers The ai_traffic parsers (`openai.rs`, `google.rs`, `request_parser.rs`) deserialize LLM request/response bodies that can be megabytes. Never use `serde_json::Value` for struct fields that hold large unconstrained JSON (tool call args, function responses, model outputs). Use `Box<serde_json::value::RawValue>` for fields that are only stringified, and remove unused fields entirely. See `/dev-rust-patterns` for the full pattern and examples. ## SSE parsing AI provider APIs (Anthropic, OpenAI, Google) use Server-Sent Events for streaming responses. The proxy parses SSE to extract model names, token counts, and cost data for telemetry. SSE parsing happens AFTER decompression. The body must be plaintext UTF-8 by the time the SSE parser sees it. ## model_calls filtering Only emit `model_calls` telemetry for actual LLM API paths (e.g., `/v1/messages`, `/v1/chat/completions`), not every request to an AI provider domain. Health checks, auth endpoints, and static assets should not create model_call rows. ## Security evaluation order 1. Network mechanics parse and normalize SNI, HTTP, DNS, model, and process facts into a `SecurityEvent`. 2. Built-in default, settings, and corp rules compile into one `SecurityRuleSet`; built-in defaults are normal late-priority rules. 3. Security plugins run by stage over the same `SecurityEvent` object: `preprocess`, rule evaluation, `postprocess`, then `logging` before ledger handoff. 4. Runtime materialization forwards allowed bytes upstream. Ledger materialization writes the sanitized/enriched event to `session.db`, logs, routes, and UI stats. ## Certificate authority - Static CA keypair: `crates/capsem-core/resources/ca/capsem-ca.key` + `crates/capsem-core/resources/ca/capsem-ca.crt` (ECDSA P-256), compiled in via `include_str!` from `net/cert_authority.rs` - Certs minted on-demand per domain, cached in `RwLock<HashMap>` - CA baked into guest rootfs via `update-ca-certificates` + certifi patch + env vars - No security value from the CA itself -- the guest is already fully sandboxed ## Provider wire format references Read these for the exact SSE format, request/response shapes, and telemetry extraction points: - `references/anthropic-wire.md` -- Anthropic Messages API (event-typed SSE, gzip gotcha) - `references/openai-wire.md` -- OpenAI Chat Completions + Responses API (data-only SSE, [DONE] sentinel) - `references/google-wire.md` -- Google Gemini (complete JSON per event, no tool call IDs, camelCase) ## Testing the proxy - Unit tests: `cargo test -p capsem-core net` (policy evaluation, SNI parsing, cert minting) - In-VM: `just exec "capsem-doctor -k network"` (TLS trust chain, port blocking, domain filtering) - Telemetry: `just exec "curl -s https://api.anthropic.com/"`, then inspect the typed timeline/security routes for the expected `net_events` evidence - Adversarial: test with blocked domains, overlapping wildcards, malformed SNI, huge request bodies