dev-mitm-proxy

# MITM Proxy The MITM proxy is the network engine's HTTPS interception boundary. It intercepts traffic from the air-gapped guest VM, normalizes it into typed facts, hands a `SecurityEvent` to the security engine, and preserves allowed runtime bytes for upstream. Treat it as a system, not a collection of hacks -- every capability must be general-purpose. ## Security boundary Network code parses transport bytes, routes traffic, and emits typed `SecurityEvent` facts. It must not broker credentials, create credential refs, run CEL/security decisions, or sanitize ledger projections. Those belong to the security engine plugin rail. Every security plugin has the same data contract: it receives a `SecurityEvent` and returns a `SecurityEvent`; the plugin stage only controls ordering (`preprocess`, `postprocess`, or `logging`). There are two materialization paths and they must never be collapsed: - **Runtime materialization** prepares bytes for the real upstream. It may resolve a broker ref back to a real credential because the protocol needs it. - **Ledger materialization** prepares the event stored in `session.db`, structured logs, route JSON, and UI stats. It must contain only broker refs, hashes, bounded previews, typed detections, and plugin execution evidence. No credential logic belongs in HTTP header formatters, DB readers, frontend transforms, debug harnesses, or route adapters. If a future change needs capture, injection, redaction, threat intel, or PII handling, implement it as a security plugin stage over `SecurityEvent -> SecurityEvent`. ## Pipeline ``` Guest curl -> iptables REDIRECT -> capsem-net-proxy (guest, port 10443) -> vsock port 5002 -> Host MITM proxy -> SNI parse -> network metadata capture -> TLS terminate (rustls, per-domain cert minted from Capsem CA) -> HTTP request parse (hyper) -> SecurityEvent -> SecurityRuleSet + plugin rail -> Forward to real upstream over TLS -> Record telemetry to session DB -> Stream response back to guest ``` ## Key source files | File | What | |------|------| | `crates/capsem-core/src/net/mitm_proxy.rs` | Async MITM proxy (rustls + hyper): TLS termination, HTTP inspection, upstream bridging | | `crates/capsem-core/src/net/cert_authority.rs` | CA loader + on-demand domain cert minting with RwLock cache | | `crates/capsem-core/src/security_engine/` | Rule/plugin/decision rail over `SecurityEvent` | | `crates/capsem-core/src/net/sni.rs` | SNI parser for TLS ClientHello | | `crates/capsem-agent/src/net_proxy.rs` | Guest-side TCP-to-vsock relay | ## Content-Encoding: the systemic rule The proxy MUST handle response decompression as a general capability. This is not optional, not per-feature. 1. Normalize `Accept-Encoding` in outgoing requests to only allow encodings we can decompress (gzip at minimum) 2. Transparently decompress response bodies before any parsing (SSE, body preview, telemetry) 3. Never strip encoding headers as a workaround -- that breaks upstream behavior **Why this matters**: Failing to handle gzip on Anthropic SSE responses caused all model/token/cost metadata to be NULL. The SSE parser received compressed garbage. This went undetected because Google's API happened to not compress SSE in testing. The fix was general-purpose decompression, not an Anthropic-specific hack. ## Serde optimization for ai_traffic parsers The ai_traffic parsers (`openai.rs`, `google.rs`, `request_parser.rs`) deserialize LLM request/response bodies that can be megabytes. Never use `serde_json::Value` for struct fields that hold large unconstrained JSON (tool call args, function responses, model outputs). Use `Box<serde_json::value::RawValue>` for fields that are only stringified, and remove unused fields entirely. See `/dev-rust-patterns` for the full pattern and examples. ## SSE parsing AI provider APIs (Anthropic, OpenAI, Google) use Server-Sent Events for streaming responses. The proxy parses SSE to extract model names, token counts, and cost data for telemetry. SSE parsing happens AFTER decompression. The body must be plaintext UTF-8 by the time the SSE parser sees it. ## model_calls filtering Only emit `model_calls` telemetry for actual LLM API paths (e.g., `/v1/messages`, `/v1/chat/completions`), not every request to an AI provider domain. Health checks, auth endpoints, and static assets should not create model_call rows. ## Security evaluation order 1. Network mechanics parse and normalize SNI, HTTP, DNS, model, and process facts into a `SecurityEvent`. 2. Profile and corp rules compile into one `SecurityRuleSet`; profile defaults are normal late-priority rules. 3. Security plugins run by stage over the same `SecurityEvent` object: `preprocess`, rule evaluation, `postprocess`, then `logging` before ledger handoff. 4. Runtime materialization forwards allowed bytes upstream. Ledger materialization writes the sanitized/enriched event to `session.db`, logs, routes, and UI stats. ## Certificate authority - Static CA keypair: `crates/capsem-core/resources/ca/capsem-ca.key` + `crates/capsem-core/resources/ca/capsem-ca.crt` (ECDSA P-256), compiled in via `include_str!` from `net/cert_authority.rs` - Certs minted on-demand per domain, cached in `RwLock<HashMap>` - CA baked into guest rootfs via `update-ca-certificates` + certifi patch + env vars - No security value from the CA itself -- the guest is already fully sandboxed ## Provider wire format references Read these for the exact SSE format, request/response shapes, and telemetry extraction points: - `references/anthropic-wire.md` -- Anthropic Messages API (event-typed SSE, gzip gotcha) - `references/openai-wire.md` -- OpenAI Chat Completions + Responses API (data-only SSE, [DONE] sentinel) - `references/google-wire.md` -- Google Gemini (complete JSON per event, no tool call IDs, camelCase) ## Testing the proxy - Unit tests: `cargo test -p capsem-core net` (policy evaluation, SNI parsing, cert minting) - In-VM: `just exec "capsem-doctor -k network"` (TLS trust chain, port blocking, domain filtering) - Telemetry: `just exec "curl -s https://api.anthropic.com/"` then `python3 build_system/scripts/doctor/check_session.py` (check net_events) - Adversarial: test with blocked domains, overlapping wildcards, malformed SNI, huge request bodies