redact-wire-capture

# Redact Wire Capture A wire capture is the highest-density leak surface in any investigation: a single `.jsonl` from a proxied session can carry the bearer token, the account email, the device hash, and the home path all in one request frame. This skill scrubs those in place with class-preserving substitutions — keeping enough of each secret's *shape* to stay analytically useful (`Bearer sk-vendor-oat01-<REDACTED>` still reads as "an OAuth bearer") — runs idempotently so re-redaction is a no-op, and finishes by asserting per file that no secret survived. ## When to Use - Exporting a MITM/proxy capture (`mitmproxy`, HAR, raw request/response logs) from a sanctioned probe into notes or a guide - Moving capture artifacts (`.jsonl`, `.json`, `.log`, `.stdout`) toward a public mirror - Sanitizing a wire dump before attaching it to a bug report or upstream disclosure - Re-running redaction on a capture dir that may already be partly scrubbed (must be safe to repeat) ## Inputs - **Required**: A capture directory containing text-mode artifacts - **Required**: The secret-class list (token prefixes, id formats, the personal identifiers to scrub) — kept private - **Optional**: An allow-list of public identifiers (marketplace/skill names, public usernames) that must be left intact - **Required**: `tools/redact-artifact.py` for the per-file scrub-and-assert step (there is no directory-wide gate; #853 records why) ## Procedure ### Step 1: Enumerate Secret Classes, Keep the Class Marker For each secret, decide what to strip and what to keep. The goal is *class-preserving* redaction: remove the secret, keep the prefix or shape that tells a reader what class it was, so the capture stays analytically legible. | Class | Synthetic shape | Redact to | Keep | |---|---|---|---| | OAuth bearer | `sk-vendor-oat01-<base64…>` | `sk-vendor-oat01-<REDACTED-BEARER>` | prefix (token class) | | API key | `sk-vendor-api03-<…>` | `sk-vendor-api-<REDACTED-KEY>` | prefix | | service id | `srv_<20+ alnum>` | `srv_<REDACTED>` | prefix | | UUID | `8-4-4-4-12 hex` | `<REDACTED-UUID>` | shape only | | email | `usr@example.com` | `<REDACTED-EMAIL>` | nothing | | home path | `/home/<user>/…` | `/home/<REDACTED-USER>/…` | path structure | | device/session hash | 64-hex under `"deviceId"` | `<REDACTED-DEVICE-HASH>` | the key, not the value | **Expected:** Every secret class has a from-shape and a to-form; the to-form preserves the class marker but not the secret. **On failure:** If a value's class is ambiguous (could be a public id or a session id), treat it as secret. A false redaction is recoverable from the private source; a false keep is a leak. ### Step 2: Apply Class-Preserving, Idempotent Substitutions In Place Scrub the text-mode files in place. Every substitution must be **idempotent** — the to-form must not re-match the from-shape, so a second run changes nothing. ```bash #!/usr/bin/env bash set -euo pipefail CAP="${1:?capture dir required}" mapfile -t FILES < <(find "$CAP" -type f \( -name '*.jsonl' -o -name '*.json' \ -o -name '*.log' -o -name '*.txt' -o -name '*.stdout' -o -name '*.stderr' \)) [ "${#FILES[@]}" -gt 0 ] || { echo "no text files under $CAP"; exit 0; } sed -i -E 's/sk-vendor-oat01-[A-Za-z0-9_-]+/sk-vendor-oat01-<REDACTED-BEARER>/g' "${FILES[@]}" sed -i -E 's/sk-vendor-api[0-9]+-[A-Za-z0-9_-]+/sk-vendor-api-<REDACTED-KEY>/g' "${FILES[@]}" sed -i -E 's/srv_[A-Za-z0-9]{20,}/srv_<REDACTED>/g' "${FILES[@]}" sed -i -E 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<REDACTED-UUID>/g' "${FILES[@]}" sed -i -E 's/[A-Za-z0-9._-]+@example\.com/<REDACTED-EMAIL>/g' "${FILES[@]}" sed -i -E 's#/home/[A-Za-z0-9_-]+/#/home/<REDACTED-USER>/#g' "${FILES[@]}" sed -i -E 's/"deviceId":"[0-9a-f]{64}"/"deviceId":"<REDACTED-DEVICE-HASH>"/g' "${FILES[@]}" ``` The `<REDACTED-…>` suffix replaces the variable part, so the pattern `[A-Za-z0-9_-]+` no longer matches `<REDACTED-BEARER>` on a second pass. That is what makes the script a safe no-op to re-run. **Expected:** After one pass, every secret is replaced by its class-preserving form; a second pass reports zero changes. **On failure:** If a second run keeps changing files, a substitution is non-idempotent (its to-form still matches its from-shape) — anchor or narrow it so the redacted form is inert. ### Step 3: Protect Public Identifiers Not everything that looks like an identifier is a secret. Public marketplace names, public skill names, and an intentionally-public username are part of the finding and must survive. Apply the allow-list *before* any broad catch-all so a generic rule does not eat a legitimate public name. **Expected:** Public identifiers on the allow-list appear unchanged in the redacted capture; only private values are scrubbed. **On failure:** If a public identifier was scrubbed, the catch-all ran too early or too broad — move the allow-list ahead of it and re-derive the redacted output from the private source. ### Step 4: Verify Through the Redaction Gate Re-run a verification pass over each scrubbed file. There is no tree-wide gate to hand the directory to, and #853 records why: a scanner asking "is this tree clean?" has no corpus to be right about, so the verification belongs inside the transform that produced the file. Redact each file through `redact-artifact`, which refuses to return output in which a listed term survives. ```bash # One `source<TAB>replacement` pair per secret you are scrubbing, plus --also-deny for any # encoding of it your mapping does not spell. The tool exits 1 if a term survives and 2 if it # cannot run at all, so a `|| exit` is a real gate rather than a hopeful one. for f in "$CAP"/*.json; do python3 tools/redact-artifact.py --type text --mapping "$CAP/../secrets.tsv" \ --also-deny "$BEARER_B64" "$f" --in-place || { echo "capture still leaks at $f; extend the secret-class list"; exit 1; } done ``` **Expected:** Every file exits 0. A non-zero exit is the gate working, and the two codes mean different things: **1** is a finding — a listed term survived the mapping. **2** is could-not-measure — an empty mapping, an unreadable or non-UTF-8 file, or markup whose structure tier examined nothing — and it must never be reported to an operator as a leak. The `||` branch below says "still leaks", so treat it as covering 1 only; on 2, fix the input and re-run. **On failure:** A surviving hit means a secret class is unhandled — add it to Step 1/Step 2, re-run the scrub from the private source, and re-verify. Never delete the offending line by hand; the next capture will reproduce it. ## Validation - [ ] Every secret class is replaced by a class-preserving form (prefix kept, secret gone) - [ ] The scrub is idempotent — a second run changes nothing - [ ] Public allow-list identifiers are intact - [ ] No UUID, token, email, home path, or device hash survives outside a `<REDACTED-…>` form - [ ] `tools/redact-artifact.py` exits 0 on every file. On the FIRST run over an unscrubbed capture the summary shows a non-zero matched count; on a re-run it shows `OUTPUT UNCHANGED`, which is the idempotence above rather than a failure - [ ] The redacted capture is reproducible from the private source (re-running yields the same result) ## Common Pitfalls - **Non-idempotent substitutions.** If the redacted form still matches the pattern, re-runs keep mutating and CI flaps. Make every to-form inert. - **Stripping the class marker.** Replacing a whole bearer with `<REDACTED>` loses the analytical signal that it *was* an OAuth bearer. Keep the prefix, drop the secret. - **Catch-all before allow-list.** A broad rule that runs first will scrub the public marketplace name you meant to keep. Allow-list first. - **Forgetting binary/secondary artifacts.** Captures often include `.har`, `.stdout`, and nested attachments. Enumerate every text-mode type, not just `.jsonl`. - **Hand-deleting leaking lines.** The capture is a function of the probe; editing the output by hand desyncs it from the source and the leak returns next run. - **Encoding ≠ redaction.** A base64'd token in a request body is still the token; the gate's structure tier must treat the decoded shape as a leak. ## Related Skills - `enforce-redaction-gate` — how to build a redaction boundary when you hold the deny-list; it teaches the two-tier design this skill's per-file assertion implements the decidable half of - `conduct-empirical-wire-capture` — produces the captures this skill scrubs; redaction is the mandatory step between capture and any public reference - `redact-for-public-disclosure` — the methodology umbrella governing what may be referenced publicly at all - `redact-visualization-for-disclosure` — the sibling transform for rendered diagrams rather than wire dumps - `security-audit-codebase` — overlapping secret-scanning techniques, oriented to live-credential leakage rather than capture sanitization