tunnel-doctor

# Tunnel Doctor Diagnose and fix conflicts when Tailscale coexists with proxy/VPN tools on macOS, with specific guidance for SSH access to WSL instances. > **Methodology base:** the general diagnostic discipline this skill builds on — evidence over assumption, falsification over confirmation, layered isolation, counter-review — lives in the **debugging-network-issues** skill. This skill is the macOS Tailscale⨯proxy *domain* layer on top of it; reach for the base skill when the symptom is *not* a known Tailscale/proxy conflict. **Ownership boundary:** this skill owns concrete local network-path operations: Tailscale, route/DNS/proxy interception, VM/WSL propagation, TUN forwarding planes, and proxy node/exit/chain capacity. Use `debugging-network-issues` as the general method when the boundary is still unknown or the symptom belongs to SSE/CDN/application protocol behavior. This keeps one concrete operator instead of forcing the user to choose between two overlapping proxy doctors. ## Target and task scope Before remote probes or changes, establish whether the target is a designated test machine, an explicitly requested colleague's working computer, or unknown. Use the owner's existing machine-purpose ledger and current task; do not log into a colleague's machine to classify it. Saved SSH access and previous successful tests do not grant ongoing use. Use designated test machines for routine testing; their unavailability does not authorize another person's computer. Keep an explicitly requested one-off test on a colleague's computer within its stated result and necessary verification. Stop there on completion, cancellation, or a need for an unapproved environment change. Move later performance, migration, and regression tests to designated test machines. A test failure does not authorize disabling that person's proxy/VPN, quitting apps, changing routes, or installing a background task. Proceed with disruptive repair there only when the user has explicitly requested that repair and its interruption. Existing authorization for owner test-machine maintenance remains valid; do not ask again for its routine steps. Apply this boundary even when invoked directly, through another Skill, after compaction, or through a peer handoff. After access is stopped, use existing records rather than another SSH probe to verify that it stopped. ## Disruptive network changes Before any authorized VPN disconnect, app termination, route cutover, or service restart, read [network_change_recovery.md](references/network_change_recovery.md). Validate prerequisites before interruption, arm bounded recovery before the first disruptive action, and verify both the repaired path and the user's original network use. A successful stop command, a detached process, or a restart placed at the end of a script is not recovery evidence. ## Conflict Layers Proxy/VPN tools on macOS create conflicts at several independent layers. Layers 1-3 affect Tailscale connectivity; Layer 4 affects SSH git operations; Layer 5 affects VM/container runtimes. TUN-state failure modes beyond this table — SSH/git connection drops, resolver stall, DIRECT split-brain — are covered in Steps 2H–2J: | Layer | What breaks | What still works | Root cause | |-------|-------------|------------------|------------| | 1. Route table | Everything (SSH, curl, browser) | `tailscale ping` | `tun-excluded-routes` adds `en0` route overriding Tailscale utun | | 2. HTTP env vars | `curl`, Python requests, Node.js fetch | SSH, browser | `http_proxy` set without `NO_PROXY` for Tailscale | | 3. System proxy (browser) | Browser only (HTTP 503) | SSH, `curl` (both with/without proxy) | Browser uses VPN system proxy; DIRECT rule routes via Wi-Fi, not Tailscale utun | | 4. SSH ProxyCommand double tunnel | `git push/pull` (intermittent) | `ssh -T` (small data) | `connect -H` creates HTTP CONNECT tunnel redundant with Shadowrocket TUN; landing proxy drops large/long-lived transfers | | 5. VM/Container proxy propagation | `docker pull`, `docker build` | Host `curl`, running containers | VM runtime (OrbStack/Docker Desktop) auto-injects or caches proxy config; removing proxy makes it worse (VM traffic via TUN → TLS timeout) | ## Diagnostic Workflow ### Step 1: Identify the Symptom Name the actual consuming command or service and its required protocols before choosing a repair. When an authorized directly reachable peer can reach the slow target, and authenticated SSH plus a representative transfer through that peer already work better, treat the gateway as an integration candidate. First inspect the consumer's effective SSH/inventory arguments and test whether its normal entry can use that path; a successful handwritten `ssh -J` does not prove integration. If the gateway carries the required workload, connect it through the existing managed transport and replay the original operation without per-request jump arguments. Verify target identity, host-key trust, content integrity, elapsed time and the declared failure/recovery behavior. Do this before further work on a native relay that is not required by the user's result. Keep exact hosts, ports and live route state in the owning infrastructure configuration. Keep unmet protocols explicit: an SSH gateway does not establish UDP or arbitrary application connectivity. Continue the relevant network repair when those are required; do not replace that acceptance with SSH success. The executing agent must judge protocol fit and user intent; no bundled checker establishes either. Determine which scenario applies: - **Public gateway returns 502 while its backend responds normally, or the dedicated SSH tunnel key is rejected as `correct key but not from a permitted host`** → read [SSH tunnel source restrictions](references/ssh_tunnel_source_restriction.md) before changing TUN/firewall rules or treating a running process as recovery. - **Browser returns HTTP 503, but `curl` and SSH both work** → System proxy bypass conflict (Step 2C) - **`local.<domain>` fails in browser/default `curl`, but direct/no-proxy request works** → Local vanity domain proxy interception (Step 2C-1) - **Tailscale ping works, SSH works, but curl/HTTP times out** → HTTP proxy env var conflict (Step 2A) - **Tailscale ping works, SSH/TCP times out** → Check route ownership (Step 2B); a matching symptom alone does not prove a route conflict. - **Tailscale IPv4 SSH times out, or an alternate peer address triggers host-key verification** → Read [same-peer IPv4/IPv6 SSH recovery](references/tailscale_address_family_ssh.md) before declaring the host offline or changing trust/network state. - **Remote dev server auth redirects to `localhost` → browser can't follow** → SSH tunnel needed (Step 2D) - **`make status` / scripts curl to localhost fail with proxy** → localhost proxy interception (Step 2E) - **`git push/pull` fails with `FATAL: failed to begin relaying via HTTP`** → SSH double tunnel (Step 2F) - **`docker build` `RUN apk/apt` fails with `Connection refused` instantly** → OrbStack transparent proxy + TUN conflict (Step 2G-1, fix: `--network host`) - **`docker pull` fails with `TLS handshake timeout`** → VM proxy misconfiguration (Step 2G-2, fix: `docker.json` with `host.internal`) - **Container healthcheck `(unhealthy)` but app runs fine** → Lowercase proxy env var leak (Step 2G-4, fix: clear `http_proxy`+`HTTP_PROXY`) - **`docker build` can't fetch base images** → VM/container proxy propagation (Step 2G) - **`docker pull` works but `docker build` still dies fetching base-image tokens (TLS/i-o timeout) on a WSL2 + Docker Desktop host, or every runbook proxy port on a Windows+v2rayN host silently died after a v2rayN upgrade** → buildkit direct-dials past the DD proxy override / v2rayN ≥7.17 LAN-port split (references/windows_host_tun_wsl_cascade.md §2026-08-23) - **`git clone` fails with `Connection closed by 198.18.x.x`** → two different mechanisms produce this; Step 2H separates them - **Every domestic/DIRECT-rule site fails at once (TLS `unexpected EOF` mid-handshake, proxy-port CONNECT returns 503, Node CLIs report `UNKNOWN_CERTIFICATE_VERIFICATION_ERROR`) while proxied overseas sites keep working** → TUN DIRECT split-brain (Step 2J) - **One hostname fails after a DNS/delegation move while other sites work** → read [single-host DNS migration](references/single-host-dns-migration.md) before diagnosing a broken forwarding plane or restarting a shared tunnel. - **Everything on macOS fails at once — domestic AND overseas, and even Step 2J's real-IP probe fails** → do not declare an outage yet; run the physical-interface discriminator in "TUN full-stall vs genuine outage" (above Step 2A) - **SSH connects but `operation not permitted`** → Tailscale SSH config issue (Step 4) - **SSH connects but `be-child ssh` exits code 1** → WSL snap sandbox issue (Step 5) - **TCP port 22 reachable (`nc -z` succeeds) but SSH fails with `kex_exchange_identification: Connection closed`** → Tailscale SSH proxy intercept on WSL (Step 5A) - **Same `kex_exchange_identification` / `Connection closed by UNKNOWN port 65535`, but there is no Tailscale — you are behind a TUN and the host is a public cloud VM whose HTTPS still works** → could be your TUN *or* the destination's firewall allowlist; they are indistinguishable from the client. Step 2K settles it out-of-band - **Every node of one proxy provider stalls identically (TCP connects, TLS `appconnect=0.000`), and restarting the local proxy app + packet-tunnel extension changes nothing** → provider-level event, not your tunnel; confirm with the provider's control plane and switch to a *different* provider's node (Step 2L) - **Every tested node of one proxy provider fails fast with TLS `unrecognized name`, while subscription refresh still works** → check the provider's account dashboard for traffic limits, billing or suspension first; the symptom does not establish the cause. Read [provider account checks](references/provider_account_suspension.md) before rebuilding tunnels or restarting proxy processes. - **`tailscale ssh` returns "not available on App Store builds"** → Wrong Tailscale distribution on macOS (Step 5B) - **Any tool using system DNS (`ssh`, `curl`, `git`) hangs ~60s before resolving, but `nslookup` returns instantly** → Stalled resolver in `getaddrinfo` chain (Step 2I) - **Windows+WSL host: everything offline at once (domestic AND overseas), WSL dead too, "even Tailscale won't come online" — and/or the user tried several recovery actions (switched NICs, toggled TUN) and can't tell which one fixed it** → Windows host TUN cascade + event-log forensics (Step 5C) - **Tailscale, DNS, route ownership, and proxy reachability all pass, but large transfers remain slow and switching the active proxy node changes throughput** → proxy node / exit / chain capacity workflow in [references/proxy_node_chain_throughput.md](references/proxy_node_chain_throughput.md) - **You want one app's bulk traffic (BaiduNetdisk downloads, Feishu/Lark uploads) off the metered proxy — or out of the status-bar counter entirely** → not a conflict; a splitting task. Use the existing local traffic-splitting workflow if one is installed and verified; this skill does not bundle a splitter, target list, replay helper or subscription-wipe guard. For mechanism diagnosis, read [references/proxy_conflict_reference.md](references/proxy_conflict_reference.md) § "Per-app traffic splitting on macOS" — which of the three mechanisms (DIRECT rule / `tun-excluded-routes` / `always-real-ip`) can work for the app depends on whether it connects to literal IPs or the system resolver, and the DNS-layer keys are inert on the macOS Catalyst build. **Key distinctions**: - SSH does NOT use `http_proxy`/`NO_PROXY` env vars. If SSH works but HTTP doesn't → Layer 2. - `curl` uses `http_proxy` env var, NOT the system proxy. Browser uses system proxy (set by VPN). If `curl` works but browser doesn't → Layer 3. - If `tailscale ping` works but regular `ping` doesn't → Layer 1 (route table corrupted). - If `ssh -T git@github.com` works but `git push` fails intermittently → Layer 4 (double tunnel). - If host `curl https://...` works but `docker pull` times out → Layer 5 (VM proxy propagation). - If `docker pull` works but `docker build` `RUN apk add` fails instantly with `Connection refused` → OrbStack transparent proxy broken by TUN (Step 2G-1). - If container healthcheck shows `(unhealthy)` but app works → lowercase `http_proxy` leaked into container (Step 2G-4). - If DNS resolves to `198.18.x.x` virtual IPs → a TUN is active, which is **not** by itself a diagnosis: under TUN every hostname resolves this way, on success as well as failure. Go to Step 2H to find out which mechanism you have. - If fake-IP connections fail but a fresh-IP `curl --resolve` succeeds, hostname-dependent resolution/forwarding is a suspect, not a proven cause. Use Step 2J for the multi-domain DIRECT failure pattern; for one recently moved domain, use [single-host DNS migration](references/single-host-dns-migration.md). - If `nc -z` succeeds on port 22 but SSH gets no banner (`kex_exchange_identification`) → Tailscale SSH proxy intercept (Step 5A). Confirm with `tcpdump -i any port 22` on the remote — 0 packets means Tailscale intercepts above the kernel. - If `nc -z` also succeeds on a port the destination does **not** serve (probe 65001) → under a TUN the local stack answers for the destination, so **no `nc -z` result on that host means anything**. Never read a port probe you have not calibrated this way. This supersedes the plain `nc -z` reading in the bullet above whenever a TUN — not just Tailscale — is in the path (Step 2K). - If the destination's own `journalctl -u ssh` shows **zero** entries for your attempts → something before `sshd` is eating them, but **not necessarily the network**: a host-local packet filter drops them just as silently. `tcpdump` on the host separates "never arrived" from "arrived and was dropped locally"; only the first implicates your TUN. Reach the machine through the cloud provider's guest-agent channel to ask, since that path does not use the port you cannot reach. Check the journal, not `auth.log`: on Ubuntu 24.04 `sshd` writes nothing to `auth.log`, so an empty file there is not evidence (Step 2K). - A cloud host enforces an inbound port **twice** — provider security group *and* host firewall (`ufw`/`firewalld`/`nftables`). Adding your IP to one leaves the other blocking, with no signal that you only did half the job (Step 2K). - If `tailscale ssh` fails with "not available on App Store builds" → install Standalone Tailscale (Step 5B). - If `nslookup <host>` is fast (<0.1s) but `dscacheutil -q host -a name <host>` takes 60s+ → a supplemental resolver in `scutil --dns` is dead (Step 2I). - If `ping <resolver-ip>` succeeds but `dig @<resolver-ip>` times out → daemon dead, `utun` interface zombied. ICMP is answered by the interface; the actual port-53 service is gone (Step 2I). - If `ssh -vvv` hangs immediately after `debug2: resolving "<host>" port <port>` and never reaches `debug1: connect to address` → DNS resolution stage, not network connect stage. This is Step 2I, not Step 2B/2H. ### Diagnosis Discipline (Read Before Committing to a Hypothesis) When symptoms point at a component (proxy, VPN, route table, DNS), **don't commit to a hypothesis from circumstantial evidence — verify with that component's own health endpoint first.** Each component has a one-line health check faster and more reliable than ruling out neighbors: | Suspected component | Authoritative health check (run this first) | |---------------------|---------------------------------------------| | HTTP proxy (Shadowrocket / Clash / Surge) | `curl -x http://127.0.0.1:<port> -m 10 https://api.github.com` returns 200 | | Tailscale daemon | `tailscale status` returns peer list (not connection error) | | A specific DNS resolver | `dig @<nameserver-ip> +tries=1 +timeout=3 example.com` <100ms | | Routing for an IP | `route -n get <ip>` shows expected interface | | Per-resolver bisection (when DNS is suspect) | The `for ns in ...; do dig @$ns ...` loop in Step 2I | | All nodes of one proxy provider dead but subscription refresh works | Provider account dashboard (suspended / traffic limit / billing) — before ANY local repair action | | An agent CLI complaining "can't connect / keeps reconnecting" (Codex etc.) | The client's own runtime log, bucketed by error shape and by hour — for Codex: `logs` table in the newest `~/.codex/logs_*.sqlite`. Errors clustered in time windows = episodic transport (keep reading here); a single failing `process_uuid` among clean ones = per-process cause (e.g. stale token), not the network at all | **Why this matters**: A symptom that matches the description of Step 2X does not, by itself, prove component X is the problem. Multiple layers can produce overlapping symptoms (a 60-second hang during `git push` could be proxy node death, fakeip route corruption, or DNS resolver stall — all plausible from the user-visible symptom alone). Reaching for the most specific verification first avoids committing to a wrong layer and chasing it down a dead end. If the failing operation involves DNS at all, **run the per-nameserver bisection from Step 2I before suspecting proxy or routing**. It rules in/out the largest single class of macOS-on-China-network failures in under 15 seconds. ### TUN Measurement Contamination (what your probes lie about while a TUN proxy is up) When a proxy tool runs in **TUN / global mode** (Shadowrocket, Clash, Surge), it intercepts traffic at the routing layer and fabricates parts of the network stack locally. Several everyday diagnostic commands then return **fabricated or misrouted numbers** — trusting them sends the whole investigation the wrong way. Know what each probe actually measures under TUN: | Probe | What it looks like | What it actually is under TUN | Trust? | |-------|-------------------|-------------------------------|--------| | `nc -z <node-ip> <port>` / raw TCP connect showing `0.00s` | "node reachable, instant" | TUN completes the TCP handshake **locally** before tunneling. `0.00s` to an overseas host is physically impossible (light alone is tens of ms each way) — you connected to the TUN, not the node. | ❌ | | `ping <host>` with near-zero loss / sub-ms RTT | "link healthy" | TUN can answer ICMP locally; loss and RTT are fabricated and uncorrelated with TCP. (Separately: ICMP ≠ TCP even with no TUN.) | ❌ | | `curl … -w '%{remote_ip}'` | "connected to peer X" | Always the local TUN endpoint (`127.0.0.1` / loopback), never the real remote peer. | ❌ | | IP-geo lookup via a **foreign** service (an `ip-api`-style endpoint) | "my egress / home IP is …" | A foreign-domain request gets routed **through the proxy**, so it reports the **exit IP**, not your real local/home IP. | ❌ for "what is my real local IP" | | IPv4-vs-IPv6 path choice, HTTP/3 / QUIC speedup | varies | TUN typically does not forward UDP/443, so QUIC never leaves. The comparison is meaningless. | ❌ | | A DIRECT-rule / domestic domain succeeding through your proxy config, cited as evidence the tunnel/exit node is alive | "the proxy has at least one working path, so the tunnel is probably fine" | DIRECT-matched traffic never enters the tunnel — it exits locally via your real last mile, identically to the proxy not running at all. Its success says nothing about the *proxied* plane. (Observed 2026-09-22: an overseas API endpoint that "worked through the proxy" during a collection-health check turned out to hit a DIRECT bypass rule; a no-proxy-at-all retest succeeded identically, proving the tunnel itself had never been exercised.) | ❌ for "is the tunnel/exit node healthy" | | `networkQuality` with no `-I` flag | "my real down/uplink capacity" | The default run rides whatever owns the default route — under TUN that is the **tunnel**, not the wire (verified 2026-10-23: the identical run moved 64.7 → 472.6 Mbps after a node switch). Good for "is my tunnel slow", wrong as a wire baseline. Whether it exercises the node at all depends on the tool's rules for the test endpoints; a number that moves when the node changes is proof it rode the tunnel. | ✅ for tunneled capacity / ❌ for physical baseline | | `curl ifconfig.me` / ipify exit IP used to name *which node* is active | "exit IP 1.2.3.4 means we're on node X" | The exit IP belongs to the *exit*, not the entry node: residential-IP products exit from ISP addresses that share nothing with the node address (measured 2026-10-23: a Comcast residential exit belonged to a "static residential" product, not to the datacenter node it entered through; full address withheld). Naming the active node from an exit IP misattributes success and failure to the wrong subscription. | ❌ for node attribution | **What you *can* trust under TUN:** - **`time_appconnect` / `time_starttransfer`** from `curl` (application-layer handshake / TTFB) — these complete only after the tunneled connection actually establishes, so they reflect the real end-to-end path. - **A capacity number from `networkQuality -s -I <physical-if>`** (`-I` = "Bind test to interface", per its own help) as the wire baseline — it bypasses the tunnel the same way curl's `--interface` does. Trust the capacity figures; do not compare the latency/responsiveness sub-metrics across binding modes — an interface-bound run has reported higher idle latency than the tunneled run on a healthy host. - **An in-region / domestic IP-geo source** for "what is my real local ISP" — an in-region domain hits the proxy's DIRECT rule and exits your real last mile (the foreign source gets tunneled and lies; see table). The same DIRECT-hit property is **not** evidence for the tunnel's own health — see the table row above; check `proxyBypassHosts`/DIRECT rule lists before crediting any single host's success to the tunnel.