smtp-via-proxy

name	smtp-via-proxy
description	Send email via Gmail SMTP and read email via Gmail IMAP through HTTP CONNECT proxy (xray/v2rayN)
version	1.2.0
category	devops
author	Hermes Agent

Email (SMTP + IMAP) via HTTP CONNECT Proxy

Access Gmail SMTP and IMAP through an HTTP CONNECT proxy when direct connections time out (e.g., behind the Great Firewall).

Problem

• Direct TCP connections to Google services (smtp.gmail.com:587, imap.gmail.com:993) time out
• himalaya CLI has no built-in proxy support
• proxychains-ng does not work on SIP-protected macOS binaries
• HTTP_PROXY env var only works for HTTP, not raw sockets
• HTTP CONNECT proxies only support implicit TLS (465/993), not STARTTLS (587)

Solution Overview

All traffic routes through a local HTTP CONNECT proxy (127.0.0.1:10808, xray/v2rayN):

• SMTP: Manual socket → HTTP CONNECT → SSL wrap, implemented in send-mail.py
• IMAP: Monkey-patch socket.create_connection so imaplib.IMAP4_SSL transparently uses the proxy

Setup

1. Proxy must be running on 127.0.0.1:10808 (xray)
2. App Password stored in macOS Keychain as himalaya-smtp (for SMTP) and himalaya-imap (for IMAP)
3. Keychain lookup: security find-generic-password -a [e-mail] -s himalaya-smtp -w

SMTP Sending

Use port 465 (implicit TLS) instead of 587 (STARTTLS). Port 465 works because the TLS handshake happens immediately without a prior cleartext STARTTLS command.

Scripts

• send-mail.py: Full SMTP conversation — TCP connect to proxy, HTTP CONNECT to smtp.gmail.com:465, SSL wrap, EHLO/AUTH/MAIL/DATA/QUIT
• send-report.py: Thin cron wrapper — pipes stdin as email body, calls send-mail.py
• send-report-587.py (in scripts/): P19 fallback — port 587 + STARTTLS via stdlib smtplib with https_proxy env var. Reuses the same keychain entry. Invoke directly when the 465 path hangs.

Usage

# Send email with body from file
python3 ~/wiki/scripts/send-mail.py --to [e-mail] --subject 'Subject' < body.txt

# Cron-compatible wrapper
python3 ~/wiki/scripts/send-report.py < report.txt

SMTP Architecture

1. socket.connect(('127.0.0.1', 10808))
2. Send wire-format: b'CONNECT smtp.gmail.com:465 HTTP/1.1\r\nHost: smtp.gmail.com:465\r\n\r\n'
3. Read HTTP/1.1 200 Connection established
4. Wrap with ssl.wrap_socket(sock)
5. SMTP conversation: EHLO, AUTH LOGIN, MAIL FROM, RCPT TO, DATA (\r\n.\r\n), QUIT

SMTP Key Lessons

Issue	Fix
Port 587 STARTTLS fails through proxy	Use port 465 (implicit TLS) instead
SMTP end-of-data marker	Must use \r\n.\r\n (CRLF-dot-CRLF)
Body newlines need conversion	Convert \n to \r\n before sending
himalaya cannot use this approach	Uses its own SMTP library without proxy hooks
recv timeout with byte-by-byte reads	Use buffered reads (recv 1024 at a time)

IMAP Reading

Monkey-patch socket.create_connection to route through the proxy before creating an imaplib.IMAP4_SSL instance. This transparently makes imaplib.IMAP4_SSL use the proxy without modifying imaplib's code.

Script Location

~/wiki/scripts/newsletter-tldr-extractor.py contains the IMAP proxy solution.

Key Function

import socket

def proxy_connect(host, port, proxy_host='127.0.0.1', proxy_port=10808, timeout=30):
    """Connect to host:port via HTTP CONNECT proxy. Returns raw socket (not SSL-wrapped)."""
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.settimeout(timeout)
    sock.connect((proxy_host, proxy_port))
    connect_req = f'CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}:{port}\r\n\r\n'
    sock.sendall(connect_req.encode())
    response = sock.recv(4096)
    if b'200' not in response:
        raise ConnectionError(f'Proxy CONNECT failed: {response[:200]}')
    # Return WITHOUT SSL wrapping — caller (IMAP4_SSL) handles that
    return sock

# Monkey-patch: substitute create_connection so IMAP4_SSL transparently proxies
orig_create_connection = socket.create_connection
def patched_create_connection(address, **kwargs):
    host, port = address
    return proxy_connect(host, port)
socket.create_connection = patched_create_connection

# Now IMAP4_SSL uses the proxy transparently
import imaplib
M = imaplib.IMAP4_SSL('imap.gmail.com', 993)

IMAP Key Details

• proxy_connect() returns the raw socket before SSL wrapping — IMAP4_SSL handles the TLS handshake itself
• Monkey-patch must happen before creating the IMAP4_SSL instance
• Port: 993 (implicit TLS)
• After connecting, M.login(), M.select(), M.search(), M.fetch() all work normally
• IMAP \\Deleted flagged messages do NOT appear in search results (even if not yet expunged)

Verifying Connectivity

import socket

s = socket.socket()
s.settimeout(10)
s.connect(('127.0.0.1', 10808))
s.sendall(b'CONNECT smtp.gmail.com:465 HTTP/1.1\r\nHost: smtp.gmail.com:465\r\n\r\n')
print(s.recv(4096))

Expected: b'HTTP/1.1 200 Connection established'

Cron Integration

Update wiki-inbox-scan cron job prompt to use send-report.py and IMAP scripts that incorporate the monkey-patch.

Fallback: STARTTLS via Environment Variables

When port 465 (implicit TLS) times out during SSL handshake through HTTP CONNECT proxy, try port 587 with environment variables:

import smtplib
import ssl
import subprocess
from email.mime.text import MIMEText

# Set proxy env vars BEFORE creating SMTP connection
import os
os.environ['http_proxy'] = 'http://127.0.0.1:10808'
os.environ['https_proxy'] = 'http://127.0.0.1:10808'

# Get password from keychain
result = subprocess.run(
    ['security', 'find-generic-password', '-a', '[e-mail]', '-s', 'himalaya-smtp', '-w'],
    capture_output=True, text=True
)
password = result.stdout.strip()

body = open('report.md').read()
msg = MIMEText(body, 'plain', 'utf-8')
msg['Subject'] = 'Subject'
msg['From'] = '[e-mail]'
msg['To'] = '[e-mail]'

context = ssl.create_default_context()
with smtplib.SMTP('smtp.gmail.com', 587, timeout=30) as server:
    server.ehlo()
    server.starttls(context=context)
    server.ehlo()
    server.login('[e-mail]', password)
    server.send_message(msg)

Key Common Pitfalls

• HTTP CONNECT proxy SSL timeout on port 465: If port 465 fails with SSL handshake timeout, try port 587 with http_proxy/https_proxy environment variables as shown above.
• Monkey-patch scope: socket.create_connection is used by many libraries; be specific about the patch lifetime (or restore after done) to avoid side effects.
• App Passwords: Gmail requires App Passwords (not regular password) when 2FA is enabled. Store in macOS Keychain, not in scripts.
• Both scripts are Python stdlib only: socket, ssl, base64, subprocess, imaplib. No pip packages needed.

P19: Port 465 implicit-TLS handshake hang on a CONNECT-200 tunnel is TRANSIENT — retry-with-backoff before falling back to 587

症状： 调 python3 ~/wiki/scripts/send-report.py < report.md 跑 send-mail.py 时：

[DEBUG] Sending CONNECT request...
[DEBUG] Proxy response: HTTP/1.1 200 Connection established
[DEBUG] TCP tunnel to smtp.gmail.com:465 established
ssl.SSLError: The handshake operation timed out
  at send-mail.py:112 — context.wrap_socket(sock, server_hostname=SMTP_HOST)

Cron context (wiki-inbox-scan-v2 等): 看到上面这个 stderr 模式，第一步就自动 invoke ~/wiki/scripts/send-report-587.py，不要先 retry 5 次再 fallback —— 587 脚本无副作用（同读 ~/wiki/cron-report.md、同发 [e-mail]、同主题），多一次 retry 至少 75s。详见 wiki-auto-pipeline-recipes SKILL.md v1.10 pitfalls 表的 P19 fallback 链说明。 HTTP CONNECT 自身成功（200 Connection established），隧道建立，但在已建立的隧道之上跑 TLS 握手时挂死。这是间歇性代理/出口网络问题——不是代码缺陷。

真实案例 (2026-06-07 10:19 CST)：

• wiki-inbox-scan-v2 cron 触发，4 次重试 × 5–30s 退避后全部复现同一错误
• 同一脚本路径在 09:58 CST 一次成功（cron-status.log:1203）
• 根因不是脚本——send-mail.py:91-112 的逻辑和凌晨 02:00 跑的成功版本逐字节相同

根因（HTTP CONNECT 代理 + implicit TLS 的特性）：

1. 代理的 CONNECT 方法在 TCP 层"开门"成功，不代表它能让 TLS 流量通过
2. 某些 xray/sing-box/v2rayN 节点会对出站 TLS 做选择性阻断（如按 SNI、按 cipher、按客户端指纹），可能在 CONNECT 后插一个"假活"隧道，对 ClientHello 静默丢弃
3. 重连到同一节点可能命中不同 routing 规则，所以重试确实可能成功（不是稳态失败）

解决（按顺序执行，每步有明确判据）：

1. 第一步：retry-with-backoff (3-5 次) — 同一脚本，同一参数：

   for i in 1 2 3 4 5; do
     python3 ~/wiki/scripts/send-report.py < ~/wiki/cron-report.md && break
     echo "[retry $i/5] sleep $((i*15))s then retry..."
     sleep $((i*15))
   done
   

   退避序列：15s → 30s → 45s → 60s → 75s（总耗时最多 ~3.5 分钟）。5 次都失败再进入 fallback。

   实际观察 (2026-06-14 10:10 CST): 3 次重试 × 20-40s 退避（for i in 1 2 3; sleep $((i*20))，总 ~1 分 40 秒）就足以判断是稳态失败而非瞬时抖动。如果 3 次以内都失败，直接进 fallback，不必等满 5 次——同一节点、同一路径的失败模式不会因为第 4、第 5 次而改变，反而浪费 ~2 分钟 cron 时间窗。

2. 第二步：port 587 STARTTLS fallback — 用 stdlib smtplib + 走 http_proxy/https_proxy 环境变量，让代理自己处理协议协商（不再需要应用层 CONNECT）。直接调用 ~/wiki/scripts/send-report-587.py（已签入 skill scripts/ 目录），它会读 ~/wiki/cron-report.md 并发送到 [e-mail]。如果需要自定义主题/收件人，下面的内联版本可直接修改：

   import os, smtplib, ssl, subprocess
   from email.mime.text import MIMEText
   
   os.environ['http_proxy']  = 'http://127.0.0.1:10808'
   os.environ['https_proxy'] = 'http://127.0.0.1:10808'
   
   password = subprocess.run(
       ['security', 'find-generic-password', '-a', '[e-mail]',
        '-s', 'himalaya-smtp', '-w'], capture_output=True, text=True
   ).stdout.strip()
   
   body = open(os.path.expanduser('~/wiki/cron-report.md')).read()
   msg = MIMEText(body, , )
   msg[] = 
   msg[]    = 
   msg[]      = 
   
   ctx = ssl.create_default_context()
    smtplib.SMTP(, , timeout=)  s:
       s.ehlo(); s.starttls(context=ctx); s.ehlo()
       s.login(, password)
       s.send_message(msg)
   

经验法则： P19 失败 = "CONNECT 200 + TLS hang" 的精确模式。如果是其他 smtp 错误（Authentication failed、530 Must issue STARTTLS、421 Service not available 等），不是 P19，是别的——查 SMTP RFC 5321 + Gmail SMTP 错误码。

预防性检查清单：

• send-report.py 失败的 stderr 第一个 200 + 第二个 ssl.SSLError 同时出现？→ P19
• 是否已经 retry 3-5 次（每次之间 ≥15s）？→ 不是再 fallback
• fallback 587 用了 smtplib + https_proxy env？→ 不是手写 socket
• 仍失败后，cron-status.log 是否包含完整 stderr？→ 是才能算"已诚实记录"
• final response 是否明确告诉用户"邮件未发送"？→ 是，避免静默丢消息