| name | smtp-via-proxy |
| description | Send email via Gmail SMTP and read email via Gmail IMAP through HTTP CONNECT proxy (xray/v2rayN) |
| version | 1.4.0 |
| category | devops |
| author | Hermes Agent |
Email (SMTP + IMAP) via HTTP CONNECT Proxy
Access Gmail SMTP and IMAP through an HTTP CONNECT proxy when direct connections time out (e.g., behind the Great Firewall).
Problem
- Direct TCP connections to Google services (
smtp.gmail.com:587, imap.gmail.com:993) time out
himalaya CLI has no built-in proxy config, but works via http_proxy/https_proxy env vars (confirmed 2026-06-18 — see P20)- proxychains-ng does not work on SIP-protected macOS binaries
- HTTP_PROXY env var only works for HTTP, not raw sockets
- HTTP CONNECT proxies only support implicit TLS (465/993), not STARTTLS (587)
Solution Overview
All traffic routes through a local HTTP CONNECT proxy (127.0.0.1:10808, xray/v2rayN):
- SMTP: Manual socket → HTTP CONNECT → SSL wrap, implemented in
send-mail.py
- IMAP: Monkey-patch
socket.create_connection so imaplib.IMAP4_SSL transparently uses the proxy
Setup
- Proxy must be running on
127.0.0.1:10808 (xray)
- App Password stored in macOS Keychain as
himalaya-smtp (for SMTP) and himalaya-imap (for IMAP)
- Keychain lookup:
security find-generic-password -a geekqjg@gmail.com -s himalaya-smtp -w
SMTP Sending
Use port 465 (implicit TLS) instead of 587 (STARTTLS). Port 465 works because the TLS handshake happens immediately without a prior cleartext STARTTLS command.
Scripts
send-mail.py: Full SMTP conversation — TCP connect to proxy, HTTP CONNECT to smtp.gmail.com:465, SSL wrap, EHLO/AUTH/MAIL/DATA/QUIT. Auto-detects HTML vs plain text (Content-Type based on body prefix). Fixed 2026-07-04: was hardcoded to text/plain — now auto-detects <!DOCTYPE/<html → text/html.send-html-mail.py (2026-07-04): HTML-only sender for Outlook-compatible HTML. Base64-encoded body, proper MIME construction, EHLO multi-line response handling. Location: ~/wiki/scripts/send-html-mail.py. Usage: python3 send-html-mail.py --to x@y.com --subject "Subject" file.htmlsend-report.py: Thin cron wrapper — pipes stdin as email body, calls send-mail.pysend-report-587.py (in scripts/): P19 fallback — port 587 + STARTTLS via stdlib smtplib with https_proxy env var. Reuses the same keychain entry. Invoke directly when the 465 path hangs.send-html-mail.py (added 2026-07-04): Full MIME-compliant HTML email sender. Uses base64 Content-Transfer-Encoding for safe Chinese character transmission. Includes proper multi-line EHLO response handling (send-mail.py's recv_line_TIMEOUT silent read silently drops continuation lines). Preferred for HTML email delivery.
Usage
python3 ~/wiki/scripts/send-report.py ~/wiki/cron-report.md
python3 ~/wiki/scripts/send-mail.py --to recipient@gmail.com --subject 'Subject' body.txt
cat ~/wiki/cron-report.md | python3 ~/wiki/scripts/send-report.py
python3 ~/wiki/scripts/send-report.py < report.txt
Key rule: Always pass the file path as a positional argument to send-report.py. The script supports it (line 14-16). Never use cat | python3 in cron mode.
SMTP Architecture
socket.connect(('127.0.0.1', 10808))- Send wire-format:
b'CONNECT smtp.gmail.com:465 HTTP/1.1\r\nHost: smtp.gmail.com:465\r\n\r\n'
- Read
HTTP/1.1 200 Connection established
- Wrap with
ssl.wrap_socket(sock)
- SMTP conversation: EHLO, AUTH LOGIN, MAIL FROM, RCPT TO, DATA (
\r\n.\r\n), QUIT
SMTP Key Lessons
| Issue | Fix |
|---|
| Port 587 STARTTLS fails through proxy | Use port 465 (implicit TLS) instead |
| SMTP end-of-data marker | Must use \r\n.\r\n (CRLF-dot-CRLF) |
| Body newlines need conversion | Convert \n to \r\n before sending |
himalaya cannot use this approach | Outdated (2026-06-18): himalaya template send works as a third fallback — see P20 |
| recv timeout with byte-by-byte reads | Use buffered reads (recv 1024 at a time) |
IMAP Reading
Monkey-patch socket.create_connection to route through the proxy before creating an imaplib.IMAP4_SSL instance. This transparently makes imaplib.IMAP4_SSL use the proxy without modifying imaplib's code.
Script Location
~/wiki/scripts/newsletter-tldr-extractor.py contains the IMAP proxy solution.
Key Function
import socket
def proxy_connect(host, port, proxy_host='127.0.0.1', proxy_port=10808, timeout=30):
"""Connect to host:port via HTTP CONNECT proxy. Returns raw socket (not SSL-wrapped)."""
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(timeout)
sock.connect((proxy_host, proxy_port))
connect_req = f'CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}:{port}\r\n\r\n'
sock.sendall(connect_req.encode())
response = sock.recv(4096)
if b'200' not in response:
raise ConnectionError(f'Proxy CONNECT failed: {response[:200]}')
return sock
orig_create_connection = socket.create_connection
def patched_create_connection(address, **kwargs):
host, port = address
return proxy_connect(host, port)
socket.create_connection = patched_create_connection
import imaplib
M = imaplib.IMAP4_SSL('imap.gmail.com', 993)
IMAP Key Details
proxy_connect() returns the raw socket before SSL wrapping — IMAP4_SSL handles the TLS handshake itself- Monkey-patch must happen before creating the
IMAP4_SSL instance
- Port: 993 (implicit TLS)
- After connecting,
M.login(), M.select(), M.search(), M.fetch() all work normally
- IMAP
\\Deleted flagged messages do NOT appear in search results (even if not yet expunged)
Verifying Connectivity
Run the diagnostic script to test all paths at once:
bash ~/.hermes/skills/devops/smtp-via-proxy/scripts/smtp-diagnostic.sh
Or test manually:
import socket
s = socket.socket()
s.settimeout(10)
s.connect(('127.0.0.1', 10808))
s.sendall(b'CONNECT smtp.gmail.com:465 HTTP/1.1\r\nHost: smtp.gmail.com:465\r\n\r\n')
print(s.recv(4096))
Expected: b'HTTP/1.1 200 Connection established'
Cron Integration
Update wiki-inbox-scan cron job prompt to use send-report.py and IMAP scripts that incorporate the monkey-patch.
Troubleshooting: SMTP Send Failures
P19: Port 465 implicit-TLS handshake hang
Symptom: CONNECT 200 success, then SSL handshake timeout.
Cause: Intermittent proxy/network issue — not code defect.
Fix: Try direct connection first (P21), then retry 3× with backoff, then fallback to port 587.
Confirmed (2026-06-30): SOCKS5 proxy also fails at the same SSL handshake step — the tunnel establishes (0500 auth, 050000017f0000012a38 connect), but ssl.wrap_socket() times out. This confirms the issue is TLS-level (not proxy-protocol-level). Increasing the socket timeout from 15s to 30s does NOT help — the handshake hangs indefinitely regardless of timeout.
Confirmed (2026-07-01): When proxy SSL handshake hangs on port 465, direct connection to smtp.gmail.com:465 works (TLSv1.3, full SMTP conversation succeeds). The proxy tunnel establishes but the TLS negotiation through it stalls — this is a proxy-side TLS forwarding issue, not a network-level SMTP block.
Diagnosis shortcut: When port 465 implicit TLS hangs through the proxy, skip retrying and go directly to direct smtplib (P21). The 3× retry with backoff wastes time on a confirmed-dead path.
P23: HTML email sent as text/plain — send-mail.py Content-Type bug (2026-07-04)
Symptom: You pipe HTML content to send-mail.py, the email sends successfully, but the recipient sees raw HTML source instead of rendered content.
Root cause: send-mail.py constructs the SMTP DATA payload with Content-Type: text/plain; charset=UTF-8 as a hardcoded string. Fixed 2026-07-04 to auto-detect <!DOCTYPE/<html prefix → set text/html.
Fix: Use send-html-mail.py instead (proper MIME: base64 Content-Transfer-Encoding, multi-line EHLO handling):
python3 ~/wiki/scripts/send-html-mail.py --to user@example.com --subject "Subject" file.html
send-html-mail.py was created 2026-07-04 specifically for Outlook-compatible HTML email delivery. It handles base64 encoding, multi-line EHLO responses, and Content-Type correctly.
P23: HTML email sent as text/plain — send-mail.py has hardcoded Content-Type
Symptom: You pipe HTML content to send-mail.py (or call it with send-report.py), the email sends successfully, but the recipient sees raw HTML source in their email client instead of rendered content.
Root cause: send-mail.py constructs the SMTP DATA payload with Content-Type: text/plain; charset=UTF-8 as a hardcoded string. It does NOT auto-detect HTML content. Fixed 2026-07-04 to auto-detect, but send-html-mail.py is the preferred path for HTML.
Fix: Use send-html-mail.py instead:
python3 ~/wiki/scripts/send-html-mail.py --to user@example.com --subject "Subject" /path/to/file.html
If you must use send-mail.py: Ensure the body starts with <!DOCTYPE or <html> (the auto-detection patch checks these prefixes). Verify with:
head -1 /tmp/report.html
Pitfall: Even with correct Content-Type: text/html, send-mail.py declares Content-Transfer-Encoding: quoted-printable for HTML content but does NOT actually QP-encode the body — the raw HTML (with Chinese characters, long lines, emoji) is sent as-is. Email clients see the QP declaration but receive non-QP content → rendering fails (raw HTML source shown, garbled text, or blank body).
Fix: Change to Content-Transfer-Encoding: 8bit (correct for raw UTF-8 HTML). QP requires quopri / base64 encoding of the body before sending. Unless you properly encode, 8bit is the only safe choice:
cte = 'quoted-printable' if content_type == 'text/html' else '8bit'
cte = '8bit'
Detection: The email arrives in the inbox but the HTML is not rendered — you see raw <html><body>...</body></html> tags instead of the rendered page, or the email appears blank. Check the email source for Content-Transfer-Encoding: quoted-printable followed by non-QP-encoded content (no = sign encoding, raw Chinese text).
Verified 2026-07-24: send-mail.py line 147 had cte = 'quoted-printable' if content_type == 'text/html' else '8bit'. Fixed to always use 8bit. All subsequent HTML emails rendered correctly.
Symptom: TLS + EHLO success, AUTH LOGIN hangs.
Fix: Same fallback chain as P19.
P21: Proxy Completely Down (2026-06-26)
Diagnosis — ALWAYS check first before retrying proxy paths:
nc -z -w 3 127.0.0.1 10808 2>&1 && echo "PROXY UP" || echo "PROXY DOWN"
If PROXY DOWN:
- Skip the entire P19/P20 retry-then-fallback chain — all proxy-dependent paths will timeout (including
send-report.py, send-report-587.py with proxy env vars, and himalaya if http_proxy/https_proxy are set)
- Use direct
smtplib (no proxy env vars) — Gmail ports 587/465 are directly reachable on this host even when the proxy is down
import smtplib, subprocess
from email.mime.text import MIMEText
password = subprocess.run(
['security', 'find-generic-password', '-a', 'geekqjg@gmail.com',
'-s', 'himalaya-smtp', '-w'], capture_output=True, text=True
).stdout.strip()
body = open('/Users/jinguo/wiki/cron-report.md').read()
msg = MIMEText(body, 'plain', 'utf-8')
msg['Subject'] = '[Hermes] Wiki 状态报告'
msg['From'] = 'jinguo <geekqjg@gmail.com>'
msg['To'] = 'geekqjg@gmail.com'
with smtplib.SMTP('smtp.gmail.com', 587, timeout=30) as server:
server.ehlo()
server.starttls()
server.ehlo()
server.login('geekqjg@gmail.com', password)
server.send_message(msg)
Why himalaya also fails when proxy is down: Himalaya respects http_proxy/https_proxy env vars. If those are set in the environment (or injected by a wrapper), himalaya routes through the dead proxy and times out. The send-report.py script uses explicit socket-level CONNECT proxy routing. Both fail when the proxy port is unreachable.
Key insight (updated 2026-06-29): On this host, direct connections to smtp.gmail.com:587 and :465 have historically been reachable when the local xray proxy is down. However, this is NOT guaranteed — on 2026-06-29, direct connections also timed out even with the proxy UP. The direct-smtplib path may fail due to ISP-level SMTP port blocking, Google rate-limiting the source IP, or transient network issues. Always test direct reachability (nc -z -w 5 smtp.gmail.com 587) before assuming P21 will work.
P22: All Paths Fail (Proxy Up + SSL Hang + Direct Timeout) — 2026-06-29
Symptom: Proxy CONNECT succeeds (HTTP 200), but SSL handshake to port 465 hangs. Port 587 via proxy also times out. Direct connections to smtp.gmail.com:587/465 timeout too. All fallback paths exhausted.
Root cause: Unclear — could be proxy exit IP blocked by Google, ISP-level SMTP port filtering, or Google throttling. The proxy is functional for HTTP/HTTPS web traffic but SMTP TLS negotiation fails through it.
Diagnosis:
nc -z -w 3 127.0.0.1 10808 && echo "PROXY UP"
nc -z -w 5 smtp.gmail.com 587 && echo "DIRECT 587 OK" || echo "DIRECT 587 BLOCKED"
nc -z -w 5 smtp.gmail.com 465 && echo "DIRECT 465 OK" || echo "DIRECT 465 BLOCKED"
python3 -c "
import socket, ssl
s = socket.socket(); s.settimeout(10)
s.connect(('127.0.0.1', 10808))
s.sendall(b'CONNECT smtp.gmail.com:465 HTTP/1.1\r\nHost: smtp.gmail.com:465\r\n\r\n')
print(s.recv(4096))
ctx = ssl.create_default_context()
try:
ctx.wrap_socket(s, server_hostname='smtp.gmail.com')
print('SSL OK')
except Exception as e:
print(f'SSL FAILED: {e}')
"
When all paths fail: Log the failure honestly in cron-status.log and save the report locally. Do NOT retry indefinitely — the issue is likely network-level and won't resolve by retrying. Flag for manual investigation (proxy node switch, ISP issue, Google account security).
⚠️ Unified Fallback Chain (MUST follow in order)
Do this FIRST, before trying ANY email-sending tool:
nc -z -w 3 127.0.0.1 10808 2>&1 && echo "PROXY UP" || echo "PROXY DOWN"
Then follow the chain:
- PROXY UP but TLS/auth flaky → try direct smtplib (P21) immediately — proxy SSL hangs while direct works (confirmed 2026-07-01). If direct fails too, retry 3× with backoff, then
send-report-587.py, then himalaya.
- PROXY DOWN → skip ALL retries and proxy-dependent paths. Use direct
smtplib (P21 above) immediately.
- PROXY UP + SSL hang + direct also fails (P22) → try
himalaya message send (bypasses proxy for SMTP — see pitfalls). If himalaya also fails, log failure, save report locally, flag for manual investigation. Do NOT retry.
- himalaya fails too → honest failure log
Anti-pattern (DO NOT): Wasting time trying himalaya, Cloudflare Email Service REST API, or other elaborate alternatives when the proxy is down. The direct smtplib approach in P21 works in ~2 seconds — use it.
Key Common Pitfalls
- HTTP CONNECT proxy SSL timeout on port 465: If port 465 fails with SSL handshake timeout, try port 587 with
http_proxy/https_proxy environment variables.
- Monkey-patch scope:
socket.create_connection is used by many libraries; be specific about the patch lifetime (or restore after done) to avoid side effects.
- App Passwords: Gmail requires App Passwords (not regular password) when 2FA is enabled. Store in macOS Keychain, not in scripts.
- Both scripts are Python stdlib only: socket, ssl, base64, subprocess, imaplib. No pip packages needed.
himalaya blocked by dead proxy: If http_proxy/https_proxy are set and proxy is down, himalaya will timeout. Either unset the env vars or use direct smtplib.- himalaya
message send folder error: If himalaya message send fails with "Folder doesn't exist" on [Gmail]/Sent Mail, this is a himalaya bug with IMAP folder alias resolution for the Sent-copy save. The email IS sent — the SMTP send completes before the IMAP folder-save fails (confirmed via --debug logs 2026-06-27: sending smtp message appears before the IMAP error). Gmail also auto-saves sent mail server-side. Verify with himalaya envelope list --folder "[Gmail]/Sent Mail". No need to fall back to smtplib for sending — only the local Sent copy is lost.
- himalaya CLI syntax: The send command is
himalaya message send (not himalaya send). It requires RFC 2822 headers (From/To/Subject) piped via stdin.
- himalaya bypasses proxy for SMTP: Unlike the custom
send-mail.py (which explicitly connects through the HTTP CONNECT proxy on port 465), himalaya connects directly to smtp.gmail.com:587 with STARTTLS — no proxy involved. This makes himalaya a reliable fallback when the proxy is flaky (but not when http_proxy/https_proxy env vars are set and proxy is down, as himalaya respects those env vars).
- Cloudflare Email Service as fallback: Only viable if CF API token is configured and the Cloudflare API endpoint is reachable. If the proxy is down, CF API calls via
wrangler will also timeout. Not a useful fallback for proxy-down scenarios.
HTML Email Template
For automated reports sent via email, use templates/daily-report.html as the base template. Outlook-compatible (table layout, inline CSS). Replace {{PLACEHOLDER}} vars with data. Colors: green (#2e7d32) OK, red (#c62828) error, orange (#e65100) warning, gray (#9e9e9e) paused.
