feishu-gateway-troubleshooting

name	feishu-gateway-troubleshooting
title	Feishu Gateway Troubleshooting
description	Diagnose and fix common Feishu (Lark) WebSocket connection issues in the Hermes gateway: keepalive ping timeouts, DNS resolution failures, proxy conflicts, SSL errors, Executor shutdown, and process conflicts.

Feishu Gateway Troubleshooting

Trigger: When user reports Feishu not responding, messages not being received, gateway disconnect errors, or when checking gateway status shows disconnection issues.

Common Error Patterns & Solutions

1. Keepalive Ping Timeout (most frequent)

Symptoms in logs:

receive message loop exit, err: sent 1011 (internal error) keepalive ping timeout

Cause: Feishu WebSocket sends keepalive pings every ~30s; if no pong received within timeout, connection drops. Happens frequently on this environment (49+ times logged), likely due to network interruptions (VPN, Wi-Fi flapping, or proxy interference).

Recovery: Automatic. The gateway retries: trying to reconnect for the 1st time -> [2nd/3rd]. Usually reconnects within 2-3 minutes. No action needed — just verify with the verification step.

2. DNS Resolution Failure

Symptoms:

Failed to resolve 'open.feishu.cn' ([Errno 8] nodename nor servname provided, or not known)

Cause: DNS temporarily unreachable (151 occurrences on this environment). Feishu API cannot be resolved -> WebSocket connection impossible.

Fix:

dig open.feishu.cn +short
nslookup open.feishu.cn 8.8.8.8

Recovery: Usually transient (DNS auto-recovers). If persistent, check /etc/resolv.conf or network config.

3. Proxy Conflict

Symptoms:

Unable to connect to proxy, Failed to establish a new connection: [Errno 61] Connection refused

Or: frequent keepalive ping timeouts and disconnections (no close frame received or sent) even though the proxy IS running.

Cause: HTTP_PROXY/HTTPS_PROXY env vars point to a proxy (eg 127.0.0.1:10808) that is not running, OR the proxy is running but NO_PROXY is not set, causing ALL traffic including localhost and WebSocket connections to be routed through the proxy.

Fix:

• Check proxy env vars: echo $HTTPS_PROXY $HTTP_PROXY $NO_PROXY
• Ensure NO_PROXY includes localhost addresses AND .feishu.cn:

  export NO_PROXY="localhost,127.0.0.1,::1,*.local,.feishu.cn"
  export no_proxy="$NO_PROXY"
  

  This prevents local services (gateway, Docker containers, Playwright) from being routed through the proxy. The .feishu.cn domain is critical — without it, the proxy's SSL certificate (self-signed or MITM) will not match open.feishu.cn and the gateway will see SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed: Hostname mismatch on all Feishu API calls.
• Either start the proxy or unset the vars before starting gateway:

  unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy
  hermes gateway stop && hermes gateway start
  

• Persistent fix: Add NO_PROXY to the proxy block in ~/.zshrc:

  if nc -z 127.0.0.1 10808 2>/dev/null; then
    export HTTPS_PROXY=http://127.0.0.1:10808
    export HTTP_PROXY=http://127.0.0.1:10808
    export NO_PROXY="localhost,127.0.0.1,::1,*.local"
    export no_proxy="$NO_PROXY"
  fi
  

PITFALL: The gateway process inherits the shell's proxy env vars at launch time. Setting NO_PROXY in ~/.zshrc only affects NEW terminal sessions. To apply to a running gateway, restart it: hermes gateway stop && hermes gateway start.

PITFALL (launchd/gateway): For the gateway launched as a launchd service, .zshrc is never sourced. The NO_PROXY fix must be applied via launchctl setenv so it propagates to all future launchd-managed processes:

launchctl setenv NO_PROXY "localhost,127.0.0.1,::1,*.local,.feishu.cn"

This persists for the current login session. After this, restart the gateway: hermes gateway restart.

PITFALL: Playwright tests against local Docker containers (localhost:8003, localhost:8004) fail silently when NO_PROXY is missing — the browser's HTTP requests go through the proxy which rejects them. This manifests as page.waitForSelector timeouts, not obvious connection errors.

4. SSL/Network EOF

Symptoms:

[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol

Cause: TLS handshake interrupted mid-flight. Rare, transient network issue.

Recovery: Automatic on next reconnect attempt. No action needed.

5. Executor Shutdown (secondary cascade)

Symptoms in error.log:

RuntimeError: Executor shutdown has been called

Often follows issues 1-4 after gateway has been running for hours.

Cause: When gateway is being stopped/restarted, messages queued for Feishu delivery fail because the event loop's executor is already shutting down. This is a normal shutdown artifact, not a real error.

What to check: If "Executor shutdown" errors are not accompanied by a recent gateway stop/start command, it may indicate the gateway process crashed. Check:

ps aux | grep "gateway run" | grep -v grep

6. Process Conflict (multiple gateways)

Symptoms:

ERROR: feishu - Another local Hermes gateway is already using this Feishu app_id (PID XXXX)

Cause: A previous gateway instance is still holding the Feishu app_id and bot token.

Fix:

hermes gateway stop && sleep 2 && hermes gateway start

7. SQLite Session Warning (harmless)

Symptoms:

Warning: Failed to create SQLite session: 'NoneType' object has no attribute 'execute'

Status: Harmless. Occurs once on startup. Does not affect Feishu connectivity or message delivery. Can be safely ignored.

8. Outdated certifi CA bundle (gateway-only SSL failures)

Symptoms: SSLCertVerificationError: certificate verify failed: Hostname mismatch for open.feishu.cn, but direct Python SSL check succeeds:

python3 -c "
import ssl, socket
ctx = ssl.create_default_context()
sock = ctx.wrap_socket(socket.socket(), server_hostname='open.feishu.cn')
sock.connect(('open.feishu.cn', 443))
print('System SSL: OK — host is reachable, cert is valid')
"

If system SSL succeeds but the gateway still reports Hostname mismatch, the gateway's bundled CA bundle (certifi) is likely outdated.

Root cause: The Hermes gateway's HTTP client loads CA certificates from the certifi package. When certifi is out of date, it may not contain the latest intermediate CA that Feishu's TLS certificate chains to. CDN-delivered services rotate their TLS infrastructure periodically, and an old certifi bundle doesn't know about the new intermediates.

Fix:

pip install --upgrade certifi
hermes gateway stop && hermes gateway start
sleep 10
grep "connected to wss.*feishu" ~/.hermes/logs/gateway.log | tail -2

Distinguish from proxy MITM (Section 3): If NO_PROXY already includes .feishu.cn and proxy env vars are not the cause, check certifi. Quick distinction:

• Proxy case: echo $HTTPS_PROXY returns a value, NO_PROXY missing .feishu.cn
• Certifi case: no proxy env vars set, or NO_PROXY is already correct, but SSL still fails

Diagnosis Procedure

Run these in order to pinpoint the issue:

Step 1: Check if gateway is running

hermes gateway status
ps aux | grep "gateway run" | grep -v grep

Step 2: Check recent Feishu connection logs

grep "connected to wss.*feishu" ~/.hermes/logs/gateway.log | tail -2
grep -E "(disconnected|keepalive ping timeout|connect failed)" ~/.hermes/logs/gateway.log | tail -5

Step 3: Check error log for cascade failures

grep -E "(ERROR|RuntimeError|Executor shutdown)" ~/.hermes/logs/gateway.error.log | tail -10

Step 4: Verify network connectivity to Feishu

dig msg-frontier.feishu.cn +short
curl -s -o /dev/null -w "%{http_code}" https://open.feishu.cn

Step 5: Check proxy interference

echo "HTTPS_PROXY=$HTTPS_PROXY HTTP_PROXY=$HTTP_PROXY"

Fix Commands (quick reference)

# Basic restart (fixes 80% of cases)
hermes gateway stop
sleep 2
hermes gateway start

# Full flush restart (if basic restart doesn't work)
hermes gateway stop
sleep 3
kill $(pgrep -f "gateway run") 2>/dev/null
hermes gateway start

Verification

After applying a fix, verify the connection established:

sleep 10
grep "connected to wss.*feishu" ~/.hermes/logs/gateway.log | tail -2

Look for the most recent line with [INFO] connected to wss://msg-frontier.feishu.cn/ws/v2...

Known Behavior

• Auto-reconnect: Gateway retries up to 3 times with ~2min intervals. Most disconnections self-heal.
• Keepalive timeout is the norm on this environment (~49 occurrences logged), not an emergency.
• The "Executor shutdown" error is always a downstream effect, not a root cause. If you see it, look for the root cause above it in the logs.
• gateway.log tracks Feishu connection events (connected/disconnected/reconnect). gateway.error.log tracks stack traces and cascade errors.