feishu-gateway-troubleshooting

name	feishu-gateway-troubleshooting
title	Feishu Gateway Troubleshooting
description	Diagnose and fix common Feishu (Lark) WebSocket connection issues in the Hermes gateway: keepalive ping timeouts, DNS resolution failures, proxy conflicts, SSL errors, Executor shutdown, and process conflicts.

Feishu Gateway Troubleshooting

Trigger: When user reports Feishu not responding, messages not being received, gateway disconnect errors, or when checking gateway status shows disconnection issues.

Common Error Patterns & Solutions

1. Keepalive Ping Timeout (most frequent)

Symptoms in logs:

receive message loop exit, err: sent 1011 (internal error) keepalive ping timeout

Cause: Feishu WebSocket sends keepalive pings every ~30s; if no pong received within timeout, connection drops. Happens frequently on this environment (49+ times logged), likely due to network interruptions (VPN, Wi-Fi flapping, or proxy interference).

Recovery: Automatic. The gateway retries: trying to reconnect for the 1st time -> [2nd/3rd]. Usually reconnects within 2-3 minutes. No action needed — just verify with the verification step.

2. DNS Resolution Failure

Symptoms:

Failed to resolve 'open.feishu.cn' ([Errno 8] nodename nor servname provided, or not known)

Cause: DNS temporarily unreachable (151 occurrences on this environment). Feishu API cannot be resolved -> WebSocket connection impossible.

Fix:

dig open.feishu.cn +short
nslookup open.feishu.cn 8.8.8.8

Recovery: Usually transient (DNS auto-recovers). If persistent, check /etc/resolv.conf or network config.

3. Proxy Conflict

Symptoms:

Unable to connect to proxy, Failed to establish a new connection: [Errno 61] Connection refused

Cause: HTTP_PROXY/HTTPS_PROXY env vars point to a proxy (eg 127.0.0.1:10808) that is not running.

Fix:

• Check proxy env vars: echo $HTTPS_PROXY $HTTP_PROXY
• Either start the proxy or unset the vars before starting gateway:

  unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy
  hermes gateway stop && hermes gateway start
  

4. SSL/Network EOF

Symptoms:

[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol

Cause: TLS handshake interrupted mid-flight. Rare, transient network issue.

Recovery: Automatic on next reconnect attempt. No action needed.

5. Executor Shutdown (secondary cascade)

Symptoms in error.log:

RuntimeError: Executor shutdown has been called

Often follows issues 1-4 after gateway has been running for hours.

Cause: When gateway is being stopped/restarted, messages queued for Feishu delivery fail because the event loop's executor is already shutting down. This is a normal shutdown artifact, not a real error.

What to check: If "Executor shutdown" errors are not accompanied by a recent gateway stop/start command, it may indicate the gateway process crashed. Check:

ps aux | grep "gateway run" | grep -v grep

6. Process Conflict (multiple gateways)

Symptoms:

ERROR: feishu - Another local Hermes gateway is already using this Feishu app_id (PID XXXX)

Cause: A previous gateway instance is still holding the Feishu app_id and bot token.

Fix:

hermes gateway stop && sleep 2 && hermes gateway start

7. SQLite Session Warning (harmless)

Symptoms:

Warning: Failed to create SQLite session: 'NoneType' object has no attribute 'execute'

Status: Harmless. Occurs once on startup. Does not affect Feishu connectivity or message delivery. Can be safely ignored.

Diagnosis Procedure

Run these in order to pinpoint the issue:

Step 1: Check if gateway is running

hermes gateway status
ps aux | grep "gateway run" | grep -v grep

Step 2: Check recent Feishu connection logs

grep "connected to wss.*feishu" ~/.hermes/logs/gateway.log | tail -2
grep -E "(disconnected|keepalive ping timeout|connect failed)" ~/.hermes/logs/gateway.log | tail -5

Step 3: Check error log for cascade failures

grep -E "(ERROR|RuntimeError|Executor shutdown)" ~/.hermes/logs/gateway.error.log | tail -10

Step 4: Verify network connectivity to Feishu

dig msg-frontier.feishu.cn +short
curl -s -o /dev/null -w "%{http_code}" https://open.feishu.cn

Step 5: Check proxy interference

echo "HTTPS_PROXY=$HTTPS_PROXY HTTP_PROXY=$HTTP_PROXY"

Fix Commands (quick reference)

# Basic restart (fixes 80% of cases)
hermes gateway stop
sleep 2
hermes gateway start

# Full flush restart (if basic restart doesn't work)
hermes gateway stop
sleep 3
kill $(pgrep -f "gateway run") 2>/dev/null
hermes gateway start

Verification

After applying a fix, verify the connection established:

sleep 10
grep "connected to wss.*feishu" ~/.hermes/logs/gateway.log | tail -2

Look for the most recent line with [INFO] connected to wss://msg-frontier.feishu.cn/ws/v2...

Known Behavior

• Auto-reconnect: Gateway retries up to 3 times with ~2min intervals. Most disconnections self-heal.
• Keepalive timeout is the norm on this environment (~49 occurrences logged), not an emergency.
• The "Executor shutdown" error is always a downstream effect, not a root cause. If you see it, look for the root cause above it in the logs.
• gateway.log tracks Feishu connection events (connected/disconnected/reconnect). gateway.error.log tracks stack traces and cascade errors.