troubleshoot

name	troubleshoot
description	Troubleshoot cortex connection failures, missing logs, unhealthy containers, restart loops, or vague "logs aren't working" reports.

Cortex Troubleshooting Skill

Diagnose cortex problems systematically. Use the binary's observability counters and existing diagnostic tooling rather than guessing — the codebase exposes most state needed to localize a failure.

Decision tree — pick the right diagnostic

Match the user's report against one of these branches and follow only that branch. Don't run every check when the failure is narrow.

Branch A — "MCP can't connect" / "Failed to reconnect" / "401 / 404 from /mcp"

Most common cause: empty / wrong $CLAUDE_PLUGIN_OPTION_SERVER_URL, mismatched $CLAUDE_PLUGIN_OPTION_API_TOKEN, or service not running.

1. Is anything listening on the MCP port? ss -tlnp | grep -E ":$CLAUDE_PLUGIN_OPTION_MCP_PORT" — if empty, the service is down → branch C
2. Is the URL Claude Code is using sane? Read ~/.claude/settings.json, find the pluginConfigs key that starts with cortex@, and inspect options.server_url — empty string is a known footgun (the .mcp.json substitution produces a literal /mcp). Check non-empty, has scheme, no trailing /mcp.
3. Does observed auth match configured auth? Run curl -sS -o /dev/null -w '%{http_code}' "$CLAUDE_PLUGIN_OPTION_SERVER_URL/mcp".
   • If $CLAUDE_PLUGIN_OPTION_NO_AUTH is true or no bearer/OAuth auth is configured, 200 or MCP protocol-level 400/405 can be normal route evidence.
   • If bearer or OAuth auth is enabled, expect 401 for an unauthenticated request.
   • If 404, the route is wrong or a different server owns that port. If connection refused, branch C.
   • If 200 while auth is intended to be enabled, flag it as an auth configuration mismatch.
4. Token roundtrip in bearer mode: curl -sS -X POST -H "Authorization: Bearer $CLAUDE_PLUGIN_OPTION_API_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' "$CLAUDE_PLUGIN_OPTION_SERVER_URL/mcp". 401 = wrong token. 200 with valid response = server fine, problem is in Claude Code's MCP client config. For OAuth mode, use the OAuth client flow instead of bearer-token curl. Note: verify the MCP protocol version string (2025-11-25) matches the current spec if this test fails unexpectedly.

Branch B — "No logs from " / "host X stopped sending" / "missing entries"

1. Does the host appear in the hosts list at all? Call MCP tool: cortex action=hosts. If host is absent, no logs ever arrived → check forwarding config on <host>. If present with old last_seen, forwarding stopped → check rsyslog/forwarder on host.
2. Is the listener actually accepting connections? ss -tlnp | grep -E ":(${CLAUDE_PLUGIN_OPTION_SYSLOG_HOST_PORT:-$CLAUDE_PLUGIN_OPTION_SYSLOG_PORT})\\b" should show our process or container port publish. From <host>: nc -zv <our_host> "${CLAUDE_PLUGIN_OPTION_SYSLOG_HOST_PORT:-$CLAUDE_PLUGIN_OPTION_SYSLOG_PORT}" should connect.
3. Recent forwarding errors on the host? ssh <host> "sudo journalctl -t rsyslogd -n 30 --no-pager" — look for omfwd errors (DNS resolution, peer closed, EOF on TCP). Common patterns we've seen: stale forwarder pointing at a dead host, idle TCP timeout flapping, missing rsyslog drop-in.
4. Drop-in present and correct? ssh <host> "cat /etc/rsyslog.d/99-cortex.conf 2>/dev/null" should contain *.* @@<our_host>:<externally reachable syslog port> (TCP), usually ${CLAUDE_PLUGIN_OPTION_SYSLOG_HOST_PORT:-$CLAUDE_PLUGIN_OPTION_SYSLOG_PORT} — if missing or wrong, push the drop-in config to the host over SSH.
5. For Docker container logs: if user expected logs from a container in $CLAUDE_PLUGIN_OPTION_FLEET_HOSTS but doesn't see them, check $CLAUDE_PLUGIN_OPTION_DOCKER_INGEST_ENABLED. If false, ingest is off entirely. If true, verify the docker-socket-proxy on that host is reachable: curl -sS http://<host>:2375/_ping should return OK.

Branch C — Service down / crashing / unhealthy

1. Get current state: docker ps --filter name=cortex --format '{{.Status}}'
2. If recently restarted / crashing — get the actual error: use logs for the last 100 lines, or run docker compose logs manually. Look for: panic messages, port-bind errors (address already in use), DB lock errors, OOM kills.
3. Common service-failure causes (ranked by frequency in this plugin's history):
   1. Port $CLAUDE_PLUGIN_OPTION_SYSLOG_PORT or $CLAUDE_PLUGIN_OPTION_MCP_PORT held by another process. First identify the owner with ss -tulpn/lsof/fuser; only kill or restart anything after the user approves the specific process and impact.
   2. Database lock (another cortex stdio process holds it). pgrep -af "cortex" to list candidates; only kill stragglers after approval.
   3. Docker image missing/stale: docker compose pull to refresh.
4. If healthcheck failing but /health works manually: Container is unhealthy because the healthcheck command inside the image is wrong/can't run. Compare image version to what you expect — docker inspect cortex | jq '.[0].Config.Image'.

Branch D — "Something's off" / vague / user doesn't know

Run a comprehensive preflight and health check: env/config, storage, ports, service status, HTTP /health, MCP actions, listener reachability, Docker ingest, and fleet rsyslog forwarding. A PASS / WARN / FAIL result narrows the problem to a specific check. Then re-enter this skill on the failing check's category.

Use observability counters

The binary exposes runtime counters via cortex action=stats and /health. Useful signals:

• total_logs not increasing → ingest pipeline is broken, not just MCP
• write_blocked: true → storage budget tripped, oldest logs being purged but can't keep up; check $CLAUDE_PLUGIN_OPTION_MAX_DB_SIZE_MB vs disk free
• phantom_fts_rows growing → retention purges aren't merging FTS5 cleanly; usually self-recovers
• last_ingest_at minutes-stale → forwarders aren't reaching us
• Newer counters in RuntimeObservability (since v0.13.0): UDP/TCP packets, ingest queue depth, writer flush failures — pull these via the /health endpoint or stats action and use to localize "ingest path" vs "writer path" failures

Don't over-fix

• For a single-host symptom, don't restart the whole stack — just fix that host's forwarder.
• For an MCP-only failure with healthy ingest, don't touch the listener config.
• If the immediate problem is a missing config, prefer redeploy over manual Docker commands.

When to escalate to the user

• After a confident diagnosis, propose the fix and ask before applying it for anything destructive: changing settings.json, killing processes, deleting files, switching deploy modes.
• If checks return inconsistent state (e.g. listener says ours, but the binary says it isn't writing), surface the inconsistency rather than guessing.
• If the failure looks like an upstream bug (panic, deadlock, repeated crash on the same input), gather the journalctl/docker-logs output and stop — don't try multiple fix attempts on suspected source-code bugs.