jetstream-operations

name

jetstream-operations

description

Use this skill whenever users are operating, troubleshooting, or monitoring a running NATS JetStream system — including diagnosing consumer lag, messages not delivering, stream-full errors, performance tuning, Prometheus metrics, Grafana dashboards, alerting rules, JetStream advisory subjects, nats CLI usage, cluster health and leader election issues, or client connection/reconnection problems. Use this skill when something is broken or slow, or when the user wants to observe their system. Do NOT use for designing new streams/consumers (use jetstream-architecture) or deploying/configuring infrastructure (use jetstream-deployment).

JetStream Operations

Troubleshoot, monitor, and tune NATS JetStream including consumer lag, delivery failures, performance optimization, and observability.

For designing new streams or consumers, defer to the jetstream-architecture skill. For deploying or configuring NATS infrastructure, defer to the jetstream-deployment skill. If the user is operating a Synadia Agent fabric, the same tooling applies: enumerate live agents with nats req '$SRV.PING.agents' '', inspect endpoints/metadata with $SRV.INFO.agents, and watch agent liveness on the heartbeat subjects agents.hb.*.*.* (an agent is offline after ~3× its advertised interval_s). Tapping agents.> shows live prompt/response traffic. For how the protocol and those subjects are defined, see the nats-agent-fabric skill.

Reference Files

Read these files when they're relevant — don't load all of them upfront:

• operations/troubleshooting.md — step-by-step diagnosis for messages not delivering, consumer lag, stream-full errors, cluster split-brain, and client disconnections. Read whenever something is broken.
• operations/performance.md — publish throughput tuning, fetch batch sizing, MaxAckPending, parallel workers, FileStorage vs MemoryStorage, OS/TCP tuning, built-in benchmark commands. Read for performance or throughput questions.
• operations/monitoring.md — NATS HTTP endpoints, Prometheus metrics, nats-exporter setup, advisory subjects, Grafana dashboard recommendations, Prometheus alert rules. Read for observability and alerting questions.
• operations/cli-reference.md — full nats CLI reference for streams, consumers, pub/sub, server commands, and diagnostic workflows. Read when the user needs specific CLI commands.

Workflow

Step 1: Identify symptoms — what is the user observing? (lag, missing messages, errors, slow throughput)

Step 2: Inspect current state — use nats CLI to examine streams, consumers, and server status.

Step 3: Diagnose root cause — match symptoms to known patterns (ack timeout, filter mismatch, resource limits).

Step 4: Apply fix — provide specific configuration changes or code fixes.

Step 5: Verify resolution — confirm the fix with CLI commands and metrics.

Step 6: Set up monitoring — recommend metrics, alerts, and advisory subscriptions to prevent recurrence.

Core Principles

• Always start diagnosis with nats stream report and nats consumer report
• Check num_ack_pending first when consumers appear stuck — it's the most common bottleneck
• Monitor JetStream advisory subjects ($JS.EVENT.ADVISORY.>) for real-time operational events
• Use nats server report jetstream to check cluster-wide JetStream health
• Set alerts on consumer pending count, not just publish rate
• Prefer nats CLI over raw API calls for operational tasks
• Always check the NATS server logs — they surface warnings before failures
• When in doubt, compare stream sequence numbers with consumer ack floor