midnight-indexer-indexer-operations

name	midnight-indexer:indexer-operations
description	This skill should be used when the user asks about running, monitoring, or troubleshooting the Midnight indexer — health checks, the /live and /ready endpoints, Prometheus metrics, OpenTelemetry tracing, node reconnection and recovery, wallet session lifecycle, caught-up status, NATS failure handling, indexer logs, or indexer deployment operations. Covers "is my indexer healthy", "why is the indexer behind the node", "indexer metrics", "indexer not catching up", and "monitoring the indexer in production".
version	0.1.0

Indexer Operations

Operational guide for running, health-checking, monitoring, and troubleshooting the Midnight indexer. For starting and stopping the local Docker stack, see midnight-tooling:devnet. For the full observability and troubleshooting reference, see references/monitoring-and-troubleshooting.md in this skill.

Deployment Recap

The indexer ships in two deployment shapes. For full detail see midnight-indexer:indexer-architecture → references/deployment-and-crates.md.

Shape	Services	Storage	Bus
Standalone	Single binary (indexer-standalone)	SQLite	In-process
Cloud	4 services: chain-indexer, wallet-indexer, indexer-api, spo-indexer	PostgreSQL	NATS

Default port: 8088 (GraphQL + health probes on the same listener).

Health Checks

The indexer exposes two HTTP probes on the GraphQL port (default 8088). Both are served by the same Axum router as the GraphQL endpoint.

Endpoint	Kubernetes role	Returns
GET /live	Liveness	200 OK — always, as long as the async runtime is responsive
GET /ready	Readiness	200 OK when caught up with the node; 503 Service Unavailable with body "indexer has not yet caught up with the node" otherwise

Source: indexer-api/src/infra/api.rs:277–309.

# Liveness — is the process alive?
curl -sf http://localhost:8088/live

# Readiness — has it caught up with the node?
curl -o - -w "\nHTTP %{http_code}\n" http://localhost:8088/ready

The /live handler runs in the async task executor; if a storage-layer deadlock parks the runtime, the liveness probe times out rather than returning 200, which lets Kubernetes terminate and recreate the pod.

Docker / Kubernetes Patterns

The official Docker Compose stack defines a HEALTHCHECK for each service that tests a running-file: the entrypoint creates /var/run/<component>/running on startup and removes it on exit (via trap … EXIT), and the healthcheck command is cat /var/run/<component>/running. The image Dockerfiles themselves embed no HEALTHCHECK. For Kubernetes — or to gate traffic on actual readiness rather than liveness — prefer the HTTP probes:

# Kubernetes
livenessProbe:
  httpGet:
    path: /live
    port: 8088
  initialDelaySeconds: 15
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /ready
    port: 8088
  initialDelaySeconds: 30
  periodSeconds: 10

# Docker Compose
healthcheck:
  test: ["CMD", "curl", "-sf", "http://localhost:8088/ready"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 60s

Caught-Up Status

"Caught up" governs whether /ready returns 200 and whether the indexer_caught_up gauge is 1.

Config key	Default	Meaning
caught_up_max_distance	10	Max blocks behind the node before caught_up = false
caught_up_leeway	5	Hysteresis — must catch within this distance before caught_up flips back to true

The formula: caught_up = (node_height − indexed_height) ≤ caught_up_max_distance, with caught_up_leeway preventing rapid oscillation at the threshold. A 503 from /ready during initial sync is normal and expected — the indexer must process all historical blocks before it reports ready.

Configuration keys live under APP__ in environment or in config.yaml. See midnight-indexer:indexer-architecture → references/configuration-reference.md for the full key listing.

Metrics and Tracing

Both are disabled by default in standalone and cloud deployments. Enable them via environment variables or config.yaml.

Prometheus Metrics

# Enable via environment
APP__TELEMETRY__METRICS__ENABLED=true   # required
APP__TELEMETRY__METRICS__ADDRESS=0.0.0.0
APP__TELEMETRY__METRICS__PORT=9000      # default

Scrape endpoint: http://<host>:9000/metrics

Metric	Type	Description
indexer_block_height	counter	Highest block the indexer has processed
indexer_node_block_height	counter	Latest block height reported by the node
indexer_caught_up	gauge	1 = caught up, 0 = behind
indexer_transaction_count	counter	Total transactions indexed
indexer_wallets_connected	gauge	Active wallet sessions
indexer_subscriptions_active	gauge	Active GraphQL subscriptions
indexer_subscriptions_rejected_total	counter	Subscriptions rejected by quota

For the full metric catalog and example Grafana queries, see references/monitoring-and-troubleshooting.md.

OpenTelemetry Tracing

APP__TELEMETRY__TRACING__ENABLED=true
APP__TELEMETRY__TRACING__OTLP_EXPORTER_ENDPOINT=http://localhost:4317  # default
APP__TELEMETRY__TRACING__SERVICE_NAME=indexer

Spans are exported via gRPC OTLP to the configured endpoint. Logs are correlated with traces — when a log record is emitted inside a span, the trace ID is included in the JSON log output.

Node Connection and Recovery

The indexer connects to the Midnight node via subxt with a built-in reconnecting RPC client using exponential backoff.

Config key	Default	Notes
reconnect_max_delay	10s	Backoff ceiling; steps are 10ms → 100ms → 1s → 10s
reconnect_max_attempts	30	≈ 5 minutes of retries before the process exits
subscription_recovery_timeout	30s	If no valid block arrives after a reconnect or duplicate event within this window, the subscription is torn down and re-established

When the node disconnects, a WARN log line "node disconnected, reconnecting" is emitted. After reconnect_max_attempts failures the process exits so the container orchestrator can restart it.

Config keys live under APP__INFRA__NODE__ (the standalone binary also has a separate APP__INFRA__SPO_NODE__ for its Cardano-side connection). See midnight-indexer:indexer-architecture → references/configuration-reference.md.

Wallet Session Lifecycle

Wallet sessions let the indexer scan shielded transactions for a specific viewing key without exposing that key on-chain.

connect(viewingKey)
  → viewing key encrypted at rest (ChaCha20-Poly1305, key from APP__INFRA__SECRET)
  → wallet-indexer scans blocks for relevant transactions
  → client subscribes via shieldedTransactions(sessionId)

disconnect(sessionId)
  → session ends, resources released

Config key	Default	Meaning
active_wallets_ttl	30m	Idle sessions are evicted after this duration
keep_wallet_alive_interval	1m	Heartbeat interval to keep long-running catch-up sessions alive

APP__INFRA__SECRET must be set to a hex-encoded 32-byte key before the indexer will accept connect mutations. Missing or invalid SECRET causes startup failure or decryption errors at session creation time.

For the GraphQL mutations (connect, disconnect) and the shieldedTransactions subscription, see midnight-indexer:indexer-graphql-api.

NATS Failure Handling (Cloud)

In cloud mode, chain-indexer → wallet-indexer and chain-indexer → indexer-api communication flows through NATS (core pub-sub — the server runs with -js but the indexer creates no JetStream streams or consumers). Publisher and subscriber components log WARN on Event::ServerError and Event::ClientError from the NATS client library; a startup ConnectError causes the service to exit so the orchestrator can retry.

For subjects, consumer groups, and failure modes, see midnight-indexer:indexer-architecture → references/nats-messaging.md.

Troubleshooting

Quick-reference for common failure modes. Full diagnostic procedures and log patterns are in references/monitoring-and-troubleshooting.md.

Symptom	Likely cause	First action
/ready returns 503 continuously	Initial sync not complete, or node unreachable	Check indexer_block_height vs indexer_node_block_height; confirm node is reachable
indexer_caught_up stuck at 0	Block processing stalled or node connection lost	Check logs for "node disconnected" or DB errors
connect mutation fails	APP__INFRA__SECRET missing or malformed	Verify the env var is set to a valid 64-char hex string
NATS startup failure (cloud)	NATS server not reachable at startup	Check APP__INFRA__PUB_SUB__URL; verify NATS container is healthy
DB migration errors	Schema version mismatch after upgrade	Check logs for sqlx migration errors; ensure the DB is accessible
Wallet sessions evicted unexpectedly	active_wallets_ttl too short for catch-up duration	Increase active_wallets_ttl or ensure the client heartbeats within the TTL
Blockfrost 402 errors (SPO indexer)	Blockfrost plan limit reached or invalid BLOCKFROST_ID	Verify APP__INFRA__SPO_NODE__BLOCKFROST_ID and plan quota

# Diagnostic checklist
# 1. Is the process alive?
curl -sf http://localhost:8088/live && echo "LIVE"

# 2. Has it caught up with the node?
curl -o - -w "\nHTTP %{http_code}\n" http://localhost:8088/ready

# 3. What metrics show?
curl -s http://localhost:9000/metrics | grep -E "indexer_(block_height|node_block_height|caught_up)"

# 4. Recent log output (standalone Docker)
docker logs --tail 50 indexer-standalone

# 5. Follow logs in real time
docker logs -f indexer-standalone

References

Name	Description	When used
references/monitoring-and-troubleshooting.md	Full metric catalog, log pattern reference, Grafana queries, and detailed diagnostic procedures	Setting up production monitoring or investigating deep operational issues
examples/running-standalone.md	Step-by-step: run the standalone indexer with health checks and metrics enabled	Getting the standalone binary operational end-to-end

Cross-References

Skill / Resource	Relevance
midnight-tooling:devnet	Starting and stopping the full local Docker stack (node + indexer + proof server)
midnight-indexer:indexer-architecture	Deployment modes, crate layout, and full configuration-reference / nats-messaging / deployment-and-crates references
midnight-indexer:indexer-graphql-api	connect / disconnect mutations and the shieldedTransactions subscription
midnight-indexer:indexer-data-model	What gets indexed, database schema, and block/transaction data shapes