| name | midnight-indexer:indexer-operations |
| description | This skill should be used when the user asks about running, monitoring, or troubleshooting the Midnight indexer — health checks, the /live and /ready endpoints, Prometheus metrics, OpenTelemetry tracing, node reconnection and recovery, wallet session lifecycle, caught-up status, NATS failure handling, indexer logs, or indexer deployment operations. Covers "is my indexer healthy", "why is the indexer behind the node", "indexer metrics", "indexer not catching up", and "monitoring the indexer in production". |
| version | 0.1.0 |
Indexer Operations
Operational guide for running, health-checking, monitoring, and troubleshooting the Midnight indexer. For starting and stopping the local Docker stack, see midnight-tooling:devnet. For the full observability and troubleshooting reference, see references/monitoring-and-troubleshooting.md in this skill.
Deployment Recap
The indexer ships in two deployment shapes. For full detail see midnight-indexer:indexer-architecture → references/deployment-and-crates.md.
| Shape | Services | Storage | Bus |
|---|
| Standalone | Single binary (indexer-standalone) | SQLite | In-process |
| Cloud | 4 services: chain-indexer, wallet-indexer, indexer-api, spo-indexer | PostgreSQL | NATS |
Default port: 8088 (GraphQL + health probes on the same listener).
Health Checks
The indexer exposes two HTTP probes on the GraphQL port (default 8088). Both are served by the same Axum router as the GraphQL endpoint.
| Endpoint | Kubernetes role | Returns |
|---|
GET /live | Liveness | 200 OK — always, as long as the async runtime is responsive |
GET /ready | Readiness | 200 OK when caught up with the node; 503 Service Unavailable with body "indexer has not yet caught up with the node" otherwise |
Source: indexer-api/src/infra/api.rs:277–309.
curl -sf http://localhost:8088/live
curl -o - -w "\nHTTP %{http_code}\n" http://localhost:8088/ready
The /live handler runs in the async task executor; if a storage-layer deadlock parks the runtime, the liveness probe times out rather than returning 200, which lets Kubernetes terminate and recreate the pod.
Docker / Kubernetes Patterns
The official Docker Compose stack defines a HEALTHCHECK for each service that tests a running-file: the entrypoint creates /var/run/<component>/running on startup and removes it on exit (via trap … EXIT), and the healthcheck command is cat /var/run/<component>/running. The image Dockerfiles themselves embed no HEALTHCHECK. For Kubernetes — or to gate traffic on actual readiness rather than liveness — prefer the HTTP probes:
livenessProbe:
httpGet:
path: /live
port: 8088
initialDelaySeconds: 15
periodSeconds: 30
readinessProbe:
httpGet:
path: /ready
port: 8088
initialDelaySeconds: 30
periodSeconds: 10
healthcheck:
test: ["CMD", "curl", "-sf", "http://localhost:8088/ready"]
interval: 30s
timeout: 5s
retries: 3
start_period: 60s
Caught-Up Status
"Caught up" governs whether /ready returns 200 and whether the indexer_caught_up gauge is 1.
| Config key | Default | Meaning |
|---|
caught_up_max_distance | 10 | Max blocks behind the node before caught_up = false |
caught_up_leeway | 5 | Hysteresis — must catch within this distance before caught_up flips back to true |
The formula: caught_up = (node_height − indexed_height) ≤ caught_up_max_distance, with caught_up_leeway preventing rapid oscillation at the threshold. A 503 from /ready during initial sync is normal and expected — the indexer must process all historical blocks before it reports ready.
Configuration keys live under APP__ in environment or in config.yaml. See midnight-indexer:indexer-architecture → references/configuration-reference.md for the full key listing.
Metrics and Tracing
Both are disabled by default in standalone and cloud deployments. Enable them via environment variables or config.yaml.
Prometheus Metrics
APP__TELEMETRY__METRICS__ENABLED=true
APP__TELEMETRY__METRICS__ADDRESS=0.0.0.0
APP__TELEMETRY__METRICS__PORT=9000
Scrape endpoint: http://<host>:9000/metrics
| Metric | Type | Description |
|---|
indexer_block_height | counter | Highest block the indexer has processed |
indexer_node_block_height | counter | Latest block height reported by the node |
indexer_caught_up | gauge | 1 = caught up, 0 = behind |
indexer_transaction_count | counter | Total transactions indexed |
indexer_wallets_connected | gauge | Active wallet sessions |
indexer_subscriptions_active | gauge | Active GraphQL subscriptions |
indexer_subscriptions_rejected_total | counter | Subscriptions rejected by quota |
For the full metric catalog and example Grafana queries, see references/monitoring-and-troubleshooting.md.
OpenTelemetry Tracing
APP__TELEMETRY__TRACING__ENABLED=true
APP__TELEMETRY__TRACING__OTLP_EXPORTER_ENDPOINT=http://localhost:4317
APP__TELEMETRY__TRACING__SERVICE_NAME=indexer
Spans are exported via gRPC OTLP to the configured endpoint. Logs are correlated with traces — when a log record is emitted inside a span, the trace ID is included in the JSON log output.
Node Connection and Recovery
The indexer connects to the Midnight node via subxt with a built-in reconnecting RPC client using exponential backoff.
| Config key | Default | Notes |
|---|
reconnect_max_delay | 10s | Backoff ceiling; steps are 10ms → 100ms → 1s → 10s |
reconnect_max_attempts | 30 | ≈ 5 minutes of retries before the process exits |
subscription_recovery_timeout | 30s | If no valid block arrives after a reconnect or duplicate event within this window, the subscription is torn down and re-established |
When the node disconnects, a WARN log line "node disconnected, reconnecting" is emitted. After reconnect_max_attempts failures the process exits so the container orchestrator can restart it.
Config keys live under APP__INFRA__NODE__ (the standalone binary also has a separate APP__INFRA__SPO_NODE__ for its Cardano-side connection). See midnight-indexer:indexer-architecture → references/configuration-reference.md.
Wallet Session Lifecycle
Wallet sessions let the indexer scan shielded transactions for a specific viewing key without exposing that key on-chain.
connect(viewingKey)
→ viewing key encrypted at rest (ChaCha20-Poly1305, key from APP__INFRA__SECRET)
→ wallet-indexer scans blocks for relevant transactions
→ client subscribes via shieldedTransactions(sessionId)
disconnect(sessionId)
→ session ends, resources released
| Config key | Default | Meaning |
|---|
active_wallets_ttl | 30m | Idle sessions are evicted after this duration |
keep_wallet_alive_interval | 1m | Heartbeat interval to keep long-running catch-up sessions alive |
APP__INFRA__SECRET must be set to a hex-encoded 32-byte key before the indexer will accept connect mutations. Missing or invalid SECRET causes startup failure or decryption errors at session creation time.
For the GraphQL mutations (connect, disconnect) and the shieldedTransactions subscription, see midnight-indexer:indexer-graphql-api.
NATS Failure Handling (Cloud)
In cloud mode, chain-indexer → wallet-indexer and chain-indexer → indexer-api communication flows through NATS (core pub-sub — the server runs with -js but the indexer creates no JetStream streams or consumers). Publisher and subscriber components log WARN on Event::ServerError and Event::ClientError from the NATS client library; a startup ConnectError causes the service to exit so the orchestrator can retry.
For subjects, consumer groups, and failure modes, see midnight-indexer:indexer-architecture → references/nats-messaging.md.
Troubleshooting
Quick-reference for common failure modes. Full diagnostic procedures and log patterns are in references/monitoring-and-troubleshooting.md.
| Symptom | Likely cause | First action |
|---|
/ready returns 503 continuously | Initial sync not complete, or node unreachable | Check indexer_block_height vs indexer_node_block_height; confirm node is reachable |
indexer_caught_up stuck at 0 | Block processing stalled or node connection lost | Check logs for "node disconnected" or DB errors |
connect mutation fails | APP__INFRA__SECRET missing or malformed | Verify the env var is set to a valid 64-char hex string |
| NATS startup failure (cloud) | NATS server not reachable at startup | Check APP__INFRA__PUB_SUB__URL; verify NATS container is healthy |
| DB migration errors | Schema version mismatch after upgrade | Check logs for sqlx migration errors; ensure the DB is accessible |
| Wallet sessions evicted unexpectedly | active_wallets_ttl too short for catch-up duration | Increase active_wallets_ttl or ensure the client heartbeats within the TTL |
| Blockfrost 402 errors (SPO indexer) | Blockfrost plan limit reached or invalid BLOCKFROST_ID | Verify APP__INFRA__SPO_NODE__BLOCKFROST_ID and plan quota |
curl -sf http://localhost:8088/live && echo "LIVE"
curl -o - -w "\nHTTP %{http_code}\n" http://localhost:8088/ready
curl -s http://localhost:9000/metrics | grep -E "indexer_(block_height|node_block_height|caught_up)"
docker logs --tail 50 indexer-standalone
docker logs -f indexer-standalone
References
| Name | Description | When used |
|---|
references/monitoring-and-troubleshooting.md | Full metric catalog, log pattern reference, Grafana queries, and detailed diagnostic procedures | Setting up production monitoring or investigating deep operational issues |
examples/running-standalone.md | Step-by-step: run the standalone indexer with health checks and metrics enabled | Getting the standalone binary operational end-to-end |
Cross-References
| Skill / Resource | Relevance |
|---|
midnight-tooling:devnet | Starting and stopping the full local Docker stack (node + indexer + proof server) |
midnight-indexer:indexer-architecture | Deployment modes, crate layout, and full configuration-reference / nats-messaging / deployment-and-crates references |
midnight-indexer:indexer-graphql-api | connect / disconnect mutations and the shieldedTransactions subscription |
midnight-indexer:indexer-data-model | What gets indexed, database schema, and block/transaction data shapes |
