midnight-indexer-indexer-architecture

name	midnight-indexer:indexer-architecture
description	This skill covers indexer architecture, components, chain-indexer, wallet-indexer, indexer-api, spo-indexer, standalone mode, cloud mode, configuration, deployment, storage, PostgreSQL vs SQLite, NATS, telemetry, or node connection. Covers "how does the indexer connect to a node", "what database does the indexer use", "how to run the indexer locally", "choosing between standalone and cloud deployment", and "indexer data flow".
version	0.1.0

Indexer Architecture

The Midnight indexer is a Rust application (edition 2024) built with async-graphql, axum, and subxt. It connects to a Midnight node via WebSocket, indexes on-chain data into a database, and exposes it through a GraphQL API.

Current version: 4.3.3 (released 2026-06-04)

Component Overview

┌──────────────────────────────────────────────────────────────────┐
│                      Midnight Indexer                            │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌────────────────┐   ┌────────────────┐   ┌────────────────┐   │
│  │ chain-indexer  │──→│       DB       │←──│  indexer-api   │   │
│  │ (node via WS)  │   │ (Postgres/     │   │  (GraphQL      │   │
│  │                │   │  SQLite)       │   │   :8088)       │   │
│  └────────────────┘   └────────────────┘   └────────────────┘   │
│                                                                  │
│  ┌────────────────┐   ┌────────────────┐   ┌────────────────┐   │
│  │ wallet-indexer │   │  spo-indexer   │   │ indexer-common │   │
│  │ (wallet        │   │ (stake pool    │   │ (shared types/ │   │
│  │  correlation)  │   │  via Blockfrost│   │  migrations)   │   │
│  └────────────────┘   └────────────────┘   └────────────────┘   │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ indexer-standalone (all-in-one with SQLite)              │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

Components

Component	Purpose
chain-indexer	Connects to node via WebSocket, subscribes to new blocks, writes indexed data to DB
wallet-indexer	Manages wallet sessions using encrypted viewing keys (APP__INFRA__SECRET), scans blocks for relevant shielded transactions
indexer-api	Serves the GraphQL API on port 8088
spo-indexer	Indexes stake pool operator data via Blockfrost
indexer-standalone	All-in-one binary bundling all components with SQLite storage
indexer-common	Shared types, database migrations, and utilities

Data Flow

Midnight Node (ws://localhost:9944)
        │
        ▼
  Chain Indexer ──→ Database (PostgreSQL or SQLite)
                         │
                         ▼
                    Indexer API (GraphQL :8088)
                         │
                         ▼
                    DApp / Wallet

Deployment Modes

Cloud Mode

• Storage: PostgreSQL 17
• Messaging: NATS for inter-component pub-sub (see NATS Messaging below)
• Components: Run chain-indexer, wallet-indexer, indexer-api, and spo-indexer as separate services
• Use case: Production deployments, high availability

Standalone Mode

• Storage: SQLite (single file)
• Components: Single indexer-standalone binary bundles everything
• Use case: Local development, devnet, testing

Deployment Mode Comparison

Aspect	Standalone	Cloud
Storage	SQLite (single file)	PostgreSQL 17
Messaging	In-process (Tokio broadcast)	NATS
Scaling	Single process	Horizontal (separate services)
Use case	Local dev, devnet, testing	Production, high availability

See also: references/deployment-and-crates.md — the seven workspace crates, which binary each deployment runs, the cloud/standalone feature gates, and the two databases (main storage DB + the separate ledger DB).

NATS Messaging (Cloud Mode)

In cloud mode, the indexer components run as separate processes and use NATS (v2.12.x) as a lightweight pub-sub bus for inter-component event notification. In standalone mode, the same pub-sub interface is implemented with in-process Tokio broadcast channels — NATS is not used.

Deep dive: references/nats-messaging.md — the Publisher/Subscriber trait abstraction and its two implementations, exact payload struct fields, the 100ms self-healing resubscribe loop, and per-component publish/subscribe wiring.

Message Types

Message	Published by	Subscribed by	Purpose
BlockIndexed	chain-indexer	wallet-indexer, indexer-api	Signals a block has been indexed; carries height, optional max_transaction_id, caught_up flag
WalletIndexed	wallet-indexer	indexer-api	Signals new relevant transactions for a wallet session; carries wallet_id
UnshieldedUtxoIndexed	chain-indexer	indexer-api	Signals unshielded UTXOs indexed for a specific address; carries address

The spo-indexer does not use NATS.

NATS subjects follow the pattern pub-sub.<MessageType> (e.g., pub-sub.BlockIndexed). Messages are JSON-serialized.

Configuration

Environment Variable	Purpose	Default
APP__INFRA__PUB_SUB__URL	NATS server address	localhost:4222
APP__INFRA__PUB_SUB__USERNAME	Authentication username	indexer
APP__INFRA__PUB_SUB__PASSWORD	Authentication password	(required, no default)
APP__INFRA__PUB_SUB__MAX_RECONNECTS	Reconnection attempts after disconnect	4

In Docker deployments, the NATS server runs with --user indexer --pass <password> -js. Connections are plaintext (no TLS).

Failure Modes

Scenario	Behavior
NATS unavailable at startup	Process exits with connection error — no retry loop on initial connect
NATS disconnects after startup	async-nats attempts up to max_reconnects (default 4) reconnections automatically
Subscriber stream completes	Self-healing outer stream resubscribes every 100ms until NATS is available
Publisher fails to publish	Error propagates up; chain-indexer stops block indexing, wallet-indexer stops wallet task
Subscriber is down when message published	Message is lost (core NATS, not JetStream); component catches up by polling the database on reconnect
Slow consumer	Logged as warning; NATS may drop messages for the slow subscriber

NATS Events Logged

Event	Log Level
Connected	DEBUG
Disconnected	WARN
LameDuckMode	WARN
Draining	WARN
Closed	WARN
SlowConsumer	WARN
ServerError	WARN
ClientError	WARN

Configuration

All configuration uses environment variables with the APP__ prefix and double-underscore nesting. The keys below are the most common; for the complete catalog (storage pooling, per-subscription batch sizes, quota, ledger DB, SPO refresh, telemetry) see references/configuration-reference.md.

Core Environment Variables

Variable	Default	Purpose
APP__APPLICATION__NETWORK_ID	undeployed	Network identifier
APP__INFRA__STORAGE__CNN_URL	/data/indexer.sqlite	SQLite path (standalone) or PostgreSQL connection URL (cloud)
APP__INFRA__NODE__URL	ws://localhost:9944	Node WebSocket endpoint
APP__INFRA__API__PORT	8088	GraphQL API port
APP__INFRA__SECRET	(required)	32-byte hex encryption secret for wallet viewing keys

Generate a secret: openssl rand -hex 32

Operational Parameters

Parameter	Value	Description
blocks_buffer	10	Number of blocks to buffer during indexing
caught_up_max_distance	10	Max blocks behind tip to consider "caught up"
active_wallets_ttl	30 minutes	Time-to-live for inactive wallet sessions
transaction_batch_size	50	Transactions processed per batch
Node reconnection	Exponential backoff, max 30 attempts	Automatic reconnection on WebSocket disconnect
Subscription recovery timeout	30 seconds	Timeout for recovering dropped subscriptions

Node Connection

The indexer connects to a Midnight node via WebSocket on port 9944 using the subxt Rust library. It subscribes to new finalized blocks and processes them sequentially, extracting transactions, contract actions, and ledger events.

Telemetry

Both telemetry systems are disabled by default.

System	Protocol	Default Port	Purpose
OpenTelemetry	OTLP	—	Distributed tracing
Prometheus	HTTP scrape	9000	Metrics collection

References

Name	Description	When used
references/deployment-and-crates.md	The seven workspace crates, each binary's role, the cloud/standalone feature gates, standalone vs cloud topology, and the main-DB / ledger-DB split	When choosing a deployment mode or understanding which component does what
references/nats-messaging.md	NATS pub-sub deep-dive: the trait abstraction, both implementations, payload fields, the self-healing subscriber loop, and per-component wiring	When debugging inter-component messaging or implementing against the pub-sub layer
references/configuration-reference.md	The complete APP__* env/config catalog from the serde structs and config.yaml files, with defaults and standalone-vs-cloud differences	When configuring any indexer component or looking up a specific key's default

Cross-References

• midnight-tooling:devnet — Manages the indexer as part of the local development stack
• midnight-indexer:indexer-operations — Running, health-checking, monitoring, and troubleshooting the indexer
• midnight-dapp-dev:midnight-sdk — Uses the indexer as a provider for contract state queries