| name | midnight-node:node-operations |
| description | Running the Midnight node, validator mode, full node, archive node, dev mode, Docker deployment, midnight-node-docker, Prometheus metrics, node monitoring, memory monitoring, graceful shutdown, P2P networking, bootnodes, node keys, port 30333, node troubleshooting, blocks not finalizing, mainchain follower errors, IPC errors, state sync issues, node logs, node peers, node health checks, start a node, deploy a node, node not syncing, node crashes, out of memory, check node status. |
| version | 0.1.0 |
Node Operations
Operational guide for running, monitoring, and troubleshooting the Midnight node. For configuration options, see midnight-node:node-configuration. For architecture internals, see midnight-node:node-architecture.
Running Modes
Validator Node
Produces blocks and participates in consensus. Requires all validator keys.
CFG_PRESET=preview midnight-node \
--validator \
--name "my-validator"
Validator nodes must have:
- AURA, GRANDPA, and CROSS_CHAIN keys configured (see
node-configuration)
- Sufficient peers for consensus participation
- Stable network connectivity and uptime
Full Node
Syncs and validates the chain without producing blocks. Serves RPC requests and provides network capacity.
CFG_PRESET=preview midnight-node \
--name "my-full-node" \
--rpc-external --rpc-cors all
Security Warning: --rpc-external --rpc-cors all exposes the RPC interface to all network interfaces without restriction. Use only for local development. In production, use a reverse proxy with TLS and restrict CORS origins.
Archive Node
Retains all historical state. Required for block explorers, indexers, and historical queries.
CFG_PRESET=preview midnight-node \
--state-pruning archive \
--blocks-pruning archive \
--name "my-archive-node"
Development Mode
Single-node ephemeral chain for local development and testing.
midnight-node --dev --tmp
Development mode automatically:
- Creates a single-validator chain
- Uses pre-funded development accounts
- Enables all RPC methods
- Uses ephemeral storage (deleted on shutdown with
--tmp)
Hardware Requirements
Minimum recommended specifications by node type:
| Node Type | CPU | RAM | Storage |
|---|
| Full node | 4 cores | 8 GB | 100 GB SSD |
| Archive node | 4 cores | 16 GB | 500 GB SSD |
| Validator | 8 cores | 16 GB | 200 GB SSD (NVMe recommended) |
These are estimates for a Substrate-based chain and may increase as chain state grows. Monitor disk usage and plan capacity accordingly.
Docker Deployment
The Midnight node is distributed as a Docker image via midnight-node-docker.
Replace <version> with the latest release tag. Check the Midnight release notes for the current version.
docker pull midnightntwrk/midnight-node:<version>
docker run -d --name midnight-node \
-p 9944:9944 \
-p 30333:30333 \
-p 9615:9615 \
-v midnight-data:/data \
-e CFG_PRESET=preview \
midnightntwrk/midnight-node:<version> \
midnight-node \
--rpc-external --rpc-cors all \
--prometheus-external
Security Warning: --rpc-external --rpc-cors all exposes the RPC interface to all network interfaces without restriction. Use only for local development. In production, use a reverse proxy with TLS and restrict CORS origins.
Port Mapping
| Port | Protocol | Purpose |
|---|
| 9944 | WebSocket | JSON-RPC API |
| 30333 | TCP | P2P networking |
| 9615 | HTTP | Prometheus metrics |
Volume Mounts
| Path | Purpose |
|---|
/data | Chain database and ledger storage |
/keys | Validator key files (validator mode only) |
These mount paths are conventions defined by the separate midnight-node-docker repository (the Compose tooling), not by the node source itself.
Monitoring
Prometheus Metrics
The node exposes Prometheus metrics on port 9615 by default.
CFG_PRESET=preview midnight-node --prometheus-external
curl http://localhost:9615/metrics
Key metrics to monitor:
| Metric | Description |
|---|
substrate_block_height | Current best and finalized block heights |
substrate_number_leaves | Number of known chain leaves (aka forks) |
substrate_sub_libp2p_peers_count | Number of connected peers |
substrate_ready_transactions_number | Transactions in the ready queue |
substrate_sync_is_major_syncing | Whether the node is in major sync (1) or not (0) |
substrate_block_verification_and_import_time | Block verification + import time (histogram) |
Deep dive: references/metrics-and-monitoring.md — the full metric catalog (with correct upstream substrate_* names), the Prometheus remote-write push (PROMETHEUS_PUSH_ENDPOINT → /api/v1/receive), the memory/storage monitors, and the one Midnight-specific metric.
Push Endpoint
The node supports pushing metrics to a Prometheus remote write receiver (HTTP POST to /api/v1/receive, targeting Thanos, Cortex, or Mimir) via the PROMETHEUS_PUSH_ENDPOINT environment variable.
PROMETHEUS_PUSH_ENDPOINT=https://thanos:9091/api/v1/receive CFG_PRESET=preview midnight-node
Memory Monitoring
The node includes a configurable memory monitoring system. When available memory drops below the configured threshold, the node initiates a graceful shutdown to prevent out-of-memory crashes.
| Parameter | Description |
|---|
memory_threshold | Required available memory in MiB; the node shuts down gracefully when available memory drops below it. Default 0 (disabled), e.g., 512 |
The node logs memory usage periodically and emits a warning before initiating shutdown, allowing the process manager (systemd, Docker, Kubernetes) to restart it cleanly.
P2P Networking
Bootnodes
Bootnodes are the initial peers the node contacts to discover the network. Each network preset includes default bootnodes.
CFG_PRESET=preview midnight-node \
--bootnodes /dns4/bootnode.example.com/tcp/30333/p2p/12D3KooW...
Node Keys
Each node has a unique libp2p identity derived from a node key. The key is auto-generated on first run and stored in the database directory.
midnight-node key generate-node-key --file /keys/node.key
CFG_PRESET=preview midnight-node --node-key-file /keys/node.key
P2P Port
The default P2P port is 30333. Ensure this port is open for inbound connections if the node should be reachable by peers.
CFG_PRESET=preview midnight-node --port 30333 --listen-addr /ip4/0.0.0.0/tcp/30333
Troubleshooting
Blocks Not Finalizing
| Symptom | Possible Cause | Resolution |
|---|
| Best block increases but finalized block is stuck | GRANDPA consensus stalled | Check peer count (system_health RPC); ensure validator keys are correct |
| Finalized block lags far behind best block | Network partition or insufficient validators | Verify bootnodes are reachable; check GRANDPA round state via grandpa_roundState |
| No blocks produced at all | AURA authority not in active set | Verify AURA key is registered; check epoch configuration |
Mainchain Follower Errors
| Symptom | Possible Cause | Resolution |
|---|
MainchainFollower connection failed | PostgreSQL db-sync unreachable | Verify db-sync is running and PostgreSQL connection string is correct |
MainchainFollower sync behind | db-sync is still syncing Cardano chain | Wait for db-sync to reach tip; check db-sync logs |
| Partner chain data stale | db-sync stopped or restarting | Restart db-sync; verify Cardano node connectivity |
IPC Errors
| Symptom | Possible Cause | Resolution |
|---|
| IPC socket connection refused | Node process crashed or socket file stale | Restart the node; remove stale socket files from data directory |
| IPC timeout | Node under heavy load | Increase system resources; check CPU and memory usage |
State Sync Issues
| Symptom | Possible Cause | Resolution |
|---|
| Sync stalls at a specific block | Corrupted state or database | Clear chain data and resync: remove database directory, restart node |
State database error | Disk full or I/O error | Check disk space; verify storage health |
| Very slow sync | Insufficient peers or slow network | Add more bootnodes; verify network connectivity; increase --db-cache |
Diagnostic Commands
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"system_health","params":[]}' \
http://localhost:9944
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"system_syncState","params":[]}' \
http://localhost:9944
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"system_peers","params":[]}' \
http://localhost:9944
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"grandpa_roundState","params":[]}' \
http://localhost:9944
Worked example: examples/diagnostics-and-deployment.md — executed system_health / system_syncState / grandpa_roundState calls with real captured output, plus Docker and systemd deployment templates.
References
| Name | Description | When used |
|---|
references/metrics-and-monitoring.md | The node's observability surface: Prometheus scrape metrics, remote-write push, memory/storage monitors, and log signals | When setting up monitoring or interpreting metrics |
Examples
| Name | Description | When used |
|---|
examples/diagnostics-and-deployment.md | Executed diagnostic RPC calls (health, sync, finality) with real output, plus Docker and systemd deployment templates | When checking node health or deploying a node |
Cross-References
midnight-node:node-configuration — Detailed configuration flags and environment variablesmidnight-node:node-rpc-api — RPC methods used for monitoring and diagnosticsmidnight-node:node-validator — Running a validator end-to-end (keys, candidacy, committee, block production)midnight-tooling:devnet — Local development stack that manages node lifecycle automatically
