devnet-runner

name

devnet-runner

description

Manage local development networks (devnets) for lean consensus multi-client testing. This skill should be used when the user asks to run a devnet, start or stop devnet nodes, spin up a local testnet, configure validator nodes, regenerate genesis files, change Docker image tags, collect or dump node logs, troubleshoot devnet issues, restart a node with checkpoint sync, run a long-lived devnet with detached containers, or perform rolling restarts to upgrade images.

Devnet Runner

Manage local development networks for lean consensus testing involving gean and peer clients (zeam, ream, lantern, ethlambda).

Prerequisites

The lean-quickstart directory must exist alongside the gean repo. If missing:

make lean-quickstart

Default Behavior

When starting a devnet, always:

1. Update validator config — Edit lean-quickstart/local-devnet/genesis/validator-config.yaml to include ONLY the nodes that will run. Remove entries for nodes that won't be started (unless the user explicitly asks to keep them). Validator indices are assigned to ALL nodes in the config; if a node is in the config but not running, its validators will miss their proposer slots. To control which nodes run, always edit this config file rather than using --node <specific>, since --node does NOT reassign validators and causes missed slots.
2. Update client image tags — If the user specifies a tag (e.g., "use dev tag for gean"), edit the relevant lean-quickstart/client-cmds/{client}-cmd.sh file to update the node_docker image tag.
3. Use run-devnet-with-timeout.sh — This script runs all nodes in the config with a timeout, dumps logs, then stops them.
4. Run for 20 slots unless the user specifies otherwise.
5. The script automatically dumps all node logs to <node_name>.log files in the gean repo root and stops the nodes when the timeout expires.

Timing Calculation

Total timeout = startup buffer + genesis offset + (slots × 4 seconds)

Component	Local Mode	Ansible Mode
Startup buffer	10s	10s
Genesis offset	30s	360s
Per slot	4s	4s

Examples (local mode):

• 20 slots: 10 + 30 + (20 × 4) = 120s
• 50 slots: 10 + 30 + (50 × 4) = 240s
• 100 slots: 10 + 30 + (100 × 4) = 440s

Quick Start (Default Workflow)

Step 1: Configure nodes — Edit lean-quickstart/local-devnet/genesis/validator-config.yaml to keep only the nodes you want to run. See references/validator-config.md for the full schema and field reference.

Step 2: Update image tags (if needed) — Edit lean-quickstart/client-cmds/{client}-cmd.sh to change the Docker image tag in node_docker. See references/clients.md for current default tags.

Step 3: Run the devnet

# Start devnet with fresh genesis, capture logs directly (20 slots = 120s)
.claude/skills/devnet-runner/scripts/run-devnet-with-timeout.sh 120

Manual Commands

All spin-node.sh commands must be run from within lean-quickstart/:

# Stop all nodes
cd lean-quickstart && NETWORK_DIR=local-devnet ./spin-node.sh --node all --stop

# Run for custom duration (e.g., 50 slots = 240s with genesis offset)
.claude/skills/devnet-runner/scripts/run-devnet-with-timeout.sh 240

# Start without timeout (press Ctrl+C to stop)
cd lean-quickstart && NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis

Command-Line Flags

Flag	Description
`--node <name	all>`
--generateGenesis	Regenerate genesis files. Implies --cleanData
--cleanData	Clean data directories before starting
--stop	Stop running nodes instead of starting them
--forceKeyGen	Force regeneration of hash-sig validator keys
--validatorConfig <path>	Custom config path (default: $NETWORK_DIR/genesis/validator-config.yaml)
--dockerWithSudo	Run docker commands with sudo

Changing Docker Image Tags

To use a specific tag for certain clients, edit the lean-quickstart/client-cmds/{client}-cmd.sh files before running.

Example: Change gean from dev to a branch tag:

# In lean-quickstart/client-cmds/gean-cmd.sh, find:
node_docker="--security-opt seccomp=unconfined gean:dev node \

# Change to:
node_docker="--security-opt seccomp=unconfined gean:my-feature-branch node \

See references/clients.md for current default images, tags, and known compatibility issues.

gean-Specific Notes

• gean uses --api-port (default 5058) and --metrics-port (default 8088) per the gean Makefile defaults. The standalone make run uses different defaults — check the lean-quickstart/client-cmds/gean-cmd.sh for the devnet defaults.
• gean expects --custom-network-config-dir, --node-key, --node-id, --data-dir, --gossipsub-port, --api-port, --metrics-port, and optionally --is-aggregator and --checkpoint-sync-url.
• To configure gean as the aggregator in a devnet, ensure --is-aggregator is set in gean-cmd.sh for exactly one gean instance.

Validator Configuration

See references/validator-config.md for the full schema, field reference, adding/removing nodes, port allocation guide, and local vs ansible deployment differences.

Log Collection

View Live Logs

docker logs gean_0           # View current logs
docker logs -f gean_0        # Follow/stream logs

Dump Logs to Files

Automatic: When using run-devnet-with-timeout.sh, logs are automatically dumped to <node_name>.log files in the gean repo root before stopping.

Single node (manual):

docker logs gean_0 &> gean_0.log

All running nodes (manual):

for node in $(docker ps --format '{{.Names}}' | grep -E '^(gean|zeam|ream|lantern|ethlambda)_'); do
  docker logs "$node" &> "${node}.log"
done

Data Directory Logs

Client-specific data and file-based logs are stored at:

lean-quickstart/local-devnet/data/<node_name>/

Common Troubleshooting

Nodes Won't Start

1. Check if containers are already running:

   docker ps | grep -E 'gean|zeam|ream|lantern|ethlambda'
   

2. Stop existing nodes first:

   cd lean-quickstart && NETWORK_DIR=local-devnet ./spin-node.sh --node all --stop
   

Nodes Not Finding Peers

1. Verify all nodes are using the same genesis:

   cd lean-quickstart && NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis
   

2. Check nodes.yaml was generated with correct ENR records

Genesis Mismatch Errors

Regenerate genesis for all nodes:

cd lean-quickstart && NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --forceKeyGen

Port Conflicts

Check if ports are in use:

lsof -i :9008  # Check gean QUIC port
lsof -i :8088  # Check gean metrics port
lsof -i :5058  # Check gean API port

Stale Containers Cause Genesis Mismatch

If you see UnknownSourceBlock or OutOfMemory deserialization errors, a container from a previous run may still be running with old genesis.

Fix: Always clean up before starting a new devnet:

docker rm -f gean_0 zeam_0 ream_0 lantern_0 ethlambda_0 2>/dev/null

Or use run-devnet-with-timeout.sh which handles cleanup automatically.

Docker Permission Issues

cd lean-quickstart && NETWORK_DIR=local-devnet ./spin-node.sh --node all --dockerWithSudo

Scripts

Script	Description
scripts/run-devnet-with-timeout.sh <seconds>	Run devnet for specified duration, dump logs to gean repo root, then stop

Long-Lived Devnets and Rolling Restarts

For persistent devnets on remote servers (e.g., ssh admin@gean-1), use detached containers instead of spin-node.sh. This allows rolling restarts to upgrade images without losing chain state.

Key points:

• Start containers with docker run -d --restart unless-stopped (not spin-node.sh)
• Rolling restart: stop one node, wait 60 seconds (gossipsub backoff), start with new image + checkpoint sync
• Restart non-aggregator nodes first, aggregator last
• Checkpoint sync URL uses gean's API port: http://127.0.0.1:<api-port>/lean/v0/states/finalized

See references/long-lived-devnet.md for the full procedure.

Reference

• references/clients.md: Client-specific details (images, ports, known issues)
• references/validator-config.md: Full config schema, field reference, adding/removing nodes, port allocation
• references/long-lived-devnet.md: Persistent devnets with detached containers and rolling restarts