| name | devnet-ops |
| description | Operate, inspect, monitor, and debug ethrex EL nodes on ethpandaops devnets (status sweeps, peers/logs, blob & fork tracking, Hive results, wipe/resync, incident history). Use whenever the user asks to check/inspect a devnet node, ssh into one, read logs, investigate a devnet error, track blob inclusion or fork schedule, see Hive conformance, deploy/wipe a node, or ask about past devnet problems. |
devnet-ops
This ethrex devnet ops + monitoring toolkit lives in the devnets_monitor/
directory of the ethrex-tooling repo (ported from the standalone ethrex-devnets
repo). It is self-contained: procedures, per-devnet history, and the dv CLI all
live here. Run every command from devnets_monitor/ (cd devnets_monitor first);
all paths below are relative to that directory.
Before any devnet operation or answering a question about a node/incident, READ:
CLAUDE.md — the working agreement (golden rules, exact data sources, the wipe
sequence).docs/devnet-ops.md — generic access & inspection procedures (SSH, inventory,
container layout, build & deploy, debug logging, wipe & resync, Dora API).
Substitute <devnet> with the target network.docs/history/<devnet>.md — per-devnet facts and incident history (roster, fork
schedule, commit map, known issues with root cause + recovery). These files are
gitignored (local-only; they embed host internals), so a freshly-cloned tree has
none. If a devnet has no history file, create one from
docs/history/_template.md as you learn facts.
The dv CLI
Run from devnets_monitor/ via uv run dv .... Read-only by default; only dv wipe
mutates (gated behind --yes). Target devnet resolves: explicit arg > $DEVNET
env > config/devnets.yaml default.
uv run dv discover <devnet> # refresh roster/forks/image from the ethpandaops repo (gh)
uv run dv status <devnet> [node] # EL build/head/peers/state@head + CL + watchtower (--json)
uv run dv peers <devnet> <node> # peer count, inbound/outbound, client mix, body-serving fails
uv run dv logs <devnet> <node> [--since 2m]
uv run dv cl <devnet> <node> [--since 3m]
uv run dv collect <devnet> [all|blobs|health|hive|forks] # into data/ethrex-devnets.sqlite
uv run dv blob <devnet> # blob inclusion per proposer + ethrex-vs-others (decay lens)
uv run dv fork <devnet> # fork schedule + EIPs + countdown
uv run dv hive <devnet> # Hive conformance summary (groups from config/devnets.yaml)
uv run dv serve # read-only dashboard at http://127.0.0.1:8099
uv run dv wipe <devnet> <node> --yes # MUTATING: recover a wedged EL
Workflow
- Read
docs/devnet-ops.md for HOW (procedures).
- Read
docs/history/<devnet>.md for WHAT/WHY — check whether the current symptom
matches a known issue before investigating from scratch; many recur.
- ALWAYS verify the live host before trusting a node name.
dv status reads the
live docker inspect execution image; a *-ethrex-* node may have been swapped
to another client.
- Default to read-only.
dv wipe and any deploy/recreate are mutating; confirm
with the user first.
Adding a devnet
- Add an entry under
devnets: in config/devnets.yaml (see config/schema.md).
uv run dv discover <devnet> to populate config/devnets/<devnet>.yaml.- Create
docs/history/<devnet>.md from docs/history/_template.md.
Maintenance
When you discover a new incident, divergence, or devnet fact, append a dated entry
to docs/history/<devnet>.md (local-only, gitignored; do not commit it). If a
procedure changed, update docs/devnet-ops.md and commit that. The fork -> EIP map
is config/eips.json (sourced via eipmcp; re-run get_hardfork to refresh).
