ops-check-balances

name

ops-check-balances

description

Internal — for Boundless team members only. Audit native ETH, market deposit, prover collateral, and distributor ZKC reserve balances for every operator-managed address (provers, distributor, order generators, signal). Use when the user wants to know which addresses need topping up, asks about the balance of provers/OGs/distributor/signal signers, says something is "running low" or "out of gas", or wants a periodic operational health check on operator wallets. Defaults to prod env (mainnets + prod testnets) — pass `--all` to also include staging.

Check Operator Balances

Audit balances on every Boundless-operated address: native ETH, market deposit (balanceOf), prover collateral (balanceOfCollateral), and the distributor's bridged-ZKC ERC20 reserve.

The list of addresses, per-role thresholds, and which checks apply (gas vs market deposit) are loaded at runtime from the runbooks repo. The skill itself contains only public protocol data (chain → market/collateral-token addresses, public RPCs).

Prerequisites

1. CLI tools: cast (from foundry), jq, python3, curl, and standard Unix utilities. python3 is used for wei → human-readable conversion (awk silently truncates large integers) and for parsing deployment.toml. curl fetches the contract registry from the public boundless repo.

2. gh CLI authenticated against boundless-xyz (required). The canonical address list and thresholds live in the private boundless-xyz/runbooks repo and are fetched at runtime via gh api — no local clone needed. Verify with:

   gh api -H "Accept: application/vnd.github.raw" \
     repos/boundless-xyz/runbooks/contents/addresses/operator_addresses.json | head -5
   

   If that fails with a 401/403, run gh auth login and ensure your account has access to boundless-xyz/runbooks.

3. network_secrets.toml at the repo root (optional). Public RPCs are usually sufficient. When a balance comes back as exactly 0 it may be rate-limited rather than actually empty — in that case retry against the private QuikNode endpoint from [networks.<chain>.rpc]. The runbook has setup instructions.

What to check

For each address in operator_addresses.json, on every chain it operates on:

1. Native ETH balance (when needs_gas: true) — cast balance --rpc-url <RPC> <addr>. Required to pay gas.
2. Market deposit (when needs_market_check: true) — cast call <market> "balanceOf(address)(uint256)" <addr>. Funds available to pay for orders (requestor) or to claim as rewards (prover).
3. Prover collateral (when role: prover) — cast call <market> "balanceOfCollateral(address)(uint256)" <addr>. Stake the prover has put up.
4. Distributor ZKC reserve (when role: distributor) — cast call <collateral_token> "balanceOf(address)(uint256)" <distributor_addr>. The distributor's raw ERC20 balance of bridged ZKC, used to top up provers' collateral. Not the same as balanceOfCollateral — that's deposited stake; this is the unstaked reserve.

Thresholds

Loaded from operator_addresses.json's thresholds block. Resolved field-by-field with priority:

per-entry override → by_role[role] → by_chain[chain] → default

ETH thresholds for distributor-managed roles (provers, OGs) come from by_chain because the distributor's ETH_THRESHOLD varies by ~3 orders of magnitude across chains (0.008 ETH on Taiko vs 10 ETH on Eth Sepolia). Audit thresholds are tied to those values: WARN ≈ 0.5 × ETH_THRESHOLD (distributor missed at least one cycle), CRIT ≈ 0.125 × ETH_THRESHOLD (auto-top-up clearly broken).

ETH thresholds for non-distributor-managed roles (distributor itself, signal signers) come from by_role and ignore the chain. Distributor's WARN matches the operational DISTRIBUTOR_ETH_ALERT_THRESHOLD of 0.5; signal signers use a smaller threshold tuned to their ~0.0001 ETH/day burn.

ZKC thresholds (prover collateral, distributor reserve) come from by_role.prover and by_role.distributor respectively, since they're uniform across chains.

See runbooks/addresses/README.md for the full table of values and rationale.

A specific entry can override any field by setting a thresholds: { eth_warn: ..., ... } block on it — only the fields you set are overridden; the rest fall through.

Public chain → contract addresses

Contract addresses (market, collateral token) are fetched at runtime from the public contracts/deployment.toml in the boundless repo on GitHub. The skill never hardcodes them — when a chain is added or addresses rotate, it's a single edit to deployment.toml and the skill picks it up on next run, no code change.

DEPLOYMENT_TOML_URL="https://raw.githubusercontent.com/boundless-xyz/boundless/main/contracts/deployment.toml"

The chain label used in operator_addresses.json must match a [deployment.<label>] section in deployment.toml. Today that's base-mainnet, taiko-mainnet, base-sepolia, ethereum-sepolia, base-sepolia-staging, taiko-staging.

The only inline data the skill keeps is the chain label → public RPC URL mapping below — RPC URLs aren't in deployment.toml and don't rotate.

Chain label	Public RPC
base-mainnet	https://mainnet.base.org
taiko-mainnet	https://rpc.taiko.xyz
base-sepolia	https://sepolia.base.org
ethereum-sepolia	https://ethereum-sepolia-rpc.publicnode.com
base-sepolia-staging	https://sepolia.base.org
taiko-staging	https://rpc.taiko.xyz

The check script

Save as /tmp/balance_check.sh and run. Defaults to prod env (mainnets + prod testnets); pass --all to also include staging.

#!/usr/bin/env bash
set -euo pipefail

# Fetch the operator address registry directly from the private runbooks repo
# via gh CLI (no local clone needed). RUNBOOKS_REF can be set to fetch from a
# specific branch/tag/sha (default: main).
RUNBOOKS_REF="${RUNBOOKS_REF:-main}"
ADDRS_FILE=$(mktemp)
trap 'rm -f "$ADDRS_FILE" "${DEPLOYMENT_TOML:-}"' EXIT
if ! gh api -H "Accept: application/vnd.github.raw" \
       "repos/boundless-xyz/runbooks/contents/addresses/operator_addresses.json?ref=$RUNBOOKS_REF" \
       > "$ADDRS_FILE" 2>/dev/null; then
  echo "ERROR: failed to fetch operator_addresses.json from boundless-xyz/runbooks" >&2
  echo "       Run 'gh auth login' and ensure your account has access to that repo." >&2
  exit 1
fi

# Default: prod only. --all also includes staging.
INCLUDE_STAGING="${INCLUDE_STAGING:-false}"
[ "${1:-}" = "--all" ] && INCLUDE_STAGING=true

# Public RPC URLs (the only inline data — they're not in deployment.toml and
# don't rotate). Function-based for macOS bash 3.2 compatibility.
chain_rpc() {
  case "$1" in
    base-mainnet)           echo "https://mainnet.base.org" ;;
    taiko-mainnet)          echo "https://rpc.taiko.xyz" ;;
    base-sepolia)           echo "https://sepolia.base.org" ;;
    ethereum-sepolia)       echo "https://ethereum-sepolia-rpc.publicnode.com" ;;
    base-sepolia-staging)   echo "https://sepolia.base.org" ;;
    taiko-staging)          echo "https://rpc.taiko.xyz" ;;
    *) echo "" ;;
  esac
}

# Fetch the canonical contract address registry from the public boundless repo
# and build a chain_label -> "market|collateral" map. Keeps the skill self-updating
# when contracts/deployment.toml changes.
DEPLOYMENT_TOML_URL="${DEPLOYMENT_TOML_URL:-https://raw.githubusercontent.com/boundless-xyz/boundless/main/contracts/deployment.toml}"
DEPLOYMENT_TOML=$(mktemp)
if ! curl -fsSL "$DEPLOYMENT_TOML_URL" -o "$DEPLOYMENT_TOML"; then
  echo "ERROR: failed to fetch $DEPLOYMENT_TOML_URL" >&2
  exit 1
fi

# Build a flat "label\tmarket\tcollateral" table by parsing the TOML once.
CHAIN_CONTRACTS=$(python3 - "$DEPLOYMENT_TOML" <<'PY'
import re, sys
t = open(sys.argv[1]).read()
zero = "0x0000000000000000000000000000000000000000"
for s in re.split(r'\n(?=\[deployment\.)', t):
    m = re.match(r'\[deployment\.([^\]]+)\]', s)
    if not m: continue
    name = m.group(1)
    market = re.search(r'^boundless-market\s*=\s*"(0x[a-fA-F0-9]{40})"', s, re.MULTILINE)
    coll   = re.search(r'^collateral-token\s*=\s*"(0x[a-fA-F0-9]{40})"', s, re.MULTILINE)
    if not market or market.group(1).lower() == zero: continue
    coll_addr = coll.group(1) if coll and coll.group(1).lower() != zero else ""
    print(f"{name}\t{market.group(1)}\t{coll_addr}")
PY
)

# label -> "market|collateral" lookup using the fetched data.
chain_contracts() {
  echo "$CHAIN_CONTRACTS" | awk -v c="$1" -F'\t' '$1==c {print $2"|"$3; exit}'
}

decode_uint() { echo "$1" | head -1 | awk '{print $1}' | grep -oE '^[0-9]+'; }
fmt_eth()     { python3 -c "v='${1:-}'; print(f'{int(v)/1e18:.6f}' if v else '?')" 2>/dev/null; }
fmt_zkc()     { python3 -c "v='${1:-}'; print(f'{int(v)/1e18:.4f}' if v else '?')" 2>/dev/null; }
flt_lt()      { python3 -c "import sys; sys.exit(0 if float('$1') < float('$2') else 1)" 2>/dev/null; }

# Build (entry × chain) tuples filtered by env, into TSV:
# label \t address \t role \t chain \t needs_gas \t needs_market \t eth_warn \t eth_crit
#
# Threshold resolution (field-by-field): per-entry override → by_role[role] → by_chain[chain] → default.
# `//` works correctly here because thresholds are numbers, not booleans (no false-collapse risk).
# But for the boolean flags `needs_gas` / `needs_market_check`, `//` would mistreat `false`,
# so we use `has()` for those.
TUPLES=$(jq -r --arg incl "$INCLUDE_STAGING" '
  . as $root
  | .addresses[]
  | . as $e
  | select($incl == "true" or .env == "prod" or .env == "all")
  | .chains[] as $c
  | (
      (($e.thresholds // {}).eth_warn)
      // (($root.thresholds.by_role[$e.role] // {}).eth_warn)
      // (($root.thresholds.by_chain[$c]    // {}).eth_warn)
      // ($root.thresholds.default.eth_warn)
      // 0.02
    ) as $eth_warn
  | (
      (($e.thresholds // {}).eth_crit)
      // (($root.thresholds.by_role[$e.role] // {}).eth_crit)
      // (($root.thresholds.by_chain[$c]    // {}).eth_crit)
      // ($root.thresholds.default.eth_crit)
      // 0.005
    ) as $eth_crit
  | [.label, .address, .role, $c,
     (if has("needs_gas") then .needs_gas else true end),
     (if has("needs_market_check") then .needs_market_check else false end),
     $eth_warn, $eth_crit]
  | @tsv
' "$ADDRS_FILE")

printf "%-36s %-12s %-22s %-12s %-13s %-12s %s\n" SERVICE ROLE CHAIN ETH DEPOSIT COLLATERAL FLAG
echo "----------------------------------------------------------------------------------------------------------------------"

while IFS=$'\t' read -r LABEL ADDR ROLE CHAIN NEEDS_GAS NEEDS_MARKET ETH_WARN ETH_CRIT; do
  RPC=$(chain_rpc "$CHAIN")
  CHAIN_INFO=$(chain_contracts "$CHAIN")
  if [ -z "$RPC" ] || [ -z "$CHAIN_INFO" ]; then
    printf "%-36s %-12s %-22s %s\n" "$LABEL" "$ROLE" "$CHAIN" "(chain not found — RPC or deployment.toml entry missing)"
    continue
  fi
  MKT="${CHAIN_INFO%%|*}"

  E="-"; D="-"; C="-"
  FLAG=""

  if [ "$NEEDS_GAS" = "true" ]; then
    E_RAW=$(cast balance --rpc-url "$RPC" "$ADDR" 2>/dev/null || true)
    E=$(fmt_eth "$E_RAW")
    if [ "$E" != "?" ] && [ "$E" != "-" ]; then
      if flt_lt "$E" "$ETH_CRIT"; then FLAG="$FLAG ETH-CRIT"
      elif flt_lt "$E" "$ETH_WARN"; then FLAG="$FLAG ETH-LOW"
      fi
    fi
  fi

  if [ "$NEEDS_MARKET" = "true" ] && [ -n "$MKT" ]; then
    D_RAW=$(decode_uint "$(cast call --rpc-url "$RPC" "$MKT" 'balanceOf(address)(uint256)' "$ADDR" 2>/dev/null || true)")
    D=$(fmt_eth "$D_RAW")
  fi

  if [ "$ROLE" = "prover" ] && [ -n "$MKT" ]; then
    C_RAW=$(decode_uint "$(cast call --rpc-url "$RPC" "$MKT" 'balanceOfCollateral(address)(uint256)' "$ADDR" 2>/dev/null || true)")
    C=$(fmt_zkc "$C_RAW")
    # Collateral thresholds are uniform; pulled from by_role.prover (or default).
    C_WARN=$(jq -r '.thresholds.by_role.prover.collateral_warn // .thresholds.default.collateral_warn // 100' "$ADDRS_FILE")
    C_CRIT=$(jq -r '.thresholds.by_role.prover.collateral_crit // .thresholds.default.collateral_crit // 50'  "$ADDRS_FILE")
    if [ "$C" != "?" ] && [ "$C" != "-" ]; then
      if flt_lt "$C" "$C_CRIT"; then FLAG="$FLAG ZKC-CRIT"
      elif flt_lt "$C" "$C_WARN"; then FLAG="$FLAG ZKC-LOW"
      fi
    fi
  fi

  printf "%-36s %-12s %-22s %-12s %-13s %-12s%s\n" "$LABEL" "$ROLE" "$CHAIN" "$E" "$D" "$C" "$FLAG"
done <<<"$TUPLES"

# Distributor bridged-ZKC reserve (one row per (distributor address × its chains))
echo
echo "=== Distributor bridged-ZKC ERC20 reserve ==="
printf "%-36s %-22s %-14s %s\n" DISTRIBUTOR CHAIN ZKC_RESERVE FLAG
echo "----------------------------------------------------------------------"

DISTRIB_TUPLES=$(jq -r --arg incl "$INCLUDE_STAGING" '
  .addresses[]
  | select(.role == "distributor")
  | select($incl == "true" or .env == "prod" or .env == "all")
  | .chains[] as $c
  | [.label, .address, $c] | @tsv
' "$ADDRS_FILE")

while IFS=$'\t' read -r LABEL ADDR CHAIN; do
  RPC=$(chain_rpc "$CHAIN")
  CHAIN_INFO=$(chain_contracts "$CHAIN")
  if [ -z "$RPC" ] || [ -z "$CHAIN_INFO" ]; then continue; fi
  TOKEN="${CHAIN_INFO##*|}"
  if [ -z "$TOKEN" ]; then continue; fi
  RAW=$(decode_uint "$(cast call --rpc-url "$RPC" "$TOKEN" 'balanceOf(address)(uint256)' "$ADDR" 2>/dev/null || true)")
  ZKC=$(fmt_zkc "$RAW")
  R_WARN=$(jq -r '.thresholds.by_role.distributor.reserve_zkc_warn // .thresholds.default.reserve_zkc_warn // 100' "$ADDRS_FILE")
  R_CRIT=$(jq -r '.thresholds.by_role.distributor.reserve_zkc_crit // .thresholds.default.reserve_zkc_crit // 50'  "$ADDRS_FILE")
  FLAG=""
  if [ "$ZKC" != "?" ] && [ "$ZKC" != "-" ]; then
    if flt_lt "$ZKC" "$R_CRIT"; then FLAG="ZKC-CRIT"
    elif flt_lt "$ZKC" "$R_WARN"; then FLAG="ZKC-LOW"
    fi
  fi
  printf "%-36s %-22s %-14s %s\n" "$LABEL" "$CHAIN" "$ZKC" "$FLAG"
done <<<"$DISTRIB_TUPLES"

Output format

Render the script's output as three separate markdown tables grouped by environment tier, in this order so operators can scan the highest-stakes rows first:

1. Prod mainnet — chains base-mainnet, taiko-mainnet. Real funds at stake; flagged rows here matter most.
2. Prod testnets — chains base-sepolia, ethereum-sepolia. Public testnet operations; flags worth checking but lower urgency.
3. Staging — chains base-sepolia-staging, taiko-staging. Internal testing; flagged rows are usually not paging-worthy unless they block an active staging change.

Within each tier, flagged rows in bold (CRIT and LOW). The Distributor's bridged-ZKC reserve table can stay as a single fourth table at the end (or split by tier too if it's noisy).

Follow with a short "action list" ordered by urgency (CRIT before LOW), with prod-mainnet items first since they map directly to operational impact. Skip the section if nothing is flagged.

Common gotchas

• Don't conflate balanceOfCollateral with balanceOf for distributors. Distributors don't deposit stake — their ZKC sits in the bridged-ERC20 directly. The script renders both: market deposit (needs_market_check: true) and the per-distributor ZKC reserve in the second table.
• Chain has both prod and staging market deployments. Chain 167000 (Taiko mainnet) and chain 84532 (Base Sepolia) each have separate prod and staging market contracts. The runbook uses chain labels like taiko-mainnet (prod) vs taiko-mainnet-staging (staging) to disambiguate; the skill's chain lookup table mirrors that.
• Decommissioned hosts may still have CloudWatch log groups (e.g. /boundless/bento/prover-01, prover-02, bento-prover-{1,2}). The audit reflects the runbook's operator_addresses.json (which mirrors infra/cw-monitoring/Pulumi.production.yaml for active provers), not historical log-group existence.
• Public RPCs sometimes return 0 when rate-limited or transiently unhealthy. If a balance comes back as exactly 0, retry once with a private RPC (see Prerequisites: network_secrets.toml's QuikNode keys) before declaring it actually empty.
• awk truncates wei to scientific notation. Always pipe through Python or printf "%.6f" for any wei→ETH math.
• cast call quoting. The function signature 'balanceOf(address)(uint256)' must be a single-quoted string; the parens are part of the type signature.

Adding a new operator address

Edit the runbook, not this skill:

1. Find the address in infra/<service>/Pulumi.l-prod-*.yaml (preferred) or ansible/inventory.yml.
2. Add a row to runbooks/addresses/operator_addresses.json (and the corresponding markdown table in runbooks/addresses/README.md) with the right role, chains, and needs_gas / needs_market_check flags.
3. The skill picks it up on the next run; no skill changes needed.

Adding a new chain

1. Add the new chain to contracts/deployment.toml in the boundless repo (the canonical contract registry). The skill picks up market + collateral-token addresses automatically on the next run.
2. If the chain has a public RPC the skill needs to query, add a row to the chain_rpc function (the only inline thing left — RPC endpoints aren't in deployment.toml).
3. Reference the new chain label in runbooks/addresses/operator_addresses.json for any operator addresses operating on that chain.