hydra-head-troubleshooter

name	hydra-head-troubleshooter
description	Hydra troubleshooting: decision tree for common issues. Maps symptoms to fixes with verification steps.
allowed-tools	["Read"]
user-invocable	true
context	["!hydra-node --version 2>&1 | head -3"]
metadata	{"openclaw":{"emoji":"🐉","requires":{"anyBins":["hydra-node","docker"],"bins":["curl"]},"install":[{"id":"brew","kind":"brew","formula":"colima docker docker-compose curl","bins":["colima","docker","docker-compose","curl"],"label":"Install Docker runtime (Colima) + Docker CLI + Compose + curl (brew)","os":["darwin","linux"]}]}}

hydra-head-troubleshooter

When to use

• Hydra Head doesn't start or no head observed
• Head exists but doesn't make progress
• Peers out of sync or disconnected
• Log messages: PeerConnected, AckSn, LogicOutcome errors

Operating rules (must follow)

• Confirm network and hydra-node version first
• Request logs from ALL participants for same time window
• Never request key contents (file paths OK)
• Output: (1) Root cause (2) Fix steps (3) Verification

Docker fallback mode

If hydra-node is not installed locally, use the wrapper script in this skill folder to run hydra-node inside Docker (Hydra upstream recommends Docker images for quickest start).

chmod +x {baseDir}/scripts/hydra-node.sh
{baseDir}/scripts/hydra-node.sh --help
{baseDir}/scripts/hydra-node.sh gen-hydra-key --output-file hydra

For full multi-node Head demos, prefer the hydra.family Docker Compose demo (it's the canonical "known-good" setup).

Quick diagnostic commands

# Check hydra-node version
hydra-node --version

# Check API health
curl -s localhost:4001/health

# Check peers
curl -s localhost:4001/peers

# Check head status
curl -s localhost:4001/status

# Check metrics (if enabled)
curl -s localhost:6001/metrics | grep hydra

Decision tree

A) "No head is observed from the chain"

Symptoms:

• Head never appears in logs
• No Init/Commit/Open progression
• Client shows no head state

Check 1: Cardano connection

# Verify cardano-node is ready
cardano-cli query tip --testnet-magic 1
# Should show current slot, not error

# Check socket exists
ls -la $CARDANO_NODE_SOCKET_PATH

Fix: Wait for cardano-node sync, verify socket path and network magic

Check 2: Scripts tx id

# Verify you're using correct scripts tx id for network
# Get from hydra-node release notes

# In hydra-node logs, look for:
grep -i "script" hydra-node.log | head -20

Fix: Use correct --hydra-scripts-tx-id for your network

Check 3: Key mismatch

# Verify cardano.sk matches what peers have as your vkey
cardano-cli key verification-key \
  --signing-key-file cardano.sk \
  --verification-key-file check.vkey

# Compare check.vkey with what you distributed

Fix: Re-exchange verification keys with all peers

---

B) "Head does not make progress"

Symptoms:

• Head exists but stuck at Init or Commit
• Snapshots not confirmed
• Commands hang or timeout

Check 1: Peer connectivity

# In logs, look for PeerConnected
grep "PeerConnected" hydra-node.log

# Check metrics
curl -s localhost:6001/metrics | grep peers_connected

Fix: Verify --peer host:port is correct and ports are reachable

Check 2: Hydra key mismatch

# Look for AckSn issues
grep -E "AckSn|LogicOutcome" hydra-node.log

# Verify hydra keys match
hydra-node verify-keys \
  --hydra-signing-key hydra.sk \
  --hydra-verification-key peer-hydra.vk

Fix: Re-exchange hydra verification keys

Check 3: Version mismatch

# All nodes must run compatible versions
hydra-node --version
# Compare across all participants

Fix: Align hydra-node versions

---

C) "Head stuck: peer out of sync"

Symptoms:

• One node accepts txs, others reject
• Snapshots stop being signed
• Ledger state diverged

Cause: Different ledger views due to config drift or missed txs

Fix: Snapshot side-loading

# 1. Get latest snapshot from healthy node
curl -s http://healthy-node:4001/snapshot > snapshot.json

# 2. Post to out-of-sync node
curl -X POST http://out-of-sync-node:4001/snapshot \
  -H "Content-Type: application/json" \
  -d @snapshot.json

Verify: Snapshots resume, all parties sign again

---

D) Mirror nodes / HA issues

Symptoms:

• SnapshotAlreadySigned in logs
• etcd quorum issues

Reality:

• SnapshotAlreadySigned is transient and harmless with mirrors
• Keep mirror count < floor(n/2) for quorum stability

Fix: Reduce mirror count or accept transient warnings

---

Log patterns to grep

Pattern	Meaning
PeerConnected	Peer connected successfully
PeerDisconnected	Peer lost connection
AckSn	Snapshot acknowledgment
LogicOutcome	Head logic result (check for errors)
SnapshotConfirmed	Snapshot finalized
HeadIsOpen	Head successfully opened
HeadIsClosed	Head closed, contestation started

Verification checklist

After any fix:

• All nodes same hydra-node version
• Scripts tx id correct for network
• cardano-node ready (query tip works)
• PeerConnected for all peers
• AckSn messages flowing
• No LogicOutcome errors

Examples

Example: Diagnose stuck head

User request: "My Hydra head won't open—stuck at commit"

Response:

# 1. Check all peers connected
grep "PeerConnected" hydra-node.log | tail -10
# Should see PeerConnected for each peer

# 2. If missing peers, check network
nc -zv peer-host 5001
# Should show "Connection succeeded"

# 3. Check all parties committed
curl -s localhost:4001/status | jq .
# Look for parties and their commit status

# 4. Check for key issues
grep -E "InvalidSignature|WrongParty" hydra-node.log

# Common fix: restart with correct peer addresses
# Verify --peer host:port matches actual peer locations

References

• hydra-head (operational guidance)
• hydra-head-operator (execution)
• reference/sources.md (doc provenance)
• hydra.family troubleshooting