| name | fix-merkle-root-mismatch |
| description | Apply fixes for merkle root mismatch issues in the relayer database. Use after running /investigate-merkle-root-mismatch to investigate and identify the mismatched entries. Requires user confirmation before any database modifications. |
Fix Merkle Root Mismatch
When to Use
Use this skill AFTER running /investigate-merkle-root-mismatch to investigate the issue.
- User request triggers:
- "Fix the merkle root mismatch on [chain]"
- "Apply the merkle tree fixes"
- "Update the relayer database with correct message IDs"
Input Parameters
| Parameter | Required | Default | Description |
|---|
origin | Yes | - | The origin chain to fix (e.g., polygonzkevm, ethereum) |
domain_id | Yes | - | Domain ID of the chain |
environment | No | mainnet3 | mainnet3 or testnet4 |
merkle_tree_insertions | Yes | - | List of corrections: [{leaf_index, message_id, block_number}] |
Prerequisites
- Investigation completed - Run
/investigate-merkle-root-mismatch first to identify mismatched entries
kubectl access to the relayer pods- Port-forward already established to relayer (port 9090)
Fix Workflow
Step 0: Find Monorepo Root
MONOREPO_ROOT=$(git rev-parse --show-toplevel)
Step 1: Verify Port-Forward is Active
Check if port 9090 is already in use:
lsof -i :9090
If not in use, start port-forward in background:
kubectl port-forward omniscient-relayer-hyperlane-agent-relayer-0 9090:9090 -n [environment] &
Wait a few seconds, then verify it's working:
curl -s "localhost:9090/merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=0&leaf_index_end=1"
Step 2: Present Changes for Confirmation
IMPORTANT: User confirmation required before executing.
Before making any changes to the relayer database, you MUST:
-
Present a summary of ALL changes to be made in a clear table format:
| Leaf Index | Correct Message ID | Block Number |
|---|
| 1477 | 0x4bb3e20db45366a6a360ad2639c5421ea622a69b89b2edb045aa97e2051529b7 | 9891832 |
| 1478 | 0xa66dbdc1874acfddf14e75e6a20dd1632e9e3206c5f5855884da0d26d8ca12fa | 9891872 |
-
Use AskUserQuestion to get explicit confirmation:
- Question: "Do you want to apply these merkle tree fixes to the relayer database?"
- Options: "Yes, apply fixes" / "No, cancel"
-
Only proceed if the user explicitly confirms.
Step 3: Apply Merkle Tree Fixes
Once confirmed, insert the correct merkle tree insertions via the relayer API:
curl -X POST \
-H 'Content-type: application/json' \
'localhost:9090/merkle_tree_insertions' \
-d '{
"merkle_tree_insertions": [
{
"chain": [domain_id],
"insertion_block_number": [block_number],
"leaf_index": [index],
"message_id": "0x..."
}
]
}'
Request schema:
{
"merkle_tree_insertions": [
{
"chain": "<domain_id as number>",
"insertion_block_number": "<block number as number>",
"leaf_index": "<leaf index as number>",
"message_id": "<0x-prefixed H256 hash>"
}
]
}
Verify response: The API should return a success response with the count of inserted entries.
Step 4: Restart Relayer
IMPORTANT: User confirmation required before executing.
Use AskUserQuestion to confirm:
- Question: "Do you want to restart the relayer to apply the database fixes?"
- Options: "Yes, restart relayer" / "No, I'll restart manually later"
Only proceed if the user explicitly confirms.
Once confirmed, restart the relayer to rebuild its in-memory merkle tree:
pnpm --dir typescript/infra exec tsx ./scripts/agents/restart-agents.ts -e [environment] --context hyperlane --role relayer
Step 5: Validate Fix
After the relayer restarts, verify the fix:
5.1: Get Latest Index
Query Grafana for the current tree size:
Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_latest_tree_insertion_index{origin="[origin]", hyperlane_deployment="[environment]"}
- startTime: now-1h
- queryType: instant
5.2: Spot-Check Roots at Key Indices
Compare validator and relayer roots at three points: first fixed index, middle, and latest.
index=[INDEX]
validator_root=$(curl -s "https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/checkpoint_${index}_with_id.json" | jq -r '.value.checkpoint.root')
relayer_root=$(curl -s "localhost:9090/merkle_proofs?domain_id=[domain_id]&leaf_index=${index}&root_index=${index}" | jq -r '.root')
echo "Index $index:"
echo " Validator: $validator_root"
echo " Relayer: 0x$relayer_root"
if [ "$validator_root" = "0x$relayer_root" ]; then echo " ✓ Match"; else echo " ❌ MISMATCH"; fi
Run this for:
- First fixed index (e.g., 37352)
- Middle index (e.g., 39000)
- Latest index (e.g., 41172)
5.3: Verify Mismatch Metric is Clear
Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_merkle_root_mismatch{origin="[origin]"}
- startTime: now-1h
- queryType: instant
Expected result: Empty result or value of 0.
5.4: Report Results
Present validation results in a table:
| Index | Location | Result |
|---|
| 37352 | First fixed | ✓ Match |
| 39000 | Middle | ✓ Match |
| 41172 | Latest | ✓ Match |
If still mismatched:
- Inform the user that additional entries may need fixing
- Suggest re-running
/investigate-merkle-root-mismatch to investigate further
API Reference
Relayer Endpoints
| Endpoint | Method | Description |
|---|
/merkle_tree_insertions | GET | List merkle tree insertions from DB |
/merkle_tree_insertions | POST | Insert/update merkle tree entries |
Request/Response Examples
POST /merkle_tree_insertions
Request:
{
"merkle_tree_insertions": [
{
"chain": 1101,
"insertion_block_number": 9891832,
"leaf_index": 1477,
"message_id": "0x4bb3e20db45366a6a360ad2639c5421ea622a69b89b2edb045aa97e2051529b7"
}
]
}
Response (success):
{
"status": "success",
"data": {
"count": 1
}
}
Common Issues
- Port-forward disconnected: Re-establish before applying fixes
- API returns error: Check the error message, verify domain_id and message_id format
- Relayer restart fails: Check kubectl access and pod status
- Fix didn't work: May have missed some mismatched indices, re-investigate
Runbook Reference
Full runbook: https://www.notion.so/hyperlanexyz/Merkle-Root-Mismatch-26a6d35200d680a2857dcd0b228d4ab7
