| name | debug-message |
| description | Debug why a Hyperlane message is not being processed. Use when given a message ID or explorer URL to investigate delivery failures, gas estimation errors, validator issues, or other processing problems. |
Debug Hyperlane Message Skill
When to Use
- User provides a message ID (e.g.,
0xa454...) or explorer URL
- Investigating why a message is stuck or not delivered
- Debugging gas estimation failures
- Understanding message processing status
Input Parameters
| Parameter | Required | Example | Description |
|---|
message_id | Yes | 0xa454559c... | The 66-character hex message ID |
explorer_url | Optional | https://explorer.hyperlane.xyz/message/0x... | Can extract message_id from URL |
Debugging Workflow
Step 1: Get Basic Message Details from Explorer
Fetch the explorer page to get origin/destination chains and basic status:
WebFetch: https://explorer.hyperlane.xyz/message/[MESSAGE_ID]
Prompt: Extract message status, origin chain, destination chain, sender, recipient, timestamp, delivery status
Key info to extract:
- Origin chain and domain ID
- Destination chain and domain ID
- Is it delivered? (if yes, no debugging needed)
- Timestamp (how old is it?)
Step 2: Search Relayer Logs for the Message
Use the gcloud CLI to find logs related to this message in the omniscient relayer:
gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="relayer" AND labels.k8s-pod/app_kubernetes_io/instance="omniscient-relayer" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent" AND "[MESSAGE_ID]"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d
Step 3: Identify the Message Status
Look for the message in PendingMessage entries. Common statuses:
| Status | Meaning | Priority |
|---|
Retry(ErrorEstimatingGas) | Gas estimation failing - contract revert | HIGH |
Retry(GasPaymentRequirementNotMet) | Insufficient gas payment | MEDIUM |
Retry(CouldNotFetchMetadata) | Validator signatures unavailable | LOW (check after 5+ min) |
FirstPrepareAttempt | Still processing, not stuck yet | LOW |
Extract status with:
grep -o "message_id: [MESSAGE_ID][^}]*" [log_output] | sort -u
Step 4: Check for Gas Payment Issues
If message is slow/stuck, search for gas payment evaluation logs:
gcloud logging read '[BASE_RELAYER_QUERY] AND "[MESSAGE_ID]" AND jsonPayload.fields.message:"Evaluating if message meets gas payment requirement"' --project=abacus-labs-dev --limit=5 --format=json --freshness=7d
Key fields in jsonPayload.fields:
current_payment.gas_amount - gas units paid for by sendertx_cost_estimate.gas_limit - gas units needed for deliverycurrent_expenditure.gas_used - gas already spent on retriespolicy - subsidy policy (e.g., fractional_numerator: 1, fractional_denominator: 2 = 50% subsidy)
If gas_amount < gas_limit, message fails with "Repreparing message: Gas payment requirement not met" and retries every ~3 minutes.
Step 5: For Gas Estimation Errors - Get the Revert Reason
If status is ErrorEstimatingGas, the actual revert reason is in jsonPayload.fields.error. Use this query and extraction:
gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="relayer" AND labels.k8s-pod/app_kubernetes_io/instance="omniscient-relayer" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent" AND "[MESSAGE_ID]" AND jsonPayload.fields.error:*' --project=abacus-labs-dev --limit=5 --format=json --freshness=1d 2>/dev/null | grep -o '"error": "[^"]*"' | head -1
The error field contains the full revert reason, e.g.:
"error": "ContractError(...JsonRpcError { code: 3, message: \"execution reverted: panic: arithmetic underflow or overflow (0x11)\", data: Some(...) }...)"
Quick extraction - pipe to extract just the revert message:
... | grep -oP 'execution reverted: [^"\\]+' | head -1
Common revert patterns:
execution reverted: panic: arithmetic underflow or overflow (0x11) - Contract math errorexecution reverted: [CUSTOM_ERROR] - Custom contract revert (decode with cast 4byte)execution reverted with hex data - Decode selector with cast 4byte 0x[first4bytes]
Step 6: Extract Message Details
From the logs, identify:
origin: Source chaindestination: Destination chain (or domain ID like 4114 for Citrea)sender: Origin contract addressrecipient: Destination contract address (the warp route or recipient)nonce: Message sequence number
Example log format:
HyperlaneMessage { id: 0x..., nonce: 162898, origin: ethereum, sender: 0x..., destination: 4114, recipient: 0x... }
Step 7: Report Findings
Summarize:
- Message ID: Full ID
- Route: Origin -> Destination (e.g., Ethereum -> Citrea)
- Status: Current processing status
- Root Cause: The actual error (e.g., contract revert reason)
- Retry Count: How many times it's been attempted
- Recommendation: What needs to happen (e.g., fix recipient contract, wait for validators)
Common Root Causes
Gas Estimation Errors
| Error | Meaning | Resolution |
|---|
panic: arithmetic underflow or overflow (0x11) | Contract math error | Bug in recipient contract |
IXERC20_NotHighEnoughLimits() | Bridge rate limit hit | Wait for limit reset |
InsufficientBalance | Not enough tokens | Fund the contract |
Unauthorized | Access control failure | Check permissions |
Metadata/Validator Issues
If CouldNotFetchMetadata persists > 5 minutes:
- Check validator status using the
debug-validator-checkpoint-inconsistency skill
- Identify if validators are behind on the origin chain
Gas Payment Issues
If GasPaymentRequirementNotMet:
- Compare
current_payment.gas_amount vs tx_cost_estimate.gas_limit
- Message will auto-retry every ~3 min until gas prices drop or subsidy kicks in
- Check
policy field for subsidy ratio (e.g., 1/2 = relayer covers 50%)
- Resolution: wait for gas prices to drop, or manually subsidize via IGP top-up
Decoding Revert Selectors
When you see hex revert data like 0x4e487b71...:
cast 4byte 0x4e487b71
Common panic codes:
0x11 - Arithmetic overflow/underflow0x12 - Division by zero0x21 - Invalid enum value0x31 - Pop on empty array0x32 - Array out of bounds
Domain ID Reference
Common domain IDs (destination field in logs):
1 - Ethereum42161 - Arbitrum10 - Optimism137 - Polygon4114 - Citrea
Check @hyperlane-xyz/registry or chain metadata for full mapping.
Example Investigation
User asks: "Why isn't message 0xa454... being processed?"
- Fetch explorer: Origin=Ethereum, Dest=Citrea, not delivered
- Query relayer logs for
0xa454...
- Find status:
Retry(ErrorEstimatingGas)
- Search error logs:
execution reverted: panic: arithmetic underflow or overflow (0x11)
- Report: "Message from Ethereum to Citrea is failing because the recipient contract (0xbd39...) on Citrea is reverting with an arithmetic overflow error. This is a bug in the destination contract that needs to be fixed by the contract owner."
