OpenSearch Diagnostics
When to use
Any OpenSearch Service investigation where the console alone is insufficient — cluster health debugging, search performance analysis, indexing troubleshooting, shard management, storage pressure, JVM tuning, access policy resolution, VPC connectivity, snapshot/restore, Dashboards issues, UltraWarm/cold tier management, ISM lifecycle, or Serverless collection problems.
Investigation workflow
Step 1 — Collect and triage
aws opensearch describe-domain --domain-name <domain>
curl -XGET "https://<endpoint>/_cluster/health?pretty"
curl -XGET "https://<endpoint>/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent,node.role"
curl -XGET "https://<endpoint>/_cat/indices?v&s=health,index&h=health,status,index,pri,rep,docs.count,store.size"
aws cloudwatch get-metric-statistics --namespace AWS/ES --metric-name ClusterStatus.red --dimensions Name=DomainName,Value=<domain> Name=ClientId,Value=<account-id> --start-time <start> --end-time <end> --period 300 --statistics Maximum
aws cloudwatch get-metric-statistics --namespace AWS/ES --metric-name FreeStorageSpace --dimensions Name=DomainName,Value=<domain> Name=ClientId,Value=<account-id> --start-time <start> --end-time <end> --period 300 --statistics Minimum
aws cloudwatch get-metric-statistics --namespace AWS/ES --metric-name JVMMemoryPressure --dimensions Name=DomainName,Value=<domain> Name=ClientId,Value=<account-id> --start-time <start> --end-time <end> --period 300 --statistics Maximum
Step 2 — Domain deep dive
curl -XGET "https://<endpoint>/_cat/shards?v&s=state,index&h=index,shard,prirep,state,docs,store,node,unassigned.reason"
curl -XGET "https://<endpoint>/_nodes/stats?pretty"
curl -XGET "https://<endpoint>/_cluster/settings?include_defaults=true&pretty"
curl -XGET "https://<endpoint>/_cat/allocation?v"
aws opensearch describe-domain-config --domain-name <domain>
aws cloudwatch get-metric-statistics --namespace AWS/ES --metric-name CPUUtilization --dimensions Name=DomainName,Value=<domain> Name=ClientId,Value=<account-id> --start-time <start> --end-time <end> --period 300 --statistics Average,Maximum
Step 3 — Detailed investigation
aws opensearch describe-domain --domain-name <domain> --query 'DomainStatus.LogPublishingOptions'
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=es.amazonaws.com --max-results 20
curl -XGET "https://<endpoint>/<index>/_settings?pretty"
curl -XGET "https://<endpoint>/<index>/_mapping?pretty"
curl -XGET "https://<endpoint>/_cluster/allocation/explain?pretty"
Read references/opensearch-guardrails.md before concluding on any OpenSearch issue.
Tool quick reference
| Tool / API | When to use |
|---|
describe-domain | Full domain configuration and status |
describe-domain-config | Domain configuration details including access policies |
_cluster/health | Cluster health status, node count, shard counts |
_cat/nodes | Node-level resource usage (heap, CPU, disk) |
_cat/indices | Index health, doc counts, storage sizes |
_cat/shards | Shard allocation, state, and unassigned reasons |
_cat/allocation | Disk allocation per node |
_nodes/stats | Detailed node statistics (JVM, OS, transport) |
_cluster/settings | Cluster-level settings including watermarks |
_cluster/allocation/explain | Why a shard is unassigned |
_nodes/hot_threads | Identify CPU-intensive operations |
_tasks | Running tasks (merges, recoveries, searches) |
_cat/recovery | Shard recovery progress |
_plugins/_ism/explain/<index> | ISM policy execution status |
Gotchas: OpenSearch Service
- Cluster health RED means at least one primary shard is unassigned: RED does not mean the cluster is down. Searches and writes to indices with all primaries assigned still work. RED means at least one index has an unassigned primary shard, so that index is partially or fully unavailable. Check
_cat/shards?v&h=index,shard,prirep,state,unassigned.reason to identify the affected index and reason.
- YELLOW means replicas are unassigned (single-node clusters are always YELLOW): YELLOW indicates all primary shards are assigned but at least one replica is not. Single-node domains are always YELLOW because replicas cannot be allocated to the same node as the primary. This is expected and not an error for single-node dev/test domains.
- JVM memory pressure > 80% causes GC pauses and potential OOM: OpenSearch uses Java heap for field data caches, query caches, and internal structures. When JVMMemoryPressure exceeds 80%, garbage collection becomes aggressive and causes latency spikes. Above 92%, the circuit breaker trips. Sustained pressure above 85% requires scaling or tuning.
- Shard count matters (aim for 10-50 GB per shard, max 1000 shards per node recommended): Oversized shards slow recovery and searches. Undersized shards waste resources with per-shard overhead (heap, file handles, cluster state). AWS recommends 10-50 GB per shard. Keep total shards per node under 1000. Total shard count = (primary shards) × (1 + replicas).
- Dedicated master nodes are critical for cluster stability (3 or 5, odd number): Dedicated master nodes manage cluster state, shard allocation, and index creation. Without them, data nodes handle master duties under load, risking instability. Always use 3 or 5 dedicated masters (odd number prevents split brain). Master nodes do not hold data.
- Storage watermarks (85% low, 90% high, 95% flood stage — blocks writes at flood): OpenSearch uses disk-based watermarks. At 85% (low), no new shards are allocated to the node. At 90% (high), shards are relocated away. At 95% (flood stage), all indices on the node become read-only (index.blocks.read_only_allow_delete). Writes are blocked until space is freed.
- Index State Management (ISM) policies for lifecycle: ISM automates index lifecycle operations (rollover, delete, snapshot, force merge, transition to warm/cold). Policies are attached to indices or index patterns. ISM runs on a configurable schedule (default 5 minutes). Failed ISM transitions require manual investigation via
_plugins/_ism/explain.
- UltraWarm is read-only (cannot write to warm/cold): UltraWarm and cold storage tiers are for read-only data. You cannot index new documents into warm or cold indices. Data must be migrated from hot to warm using ISM or the migration API. Warm indices can be searched but not written to. Cold indices must be moved back to warm before searching.
- Access policies are resource-based (like S3 bucket policies): OpenSearch access policies are JSON resource-based policies attached to the domain. They control who can access the domain endpoint and which actions are allowed. They work alongside IAM policies. An explicit deny in either policy blocks access. IP-based conditions are common for non-VPC domains.
Anti-hallucination rules
- Always cite specific domain configurations, cluster health output,
_cat API results, CloudWatch metrics, or node stats as evidence.
- Never confuse cluster health RED (unassigned primaries) with cluster down. RED clusters can still serve requests for indices with assigned primaries.
- Never suggest writing to UltraWarm or cold storage indices. These tiers are strictly read-only. Data must be migrated from hot tier.
- Never claim VPC domains have public endpoints. VPC domains are only accessible from within the VPC or via VPN/peering/proxy.
- OpenSearch Serverless uses collections, OCUs, and different APIs — never apply managed domain concepts (shards, nodes, JVM) to Serverless.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
34 runbooks
| Category | IDs | Covers |
|---|
| A — Cluster Health | A1-A4 | RED cluster, YELLOW cluster, split brain, master node issues |
| B — Performance | B1-B4 | Search latency, indexing throughput, JVM memory pressure, GC pauses |
| C — Storage | C1-C3 | Disk watermarks, storage full, UltraWarm/cold tier issues |
| D — Shards | D1-D3 | Unassigned shards, shard imbalance, too many shards |
| E — Indexing | E1-E3 | Indexing failures, mapping conflicts, bulk indexing errors |
| F — Access & Security | F1-F3 | Access policy issues, fine-grained access control, VPC connectivity |
| G — Snapshots | G1-G3 | Snapshot failures, restore issues, repository configuration |
| H — Dashboards | H1-H2 | Dashboards access, visualization errors |
| I — Serverless | I1-I2 | Collection issues, capacity/scaling |
| J — ISM & Lifecycle | J1-J2 | ISM policy failures, index rollover |
| Z — Catch-All | Z1 | General troubleshooting |
