analytics-opensearch-expertise

# OpenSearch Domain Health Assessment ## Overview This skill performs a comprehensive, read-only health assessment of Amazon OpenSearch Service domains using AWS APIs available through the customer's local AWS profile. It produces a structured report with findings and actionable recommendations. > **Scope:** CLI-agent compatible. No external packages, no IDE workspace, no human review gates. ## Trigger Keywords `opensearch`, `opensearch health`, `domain health`, `cluster review`, `opensearch review`, `opensearch assessment`, `opensearch check`, `open search`, `opensearch performance`, `opensearch security`, `opensearch cost` ## Prerequisites - AWS CLI profile configured with read-only access to the target account - IAM permissions required: - `es:DescribeDomain` - `es:DescribeDomainHealth` - `es:GetCompatibleVersions` - `es:DescribeReservedInstances` - `es:ListDomainNames` - `cloudwatch:GetMetricData` ## Execution Flow ### Input **Required:** Domain ARN (e.g., `arn:aws:es:us-west-2:123456789012:domain/my-domain`) Parse the ARN to extract: - **Region:** field 4 (e.g., `us-west-2`) - **Account ID:** field 5 (e.g., `123456789012`) - **Domain name:** after `domain/` in field 6 **Fallback / Target Resolution (apply these rules exactly — never silently substitute a target):** 1. **Exact match:** the ARN (or name + region) resolves via `DescribeDomain` → proceed with the assessment. 2. **No domain specified:** run `ListDomainNames` in the user's likely region(s) (cross-region scan if needed) and **present the discovered domains as options for the user to choose**. Do not pick one yourself unless exactly one domain exists in the entire account — state that you did so. 3. **Specified but NOT found:** search for candidates (`ListDomainNames` in the ARN's region, then other regions). Then **STOP and confirm — do NOT assess**: - Report clearly: "Domain {given} not found in {region}." - List the closest candidate(s) with their regions (e.g., name differs by a suffix, or same name in another region). - Ask the user to confirm which domain to assess. **Never run the assessment against a domain the user did not name without their explicit confirmation** — in accounts with similarly-named sibling domains (e.g., `prod-es-content` / `prod-es-users`), a silently substituted target produces a confident report about the wrong domain, which is worse than an error. 4. If the user provides only a domain name with no region, ask for the region (or apply rule 3's candidate search). ### Steps ``` 1. Parse domain ARN → region, account, domain name 2. Resolve the target (Target Resolution rules above) — confirm with the user if not an exact match 3. Describe domain configuration (DescribeDomain) 4. Describe domain health (DescribeDomainHealth) 5. Pull CloudWatch metrics (last 24h) 6. Check upgrade eligibility (GetCompatibleVersions; ListVersions fallback) 7. Check reserved instance coverage (DescribeReservedInstances) 8. Analyze findings against thresholds 9. Load the Remediation Reference — read_skill_resource(skill_id='analytics-opensearch-expertise', path='references/remediation-reference.md') — MANDATORY before writing recommendations 10. Generate report with recommendations ``` ## API Compatibility (DevOps Agent tooling) The DevOps Agent AWS tool exposes the LEGACY Elasticsearch-compatible client. Call legacy operation names FIRST; fall back to the modern names only if the legacy call fails: | Purpose | Try first (legacy) | Modern equivalent | |---|---|---| | Domain config | `describe_elasticsearch_domain` | `describe_domain` | | Version catalog | `list_elasticsearch_versions` | `list_versions` | | Upgrade targets | `get_compatible_elasticsearch_versions` | `get_compatible_versions` | | Reserved instances | `describe_reserved_elasticsearch_instances` | `describe_reserved_instances` | - IAM note: the execution role's managed policy grants `es:Describe*` and `es:List*` — Describe/List operations succeed under either naming; `Get*` actions (BOTH GetCompatible variants) are typically denied. This is exactly why the 1.4 ListVersions fallback exists. - `DescribeDomainHealth` has NO legacy equivalent. If it fails, do NOT skip checks 2.2/2.3 — use CloudWatch instead: `Shards.active` (Maximum) = active shards, `Shards.unassigned` (Maximum) = unassigned, TotalShards ≈ active + unassigned; `ClusterStatus.green/yellow/red` for 1.1/3.7. Never report the checks as SKIPPED when this fallback is available. - All CloudWatch metrics in this skill: namespace `AWS/ES`, dimensions `DomainName` and `ClientId` (the account ID). ## Checks & Decision Logic ### Category 1: Cluster Health & Configuration #### 1.1 Cluster Status - **API:** `DescribeDomainHealth` - **Field:** `ClusterHealth` (green | yellow | red) - **Logic:** - GREEN → ✅ All primary and replica shards assigned - YELLOW → ⚠️ All primaries assigned, some replicas unassigned - RED → 🔴 Some primary shards unassigned — data loss risk #### 1.2 Node Count & AZ Balance - **API:** `DescribeDomain` → `ClusterConfig` - **Fields:** `InstanceCount`, `ZoneAwarenessEnabled`, `AvailabilityZoneCount` - **Logic:** - If `ZoneAwarenessEnabled == true`: - `InstanceCount % AvailabilityZoneCount != 0` → ⚠️ "Data node count ({count}) is not a multiple of AZ count ({az_count}). This causes uneven shard distribution across AZs, reducing fault tolerance." - Recommendation: "Adjust data node count to a multiple of {az_count} (e.g., {nearest_multiple})." - If `ZoneAwarenessEnabled == false` and `InstanceCount > 1`: - ⚠️ "Multi-AZ is not enabled despite having {count} data nodes. Single-AZ deployment risks full cluster unavailability during AZ failure." #### 1.3 Dedicated Master Configuration - **API:** `DescribeDomain` → `ClusterConfig` - **Fields:** `DedicatedMasterEnabled`, `DedicatedMasterType`, `DedicatedMasterCount` - **Logic:** - `DedicatedMasterEnabled == false` AND `InstanceCount >= 3` → ⚠️ "No dedicated master nodes. Clusters with 3+ data nodes should use dedicated masters for stability." - `DedicatedMasterCount == 2` → ⚠️ "Even number of master nodes ({count}) risks split-brain. Use 3 dedicated master nodes." - `DedicatedMasterCount < 3` → ⚠️ "Fewer than 3 master nodes reduces fault tolerance." #### 1.4 Engine Version & Upgrade Eligibility - **API:** `GetCompatibleVersions` with `DomainName` - **Fallback (use when `GetCompatibleVersions` returns AccessDeniedException — do NOT skip):** call `ListVersions` and derive upgrade candidates using these two rules: 1. Direct upgrade candidates = versions in `ListVersions` with the SAME major version as the domain and a LATER minor version (e.g., a 2.11 domain → every later 2.x in the region). 2. Next-major path = via the LATEST available minor of the current major (compute it from `ListVersions`; never hard-code a specific version number). Label fallback results as "derived upgrade candidates (ListVersions + documented upgrade-path rules)" and cite https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html for the reader. Only mark 1.4 SKIPPED if BOTH APIs are denied. **Note placement:** the fallback provenance note belongs INSIDE the 1.4 finding ONLY — do NOT surface it in the Executive Summary or any report-level note box, and do NOT name API errors (e.g., AccessDeniedException) anywhere in the report. Phrase it customer-friendly: "Upgrade candidates were derived from the available version listings and are not API-confirmed — verify eligible target versions in the console before planning an upgrade." Render candidates as a SET of direct single-hop targets (comma-separated) — NEVER as an arrow chain (NOT "2.13 → 2.15 → 2.19"): every listed same-major version is directly reachable in ONE hop. Any upgrade recommendation MUST propose a single hop to the latest same-major version (plus one more hop for a major jump). Do NOT invent sequential multi-hop plans or per-hop time estimates. - **Logic:** - If compatible target versions exist → ℹ️ "Current version: {current}. Upgrades available: {targets}." - If current version is more than 2 releases behind the latest same-major version → ⚠️ "Consider upgrading for performance improvements and security patches." - **Counting rule (apply exactly):** "releases behind" = the COUNT of later same-major versions returned by `ListVersions`/`GetCompatibleVersions` (e.g., 2.11 with {2.13, 2.15, 2.17, 2.19} available = 4 releases behind) — NEVER the numeric delta between version numbers (2.19 − 2.11 is NOT "8 versions behind"). ### Category 2: Storage & Shard Strategy #### 2.1 EBS Configuration - **API:** `DescribeDomain` → `EBSOptions` - **Fields:** `VolumeType`, `VolumeSize`, `Iops`, `Throughput` - **Logic:** - `VolumeType == "gp2"` → ⚠️ "Using gp2 volumes. gp3 offers better price/performance with configurable IOPS and throughput." - Report total cluster storage: `VolumeSize × InstanceCount` - **API (throttle detection):** CloudWatch `IopsThrottle`, `ThroughputThrottle`, `ReadIOPSMicroBursting`, `WriteIOPSMicroBursting`, `ReadThroughputMicroBursting`, `WriteThroughputMicroBursting` (Maximum statistic, last 7 days) - **Logic (throttle detection):** - `IopsThrottle` Max > 0 OR `ThroughputThrottle` Max > 0 in the window → ⚠️ "EBS IOPS/throughput throttling detected. Increase provisioned `Iops`/`Throughput` on gp3 via `UpdateDomainConfig`." - Throttle metrics 0 but any microbursting metric > 0 on multiple days → ℹ️ "Workload bursts above the provisioned EBS baseline; monitor — sustained growth will lead to throttling. Consider raising gp3 `Iops`/`Throughput`." - No datapoints (non-EBS domain, e.g. instance-store or OR remote-store instance families) → skip this sub-check silently; do NOT mark check 2.1 SKIPPED. #### 2.2 Shard Count & Density - **API:** `DescribeDomainHealth` - **Fields:** `TotalShards`, `ActiveShards` - **Compute:** `shards_per_node = TotalShards / InstanceCount` - **Version-dependent limits (from `DescribeDomain` → `EngineVersion`):** - OpenSearch ≤ 2.15: Hard limit = **1,000 shards/node** - OpenSearch 2.17+: Hard limit = **1,000 shards per 16 GB of data node heap** - **Heap estimation (MUST use the real instance type, not an assumed value):** estimate per-node JVM heap from the data node instance type: `heap_GiB ≈ min(32, instance_RAM_GiB / 2)` (e.g., t3.medium.search 4 GiB RAM → ~2 GiB heap; r6g.xlarge.search 32 GiB RAM → ~16 GiB heap). **Instance RAM reference (GiB):** t3.small=2 · t3.medium=4 · c6g.large=4 · m6g.large=8 · c6g.xlarge=8 · r6g.large=16 · m6g.xlarge=16 · c6g.2xlarge=16 · r6g.xlarge=32 · m6g.2xlarge=32 · r6g.2xlarge=64 (heap caps at 32 GiB). For unlisted types use the EC2 equivalent's memory — never guess silently; name the assumed RAM in the report. - **Best-practice threshold:** `bp_limit = 25 × heap_GiB` shards/node. - **Logic (apply these exact thresholds — do not substitute your own; the verdict tier is determined ONLY by which line matches, never by judgment):** - `shards_per_node > hard_limit` → 🔴 "Exceeds hard shard limit for engine version {version}." - `shards_per_node > 4 × bp_limit` → 🔴 "Shard density ({shards_per_node}/node) is more than 4× the best-practice limit ({bp_limit}/node for ~{heap_GiB} GiB heap). Severe heap/GC and recovery risk." (This tier is 🔴, not ⚠️ — do not downgrade.) - `shards_per_node > bp_limit` → ⚠️ "Shard density ({shards_per_node}/node) exceeds best-practice threshold of 25 shards/GiB heap ({bp_limit}/node for ~{heap_GiB} GiB heap)." - `shards_per_node > 0.5 × bp_limit` → ℹ️ "Moderate shard density. Monitor JVM heap pressure." - `shards_per_node <= 0.5 × bp_limit` → ✅ PASS. (Verbatim: at or below half the best-practice limit the verdict is PASS, never ℹ️ — low density is not a finding.) #### 2.3 Active Shards vs. Total Shards - **API:** `DescribeDomainHealth` - **Fields:** `ActiveShards`, `TotalShards` - **Expected:** `ActiveShards == TotalShards` (all shards placed and serving) - **Logic:** - `ActiveShards < TotalShards`: - Delta = `TotalShards - ActiveShards` - ⚠️ "{delta} shards are not active. Non-active shards may be:" - "• **Initializing** — being allocated after a node restart or scaling event (transient, typically resolves in minutes)" - "• **Relocating** — being moved during a rebalance operation (transient)" - "• **Unassigned** — cannot be placed on any node (persistent problem). Common causes: insufficient disk space, allocation filtering rules, AZ awareness constraints with unbalanced node counts, or max shards per node limit reached." - Recommendation: "Check `_cluster/allocation/explain` for unassigned shard root cause." #### 2.4 Free Storage Space - **API:** CloudWatch `FreeStorageSpace` (Minimum statistic, last 24h) - **CRITICAL — the Minimum statistic is PER-NODE (the lowest single node's free MB), NOT a cluster-wide total. Compute free % against ONE node's volume, never the cluster total:** `free_pct = FreeStorageSpace_min_MB / (VolumeSize_GiB × 1024) × 100` (Dividing the per-node Minimum by `VolumeSize × InstanceCount` understates free space by a factor of InstanceCount — this is wrong. Cross-check: the Sum statistic ÷ (VolumeSize × InstanceCount × 1024) should give approximately the same percentage.) - **Note:** usable space per node is less than the raw volume size due to OS/service reserved overhead — an empty node typically shows ~72% free, not ~97%. - **Logic (apply these exact thresholds verbatim):** - `free_pct < 10%` → 🔴 "Free storage below 10%. High watermark may trigger read-only mode." - `free_pct < 20%` → ⚠️ "Free storage below 20%. Approaching low watermark. Plan capacity increase." - `free_pct < 30%` → ℹ️ "Storage utilization above 70%. Monitor trend." - `free_pct >= 30%` → ✅ PASS. - If the metric returns NO datapoints (new domain, or configuration change in progress) → SKIPPED "FreeStorageSpace metric unavailable" — never silently omit the check. ### Category 3: Performance Metrics > **Note on percentiles:** CloudWatch metrics for OpenSearch (`SearchLatency`, `IndexingLatency`) report aggregate values, not per-request percentiles. We use the **Maximum** statistic over 1-minute periods as a "worst-case" proxy. True p50/p99 latency requires OpenSearch slow-log analysis or UltraWarm query insights, which is outside the scope of API-only checks. **Time range:** Last 24 hours, 1-minute period granularity. #### 3.1 JVM Memory Pressure - **Metric:** `JVMMemoryPressure`, Statistic: `Maximum` — **the verdict MUST be computed from the Maximum statistic. Never substitute Average (an idle cluster commonly shows Avg ~43% while Max sustains ~74% — these produce different verdicts). Report Average only as context.** - **Logic:** - Max > 92% → 🔴 "JVM heap pressure critically high ({value}%). Cluster is at risk of circuit breaker trips and OOM. Consider scaling up instance type or reducing shard count." - Max > 80% → ⚠️ "JVM heap pressure elevated ({value}%). Approaching GC thrashing threshold." - Max > 70% → ℹ️ "JVM heap pressure moderate. Monitor trend." #### 3.2 CPU Utilization - **Metric:** `CPUUtilization`, Statistic: `Average` (sustained) and `Maximum` (spikes) - **Logic:** - Avg > 80% sustained (>50% of datapoints) → ⚠️ "Sustained high CPU. Cluster may be undersized for current workload." - Max > 95% → ℹ️ "CPU spikes observed. Check for expensive queries or bulk indexing bursts." #### 3.3 Search Latency - **Metric:** `SearchLatency`, Statistic: `Maximum` (worst-case proxy) - **Logic:** - Max > 500ms → ℹ️ "Worst-case search latency {value}ms — elevated spikes observed. Investigate slow queries if user-facing impact is reported." - Report average as baseline context. - **Informational only — search latency NEVER renders ⚠️ or 🔴.** The 500ms marker is a heuristic: AWS does not define a canonical latency threshold; acceptable latency is workload-dependent, so frame findings against the customer's own baseline. #### 3.4 Indexing Latency - **Metric:** `IndexingLatency`, Statistic: `Maximum` (worst-case proxy) - **Logic:** - Max > 500ms → ℹ️ "Worst-case indexing latency {value}ms. Check merge pressure or refresh interval if ingestion throughput is impacted." - Report average as baseline context. - **Informational only — indexing latency NEVER renders ⚠️ or 🔴.** Heuristic marker; frame against the workload's own baseline. #### 3.5 Search & Indexing Rate - **Metrics:** `SearchRate` (Sum), `IndexingRate` (Sum) - **Purpose:** Workload characterization — reported for context, no threshold alerts.