sap-operational-health

name	sap-operational-health
description	Unified health dashboard for SAP systems. Checks AMS provider health, data freshness, VM power state, CPU/memory/disk metrics, accelerated networking, proximity placement, Resource Health, and alert coverage. Traffic-light output per layer. Enriched with live VM data when command proxy is available.
tools	["ExecutePythonCode","RunAzCliReadCommands","GetArmResourceAsJson","QueryLogAnalyticsByWorkspaceId","GetMetricTimeSeriesElementsForAzureResource","PlotBarChart","PlotHeatmap"]

Environment Configuration

All environment-specific values (subscription ID, AMS workspace ID, proxy URLs, API keys, SAP landscape) are provided via the Team Onboarding instructions. The agent reads these from the onboarding context at runtime. Do not hardcode environment values in this skill.

Authentication: Use the agent's built-in tools (RunAzCliReadCommands, GetArmResourceAsJson, QueryLogAnalyticsByWorkspaceId, GetMetricTimeSeriesElementsForAzureResource) for Azure API calls. These authenticate automatically via the agent's Managed Identity.

Data Reuse (AAU Optimization): Before calling any API or proxy, check if the data was already retrieved earlier in this conversation. Reuse landscape registry, VM power states, config files, and AMS query results from context. Do not re-fetch data that is already available.

Config reads & proxy fallback: Stored SAP/OS configs are read directly from the sap-configs blob container using the agent's own Managed Identity (--auth-mode login) — there is no config proxy. The MCP command proxy is optional and runs only live VM commands; if it is not deployed or errors (timeout, 5xx, unreachable), continue with stored blob configs + Azure-native sources (AMS, ARM API, Azure Monitor). Never block the skill on the proxy.

Telemetry source adaptation: HANA/OS/SAP-app depth depends on which telemetry source is listed in ## Deployed Infrastructure. Adapt automatically — never hard-fail because a specific source is absent:

• AMS (Azure Monitor for SAP) listed → use the AMS KQL tables below (deepest HANA/OS/cluster signals).
• No AMS, but an SAP APM connector is listed (SAP Cloud ALM / Focus Run / Dynatrace) → invoke the SAP APM Connector for SAP-app/HANA signals; use Azure Monitor platform metrics for VM/infra.
• No SAP telemetry source at all → use Azure platform metrics only (VM Insights, Azure Monitor, Resource Health) for VM-level CPU/memory/disk/network, and disclose in the header: "SAP-application-level telemetry is not configured; results use Azure platform metrics. Add AMS or an APM connector for HANA/SAP depth."
• "Health check for AB1" / "Status of all systems"
• "Any CPU or memory pressure on AB3 VMs?"
• "Are there network issues between DB and App servers?"
• "Check if AB1 VMs have accelerated networking"

Topology handling (all 8 system types)

Read the target system's architecture (scale-up|scale-out) and deployment (standalone|distributed|high-availability|disaster-recovery) from the inventory and adapt — never assume AB1's single-VM shape:

• scale-out → the HANA DB spans master + worker (+ standby) nodes. Group every HANA/OS KQL by HOST_s and report each DB node's L2/L4 health individually; flag worker imbalance and standby readiness. scale-up = one active DB node per replica.
• standalone / distributed → no Pacemaker/HSR — set L3 Cluster = ⚪ N/A and skip the Pacemaker query (never error).
• high-availability → run the L3 Pacemaker + HSR checks.
• disaster-recovery → also show the cross-region DR replica (ha_role: dr, dr.region) as a separate health line with its async HSR status; don't fold it into the primary site.

Data Sources

Source	Primary/Fallback	Freshness	What It Provides
Azure Monitor Metrics	Primary	Real-time (1 min)	VM CPU, memory, disk IOPS/latency, network
Azure Resource Health	Primary	Real-time	Platform events (planned/unplanned maintenance)
Azure VM Power State	Primary	Real-time	Running/stopped/deallocated
AMS Log Analytics	Primary	2-5 min	HANA availability, OS metrics, Pacemaker state, SAP processes
Activity Log	Primary	Real-time	Recent changes
sap-configs blob (agent MI, direct)	Primary for stored configs	Cron (weekly)	ethtool output, VM configs, accelerated networking — read directly with the agent's own Managed Identity (az storage blob ... --auth-mode login); no proxy
MCP command proxy (run_batch, optional)	Live enrichment only	Live	Freshest ethtool / PPG / live VM state — only if the MCP command proxy is configured

Authentication

IMPORTANT — Azure API Access: Do NOT use IMDS tokens (169.254.169.254) or ManagedIdentityCredential — they are not available in the agent sandbox. Instead:

• For Azure Resource Manager queries: Use the built-in GetArmResourceAsJson or RunAzCliReadCommands tools
• For Log Analytics queries: Use the built-in QueryLogAnalyticsByWorkspaceId tool
• For metrics: Use the built-in GetMetricTimeSeriesElementsForAzureResource tool
• For live VM commands: invoke the SAP Command Runner skill (the sap-sre-proxy MCP connector) — never make direct HTTP or proxy calls

# Use the built-in tools above for Azure API access.
import json
from datetime import datetime, timedelta, timezone

# SUB_ID: Use subscription_id from Team Onboarding
# AMS_WORKSPACE_ID: Use ams_workspace_id from Team Onboarding (only if AMS is listed)

Getting VM data (adaptive — no dead endpoints):

• Stored configs (accelerated networking, ethtool, VM configs) — read directly from the sap-configs blob with the agent's own MI: RunAzCliReadCommands → az storage blob download --auth-mode login --account-name <sa> --container-name sap-configs --name <SID>/<host>/latest/.... Available whenever a Storage Account is listed.
• Live VM state (freshest ethtool / PPG / uptime) — only when the MCP command proxy is configured: invoke the SAP Command Runner skill (run_batch) with the health commands below. If it is not configured or errors, silently use the stored blob configs. Never block on it.
• Landscape inventory — read from the agent Knowledge Base (primary) or the sap-configs blob directly. There is no /api/registry endpoint — the proxy is MCP (live commands only).

Standard commands for health checks:

HEALTH_COMMANDS = [
    {"id": "ethtool", "cmd": "ethtool -i eth0 2>/dev/null | grep driver"},
    {"id": "accel_net", "cmd": "ethtool -i eth0 2>/dev/null | grep -q 'driver: mlx' && echo 'enabled' || echo 'disabled'"},
    {"id": "uptime", "cmd": "uptime"},
    {"id": "free_mem", "cmd": "free -m | grep Mem"},
    {"id": "df_usage", "cmd": "df -h /hana/data /hana/log /hana/shared 2>/dev/null || df -h /"},
]

Health Dashboard — 5 Layers

Layer 1: Azure Infrastructure (ARM API + Azure Monitor)

Check	Source	GREEN	AMBER	RED
VM Power State	ARM instance view	Running	—	Stopped/Deallocated
CPU % (avg 1h)	Azure Monitor	<70%	70-90%	>90%
Memory % (avg 1h)	Azure Monitor	<80%	80-95%	>95%
Data Disk IOPS %	Azure Monitor	<70%	70-90%	>90% (throttled)
Network Packets Dropped	Azure Monitor	0	1-100	>100
Accelerated Networking	ARM NIC	Enabled	—	Disabled
Proximity Placement	ARM PPG	In PPG	—	No PPG
Resource Health	Resource Health API	Available	Degraded	Unavailable

Layer 2: Guest OS (AMS)

Run Prometheus_OSExporter_CL | getschema FIRST — the node_exporter column names below are a template and vary by AMS collector version; bind to the real columns before running.

Prometheus_OSExporter_CL
| where TimeGenerated > ago(15m)
| summarize latest_cpu=max(node_cpu_seconds_total_d), latest_mem=max(node_memory_MemTotal_bytes_d - node_memory_MemAvailable_bytes_d) by Computer

Layer 3: Pacemaker Cluster (AMS — only when deployment is high-availability or disaster-recovery; skip for standalone/distributed)

Run Prometheus_HaClusterExporter_CL | getschema FIRST — the ha_cluster_* column names below are a template and vary by AMS collector version; bind to the real columns before running.

Prometheus_HaClusterExporter_CL
| where TimeGenerated > ago(15m)
| summarize nodes_online=dcountif(ha_cluster_pacemaker_nodes_d, ha_cluster_pacemaker_nodes_d == 1) by ClusterName_s

Layer 4: HANA Database (AMS)

Run getschema first — column names vary by AMS version. HANA tables key on sapsid_s (not SID_s); availability fields are SYSTEM_ACTIVE_s / DATABASE_ACTIVE_s / HOST_ACTIVE_s.

SapHana_SystemAvailability_CL
| where TimeGenerated > ago(15m)
| summarize arg_max(TimeGenerated, *) by sapsid_s, HOST_s
| project sapsid_s, HOST_s, SYSTEM_ACTIVE_s, DATABASE_ACTIVE_s, DATABASE_NAME_s

Layer 5: SAP Application (AMS — only if the NetWeaver provider is configured)

Optional provider. If SapNetweaver_GetProcessList_CL doesn't exist in the workspace, skip Layer 5 and note the NetWeaver provider isn't configured.

SapNetweaver_GetProcessList_CL
| where TimeGenerated > ago(15m)
| summarize arg_max(TimeGenerated, *) by sapsid_s, instanceNr_s, name_s
| project sapsid_s, instanceNr_s, name_s, dispstatus_s

Output Format

Traffic-light dashboard per system:

AB1 — Overall: 🟢 GREEN
  L1 Infrastructure: 🟢 (CPU 23%, Mem 45%, Disk 12%, AccelNet ✓, PPG ✓)
  L2 Guest OS:       🟢 (CPU 18%, Mem 42%, Swap 0%)
  L3 Cluster:        ⚪ N/A (no HA)
  L4 HANA:           🟢 (ACTIVE, DB1 online)
  L5 SAP App:        🟡 (Instance 02 dispatcher GRAY)
  Alerts:            🟢 No active alerts for AB1

Alert Filtering — CRITICAL

When checking alerts for a specific SAP system, ALWAYS filter by that system's provider instance tag.

Alert rules in AMS are scoped to the shared Log Analytics workspace, which monitors ALL SAP systems (AB1, HSO, etc.). If you query alerts without filtering, you will see alerts from OTHER systems — causing false positives.

How to filter:

1. When querying fired alerts via RunAzCliReadCommands, filter by the sap-system or profile-id tag on the alert rule
2. Use this pattern:

az monitor scheduled-query list --resource-group mrg-sapmon-abb --subscription {SUB_ID} --query "[?tags.\"sap-system\"=='{SID}']"

3. Or when listing fired alert instances, cross-reference the alert rule name against the system being checked

Provider instance mapping (from Team Onboarding):

System	AMS Provider Instance	Alert Tag
AB1	sap-hana-pr-AB1	sap-system: AB1
HSO	hana-pr-HSO	profile-id: hana-pr-HSO

NEVER show alerts from system X when the user asks about system Y. If unsure which system an alert belongs to, check the alert rule's KQL query for the PROVIDER_INSTANCE_s filter or the alert rule's tags.