acm-load-analysis

name	acm-load-analysis
description	Analyze test results from acm-deploy-load.py or acm-telco-core-load.py. Produces resource consumption reports from a single result or side-by-side comparisons of two results. Covers timing, hub cluster CPU/memory/network/disk/etcd, component breakdowns, pod health, and operational flags.
when_to_use	analyze test results, compare test runs, resource consumption report, hub cluster analysis, test result comparison, ODF vs No-ODF comparison, review test data, review results data
argument-hint	[result-dir] or [result-a] vs [result-b]

ACM Load Test Result Analysis

Analyze and compare test results from acm-deploy-load.py and acm-telco-core-load.py.

When to use

• Single result analysis: user points to a result directory and wants a full resource consumption breakdown (CPU, memory, network, disk, etcd, components)
• Two-result comparison: user has two result directories and wants to know what changed between them (e.g., ODF vs No-ODF, different cluster counts, version upgrades)
• Timing-focused analysis: user wants deployment duration breakdowns, per-cluster install timing percentiles (ICI, CGU, CI), or peak concurrency numbers
• Pod health check: user wants to know if any pods crashed, OOM'd, or error'd

Workflow

Follow these steps in order. Do not skip the identification or inventory steps — the directory structure determines what data is available.

Step 1: Identify the Result(s)

Determine if this is a single-result analysis or a two-result comparison.

For each result directory, read report.txt to identify:

• Which tool produced the result — check the report title:
  • acm-deploy-load Report Card → acm-deploy-load
  • acm-telco-core-load Report Card → acm-telco-core-load
• Test metadata: ACM version, Hub OCP version, Deployed OCP version, cluster count, deployment method, success/failure rates
• Phase structure: phase durations, start/end timestamps
• Workload parameters: batch size, interval, WAN emulation, clusters per app

Step 2: Inventory Available Data

List the result directory to find what data is present:

Prometheus analysis directories (contain resource consumption stats):

• Phase directories: phase1-*, phase2-*, phase3-* (current naming) or idle-baseline-*, cluster-deployment-*, soak-baseline-* (older naming). Generated internally by acm-deploy-load.py / acm-telco-core-load.py when start-delay or end-delay is long enough (>= 15 minutes) — not only for capacity/soak runs
• Full-test Prometheus analysis generated by the orchestration scripts (not per-phase analysis):
  • acm-deploy-load: deploy-pa-* (from interval-ztp-install-all.sh)
  • acm-telco-core-load: acm-telco-load-hub-* (from acm-telco-core-load.sh)
• deploy-pa-all-* directories are generated ad-hoc manually and should be excluded unless the user specifically requests them

Workload timing files (at result top level):

• deploy-time-* — install/deployment milestones and peak concurrency
• imageclusterinstalls-*.stats — IBI per-cluster install timing
• clusterinstances-*.stats — ClusterInstance provisioning timing
• clustergroupupgrades-*.stats — CGU per-cluster policy timing
• agentclusterinstalls-*.stats — AI per-cluster install timing

Other data:

• monitor_data.csv — time-series workload state (1-minute intervals)
• install-data/ — pod snapshots, events, detailed CRs, failure data

Step 3: Read Data — Standardized Metrics

Read references/metrics-and-units.md for the canonical list of metrics, their stats file locations, units, and which statistic to extract.

Core rules — no exceptions:

• CPU: use P95 (95th percentile row in stats file)
• Memory: use Max
• Network: use P95
• Disk IOPS (read and write): use P95
• Disk throughput (read and write): use P95
• Disk usage: use Max
• etcd latency (backend commit, WAL fsync, peer RTT): use P99
• etcd DB size: use Max
• Pod/object counts: use Max
• PVC usage: use Max
• Never use mean/average for any resource metric

Unit rules — always verify:

• CPU: cores (no conversion)
• Memory: GiB (bytes / 1024^3)
• Network: check the stats file for a Unit: header (first line). If present, use the stated unit. If absent (old results), values are in MiB/s — convert to Mbps (× 8.388608). Use Gbps when value exceeds 1000 Mbps.
• Disk usage: GB (bytes / 1000^3) — decimal, NOT binary
• Disk throughput: MB/s (bytes / 1000^2) — decimal
• etcd DB size: GB (bytes / 1000^3) — decimal
• PVC usage: GB (bytes / 1000^3) — decimal

For full unit conversion details and stats file parsing, see .claude/skills/shared-references/stats-file-format.md.

Stats file location rule: Stats files are always at {component}/stats/{metric}.stats — never at {component}/{metric}.stats. The stats/ subdirectory is mandatory.

Step 4: Analyze — Single Result

For a single result, produce these sections:

4a. Test Summary

From report.txt:

• Versions (ACM, Hub OCP, Deployed OCP)
• Method and scale (cluster count, deployment method)
• Success rates (cluster install, managed, DU profile)
• Phase durations

4b. Timing Analysis

From deploy-time-*:

• Cluster Install Duration, DU Profile Duration, Full Duration
• Peak Cluster Installing, Peak DU Applying, Peak Concurrency

From workload stats files (these are plaintext, NOT pandas format — read references/result-directory-layout.md for format):

• ICI per-cluster: p50, p95, p99, max (seconds)
• CGU per-cluster: p50, p95, p99, max (seconds)
• ClusterInstance total duration: p50, p95, p99, max (seconds)
• ACI per-cluster (if AI method): p50, p95, p99, max (seconds)

4c. Hub Cluster Resource Consumption

From the Prometheus analysis directory (phase directories, or full-test analysis: deploy-pa-* for acm-deploy-load, acm-telco-load-hub-* for acm-telco-core-load):

Always present cluster-level first, then per-node as drill-down detail.

Cluster-level (single aggregate column) — present first:

• CPU P95 (cores), App CPU P95 (cores)
• Memory Max (GiB), App Memory Max (GiB)
• Network Receive P95, Transmit P95 (Mbps or Gbps)
• Non-terminated Pods Max (count)

Per-node (one column per node in stats files) — present after cluster-level:

• CPU P95 (cores)
• Memory Max (GiB)
• Network Receive P95, Transmit P95 (Mbps)
• Disk IOPS Read P95, Write P95 per partition (IOPS)
• Disk Throughput Read P95, Write P95 per partition (MB/s)
• Disk Usage Max per partition (GB)
• Non-terminated Pods Max (count)

4d. Component Breakdown

For each detected component directory in the Prometheus analysis:

• CPU P95 (cores)
• Memory Max (GiB)
• Network Receive P95 (Mbps)
• Network Transmit P95 (Mbps)

Present as a table sorted by Memory Max descending.

4e. etcd Health

• DB Size Max (GB) — flag if approaching 8.59 GB (the 8 GiB binary quota)
• DB Size In Use Max (GB)
• Backend Commit Duration P99 (seconds) — flag if > 0.025s
• WAL Fsync Duration P99 (seconds) — flag if > 0.010s
• Peer Round-Trip Time P99 (seconds)
• Leader Elections Max (count) — flag if > 0

4f. Storage

• PVC Usage Max per namespace (GB)
• If ODF detected (odf/ subdir exists): Ceph used capacity Max (GB), Ceph total capacity (TB), Ceph IOPS read/write P95

4g. Pod Health

Check install-data/ for:

• Pods in CrashLoopBackOff or Error state
• Containers with OOMKilled restart reason
• Report any found with pod name, namespace, and restart count

4h. Operational Flags

• etcd DB size approaching quota (> 6 GB of 8.59 GB limit)
• Non-terminated pod count approaching max-pods limit (250 default)
• Node memory usage exceeding 80% of installed RAM
• Cluster install or DU profile failures (> 0%)

Step 5: Analyze — Two-Result Comparison

Read references/comparison-methodology.md for the full methodology.

Produce the same sections as single-result, but in side-by-side format with delta (absolute and percentage) columns.

Phase matching — MANDATORY when both results have phase directories:

When both results have phase directories, the report MUST include:

1. A separate comparison section per phase — Phase 1 (Idle), Phase 2 (Deployment), Phase 3 (Soak), etc. Each phase section contains per-node resources, cluster-level resources, component data (if available for that phase), etcd, and any ODF/Ceph data.
2. A full-test overview section using the deploy-pa-* or acm-telco-load-hub-* directories — this provides the aggregate view across the entire test and includes data not broken out per-phase (disk I/O, component breakdown).
3. A cross-phase trends summary in Key Differences — highlight how metrics change across phases (e.g., memory growth from idle to soak, network peaks during deployment).

Do NOT produce only a full-test comparison when phase data exists. The phases reveal workload-specific differences (idle overhead vs deployment peak vs sustained soak) that full-test aggregates obscure.

When phase directories are absent from one or both results:

• If neither has phase directories, compare full-test analysis directories only
• If only one has phases, compare full-test analysis directories from both

Significance thresholds (resource metrics — CPU, memory, network, disk):

• < 5%: within run-to-run variance, not significant
• 5-10%: minor difference, note but don't emphasize
• 10-25%: notable difference, highlight

• 25%: significant difference, call out explicitly

Significance thresholds (timing metrics — ICI, CGU, CI, deploy durations):

• < 5%: run-to-run variance
• 5-15%: minor difference

• 15%: notable difference, highlight

Causation rule: Report observations, not theories. State what differs and by how much. Do not assert why unless the test design explicitly isolates that variable.

Step 6: Generate Report

Output as structured markdown. Every table must include units in column headers.

Write the report to files in the results/ directory — produce both a markdown (.md) and a plain text (.txt) version:

• Single result: results/analysis-{result-suffix}.md and .txt (e.g., results/analysis-odf-ibi-extended-12.md)
• Comparison: results/comparison-{suffixA}-vs-{suffixB}.md and .txt (e.g., results/comparison-noodf-ibi-extended-10-vs-odf-ibi-extended-11.md)

The markdown version uses standard markdown tables. The plain text version uses aligned columns with ASCII separators (---, ~~~, ===) for terminal and email readability.

Derive the suffix from the result directory name — use the portion after the timestamp-method prefix (the human-readable label at the end). Display the report content in the conversation and save both files.

Important considerations

• Never fabricate data. Every value in a report must come from an actual file that was read during analysis. If a stats file is missing, unreadable, or does not contain the expected row, omit that metric from the report and note it as unavailable — never fill in a plausible value. If information needed for the report (e.g., hardware specs, test parameters, storage backend type) cannot be determined from the test artifacts, ask the user rather than guessing.

• Never assume components. Check which component directories actually exist in the Prometheus analysis directory. Not all runs will have ODF, LSO, AAP, etc.

• Cluster vs node distinction. Cluster stats (cluster/stats/) sum all container activity across the cluster — they will be much larger than node stats (node/stats/) which are per-physical-host NIC/CPU/memory. Never confuse the two. Report per-node stats for sizing recommendations; cluster stats for total capacity context. For network specifically, cluster-level measures total container network I/O including intra-node pod-to-pod traffic that never leaves the host (e.g., API server to etcd, Prometheus scraping, OVN internal, Ceph replication between co-located OSDs). Add a note below cluster-level resource tables in reports explaining this distinction.

• Component memory file selection. A component directory may contain multiple memory stats files (sub-groupings, per-pod files). Always use mem-{component}.stats for the component total. Sub-component files will undercount.

• Non-terminated pod count. Use cluster/stats/nonterm-pods-cluster.stats or node/stats/nonterm-pods-node.stats for schedulable pod count. Do NOT use resource/stats/pods.stats — it includes terminated/completed pods and will overcount against the max-pods limit.

• etcd DB size quota. The default etcd quota is 8 GiB binary = 8.59 GB decimal. Stats files report in GB (decimal). Compare observed values against 8.59 GB, not 8.00.

• Phase directories are not only for soak/capacity runs. They are present whenever start-delay or end-delay exceeds ~15 minutes. Don't assume phase presence implies a particular test type.

• Tool identification. Read the report.txt title line (acm-deploy-load Report Card vs acm-telco-core-load Report Card) to determine which tool produced the result. Do not guess from the directory name.

• ODF/Ceph data availability. Ceph metrics are only present in Prometheus analysis directories where the ODF component was detected. Check for the odf/ subdirectory before attempting to read Ceph stats. Ceph stats will NOT exist in phase directories unless ODF was installed on the hub cluster.

• Workload timing stats format. The ICI, CGU, CI, and ACI .stats files at the result top level use a plaintext format (not pandas describe output). Read references/result-directory-layout.md for the exact format.