| name | chaos-analysis |
| description | Analyze recent E2E Chaos runs for packet loss and downtime, attribute both to fault actions, compare nights, and derive concrete smoothness measures. Use when asked to analyze chaos/E2E runs, investigate packet loss or failover downtime, or check whether a change made the agent smoother. |
Chaos run smoothness analysis
Aggregate the chaos-run records of several E2E Chaos runs, attribute
packet loss and downtime to fault actions, and turn the worst offenders
into concrete, issue-ready measures.
1. Collect the runs
gh run list --workflow "E2E Chaos" --limit 10 \
--json databaseId,conclusion,createdAt,event
Pick the runs to compare (default: the last 4 with artifacts — artifact
retention is 7 days). Download each into its own directory:
for run in <id...>; do
gh run download "$run" --repo osism/ovn-network-agent --dir e2e-runs/"$run" &
done; wait
Nightly runs hold one artifact per profile (6); a dispatch run holds one.
2. Aggregate
python3 .claude/skills/chaos-analysis/analyze.py e2e-runs/* > report.md
python3 .claude/skills/chaos-analysis/analyze.py e2e-runs/<one-run>
--format json emits the same aggregation for scripted drill-downs.
Always ALSO run the per-night breakdown for each run: the pooled table
mixes seeds, and a fix shows up as one night improving, not as the pool
improving. Correlate nights with git log — nightlies run ~06:00 UTC,
so a fix merged in the evening first shows in the next morning's run.
3. Read the numbers — semantics that matter
- Per recovery event,
from_inject_ms (per probe) is downtime measured
from fault injection; from_restore_ms is the part after the fault was
lifted. The hold (hold_ms on the decision journal event, 0–45 s
by action) is the fault window itself.
- Downtime ≈ hold + ~1–3 s, per probe means that traffic class never
failed over and waited for the node to come back. That is the smell to
chase: with three gateways, loss should end at failover (a few
seconds), not at restore.
from_restore_ms (residual) is the tail the recovery path owns —
the agent's reaction plus BGP re-establishment after the node returns.- Probe classes:
fip-vm* = FIPs on the flat (Geneve-backed) provider
network, fip-vlan10* = FIPs on VLAN provider networks, pf-vip =
DNAT port-forward VIP, api-vip = LB VIP. Which classes go dark in an
event tells you which networks' CR ports the target chassis owned.
check-error events with ovn-sbctl --timeout=5 during sb-pause
holds are harness-side sweep noise, not agent regressions.- Actions whose median worst downtime is 0 with occasional small
after
loss windows (route drops, churn, pauses) are healthy.
4. Known-good baseline (2026-07-30, runs 30288259170..30518214683)
- Overall probe loss 5.13 % pooled; best night 0.4–1.9 % per profile.
- Fast and fine: all pause/churn/drop actions converge in ~100–300 ms
with no downtime.
- Standing offenders (each ends ~1–3 s after restore, i.e. no failover):
gateway-kill/double-failover up to 42 s on VLAN+PF probes,
controller-restart ~15 s, agent-terminate ~19 s,
upstream-bgp-restart ~15–19 s on every probe,
config-flip/gateway-restart 6–14 s.
- A consistent ~2.4–3.2 s residual follows nearly every restore.
Regressions are deviations from this shape, not from zero.
5. Derive measures and file issues
For each offender, name the mechanism before proposing a fix: replay it
first (make e2e-chaos CHAOS_FLAGS="-seed <seed> -profile <profile> -duration 10m" — seed and profile are in the report/summary.json) when
the mechanism is unclear. Then propose one measure per root cause, each
with: the measured cost (seconds of downtime × frequency), the affected
probe classes, the suspected code/config surface, and the replay line.
File issues per the user's convention: a few independent, fully-detailed
issues (not one plan), gh issue create with measured evidence and
acceptance criteria phrased against these metrics (e.g. "worst-probe
downtime for gateway-kill on the owner drops below 5 s").
