on-call-sla-troubleshooting

name	On-Call SLA Troubleshooting
description	SLA path failure workflow — read sla_paths.json, traceroute-first localization, ECMP handling, protocol triage

On-Call SLA Troubleshooting Skill

Scope

On-Call mode investigation guide — reading sla_paths.json, traceroute-first localization, ECMP handling, protocol triage order.

Terminology

• Primary On-Call session: Triggered directly by a new SLA failure event. Full workflow: Steps 0 through 3, session closure, Jira, lessons. Runs in non-interactive print mode — exits autonomously after presenting the summary.
• Deferred failures: SLA Down events that arrive while a primary session is active. After the session ends, watcher.py documents them automatically (Jira comment + Discord embed). No second agent session is spawned.

Before starting: Read cases/lessons.md per CLAUDE.md guidelines — past lessons often shortcut diagnosis.

---

Step 0: Read the sla_paths.json Entry for the Failed Path

Use the Read tool to read the file sla_paths/paths.json (this is a local file — do NOT use readMcpResource).

Locate the path entry matching the source device from the log event. Extract these key fields:

• source_device: the device that generated the SLA failure event
• destination_ip: IP being pinged by the SLA
• scope_devices: ALL devices you may need to investigate
• ecmp: if true, TWO paths exist — both must be checked
• ecmp_node: the device where the path splits
• ecmp_next_hops: the two next-hop devices after the split

---

Step 1: Traceroute from Source Device (Always First)

If source_ip is available in the paths.json entry, always pass it:

traceroute(device=<source_device>, destination=<destination_ip>, source=<source_ip>)

Using source_ip forces the traceroute onto the monitored path. If the source interface is down, the traceroute fails immediately at the source device — correctly localizing the issue without alternate-path confusion.

MANDATORY scope check — do this before applying any outcome below:

Compare EVERY hop in the traceroute output against the scope_devices list from Step 0. Resolve IP addresses to device names using the inventory or context you already have. If ANY intermediate hop is a device NOT in scope_devices, this is an off-path transit — even if the traceroute reached the destination. The path is using an alternate route, NOT the monitored path. Apply the third outcome below.

Do NOT treat a completed traceroute as "successful" if it transited an off-scope device.

Read the output:

• Path stops at hop N (or fails at source): issue is on or before that hop → proceed to Step 2.5
• Timeout at first hop: issue is on the source device itself (interface down, OSPF/BGP neighbor lost, no default route) → proceed to Step 2.5
• Path transits a device NOT in scope_devices: routing anomaly on the last in-scope hop. Do NOT investigate the off-path device. Identify the last hop that IS in scope_devices, run get_routing(<that_device>, prefix=<destination_ip>) to confirm it is routing toward the off-path device. Whether the route is present or absent, treat that in-scope device as the breaking hop and proceed immediately to Step 2.5.
• Full path to destination AND all hops within scope_devices: do NOT conclude "transient" yet — go to Step 1a

Step 1a: Source-Device Sanity Check (when traceroute succeeds)

Even if the traceroute completes, the SLA was triggered for a reason. Verify the source device's local state with exactly two queries:

get_interfaces(device=<source_device>)
get_ospf(device=<source_device>, query="neighbors")   ← run on source_device, NOT the next-hop

Critical: Always query routing protocol neighbors on the source_device first. Querying the next-hop device may show healthy adjacencies even when the source-side interface is down.

Branch A — Traceroute completed, source state healthy (source interface Up/Up AND source device's expected routing neighbors present AND all traceroute hops within scope_devices):

A completed traceroute does NOT automatically mean the path is recovered. The SLA may have recovered via a failover/alternate route while the primary defined path still has a failure. Determine which case applies:

Compare the observed traceroute hops against the expected path from Principle 1 (INTENT.json + paths.json):

Case	Traceroute Result	Action
Truly transient	Hops follow the expected path exactly (same transit devices as the expected path)	Present summary table below, log to Jira as transient/self-resolved (resolution="Won't Fix"), proceed to session closure
Failover recovery	Hops reach the destination but via different devices than the expected path (alternate ABR, alternate ISP, different next-hops)	Primary path still broken — treat the first device where the trace diverges from the expected path as the breaking hop. Proceed to Step 2.5 to investigate why traffic is bypassing the expected path

Present this summary table (fill in actual values):

| Check | Result | Status |
|-------|--------|--------|
| Traceroute to <destination_ip> | Full path, all hops respond | ✓ |
| Source interface (<source_interface>) | Up/Up | ✓ |
| Source device routing neighbors | All expected neighbors present | ✓ |
| Path follows expected route | <Yes — transient / No — via <alternate_device> — investigate> | ✓/✗ |

Branch B — Issue still present (source interface down OR expected neighbor missing):

This is the root cause. Proceed directly to Step 2.5.

---

Step 2: ECMP Handling

If the path entry has "ecmp": true, the traceroute shows ONE of the two paths. The other may also be broken.

1. Both paths broken? If the path still fails after a fix, re-run traceroute with source=<source_ip>. If it still fails or takes a different failing path, an independent failure exists on the alternate path — return to Step 1 with the new traceroute result.
2. One path fixed — verify alternate: get_routing(ecmp_node, prefix=<destination>) — expect 2 equal-cost entries. If only 1 entry, the alternate path is not restored.
3. Path identification: The SLA probe may use either ECMP path. Always use source=<source_ip> in traceroute to force the probe's perspective and avoid alternate-path confusion.

Proceed to Step 2.5 after identifying the breaking hop via traceroute. If the traceroute from Step 1 completed to the destination, at least one ECMP path is functional — then proceed to Step 2.5 to investigate the alternate path's devices if ECMP verification shows only one RIB entry.

---

Step 2.5: Basic Operational Checks (Mandatory — Run Before Anything Else)

A missing route on the breaking hop is NOT a reason to investigate other devices. It is a reason to check the breaking hop's own state first.

get_interfaces(device=<breaking_hop>)
get_ospf(device=<breaking_hop>, query="neighbors")    ← or get_bgp per Step 3 triage table

Decision gate:

Result	Action
Interface down (admin or line-protocol)	Root cause found. Present findings table. Stop.
No neighbors / fewer neighbors than expected	Go directly to the Adjacency Checklist in the protocol skill. Do NOT investigate downstream devices.
All neighbors FULL, all interfaces Up/Up	Proceed to Step 3. Issue is in LSDB, RIB, or policy layer.

Do not leave the breaking hop to investigate downstream devices in the path. Missing adjacencies on the breaking hop explain missing routes everywhere downstream. Investigating downstream only confirms the problem cascaded — it never finds the root cause. Timer mismatch, passive interface, and area mismatch are caught in one query.

One fix per layer (Principle 7): If the decision gate above identifies an interface-layer issue (admin-down, line-protocol down), propose only the interface fix — even if investigation also revealed higher-layer issues (missing BGP config, wrong route-maps, etc.). Verify first. If the SLA path is still broken after the interface fix, re-run traceroute and start a new investigation cycle for the next layer.

---

Step 3: Protocol Triage — Which Skill to Read Next

Map the breaking hop to its protocol:

Breaking Hop Device	Protocol to Investigate	Skill to Read
A1C, A2C (source)	OSPF Area 1 stub (to C1C or C2C ABR)	skills/ospf/SKILL.md
C1C, C2C	OSPF Area 0 core + ABR	skills/ospf/SKILL.md
E1C, E2C	OSPF Area 0 + BGP AS1010 + NAT	If OSPF neighbors down → skills/ospf/SKILL.md; if BGP to ISP down → skills/bgp/SKILL.md; NAT issue → skills/routing/SKILL.md
X1C	BGP AS2020 (to IAN + IBN)	skills/bgp/SKILL.md
IAN, IBN	BGP AS4040/AS5050 (ISP edge — fully managed)	skills/bgp/SKILL.md

---

Time Efficiency Rules

• Localize first, don't investigate all: traceroute narrows to 1-2 devices max before running protocol tools
• ECMP: check both paths before concluding the issue is fixed
• Don't re-check devices that are not on the scope_devices list: out-of-scope devices won't affect this SLA path
• Non-scope hop in traceroute: stay in scope: if traceroute exits scope_devices, do NOT query or investigate the off-path device. Find the last in-scope hop, run get_routing on it, then go to Step 2.5.
• No route on breaking hop → check its neighbors, not downstream devices: a missing route means "run Step 2.5 on this device now". Missing adjacencies on the breaking hop explain missing routes across the entire downstream path.
• Step 2.5 before any protocol skill section: zero neighbors is an adjacency problem (timers, passive, area, auth). Resolve it on the breaking hop before consulting LSDB, redistribution, or area-type sections.

---

Presenting Findings

Always present your analysis summary in a Markdown table before proposing a fix:

Finding	Detail	Status
Traceroute result	Stopped at hop N — device X	✗
Interface / neighbor state	e.g. Ethernet3 admin down	✗
Root cause	Brief description	✗

Use ✓ for healthy items and ✗ for the identified issues. This lets the user scan the summary instantly before approving any configuration change.

---

Step 4: Approval, Remediation & Session Closure

After presenting findings, follow this sequence exactly:

1. Call assess_risk — assess_risk(devices=<affected>, commands=<fix_commands>). Include the risk level in the findings table.

2. Call request_approval — request_approval(issue_key, summary, findings, commands, devices, risk_level). This posts an approval embed to Discord (if configured) and writes an audit record that push_config requires before it will execute.

3. Handle the return value:

   Decision	Action
   "approved"	Proceed to push
   "rejected"	Call post_approval_outcome(message_id=..., decision="rejected", decided_by=<username>), log to Jira, go to session closure
   "expired"	Call post_approval_outcome(message_id=..., decision="expired") to post expiry outcome to Discord. Log to Jira that the fix could not be applied (approval expired), then go to session closure. Never push.
   "skipped"	Discord not configured. The code-level gate blocks push_config (no APPROVED record). Log to Jira that no approval channel is configured and go to session closure. Never push.

4. NEVER call push_config without approval — push_config enforces this at the code level and will return an error if no approved record exists or if the device list does not match the approval.

5. After approval: call push_config(devices=..., commands=...), then verify the fix resolved the issue (run traceroute and/or get_<protocol> to confirm). Then call post_approval_outcome(message_id=..., decision="approved", decided_by=<username>, verified=<True|False>, verification_detail=<brief_result>).

5a. Push failure retry — when push_config returns errors on one or more devices:

• Diagnose the error. Common causes: wrong AS/area number, syntax error, transport failure.
• For partial failures (some devices succeeded, some failed), scope the retry to only the failed devices.
• Call post_approval_outcome(message_id=..., decision="approved", decided_by=<username>, verified=False, verification_detail="push failed: <brief error>") for the failed attempt.
• Correct the commands and call request_approval again with corrected commands and the failed-device list only.
• On approval, call push_config with the corrected commands and failed devices only. Continue to verification.
• Cap: 2 retries max (3 total push attempts). After the third failure, log to Jira and go to session closure.
• Each retry cycle produces its own Discord outcome embed — expected behavior.

6. Session closure: follow CLAUDE.md On-Call Session Closure — log to Jira, evaluate cases/lessons.md, present summary, then exit autonomously. The watcher resumes monitoring immediately. Deferred failures (if any) are documented by the watcher — no agent action required.

---

Jira Updates (On-Call)

Follow the Jira comment workflow in CLAUDE.md → Case Management. Use the cases/case_format.md structure for all comments. If no issue key is present, skip all Jira calls silently.