diag

name	diag
description	Analyzes VergeOS diagnostic bundles to surface storage, network, hardware, container, and Linux system issues. Use when handed a diag bundle or asked to investigate a VergeOS system's health. Applies to "analyze this diag bundle", "what's wrong with this VergeOS system", "check vSAN health", "investigate node connectivity", "diagnose this cluster", "why is this node throttling". Returns a consolidated RESULTS.md with critical issues, warnings, capacity summary, and recommended actions.
argument-hint	[path/to/bundle]
allowed-tools	Read, Grep, Glob, Bash(python3 *), Bash(for *), Bash(while *), Bash(xzcat *), Bash(zcat *), Bash(tar *), Bash(ls *), Bash(find *), Task

VergeOS Diagnostic Analyzer

Analyze VergeOS diagnostic bundles using parallel subagents to identify issues with storage, networking, hardware, and system health.

Arguments

• $ARGUMENTS: Optional path to extracted diagnostic bundle. Defaults to current directory if not specified.

Quick Start

1. Extract the diagnostic bundle: tar -xzf [SYSTEMNAME]_diags_[DATE]_[TIME].tar.gz
2. Run /diag from within the extracted directory, or /diag /path/to/bundle

Bundle Structure

Diagnostic bundles are named [SYSTEMNAME]_diags_[YYYYMMDD]_[HHMMSS].tar.gz and contain:

• Point-in-time snapshots organized by node (e.g., node1/, node2/, ...)
• The finished file confirms collection completed successfully
• Logs are retained 45 days before automatic deletion

Per-Node Directory Contents

Directory/File	Contents
vsan/	tierstatus.txt, journalstatus.txt, clusterusage.txt, devicestatus.txt, devicelist.txt, clusterrates.txt, fuseinfo.txt, integcheckstatus.txt, cacheinfo.txt, currentmaster.txt, repairstatus.txt, runningconf.txt, vsan.conf, nodelist.txt, deviceusage.txt
smart/	Per-drive SMART data (nvme0n1.txt, sda.txt, etc.)
network/	ifconfig.txt, ipaddr.txt, ybfabric.txt, nicinfo.txt, arp.txt, iproute.txt, lldpneighbors.txt, bridgefdb.txt, bond* files
lsblk/	Per-drive JSON files (lsblk_sda.json, lsblk_nvme0n1.json, etc.)
container_logs/	Per-container subdirs (vnet1/, vnet2/, vnet3/, nfstap25/, etc.) with messages files
proc/	meminfo.txt, vmstat.txt, diskstats.txt, interrupts.txt
sysstat/	sar binary (sa*) and text (sar*) files
boot/	ybos-version.txt, install-settings.txt, network-config.txt
kernel_logs/	Kernel ring buffer output

Top-Level Bundle Files

File	Contents
system-settings.json	Cloud name, domain, NTP, MTU config
alarms.json	Active system alarms
alarm_history.json	Recent alarm history
yottabyte-boot.log	VergeOS boot sequence log
boot-syslog	Early kernel/init messages
containerslist.txt	Container inventory with states
db_rowcounts.txt	Database table row counts
db_sizes.txt	Database table sizes
*-logs.tgz	Timestamped log archives from /vsan/vol/nodes/

Analysis Workflow

This skill's bundled files (the scripts/ folder and the *.md reference files) live in its base directory — the path shown as "Base directory for this skill" when the skill loads. Refer to that absolute path as SKILL_DIR throughout. Using SKILL_DIR keeps the skill working whether it's installed as a Claude Code plugin (loaded from a cache directory) or run from a local checkout.

Step 1: Verify Bundle and Determine Path

Check the finished file exists to confirm diagnostics completed. Set the bundle path:

• If $ARGUMENTS provided, use that path
• Otherwise, use current directory

Step 1.1: Scrub PII from Bundle

MANDATORY — Before reading ANY files from the bundle, scrub it. Customer diagnostic bundles contain identifying information (domains, emails, login IPs, session tokens, public IPs) that must not be read or analyzed in raw form. Run:

python3 "$SKILL_DIR/scripts/scrub-bundle.py" "<BUNDLE_PATH>"

This copies the bundle to <BUNDLE_PATH>/tmp/<BUNDLE_NAME>/, scrubs the copy in place (replacing customer-identifying data with numbered placeholders), and prints the scrubbed copy's path to stdout — the original bundle is never modified. It reuses an existing scrubbed copy if one is already present (.pii-scrubbed marker), so it's safe to re-run.

Capture the path printed on stdout and set BUNDLE_PATH to it. Use the scrubbed copy path for ALL subsequent steps.

Step 1.5: Read System Context

Before launching agents, read from any node directory:

1. system-settings.json → extract cloud_name (system identity)
2. boot/ybos-version.txt → VergeOS version
3. alarms.json → active alarm count and severity summary
4. List node directories to determine cluster size

Pass this context (cloud_name, version, node count) to all subagents.

Step 2: Launch Parallel Analysis Agents

Spawn these subagents IN PARALLEL using the Task tool with model: sonnet:

Agent 1: Storage Analyzer

Analyze VergeOS vSAN storage health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].

For EACH node directory, check these vSAN files in priority order:
1. vsan/tierstatus.txt - Per-tier: working=false CRITICAL, redundant=false WARNING, bad_drives>0, repairs>0, progress
2. vsan/journalstatus.txt - status (idle/collapsing=OK, active=check), alive=1, redundant=1, curmaster, tier lag
3. vsan/clusterusage.txt - Per-tier used_pct: >=85% WARNING, >=91% CRITICAL (throttling starts)
4. vsan/repairstatus.txt - All device values should be 0. Non-zero = active repair
5. vsan/devicestatus.txt - Check rd_errs, wr_errs, mismatch, oos_blocks (all should be 0).
   Latency: lat_w_avg_full >1000us WARNING, >5000us CRITICAL. throttle>0 = active throttling
6. vsan/clusterrates.txt - write_throttle != "unlimited" = throttled. wrate<1KB/sec CRITICAL
7. vsan/fuseinfo.txt - throttled_reads>0 or throttled_writes>0 = active throttling
8. vsan/integcheckstatus.txt - bad_blocks, bad_files, bad_dirs should all be 0
9. vsan/cacheinfo.txt - Calculate hit rate: cache_hits/(cache_hits+cache_misses). <50% WARNING
10. vsan/currentmaster.txt - Report master node identity
11. smart/*.txt - NVMe: available_spare<10% CRITICAL, media_errors>0 CRITICAL, percentage_used>95% CRITICAL.
    SATA: reallocated>0, pending>0, uncorrectable>0 all CRITICAL

Cross-node: Compare tier status and device health across all nodes for inconsistencies.
Context: tier 0 working=false with tier 3 working=true may be metadata-only issue, not full outage.

Report: CRITICAL first, WARNINGS, INFO. Include specific values found.
Reference: $SKILL_DIR/storage.md

Agent 2: Network Analyzer

Analyze VergeOS network health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].

For EACH node directory:
1. network/ybfabric.txt - Parse JSON. Per peer node:
   - confirmed=false CRITICAL (path not verified, connectivity lost)
   - confirmed=true = healthy bidirectional communication
   - Score maps to NIC link speed: 10Gbps=20, 25Gbps=50, 50Gbps=100, 100Gbps=200.
     Determine expected max from nicinfo.txt link speed. Score below max = degraded WARNING
   - Missing peer nodes = cluster partition CRITICAL
   - syncing_time=true WARNING (should be false after boot)
   - Check vxlans[] for VXLAN tunnel assignments
2. network/ifconfig.txt - All interfaces:
   - Bond: errors>0, calculate drop rate vs total packets (>0.1% = investigate)
   - vx2: must be UP,BROADCAST,RUNNING. MTU should be NIC MTU - 50
   - br2: core fabric bridge
3. network/nicinfo.txt - NIC model, driver, link speed. All fabric NICs should match speed
4. network/bond* - Mode (802.3ad/active-backup), slave status, miimon setting
5. network/ipaddr.txt - Verify fabric IPs present (172.16.x.x or configured range)
6. network/bridgefdb.txt - VXLAN FDB entries. Missing peers or stale entries
7. network/lldpneighbors.txt - Physical switch topology verification
8. network/arp.txt - Incomplete entries = connectivity issues

Cross-node: Compare fabric scores — one node lower = that node's NIC/cable issue.
Context: Core network 100.96.0.x/24, VXLAN VNI=2 UDP 4789, ybfabric port 14202.

Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/network.md

Agent 3: Hardware Analyzer

Analyze hardware health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].

For EACH node directory:
1. ipmisensor.txt - CPU >85C CRITICAL, >70C WARNING. Drives >51C CRITICAL, >45C WARNING.
   Fans RPM=0 CRITICAL. Voltages out of range CRITICAL.
2. ipmisel.txt - Flag: uncorrectable ECC, thermal events, PSU failures, fan failures
3. smart/*.txt - NVMe: Available Spare <10% CRITICAL, Media Errors >0 CRITICAL,
   Percentage Used >95% CRITICAL, Unsafe Shutdowns (high = power issues).
   SATA: Reallocated >0, Pending >0, Offline Uncorrectable >0 CRITICAL.
4. ras-edac-info.txt - ECC: correctable frequent WARNING, uncorrectable CRITICAL. ID affected DIMM.
5. dmidecode.txt - Memory config, CPU model/count
6. lspci.txt - PCI inventory, GPU presence, NVMe controllers
7. proc/meminfo.txt - MemAvailable <15% CRITICAL. Note: high Mlocked is NORMAL (VM hosts).
   No swap is by design on VergeOS.
8. sensors.txt - Current thermal readings cross-reference with IPMI
9. lsblk/*.json - Per-drive partition layout (p1=boot, p2=vSAN data)

Cross-node: Compare hardware configs for heterogeneity.
Node types: Controller (DB, PXE), Compute (workload), Storage+Compute (hyperconverged).

Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/hardware.md

Agent 4: Linux System Analyzer

Analyze Linux system health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].

IMPORTANT VergeOS context:
- /ybos at 100% is NORMAL (read-only loop mount)
- High Mlocked/Unevictable memory is NORMAL (VMs pinned in RAM)
- No swap is by design
- Root filesystem is overlay on tmpfs (8GB)

For EACH node directory:
1. df.txt - overlay / >85% WARNING, >90% CRITICAL. /vsan usage. IGNORE /ybos 100%.
2. free.txt + proc/meminfo.txt - MemAvailable <20% WARNING, <15% CRITICAL (factor out Mlocked)
3. psaux.txt - Must have: ybvsan, ybfabric. Controller nodes: appserver.
   D-state processes = stuck I/O. Zombie count. High CPU qemu = normal VM workload.
4. syslog + syslog.1 - Search: "oom", "out of memory", "killed process", "segfault",
   "ybvsan", "ybfabric", "appserver", "hard reset", "clock.*jump"
5. yottabyte-boot.log - VergeOS boot: ybfabric start, vSAN mount, kernel selection errors
6. boot-syslog - Early boot issues
7. sysstat/sar* - %iowait sustained >20% = storage bottleneck, CPU trends
8. proc/vmstat.txt - oom_kill count
9. lsmod.txt - Verify: vfio, kvm, bonding, vxlan, overlay

Systemd order: yb-config → basic.target → vsan.service → appserver.service
Watchdog: vSAN crash → syslog archived → hard reboot (unless /boot/.no-vsan-reboot)

Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/linux.md

Agent 5: Container/Service Analyzer

Analyze VergeOS container services across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].

For EACH node directory:
1. containerslist.txt - All containers. Expected: vnet1 (DMZ), vnet2 (Core), vnet3 (External).
   Additional are user-created. Identify each by container_name.txt in container_logs/ subdirs.
2. container_logs/vnet1/messages (DMZ) - dnsmasq: "no servers found" = upstream DNS issue,
   DHCP pool exhaustion, frequent restarts. Pool default: 100.64.191.254-100.64.255.254
3. container_logs/vnet2/messages (Core) - appserver errors, database issues, API timeouts
4. container_logs/vnet3/messages (External) - routing failures, NAT issues, upstream connectivity
5. container_logs/*/messages - All other containers. Read container_name.txt to identify purpose.
6. alarms.json - Active alarms. Owner types: vms/N, nodes/N, vnets/N, system.
   Levels: info, warning, error, critical. Flag critical/error alarms.
7. alarm_history.json - Recent alarm patterns (if present)
8. system-settings.json - cloud_name, domain, NTP config for context

Container architecture: each vnet = LXC container with own network namespace.
Services: nftables, dnsmasq, BIND9, strongSwan, WireGuard, FRRouting, HAProxy.
Writable mounts: root(1MB), tmp(128MB), var/log(5MB), run(1MB) — cleared on restart.

Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/containers.md

Step 3: Consolidate Findings

After all agents complete, consolidate their reports into a unified summary:

## Diagnostic Summary for [CLOUD_NAME]

### System Identity
- **Cloud Name**: [from system-settings.json]
- **VergeOS Version**: [from boot/ybos-version.txt]
- **Nodes**: [count and list]
- **vSAN Master**: [from currentmaster.txt]

### Active Alarms
[Summary from alarms.json — count by severity, list critical/error alarms]

### Critical Issues (Immediate Action Required)
[List all CRITICAL findings from all agents]

### Warnings (Address Soon)
[List all WARNING findings from all agents]

### Storage Capacity Summary
| Tier | Used | Max | Used% | Status |
|------|------|-----|-------|--------|
[Per-tier from clusterusage.txt]

### Observations
[List all INFO findings from all agents]

### Recommended Actions
1. [Prioritized action items]
2. ...

Step 4: Write Results and Finish

After consolidating the report:

1. Write RESULTS.md — Save the consolidated summary to [BUNDLE_PATH]/RESULTS.md using the Write tool
2. Create finished file — Create an empty [BUNDLE_PATH]/finished file to mark analysis as complete
3. Output the report — Display the full RESULTS.md content to the user so they can review it directly in the conversation

Reference Files

• storage.md - vSAN storage analysis
• network.md - Network diagnostics
• hardware.md - IPMI, SMART, hardware
• linux.md - Linux system and VergeOS-specific checks
• logs.md - Log analysis patterns
• containers.md - Container services and alarms

Key Thresholds

Metric	Warning	Critical
Storage usage (tier)	85%	91%+ (throttling)
CPU Temperature	70C	85C+
Drive Temperature	45C	51C+
Memory available	<20%	<15%
Overlay disk space (/)	>85%	>90%
SMART reallocated	>0	>1
Drive wear	>90%	>95%
Load average	>CPU count	>2x CPU count
Fabric score	Below link-speed max	confirmed=false
Write latency (lat_w_avg_full)	>1000µs	>5000µs
Device rd_errs/wr_errs	—	>0
Cache hit rate	<70%	<50%

Fabric score reference (score = link speed indicator, NOT fixed at 200):

NIC Link Speed	Expected Max Score
10 Gbps	20
25 Gbps	50
50 Gbps	100
100 Gbps	200

Base OS Context

VergeOS runs on Linux with:

• Upstream kernel (not modified)
• Debian packages for non-Verge components
• /ybos is a read-only loop mount (100% usage is NORMAL)
• Root filesystem is overlay on tmpfs (8GB)
• No swap by design
• Critical processes: ybvsan, ybfabric, appserver (controller only), qemu-system-x86_64 (VMs)

Compressed Log Handling

xzcat syslog.xz | grep -i error
zcat syslog.2.gz | grep -i warning