| name | diag |
| description | Analyzes VergeOS diagnostic bundles to surface storage, network, hardware, container, and Linux system issues. Use when handed a diag bundle or asked to investigate a VergeOS system's health. Applies to "analyze this diag bundle", "what's wrong with this VergeOS system", "check vSAN health", "investigate node connectivity", "diagnose this cluster", "why is this node throttling". Returns a consolidated RESULTS.md with critical issues, warnings, capacity summary, and recommended actions. |
| argument-hint | [path/to/bundle] |
| allowed-tools | Read, Grep, Glob, Bash(python3 *), Bash(for *), Bash(while *), Bash(xzcat *), Bash(zcat *), Bash(tar *), Bash(ls *), Bash(find *), Task |
VergeOS Diagnostic Analyzer
Analyze VergeOS diagnostic bundles using parallel subagents to identify issues with storage, networking, hardware, and system health.
Arguments
$ARGUMENTS: Optional path to extracted diagnostic bundle. Defaults to current directory if not specified.
Quick Start
- Extract the diagnostic bundle:
tar -xzf [SYSTEMNAME]_diags_[DATE]_[TIME].tar.gz
- Run
/diag from within the extracted directory, or /diag /path/to/bundle
Bundle Structure
Diagnostic bundles are named [SYSTEMNAME]_diags_[YYYYMMDD]_[HHMMSS].tar.gz and contain:
- Point-in-time snapshots organized by node (e.g.,
node1/, node2/, ...)
- The
finished file confirms collection completed successfully
- Logs are retained 45 days before automatic deletion
Per-Node Directory Contents
| Directory/File | Contents |
|---|
vsan/ | tierstatus.txt, journalstatus.txt, clusterusage.txt, devicestatus.txt, devicelist.txt, clusterrates.txt, fuseinfo.txt, integcheckstatus.txt, cacheinfo.txt, currentmaster.txt, repairstatus.txt, runningconf.txt, vsan.conf, nodelist.txt, deviceusage.txt |
smart/ | Per-drive SMART data (nvme0n1.txt, sda.txt, etc.) |
network/ | ifconfig.txt, ipaddr.txt, ybfabric.txt, nicinfo.txt, arp.txt, iproute.txt, lldpneighbors.txt, bridgefdb.txt, bond* files |
lsblk/ | Per-drive JSON files (lsblk_sda.json, lsblk_nvme0n1.json, etc.) |
container_logs/ | Per-container subdirs (vnet1/, vnet2/, vnet3/, nfstap25/, etc.) with messages files |
proc/ | meminfo.txt, vmstat.txt, diskstats.txt, interrupts.txt |
sysstat/ | sar binary (sa*) and text (sar*) files |
boot/ | ybos-version.txt, install-settings.txt, network-config.txt |
kernel_logs/ | Kernel ring buffer output |
Top-Level Bundle Files
| File | Contents |
|---|
system-settings.json | Cloud name, domain, NTP, MTU config |
alarms.json | Active system alarms |
alarm_history.json | Recent alarm history |
yottabyte-boot.log | VergeOS boot sequence log |
boot-syslog | Early kernel/init messages |
containerslist.txt | Container inventory with states |
db_rowcounts.txt | Database table row counts |
db_sizes.txt | Database table sizes |
*-logs.tgz | Timestamped log archives from /vsan/vol/nodes/ |
Analysis Workflow
This skill's bundled files (the scripts/ folder and the *.md reference files)
live in its base directory — the path shown as "Base directory for this skill"
when the skill loads. Refer to that absolute path as SKILL_DIR throughout. Using
SKILL_DIR keeps the skill working whether it's installed as a Claude Code plugin
(loaded from a cache directory) or run from a local checkout.
Step 1: Verify Bundle and Determine Path
Check the finished file exists to confirm diagnostics completed. Set the bundle path:
- If
$ARGUMENTS provided, use that path
- Otherwise, use current directory
Step 1.1: Scrub PII from Bundle
MANDATORY — Before reading ANY files from the bundle, scrub it. Customer diagnostic
bundles contain identifying information (domains, emails, login IPs, session tokens,
public IPs) that must not be read or analyzed in raw form. Run:
python3 "$SKILL_DIR/scripts/scrub-bundle.py" "<BUNDLE_PATH>"
This copies the bundle to <BUNDLE_PATH>/tmp/<BUNDLE_NAME>/, scrubs the copy in place
(replacing customer-identifying data with numbered placeholders), and prints the
scrubbed copy's path to stdout — the original bundle is never modified. It reuses an
existing scrubbed copy if one is already present (.pii-scrubbed marker), so it's safe
to re-run.
Capture the path printed on stdout and set BUNDLE_PATH to it. Use the scrubbed copy
path for ALL subsequent steps.
Step 1.5: Read System Context
Before launching agents, read from any node directory:
system-settings.json → extract cloud_name (system identity)boot/ybos-version.txt → VergeOS versionalarms.json → active alarm count and severity summary- List node directories to determine cluster size
Pass this context (cloud_name, version, node count) to all subagents.
Step 2: Launch Parallel Analysis Agents
Spawn these subagents IN PARALLEL using the Task tool with model: sonnet:
Agent 1: Storage Analyzer
Analyze VergeOS vSAN storage health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].
For EACH node directory, check these vSAN files in priority order:
1. vsan/tierstatus.txt - Per-tier: working=false CRITICAL, redundant=false WARNING, bad_drives>0, repairs>0, progress
2. vsan/journalstatus.txt - status (idle/collapsing=OK, active=check), alive=1, redundant=1, curmaster, tier lag
3. vsan/clusterusage.txt - Per-tier used_pct: >=85% WARNING, >=91% CRITICAL (throttling starts)
4. vsan/repairstatus.txt - All device values should be 0. Non-zero = active repair
5. vsan/devicestatus.txt - Check rd_errs, wr_errs, mismatch, oos_blocks (all should be 0).
Latency: lat_w_avg_full >1000us WARNING, >5000us CRITICAL. throttle>0 = active throttling
6. vsan/clusterrates.txt - write_throttle != "unlimited" = throttled. wrate<1KB/sec CRITICAL
7. vsan/fuseinfo.txt - throttled_reads>0 or throttled_writes>0 = active throttling
8. vsan/integcheckstatus.txt - bad_blocks, bad_files, bad_dirs should all be 0
9. vsan/cacheinfo.txt - Calculate hit rate: cache_hits/(cache_hits+cache_misses). <50% WARNING
10. vsan/currentmaster.txt - Report master node identity
11. smart/*.txt - NVMe: available_spare<10% CRITICAL, media_errors>0 CRITICAL, percentage_used>95% CRITICAL.
SATA: reallocated>0, pending>0, uncorrectable>0 all CRITICAL
Cross-node: Compare tier status and device health across all nodes for inconsistencies.
Context: tier 0 working=false with tier 3 working=true may be metadata-only issue, not full outage.
Report: CRITICAL first, WARNINGS, INFO. Include specific values found.
Reference: $SKILL_DIR/storage.md
Agent 2: Network Analyzer
Analyze VergeOS network health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].
For EACH node directory:
1. network/ybfabric.txt - Parse JSON. Per peer node:
- confirmed=false CRITICAL (path not verified, connectivity lost)
- confirmed=true = healthy bidirectional communication
- Score maps to NIC link speed: 10Gbps=20, 25Gbps=50, 50Gbps=100, 100Gbps=200.
Determine expected max from nicinfo.txt link speed. Score below max = degraded WARNING
- Missing peer nodes = cluster partition CRITICAL
- syncing_time=true WARNING (should be false after boot)
- Check vxlans[] for VXLAN tunnel assignments
2. network/ifconfig.txt - All interfaces:
- Bond: errors>0, calculate drop rate vs total packets (>0.1% = investigate)
- vx2: must be UP,BROADCAST,RUNNING. MTU should be NIC MTU - 50
- br2: core fabric bridge
3. network/nicinfo.txt - NIC model, driver, link speed. All fabric NICs should match speed
4. network/bond* - Mode (802.3ad/active-backup), slave status, miimon setting
5. network/ipaddr.txt - Verify fabric IPs present (172.16.x.x or configured range)
6. network/bridgefdb.txt - VXLAN FDB entries. Missing peers or stale entries
7. network/lldpneighbors.txt - Physical switch topology verification
8. network/arp.txt - Incomplete entries = connectivity issues
Cross-node: Compare fabric scores — one node lower = that node's NIC/cable issue.
Context: Core network 100.96.0.x/24, VXLAN VNI=2 UDP 4789, ybfabric port 14202.
Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/network.md
Agent 3: Hardware Analyzer
Analyze hardware health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].
For EACH node directory:
1. ipmisensor.txt - CPU >85C CRITICAL, >70C WARNING. Drives >51C CRITICAL, >45C WARNING.
Fans RPM=0 CRITICAL. Voltages out of range CRITICAL.
2. ipmisel.txt - Flag: uncorrectable ECC, thermal events, PSU failures, fan failures
3. smart/*.txt - NVMe: Available Spare <10% CRITICAL, Media Errors >0 CRITICAL,
Percentage Used >95% CRITICAL, Unsafe Shutdowns (high = power issues).
SATA: Reallocated >0, Pending >0, Offline Uncorrectable >0 CRITICAL.
4. ras-edac-info.txt - ECC: correctable frequent WARNING, uncorrectable CRITICAL. ID affected DIMM.
5. dmidecode.txt - Memory config, CPU model/count
6. lspci.txt - PCI inventory, GPU presence, NVMe controllers
7. proc/meminfo.txt - MemAvailable <15% CRITICAL. Note: high Mlocked is NORMAL (VM hosts).
No swap is by design on VergeOS.
8. sensors.txt - Current thermal readings cross-reference with IPMI
9. lsblk/*.json - Per-drive partition layout (p1=boot, p2=vSAN data)
Cross-node: Compare hardware configs for heterogeneity.
Node types: Controller (DB, PXE), Compute (workload), Storage+Compute (hyperconverged).
Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/hardware.md
Agent 4: Linux System Analyzer
Analyze Linux system health across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].
IMPORTANT VergeOS context:
- /ybos at 100% is NORMAL (read-only loop mount)
- High Mlocked/Unevictable memory is NORMAL (VMs pinned in RAM)
- No swap is by design
- Root filesystem is overlay on tmpfs (8GB)
For EACH node directory:
1. df.txt - overlay / >85% WARNING, >90% CRITICAL. /vsan usage. IGNORE /ybos 100%.
2. free.txt + proc/meminfo.txt - MemAvailable <20% WARNING, <15% CRITICAL (factor out Mlocked)
3. psaux.txt - Must have: ybvsan, ybfabric. Controller nodes: appserver.
D-state processes = stuck I/O. Zombie count. High CPU qemu = normal VM workload.
4. syslog + syslog.1 - Search: "oom", "out of memory", "killed process", "segfault",
"ybvsan", "ybfabric", "appserver", "hard reset", "clock.*jump"
5. yottabyte-boot.log - VergeOS boot: ybfabric start, vSAN mount, kernel selection errors
6. boot-syslog - Early boot issues
7. sysstat/sar* - %iowait sustained >20% = storage bottleneck, CPU trends
8. proc/vmstat.txt - oom_kill count
9. lsmod.txt - Verify: vfio, kvm, bonding, vxlan, overlay
Systemd order: yb-config → basic.target → vsan.service → appserver.service
Watchdog: vSAN crash → syslog archived → hard reboot (unless /boot/.no-vsan-reboot)
Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/linux.md
Agent 5: Container/Service Analyzer
Analyze VergeOS container services across ALL nodes in [BUNDLE_PATH].
VergeOS version: [VERSION]. System: [CLOUD_NAME].
For EACH node directory:
1. containerslist.txt - All containers. Expected: vnet1 (DMZ), vnet2 (Core), vnet3 (External).
Additional are user-created. Identify each by container_name.txt in container_logs/ subdirs.
2. container_logs/vnet1/messages (DMZ) - dnsmasq: "no servers found" = upstream DNS issue,
DHCP pool exhaustion, frequent restarts. Pool default: 100.64.191.254-100.64.255.254
3. container_logs/vnet2/messages (Core) - appserver errors, database issues, API timeouts
4. container_logs/vnet3/messages (External) - routing failures, NAT issues, upstream connectivity
5. container_logs/*/messages - All other containers. Read container_name.txt to identify purpose.
6. alarms.json - Active alarms. Owner types: vms/N, nodes/N, vnets/N, system.
Levels: info, warning, error, critical. Flag critical/error alarms.
7. alarm_history.json - Recent alarm patterns (if present)
8. system-settings.json - cloud_name, domain, NTP config for context
Container architecture: each vnet = LXC container with own network namespace.
Services: nftables, dnsmasq, BIND9, strongSwan, WireGuard, FRRouting, HAProxy.
Writable mounts: root(1MB), tmp(128MB), var/log(5MB), run(1MB) — cleared on restart.
Report: CRITICAL first, WARNINGS, INFO.
Reference: $SKILL_DIR/containers.md
Step 3: Consolidate Findings
After all agents complete, consolidate their reports into a unified summary:
## Diagnostic Summary for [CLOUD_NAME]
### System Identity
- **Cloud Name**: [from system-settings.json]
- **VergeOS Version**: [from boot/ybos-version.txt]
- **Nodes**: [count and list]
- **vSAN Master**: [from currentmaster.txt]
### Active Alarms
[Summary from alarms.json — count by severity, list critical/error alarms]
### Critical Issues (Immediate Action Required)
[List all CRITICAL findings from all agents]
### Warnings (Address Soon)
[List all WARNING findings from all agents]
### Storage Capacity Summary
| Tier | Used | Max | Used% | Status |
|------|------|-----|-------|--------|
[Per-tier from clusterusage.txt]
### Observations
[List all INFO findings from all agents]
### Recommended Actions
1. [Prioritized action items]
2. ...
Step 4: Write Results and Finish
After consolidating the report:
- Write RESULTS.md — Save the consolidated summary to
[BUNDLE_PATH]/RESULTS.md using the Write tool
- Create finished file — Create an empty
[BUNDLE_PATH]/finished file to mark analysis as complete
- Output the report — Display the full RESULTS.md content to the user so they can review it directly in the conversation
Reference Files
Key Thresholds
| Metric | Warning | Critical |
|---|
| Storage usage (tier) | 85% | 91%+ (throttling) |
| CPU Temperature | 70C | 85C+ |
| Drive Temperature | 45C | 51C+ |
| Memory available | <20% | <15% |
| Overlay disk space (/) | >85% | >90% |
| SMART reallocated | >0 | >1 |
| Drive wear | >90% | >95% |
| Load average | >CPU count | >2x CPU count |
| Fabric score | Below link-speed max | confirmed=false |
| Write latency (lat_w_avg_full) | >1000µs | >5000µs |
| Device rd_errs/wr_errs | — | >0 |
| Cache hit rate | <70% | <50% |
Fabric score reference (score = link speed indicator, NOT fixed at 200):
| NIC Link Speed | Expected Max Score |
|---|
| 10 Gbps | 20 |
| 25 Gbps | 50 |
| 50 Gbps | 100 |
| 100 Gbps | 200 |
Base OS Context
VergeOS runs on Linux with:
- Upstream kernel (not modified)
- Debian packages for non-Verge components
/ybos is a read-only loop mount (100% usage is NORMAL)- Root filesystem is overlay on tmpfs (8GB)
- No swap by design
- Critical processes:
ybvsan, ybfabric, appserver (controller only), qemu-system-x86_64 (VMs)
Compressed Log Handling
xzcat syslog.xz | grep -i error
zcat syslog.2.gz | grep -i warning
