| name | troubleshoot-lvm |
| description | Use when diagnosing issues with LVM (Linux Logical Volume Manager): space exhaustion, device loss, silent redundancy loss, metadata corruption, or snapshot collapse. Queries Netdata via MCP for physical volume accessibility, lv activation state, volume group free space, thin pool data usage, mirror/raid synchronization status, applies the diagnostic tree from the Netdata operator playbook, and recommends remediation. |
| version | 0.1.0 |
| author | Netdata |
| license | Apache-2.0 |
| tags | ["netdata","troubleshoot","mcp","lvm"] |
Troubleshoot LVM (Linux Logical Volume Manager)
When to use this skill
- Space exhaustion: VG free extents gone, or thin pool data/metadata full. Results in failed
writes (thin), failed operations (snapshots, extends), or application
crashes.
- Device loss: PV disappears (disk failure, SAN path loss, cable issue). VG becomes partial.
Linear/striped LVs on missing PV are immediately inaccessible; mirrored/RAID LVs
degrade.
- Silent redundancy loss: Mirror leg fails, RAID component fails. System continues operating on
remaining leg(s). No visible errors until second failure causes
complete data loss.
- Metadata corruption: Crash during metadata update, bad blocks in PV metadata area. Can make
entire VG unreadable and all LVs inaccessible.
- Snapshot collapse: Heavy writes on snapshotted origin fill COW exception store. Snapshot
invalidates instantly. Origin I/O latency spikes from COW overhead before
invalidation.
- Thin pool strangulation: Pool approaching full triggers aggressive block reclaim. I/O latency
spikes. At 100% data: writes queue for 60 seconds (default) then fail
with errors. At 100% metadata: pool can become corrupted, and
recovery is not guaranteed.
- Any time the user reports a LVM (Linux Logical Volume Manager) service behaving outside its
expected envelope (elevated errors, latency, saturation, resource exhaustion, or unexpected
restarts).
- An on-call engineer is paging on a Netdata alert tied to a LVM (Linux Logical Volume Manager)
instance and wants a structured triage path.
Key facts
- This skill wraps the Netdata operator playbook for LVM (Linux Logical Volume Manager). It does not
replace the playbook; it routes a coding agent through MCP queries against the same signals the
playbook relies on.
- LVM is a storage virtualization layer between physical block devices and
filesystems/applications. It does not store data; it tells the kernel's device-mapper (dm)
subsystem how to map I/O from virtual block devices to physical extents on real disks.
- The playbook decomposes LVM (Linux Logical Volume Manager) health into 3 signal domains:
Availability Domain, Saturation Domain, Replication / Consistency Domain. Each domain maps to one
rule file in this skill.
- Dominant failure archetypes the playbook calls out: Space exhaustion; Device loss; Silent
redundancy loss; Metadata corruption; Snapshot collapse.
- Netdata observes the signals listed in the rule files via its native collectors, plus any
OpenTelemetry-shipped metrics that your LVM (Linux Logical Volume Manager) instrumentation adds.
Both paths end at the same MCP query surface.
- Netdata's lvm collector emits 2 context(s) under
lvm.*. The rule files enumerate which contexts
surface which domain; the Verification section below names the load-bearing ones explicitly.
Step-by-step
- Confirm the LVM (Linux Logical Volume Manager) service is up. Query Netdata via MCP with
list_nodes and filter by the host running the target. A missing node means the symptom is at
the network or orchestrator layer, not inside the service.
- Pull the last 15 minutes of signals for the target. Use
query_metrics against the contexts
listed in the domain rule files. Run find_anomalous_metrics in parallel over the same window;
anomalies frame which rule file to read first.
- Check for Space exhaustion. VG free extents gone, or thin pool data/metadata full. Results in
failed writes (thin), failed operations (snapshots, extends), or application crashes. Inspect the
rule file whose signals move first for this mode.
- Check for Device loss. PV disappears (disk failure, SAN path loss, cable issue). VG becomes
partial. Linear/striped LVs on missing PV are immediately inaccessible; mirrored/RAID LVs
degrade. Inspect the rule file whose signals move first for this mode.
- Check for Silent redundancy loss. Mirror leg fails, RAID component fails. System continues
operating on remaining leg(s). No visible errors until second failure causes complete data loss.
Inspect the rule file whose signals move first for this mode.
- Check for Metadata corruption. Crash during metadata update, bad blocks in PV metadata area.
Can make entire VG unreadable and all LVs inaccessible. Inspect the rule file whose signals move
first for this mode.
- Check for Snapshot collapse. Heavy writes on snapshotted origin fill COW exception store.
Snapshot invalidates instantly. Origin I/O latency spikes from COW overhead before invalidation.
Inspect the rule file whose signals move first for this mode.
- Correlate with host-level signals (
system.cpu.utilization, system.memory.usage,
system.disk.io_time). Many service-level failures have a host-resource precursor.
- Apply the remediation hinted at in the matching rule file or the operator playbook. Re-run the
MCP queries from the Verification section to confirm the signals returned to expected ranges. A
fix that does not move the signal back is not a fix.
Handy MCP call templates
# Discover metrics from LVM (Linux Logical Volume Manager)
list_metrics with q="lvm"
# Pull a specific context over the last window
query_metrics with context="lvm.lv_data_space_utilization", relative_window=-15m
# Rank anomalies for the service or host
find_anomalous_metrics with node=<host> and context_pattern="lvm.*"
# Correlate a known problem context with others
find_correlated_metrics around the incident window
# Show current alert state
list_raised_alerts scoped to the node
Common mistakes
- Treating LVM (Linux Logical Volume Manager) as a generic HTTP or process health check. LVM (Linux
Logical Volume Manager) has specific failure archetypes (see Key facts) that generic checks miss.
- Stopping at the first anomalous metric. Several archetypes produce correlated spikes; use
find_correlated_metrics to widen the search before concluding a root cause.
- Quoting percentile latency without the sample count. Low traffic plus a single slow request moves
p99 by seconds.
- Reading dashboards for a window shorter than the failure's fingerprint. Slow-brew failures (queue
growth, bloat, memory fragmentation) need 30+ minutes of data to see the trend.
- Skipping the host-level correlation. A process-level fix for a noisy-neighbour problem does not
hold.
- Assuming alert thresholds are tuned for your workload. Tune against observed LVM (Linux Logical
Volume Manager) traffic before escalating an alert configuration issue.
Verification
Run these MCP queries against the Netdata instance that sees the LVM (Linux Logical Volume Manager)
service. Every context listed below is a real Netdata chart name; the agent does not need to guess.
1. list_metrics filtered by q="lvm" (returns every lvm.* context Netdata sees)
2. query_metrics with contexts=[lvm.lv_data_space_utilization, lvm.lv_metadata_space_utilization] and relative_window=-30m
3. find_anomalous_metrics filtered by node=<host> and context_pattern="lvm.*"
Load-bearing contexts for this service:
lvm.lv_data_space_utilization: Logical volume space allocated for data (%). Dimensions:
utilization.lvm.lv_metadata_space_utilization: Logical volume space allocated for metadata (%). Dimensions:
utilization.
A clean result means every context is within its expected band and the find_anomalous_metrics list
is empty or contains only already-acknowledged items. If the fix was real, re-running the same
queries 10 minutes after applying it will show a clean result. If it does not, revert and look
deeper.
When the fix does not hold
If signals drift back into the anomalous range within 30 minutes of a remediation, the cause was
deeper than the applied change. Typical misdiagnoses for LVM (Linux Logical Volume Manager):
- Host-resource pressure masquerading as application bug.
- Dependent service (DB, cache, upstream) causing a secondary symptom in the instrumented service.
- Configuration change that was never reloaded (some subsystems only pick up config on full
restart).
Escalate by widening the query window: 2-6 hours instead of 15 minutes. Slow-moving causes are
invisible at triage window sizes.
References
