| name | backup-recovery |
| description | Diagnose HX Lab data incidents, determine the last known-good time, and advise the smallest safe recovery using local Proxmox snapshots, TrueNAS disaster recovery, or PostgreSQL-aware recovery. |
Backup and Recovery
Use for data loss, corruption, accidental deletion, bad deployments, PostgreSQL incidents, ZFS recovery, and disaster recovery.
This skill is advisory only.
Never execute backup, restore, rollback, database recovery, SSH, ZFS, Docker, Proxmox, or TrueNAS commands.
The agent may inspect repository configuration and analyze evidence provided by the user. For runtime state, tell the user what command or UI information is needed and analyze the returned output.
Decision flow
Incident
↓
When did it start?
↓
What is affected?
↓
What recovery point is safe?
↓
Is it available locally?
├─ Yes → advise local recovery
└─ No → advise TrueNAS recovery
Always:
- diagnose the incident
- establish the incident window
- determine the smallest affected scope
- choose the newest recovery point clearly before the incident
- recommend the smallest safe recovery
- tell the user exactly what to run or select
Never perform the recovery yourself.
Storage model
Live persistent data:
rpool/swarm
Backup path:
rpool/swarm
↓ hourly Proxmox ZFS snapshot
rpool/swarm@auto-YYYY-MM-DD_HH-MM
↓ TrueNAS scheduled PULL over SSH
tank/backups/swarm
Local Proxmox snapshots are for short-term recovery.
TrueNAS is the off-host recovery layer.
TrueNAS is not runtime storage.
1. Diagnose the incident
Determine what failed.
Prefer the smallest scope:
file
directory
application data
application
PostgreSQL rows/tables
PostgreSQL database
PostgreSQL instance
rpool/swarm
Proxmox storage/host
Investigate available evidence:
- symptoms
- logs supplied by the user
- deployment/change history
- timestamps
- filesystem evidence
- database symptoms
- user actions
- last known successful operation
Do not recommend rollback before understanding the incident.
Do not restore the whole dataset for an incident that can be recovered more narrowly.
2. Determine when
Build a timeline:
last known good
↓
incident introduced
↓
incident detected
Choose the newest recovery point that clearly predates the incident.
Example:
10:00 snapshot safe
11:00 snapshot safe
11:37 bad deployment
12:00 snapshot unsafe
12:24 incident found
Recommendation:
11:00
Do not choose 12:00 simply because it is newer.
If the exact incident time is uncertain:
- state the uncertainty
- identify the suspected incident window
- choose a recovery time safely before that window
Never present an inferred time as confirmed.
3. Choose recovery scope
Prefer recovery in this order:
selective file recovery
↓
application-level recovery
↓
PostgreSQL-aware recovery
↓
local ZFS rollback
↓
TrueNAS disaster recovery
Use the smallest method capable of fixing the incident.
Avoid restoring unrelated data.
4. Determine local availability
Local snapshots use:
rpool/swarm@auto-%Y-%m-%d_%H-%M
The agent must not run commands to inspect them.
If local snapshot availability is unknown, tell the user to provide the output of the repository's read-only snapshot/status workflow.
If none exists, advise a read-only inspection command such as:
zfs list -t snapshot -r rpool/swarm
Do not execute it.
Use the returned output to determine whether a safe local recovery point exists.
5. Local recovery
Use local recovery when a safe recovery point is confirmed to exist on Proxmox.
Recommend:
task restore
Do not run it.
Tell the user which recovery point to select.
Example recommendation:
Incident
Application data became invalid after the 11:37 deployment.
When
Last known good: 11:00
Incident introduced: approximately 11:37
Impact
Application data
Recovery point
auto-2026-08-25_11-00
Availability
Confirmed local on Proxmox.
Recommended solution
Local rollback.
Why
11:00 is the newest confirmed snapshot before the incident.
Action
Run:
task restore
Select:
auto-2026-08-25_11-00
Before recommending a full rollback, warn that workloads using rpool/swarm must be stopped or safely quiesced.
If rollback could remove newer snapshots/data, warn the user before they proceed.
Never run the restore command.
6. Selective recovery
If only a file or directory is affected, recommend selective recovery when possible.
State:
- affected path
- safe recovery time
- why full dataset rollback is unnecessary
Do not recommend whole-rpool/swarm rollback unless required.
7. TrueNAS recovery
Use TrueNAS when:
- the safe recovery point is no longer available locally
rpool/swarm is lost- Proxmox storage failed
- the host/storage was rebuilt
The agent has no access to TrueNAS.
Do not claim a TrueNAS snapshot exists, is healthy, or is available.
Determine only the safe recovery time.
Example:
Last known good:
11:00
Incident window:
11:37–12:10
Recommended safe recovery boundary:
11:00 or earlier
Then advise the user to open:
TrueNAS
→ Data Protection
→ Replication Tasks
→ swarm-restore-truenas-to-pve1
Tell the user:
Select the newest available snapshot at or before the recommended safe recovery time, then run the PUSH restore.
Example:
Recommended safe recovery time:
2026-08-25 11:00
Action:
Open TrueNAS:
Data Protection
→ Replication Tasks
→ swarm-restore-truenas-to-pve1
Choose the newest available snapshot at or before 11:00 and run the PUSH restore.
Never recommend a snapshot inside or after the suspected incident window.
If the user finds no snapshot at or before the safe time, ask them to provide the available older recovery points and reassess.
Do not generate:
- TrueNAS API calls
- SSH commands
zfs sendzfs receive- custom replication commands
The TrueNAS restore task remains manual.
For recovery testing, prefer restoring to:
rpool/restore-test
rather than overwriting live data.
8. PostgreSQL incidents
Do not default PostgreSQL incidents to ZFS rollback.
Diagnose the database scope first.
Prefer surgical recovery:
rows/tables
↓
single database
↓
PostgreSQL instance
↓
filesystem recovery
↓
TrueNAS recovery
Examples:
Deleted or incorrect rows/tables
Recommend PostgreSQL-aware surgical recovery.
Do not rollback the entire filesystem unless no narrower recovery exists.
Single database affected
Recommend restoring only that database when possible.
PostgreSQL instance damaged
Recommend PostgreSQL-aware instance recovery.
Storage lost
Recommend:
restore storage
↓
restore PostgreSQL from database-aware backup
↓
verify database
↓
recover dependent applications
The simple PostgreSQL backup strategy is logical backups such as pg_dump.
Only recommend WAL/PITR when point-in-time recovery is actually required and implemented.
Never claim PostgreSQL is recovered merely because ZFS data was restored.
9. Response format
Finish every incident analysis with:
Incident
<what failed>
When
Last known good: <time>
Incident window: <time/range>
Impact
<smallest affected scope>
Safe recovery time
<time>
Availability
Local Proxmox | TrueNAS required | Database backup | Unknown
Recommended solution
<smallest safe recovery method>
Why
<short justification>
Action
<exact command the user should run or TrueNAS UI path they should use>
If evidence is insufficient, do not guess.
Instead report:
Missing evidence
<what is needed>
Action
<read-only command/UI information the user should provide>
Hard rules
- Advise only; never execute recovery.
- Never run operational commands.
- Diagnose before recommending rollback.
- Determine the incident window first.
- Use the newest recovery point clearly before the incident.
- Prefer the smallest recovery scope.
- Never assume the newest snapshot is safe.
- Never invent recovery points.
- Never claim local snapshot availability without evidence.
- Never claim TrueNAS snapshot availability without user evidence.
- Never invent Task commands.
- Prefer repository Task workflows over raw commands.
- Never rollback active persistent workloads.
- Never make TrueNAS a runtime dependency.
- Never put TrueNAS credentials on Proxmox.
- Never schedule TrueNAS restore replication.
- Never treat ZFS snapshots as the sole PostgreSQL recovery mechanism.
- Never claim recovery success without evidence.
