storage-backup

name	storage-backup
description	Use when asking about storage classes, Velero backups, backup schedules, ZFS volumes, or disaster recovery patterns.

Storage and Backup

Overview

The homelab uses ZFS-backed storage classes via OpenEBS and Velero for backup to Cloudflare R2 (S3-compatible). Volumes are automatically labeled for backup based on size.

Storage Classes

Note: K8s storage class names are legacy and don't match actual hardware:

• zfs-ssd → backed by NVMe SSDs
• zfs-hdd → backed by SATA SSDs (not HDDs)

zfs-ssd (NVMe SSD)

For config files, databases, and performance-critical data.

import { NVME_STORAGE_CLASS } from "../misc/storage-classes.ts";

// Use via construct
const volume = new ZfsNvmeVolume(chart, "myapp-pvc", {
  storage: Size.gibibytes(8),
});

// Or directly in Helm values
persistence: {
  storageClass: NVME_STORAGE_CLASS,  // "zfs-ssd"
  size: "8Gi",
}

Characteristics:

• Pool: zfspv-pool-nvme
• Provisioner: zfs.csi.openebs.io
• Compression: off
• Dedup: off
• Record size: 128k

zfs-hdd (SATA SSD)

For media files and large downloads. Despite the name, this is backed by SATA SSDs (not HDDs).

import { SATA_STORAGE_CLASS } from "../misc/storage-classes.ts";

const mediaVolume = new ZfsSataVolume(chart, "media-pvc", {
  storage: Size.tebibytes(4),
});

Characteristics:

• Pool: zfspv-pool-hdd (legacy name - actually SATA SSDs)
• Same provisioner and settings as NVMe

Automatic Backup Labeling

The ZfsNvmeVolume and ZfsSataVolume constructs automatically label PVCs for Velero:

// In zfs-nvme-volume.ts / zfs-sata-volume.ts
const shouldBackup = props.storage.toKibibytes() < Size.gibibytes(200).toKibibytes();

metadata: {
  labels: {
    "velero.io/backup": shouldBackup ? "enabled" : "disabled",
    "velero.io/exclude-from-backup": shouldBackup ? "false" : "true",
  },
},

Volume Size	Backup Status	Rationale
< 200 GB	Enabled	Config, databases, important data
>= 200 GB	Disabled	Large media (too big to backup efficiently)

Velero Backup Schedules

Defined in src/cdk8s/src/resources/velero-schedules.ts:

Schedule	Frequency	Retention	Description
6hourly	Every 6 hours	3 days (12 backups)	Frequent recovery points
3daily	Every 3 days	216 days (72 backups)	Medium-term retention
weekly	Monday 3:45 AM	49 days (7 weeks)	Weekly snapshots
monthly	2nd of month	120 days (4 months)	Monthly archives
quarterly	Every 3 months	90 days	Long-term archives

Schedule Configuration

export type VeleroScheduleConfig = {
  id: string;
  name: string;
  cronSchedule: string;
  ttl: string;
  backupType: string;
  description: string;
};

const VELERO_SCHEDULE_CONFIGS: VeleroScheduleConfig[] = [
  {
    id: "velero-backup-6hourly",
    name: "6hourly-backup",
    cronSchedule: "15 */6 * * *",
    ttl: "72h",
    backupType: "6hourly",
    description: "Every 6 hours - FULL BACKUPS",
  },
  // ... other schedules
];

Backup Storage Configuration

Cloudflare R2 (Primary)

configuration: {
  backupStorageLocation: [
    {
      name: "default",
      bucket: "homelab",
      provider: "aws",
      prefix: "torvalds/backups/",
      config: {
        region: "auto",
        s3Url: "https://xxx.r2.cloudflarestorage.com",
      },
    },
  ],
}

ZFS Volume Snapshots

volumeSnapshotLocation: [
  {
    name: "zfspv-incr",
    provider: "openebs.io/zfspv-blockstore",
    config: {
      bucket: "homelab",
      incrBackupCount: "15",
      fullBackupPrefix: "zfspv-full",
      backupPathPrefix: "zfspv-incr",
      prefix: "torvalds/zfs/",
      region: "auto",
      s3Url: "https://xxx.r2.cloudflarestorage.com",
    },
  },
],

Verifying Backup Coverage (Operational)

Volume data is backed up by the openebs zfs-localpv Velero plugin (per-PVC zfs send → Cloudflare R2), NOT by Velero kopia/restic FSB or CSI data-mover. So do not judge backup coverage by looking for a Velero node-agent daemonset, BackupRepositories, DataUploads, or podvolumebackups — those are absent by design.

• Read the data streams with the r2 aws profile: aws s3 ls s3://homelab/zfspv-incr/backups/ --profile r2
• Layout: zfspv-incr/backups/<velero-backup-name>/torvalds/zfs/-<backup>-pvc-<uuid>-<backup> (data stream) plus a sibling ...zfsvol (~1.4 KiB header) per volume. A complete set ≈ 92 objects / ~1.4–5 GB; a metadata-only stub ≈ 46 objects / ~79 KB means zfs send failed (e.g. dead/suspended pool).
• K8s manifests are separate, under torvalds/backups/ (standard Velero).
• The backup counter is status.volumeSnapshotsAttempted (~46 PVCs), NOT csiVolumeSnapshotsAttempted.
• ⚠️ Velero can mark a backup Failed even when the full data uploaded (one chronic single-volume error fails the whole backup's phase) — verify by R2 object size, not by Velero phase.
• The single-disk zfspv-pool-nvme has no redundancy: if it suspends (e.g. an NVMe controller hang), zfs send stops and the newest R2 backup may be tens of minutes stale.

Manual Backup Operations

Create On-Demand Backup

velero backup create manual-backup-$(date +%Y%m%d) \
  --selector velero.io/backup=enabled \
  --snapshot-volumes

Restore from Backup

# List backups
velero backup get

# Restore specific backup
velero restore create --from-backup 6hourly-backup-20240115120000

Check Backup Status

velero backup describe 6hourly-backup-20240115120000
velero backup logs 6hourly-backup-20240115120000

Excluding Large Volumes

For volumes >= 200GB that shouldn't be backed up:

// Automatic via construct
const mediaVolume = new ZfsSataVolume(chart, "media-pvc", {
  storage: Size.tebibytes(4),  // >= 200GB, auto-excluded
});

// Manual exclusion
metadata: {
  labels: {
    "velero.io/backup": "disabled",
    "velero.io/exclude-from-backup": "true",
  },
},

Storage Class Definitions

From src/cdk8s/src/misc/storage-classes.ts:

new KubeStorageClass(chart, "host-zfs-ssd", {
  metadata: { name: "zfs-ssd" },
  provisioner: "zfs.csi.openebs.io",
  allowVolumeExpansion: true,
  reclaimPolicy: "Retain",
  parameters: {
    fstype: "zfs",
    poolname: "zfspv-pool-nvme",
    compression: "off",
    dedup: "off",
    recordsize: "128k",
    shared: "yes",
  },
  volumeBindingMode: "WaitForFirstConsumer",
});

Velero Helm Values

From src/cdk8s/src/resources/argo-applications/velero.ts:

const veleroValues: HelmValuesForChart<"velero"> = {
  metrics: {
    serviceMonitor: { enabled: true },
  },
  initContainers: [
    {
      name: "velero-plugin-for-aws",
      image: `velero/velero-plugin-for-aws:${versions["velero/velero-plugin-for-aws"]}`,
      volumeMounts: [{ mountPath: "/target", name: "plugins" }],
    },
    {
      name: "velero-plugin-openebs",
      image: `openebs/velero-plugin:${versions["openebs/velero-plugin"]}`,
      volumeMounts: [{ mountPath: "/target", name: "plugins" }],
    },
  ],
  configuration: {
    backupStorageLocation: [...],
    volumeSnapshotLocation: [...],
  },
};

Key Files

• src/cdk8s/src/misc/storage-classes.ts - Storage class definitions
• src/cdk8s/src/misc/zfs-nvme-volume.ts - SSD volume construct
• src/cdk8s/src/misc/zfs-sata-volume.ts - HDD volume construct
• src/cdk8s/src/resources/velero-schedules.ts - Backup schedule config
• src/cdk8s/src/resources/argo-applications/velero.ts - Velero deployment
• src/cdk8s/src/resources/argo-applications/openebs.ts - OpenEBS CSI driver