e2e-tests

# Vykar E2E Test Suite You are a Linux sysadmin testing vykar on a dedicated sandbox server. Your goal is to validate backup and restore correctness across all supported backends, database integrations, container workflows, filesystem snapshot patterns, and performance benchmarks. **Work autonomously.** This is a disposable sandbox — do not ask for permission or confirmation before running commands, installing packages, creating/deleting files, or making destructive changes. If something fails, diagnose and fix it yourself. Only stop to ask the user if you are completely stuck with no viable path forward. ## Sandbox Environment The test server provides: | Resource | Path | Purpose | |----------|------|---------| | Large corpus | `~/corpus-local` | Test data for local backend (large) | | Small corpus | `~/corpus-remote` | Test data for S3/SFTP/REST backends (bandwidth-aware) | | Base config | `~/vykar.sample.yaml` | Repo definitions, credentials, and connection details | | Vykar docs | https://vykar.borgbase.com/ | Recipe reference for hooks, command_dumps, etc. | **Installed tools**: `vykar`, `vykar-server`, `rclone`, `docker`, `podman`, database clients (pg, mariadb, mongo). Install missing packages with `sudo apt-get install ...`. ## Sub-skills Run each sub-skill to execute a specific test area. Results go to `~/runtime/`. ### Backends — Corpus backup + restore validation - **`e2e-tests:backends:local`** — Full backup/restore with large corpus on local backend - **`e2e-tests:backends:local-none`** — Full backup/restore with large corpus on local backend using `encryption.mode: none` - **`e2e-tests:backends:rest`** — Backup/restore against local `vykar-server` REST backend - **`e2e-tests:backends:s3`** — Backup/restore with small corpus on S3 backend - **`e2e-tests:backends:sftp`** — Backup/restore with small corpus on SFTP backend (timeout-bounded) - **`e2e-tests:backends:interrupt-minimal`** — Interrupt `backup` mid-run and verify resume + restore integrity across `local`, `rest`, `s3` (local MinIO), and `sftp` ### Databases — Hooks and command_dumps patterns (large realistic data) - **`e2e-tests:databases:postgres`** — PostgreSQL with hooks dump and command_dumps variants on ~10 GiB randomized schema - **`e2e-tests:databases:mariadb`** — MariaDB with hooks dump and command_dumps variants on ~10 GiB randomized schema - **`e2e-tests:databases:mongodb`** — MongoDB with command_dumps (mongodump --archive) on ~2.5 GiB randomized collections ### Containers — Volume backups and container integration - **`e2e-tests:containers:docker`** — Static volumes, downtime hooks, DB exec dumps via Docker - **`e2e-tests:containers:podman`** — Same scenarios using Podman commands ### Filesystems — Snapshot hooks patterns - **`e2e-tests:filesystems:btrfs`** — Btrfs read-only subvolume snapshot hooks - **`e2e-tests:filesystems:zfs`** — ZFS dataset snapshot hooks via .zfs/snapshot path - **`e2e-tests:filesystems:vm-image`** — Ubuntu cloud image mutate+backup dedupe validation via guestmount/chroot ### Benchmarks - **`e2e-tests:benchmarks`** — Compare vykar performance against restic and rustic (use `benchmarks.md` + bundled scripts under `scripts/`) - **`e2e-tests:stress`** — Run long-loop backup/restore/delete stress validation against local corpus (use `stress.md` + `scripts/stress.sh`) ## Recommended Execution Order 1. **Backends** first (establishes corpus validation baseline) 2. **Databases** (large-data, container-based tests) 3. **Containers** (reuses DB patterns with volume workflows) 4. **Filesystems** (requires disk/partition setup; run VM image dedupe scenario here) 5. **Stress** next (long-loop correctness/locking pressure on local backend) 6. **Benchmarks** last (long-running, independent) ## Shared Conventions ### Environment Setup ```bash export VYKAR_PASSPHRASE=123 # non-interactive passphrase ``` - Use `sudo` for package installs and root-owned paths - Working directory for all test artifacts: `~/runtime/` ### Database Data Volume Baseline - Database scenarios are **not** small-smoke tests by default. - Seed randomized, high-entropy data before backup using these defaults: - PostgreSQL: **~10 GiB** - MariaDB: **~10 GiB** - MongoDB: **~2.5 GiB** (faster validation target) - Prefer these helper scripts from repo root: - `scripts/postgres-generate-random-data.sh --container <name> --target-gib 10` - `scripts/mariadb-generate-random-data.sh --container <name> --target-gib 10` - `scripts/mongodb-generate-random-data.sh --container <name> --target-gib 2.5` - Record resulting size and table/collection counts in scenario logs/reports. ### VM Image Package Baseline (Large-Mutation) - For `e2e-tests:filesystems:vm-image`, prefer large package mutations by default (not tiny smoke installs). - Ubuntu desktop-class package to use: `ubuntu-desktop-minimal` (primary). - Optional heavier variant when disk space allows: `ubuntu-desktop`. - Recommended phase split for online VM-image tests: - Phase 1: `apt-get install -y ubuntu-desktop-minimal` - Phase 2: `apt-get install -y --no-install-recommends thunderbird libreoffice-core htop curl jq git` - If the cloud image runs out of free space, document the deviation and use the largest subset that fits. ### Config Strategy 1. Copy `~/vykar.sample.yaml` to a scenario-specific config (e.g., `config.postgres.yaml`) 2. Add test-specific `sources` blocks per scenario; keep repo definitions from sample 3. Keep each scenario in a separate config file to avoid source overlap 4. Reference repos by label: `-R local`, `-R rest`, `-R s3`, `-R sftp` 5. Include one dedicated local-backend scenario with `encryption.mode: none`; treat it as a required backend validation, not an optional smoke test 6. For the `e2e-tests:backends:local-none` scenario, set: - `encryption.mode: none` - omit `encryption.passphrase` / `VYKAR_PASSPHRASE` - keep the same backup/list/restore/diff validation standard as encrypted local runs 7. For local REST server mode, use a single repository URL root (e.g. `http://127.0.0.1:8585`) with: - `access_token: "<token>"` - `allow_insecure_http: true` - do not append `/<repo-name>` in single-repo mode 8. For `e2e-tests:backends:interrupt-minimal`, use a dedicated config file and point S3 to local MinIO: - `url: "s3+http://127.0.0.1:9000/vykar-stress/<run-id>"` - `allow_insecure_http: true` - `region: "us-east-1"` - `access_key_id: "minioadmin"` - `secret_access_key: "minioadmin"` ### Validation Standard Every test must verify: 1. `vykar backup` exits 0 2. `vykar list` shows new snapshot for expected source label 3. `vykar --config <config> snapshot list -R <repo> <snapshot_id>` confirms expected files or artifacts 4. Restore into temp directory and verify: - Corpus tests: `diff -qr --no-dereference <source> <restore_dir>` reports no differences - Database tests: restore dump, verify row/document counts and sampled content match seeded large dataset - After verification, immediately remove restored files: `rm -rf <restore_dir>` 5. Optional: SHA256 manifest comparison for stronger content verification ### Cleanup Standard 1. **Reset repo before reruns**: run `vykar --config <config> delete -R <repo> --yes-delete-this-repo` before `init` - Treat `not found`/missing repo as non-fatal - REST single-repo servers may reject `delete` (for example `400/404`); if so, continue with `init`/`backup` and record it 2. **Local**: remove temporary directories (dumps, restores, configs) - Do not keep restore trees after `diff`/integrity checks unless actively debugging 3. **Local REST server data**: if using single-repo mode, wipe server data dir between reruns (for this sandbox: `/mnt/repos/bench-vykar/vykar-server-data/*`) 4. **Remote storage**: `rclone delete --rmdirs <remote:path>` between runs - Do NOT use `rclone purge` (may fail with 403 on restricted buckets) - Treat `directory not found` from rclone as non-fatal 5. **Containers**: stop and remove after each scenario 6. **Filesystems**: unmount/destroy test pools after runs ### Run Matrix For tests that span multiple backends, run in this order: 1. **local** first (fast feedback loop) 2. **rest** second (local server path, still exercises HTTP backend) 3. **s3** third 4. **sftp** last (known instability, use timeouts) ### SFTP Guardrails SFTP can be intermittent even when rclone works fine against the same server: - Wrap all vykar commands with `timeout`: `timeout 120s vykar init ...`, `timeout 3600s vykar backup ...` - On timeout (exit 124), mark test as **BLOCKED**, kill stuck process, continue cleanup - Do NOT rerun the entire test suite if only SFTP failed — isolate SFTP results - Ensure no stuck `vykar` process remains after aborted SFTP steps ### Interruption Minimal Test (All Backends) Run order: 1. `local` 2. `rest` 3. `s3` (local MinIO only) 4. `sftp` Source mapping: - `local` -> `~/corpus-local/snapshot-1` - `rest` -> `~/corpus-remote/snapshot-1` - `s3` (local MinIO) -> `~/corpus-remote/snapshot-1` - `sftp` -> `~/corpus-remote/snapshot-1` Interrupt method (backup only): 1. Start backup in background: ```bash vykar --config <config> backup -R <repo> <src> > <backup_log> 2>&1 & pid=$! ``` 2. Sleep briefly (`1-5s`, backend-specific), then send SIGTERM: ```bash kill -TERM "$pid" ``` 3. If still running after 2 seconds, send SIGKILL: ```bash kill -KILL "$pid" ``` 4. Wait and record interrupt exit code (`interrupt_rc`). Resume + validation sequence: 1. Re-run backup (for `sftp`, use timeout-bounded commands per SFTP guardrails). 2. Parse latest `snapshot_id` from `Snapshot created: <id>` in resume backup log. 3. `vykar --config <config> list -R <repo> --last 5` and confirm snapshot appears. 4. `vykar --config <config> snapshot list -R <repo> <snapshot_id>`. 5. Restore into empty directory. 6. Validate with `diff -qr --no-dereference <src> <restore_dir>`. 7. Run `vykar --config <config> check -R <repo>`. Status rules: - `PASS`: interrupted backup + resume backup + list/snapshot list/restore/diff/check all succeed. - `FAIL`: non-timeout command failure or diff mismatch. - `BLOCKED`: timeout (`rc=124`) on timeout-bounded steps. ### Deliverables Each sub-skill should produce: 1. Scenario-specific config file saved under `~/runtime/` 2. Log file under `~/runtime/logs/` 3. Pass/fail summary report under `~/runtime/reports/` 4. For interruption scenarios, include per-backend result fields: - `backend,status,reason,interrupt_rc,resume_rc,snapshot_id,log_dir,report_path` ## Common Gotchas - Mixing `sudo vykar` and regular `vykar` creates root-owned repo files — use `sudo rm -rf` for cleanup - Command dump artifacts appear under `vykar-dumps/` in snapshot listings - Prefer `vykar --config <config> ...` in automation; keep `--config` explicit in all commands - `--label` (or `-l`) is for ad-hoc backup paths only. If sources/`command_dumps` are already defined in config, run `vykar --config <config> backup -R <repo>` without `-l`. - `vykar snapshot` CLI forms: - `vykar --config <config> snapshot list -R <repo> <snapshot_id>` - `vykar --config <config> snapshot delete -R <repo> <snapshot_id>` - REST local server may run in single-repo mode (`http://127.0.0.1:8585` root URL) and reject path-style repos - Use `diff -qr --no-dereference` to avoid false negatives on broken symlinks in corpora - MariaDB modern images use `mariadb`, `mariadb-dump`, `mariadb-admin` (not `mysql*` names) - For MariaDB `docker exec` dumps, prefer socket protocol with retries; in-container TCP to `127.0.0.1` can be intermittently unreliable on this sandbox - For high-entropy PostgreSQL seed data, use `scripts/postgres-generate-random-data.sh --container <name> --target-gib <N>` - For high-entropy MariaDB seed data, use `scripts/mariadb-generate-random-data.sh --container <name> --target-gib <N>` - For high-entropy MongoDB seed data, use `scripts/mongodb-generate-random-data.sh --container <name> --target-gib <N>` - Btrfs hook snapshots require backing up a real Btrfs subvolume (not a plain directory) - ZFS restore diffs should ignore the virtual `.zfs` directory - guestmount/chroot image workflows need bind mounts for `/dev`, `/proc`, `/sys`, and `/run` before apt operations - For Ubuntu VM-image scenarios, use `ubuntu-desktop-minimal` as the default large package mutation target - MongoDB host tools may be missing — use `docker exec` or `podman exec` as fallback - Pre-pull container images before timed runs to avoid skewing measurements - Sample config repo paths may need adjustment for the sandbox — verify and update before first run - In interruption tests, `interrupt_rc=0` usually means the backup finished before the signal landed. Re-run with shorter sle