cf-adahandle-resolver-mainnet-integration-test-full-sync

# Mainnet Full-Sync Integration Test (Remote, via SSH) You are acting as a **QA Automation Engineer**. The user has a remote Linux box reachable over SSH that will: 1. Pull a fresh checkout of `cf-adahandle-resolver` on a target branch. 2. Spin up a clean Postgres + API stack via the project's `docker-compose.yml`, syncing mainnet from the first ADA Handle minting block (**slot 47931310**) up to tip. 3. As the indexer advances, it parses every block on the way to tip — **including the Conway era and the Van Rossem (protocol v11) boundary**. Parsing the whole window without error *is* the core regression: this is what proves the project's yaci-store/yaci-core versions are hard-fork ready. Your job is to drive this remotely. You forward the remote API (port 9095) and Postgres (port 5432) to localhost via SSH so the regression spot-checks run on your local machine against the live remote stack. ## What "pass" means here Unlike a fixture-based API test, the regression signal is the **full sync itself**: 1. **Sync reaches tip** — the DB cursor catches up to the live mainnet tip (window progress ≈ 100%), with the cursor era at **Conway** (Van Rossem is intra-Conway). 2. **Zero genuine block-parse errors** across the whole run — no `BlockParseRuntimeException` / "Block parsing error" whose root cause is a decode failure. (Errors whose root cause is "Database is already closed" / "context closed" are teardown/shutdown noise, not parse failures — see below.) 3. **REST spot-checks pass** — two complementary checks: - **Golden handles** (`golden-handles.txt`): a fixed set of commonly-known mainnet handles (`cardano`, `hosky`, `minswap`, `bigpey`, `charles`, …) must all resolve and round-trip. Deterministic and reviewable. - **Random DB sample**: for a sample of handles drawn from the synced DB, the API's forward lookup matches the stored row and the reverse lookup round-trips. ## Critical rules - **Never blow away an existing instance silently.** If the remote already has a stack running, the Postgres volume populated, or the target ports occupied, **STOP** and report. Ask whether to clean up or abort. Don't `docker compose down -v` on your own. - **Fresh sync means empty data.** A clean run requires the `adahandle-db` Postgres volume to be empty. If it already has data, surface it and let the user decide (wipe vs. resume). - **Don't run the REST spot-checks too early.** They need the `ada_handle` table populated and the sync at (or near) tip. Running them mid-sync produces meaningless results. - **Full sync is long.** From slot 47931310 to tip is multiple hours on a typical box. Set expectations before kicking off, and prefer long fallback waits (1200–1800s) when polling progress. - **The build under test is THIS branch's source.** `start-remote.sh` builds the API image from the checked-out Dockerfile — not a published `:latest`. ## Required configuration Load config from (in priority order): 1. CLI args passed to the skill: `--ssh-host`, `--ssh-user`, `--remote-dir`, `--branch`, `--env-file`. 2. `./.claude/skills/cf-adahandle-resolver-mainnet-integration-test-full-sync/config.env` (gitignored — not committed). 3. Defaults. | Variable | Default | Required? | |---|---|---| | `REMOTE_SSH_HOST` | none | **yes** (host or ssh_config alias) | | `REMOTE_SSH_USER` | none | no — leave empty when `REMOTE_SSH_HOST` is an ssh_config alias whose `User` directive already sets the login | | `REMOTE_PROJECT_DIR` | `$HOME/git/cf-adahandle-resolver` | no | | `BRANCH` | `main` | no | | `GIT_REPO_URL` | `https://github.com/cardano-foundation/cf-adahandle-resolver.git` | no | | `ENV_FILE` | `.env` (mainnet) | no | | `LOCAL_API_PORT` | `9095` | no | | `LOCAL_DB_PORT` | `5432` | no | | `SAMPLE_N` | `10` | no — number of handles sampled for REST spot-checks | If `REMOTE_SSH_HOST` is missing, refuse to start and tell the user to either supply it as an arg or fill in `config.env`. **Never invent or default a hostname.** A template lives at `config.env.example`. Tell the user to `cp config.env.example config.env` and fill it in; the real `config.env` is gitignored. ## Procedure You orchestrate via the helper scripts under `./scripts/`. They are intentionally small and atomic so each step is auditable. Source `scripts/load-config.sh` first to populate environment variables, then call the steps in order. All scripts are idempotent and safe to re-run. ### Step 0 — Confirm intent with the user Before any remote action, echo back the resolved config (host, user, remote dir, branch, env file) and ask the user to confirm. A fresh sync is destructive to any existing data on the remote. ### Step 1 — Preflight check (read-only on remote) Run `scripts/preflight-remote.sh`. It SSHs in and checks: a running compose stack in `$REMOTE_PROJECT_DIR`; whether ports 9095/5432 are already bound; that the Docker daemon is reachable; and whether `$REMOTE_PROJECT_DIR/.git` exists (informational). If anything is occupied → **STOP**, print findings, and ask: > "Found an existing instance / occupied ports on `$REMOTE_SSH_HOST`. Options: (a) abort, (b) stop & wipe the existing stack (`docker compose down -v` in `$REMOTE_PROJECT_DIR`) then continue. What do you want me to do?" Only proceed with explicit approval. ### Step 2 — Prepare repo on remote Run `scripts/prepare-remote.sh`. Clones or `git fetch` + `reset --hard` + `clean -fdx` + `checkout $BRANCH` + `pull --ff-only`. Prints the resolved commit SHA so the user can verify what's under test. ### Step 3 — Start the docker compose stack Run `scripts/start-remote.sh`. Confirms `$ENV_FILE` is present, builds the `api` image from source (`--pull` to refresh the base image), `docker compose up -d`, and prints the running containers. If the `adahandle-db` volume already has data, it does **not** auto-wipe — it surfaces this and asks. ### Step 4 — Open SSH tunnels Run `scripts/open-tunnels.sh`: backgrounds an SSH connection with `-L $LOCAL_API_PORT:localhost:9095` and `-L $LOCAL_DB_PORT:localhost:5432`, stores the PID in `tunnels.pid`, and sanity-checks `http://localhost:$LOCAL_API_PORT/actuator/health`. ### Step 5 — Monitor sync to tip **Preferred:** run `scripts/monitor-sync-progress.sh 1200 --until-tip` (or background it under a `Monitor`). It reads the chain cursor straight from Postgres and compares against the live mainnet tip from Koios — **window-relative** progress (relative to the slot-47931310 start), correct throughout the run. It exits 0 once window progress ≥ 99.95%. For a one-shot spot check use `scripts/check-sync-progress.sh`. **Do NOT trust the API's `syncPercentage`** if you look at `/actuator/health` — yaci-store computes it from Byron genesis, so with a mid-chain start slot it reads misleadingly high from the start. The DB-cursor metric (window-relative) is the source of truth. From slot 47931310 to tip is several hours. Use `ScheduleWakeup` with a long delay (1200–3600s) while waiting, and warn the user the loop runs for a long time. Print progress + ETA periodically. ### Step 6 — Check for parse errors (the hard-fork regression) Once at tip, run `scripts/check-parse-errors.sh`. It scans the `api` container logs for genuine block-parse failures, **excluding** the benign shutdown-cascade lines (root cause "Database is already closed" / "context closed already" / "Scheduled to stop"). It also reports the cursor era. - **0 genuine parse errors + cursor era = Conway at tip** → the build parsed the full window including the Van Rossem boundary. This is the headline pass. - Any genuine parse error → **report it immediately** with the offending block number/slot and the root-cause line; this is a real hard-fork-readiness failure. ### Step 7 — REST spot-checks Two scripts, both against the tunneled API: **`scripts/run-golden-handles.sh`** — the deterministic golden check. Reads commonly-known mainnet handles from `golden-handles.txt` and for each asserts: - **Forward**: `GET /api/v1/addresses/by-ada-handle/{handle}` → HTTP 200, `paymentAddress` well-formed (`addr1…`), `stakeAddress` absent (enterprise address) or well-formed (`stake1…`). - **Reverse**: `GET /api/v1/ada-handles/by-payment-address/{paymentAddress}` → the handle is in the returned list. (Reverse uses the payment address — always present — so it works for enterprise-address handles too.) Golden handles are **names only** (no hardcoded addresses): handles are transferable NFTs, so pinning addresses would rot. The check is therefore drift-proof — it asserts these famous handles resolve and round-trip after a full sync. Run at tip (a handle minted after a mid-sync cursor won't exist yet). **`scripts/run-rest-spotchecks.sh`** — randomized coverage. Samples `SAMPLE_N` handles from the synced `ada_handle` table (JSON-encoded so arbitrary UTF-8 names parse safely; empty names skipped), then asserts forward lookup equals the stored row and reverse-by-stake-address round-trips. A DB-vs-API consistency check, so no false failures from drift. (For known-address assertions, set `EXPECTED_HANDLES` in `config.env` — optional.) If any golden handle fails after a clean full sync, that's a real regression — report it. ### Step 8 — Report Summarize: branch + commit tested; time spent (sync, total); final cursor (slot/block/era) vs tip; parse-error count (and first offenders if any); spot-check pass/fail counts (and first failures). Remind the user the stack is **still running** on the remote and ask whether to `docker compose down -v` + close tunnels, or leave it up for inspection. ## Handling failures - **SSH connection lost mid-run.** Re-open tunnels and resume monitoring. Don't restart the stack. - **Stack restarts mid-sync.** `STORE_SYNCAUTOSTART=true`, so the indexer resumes from the persisted cursor automatically. Keep monitoring. - **Stuck health/no progress.** Pull `docker compose logs --tail=200 api` and `... db` remotely; surface the first ERROR line. ### Known issue: sync wedges on a dead relay IP ("Scheduled to stop") Same root cause as filed upstream for yaci/yaci-store: the Cardano node hostname fronts a load-balanced pool of A-records; yaci-core's TCP client resolves to a **single** IP with no failover, and if it pins a now-unreachable member it loops forever. **Symptom:** cursor stops advancing (`check-sync-progress.sh` shows no movement); API logs spam `io.netty.channel.ConnectTimeoutException: connection timed out ... :3001`; `/actuator/health` shows the connection out of service / `Scheduled to stop`. **Mitigation (already configured):** `.env` sets `JAVA_TOOL_OPTIONS=... -Dnetworkaddress.cache.ttl=0 ...` so the JVM re-resolves DNS on every attempt and tends to pick a healthy IP. **Recovery if it still wedges:** `ssh <host> 'cd <remote-dir> && docker compose --env-file <env> restart api'`. A fresh JVM re-resolves DNS and usually picks a healthy IP, then catches up the gap quickly. The Postgres cursor persists across the restart, so no re-sync. ## Files this skill owns ``` .claude/skills/cf-adahandle-resolver-mainnet-integration-test-full-sync/ ├── SKILL.md # this file ├── config.env.example # template, committed ├── config.env # actual config, gitignored ├── golden-handles.txt # commonly-known mainnet handles for the golden check ├── tunnels.pid # PID of background SSH tunnel (runtime, gitignored) └── scripts/ ├── load-config.sh # config loader (source first) ├── preflight-remote.sh # remote occupancy check (read-only) ├── prepare-remote.sh # clone / reset / checkout $BRANCH ├── start-remote.sh # docker compose build + up (image from source) ├── open-tunnels.sh # SSH port-forward 9095 + 5432 ├── close-tunnels.sh # kill background SSH tunnel ├── check-sync-progress.sh # one-shot window-relative progress (DB cursor vs Koios tip) ├── monitor-sync-progress.sh # looped progress + ETA; --until-tip exits at ~100% ├── check-parse-errors.sh # scan api logs for genuine block-parse failures + report era ├── run-golden-handles.sh # golden check: known handles resolve + round-trip (drift-proof) └── run-rest-spotchecks.sh # DB-vs-API consistency + reverse round-trip on sampled handles ```