harness-migration

# Harness Migration First confirm a broken `harness/` ledger actually has a generation mismatch rather than some other fault. If it does, replay it into a freshly initialized current-format repository; the source is never written to. Replay is the only supported migration path — there is no in-place repair tool. **This skill assumes nothing is installed.** It fetches the current source into a throwaway directory and runs everything from there. A Harness installation already on the machine — including a running daemon — is left untouched. ## Before anything: confirm this is the right document This skill is versioned in the `harness-anything` repository as `skills/harness-migration/SKILL.md` on **`main`**, and that branch is the authority. **Read it from a git ref, never from whatever working tree happens to be at hand** — a checkout parked on another branch may not carry this file at all, or may carry an older revision, and neither difference is visible once the text is in front of you. ```bash git -C <any-checkout-of-harness-anything> show origin/main:skills/harness-migration/SKILL.md ``` A machine may also carry a separately maintained skill with a similar name — `harness-ledger-migration` is the one that exists today, and it is an older, **different** document rather than an alias. Following it instead is a silent wrong turn. The front matter settles it: this skill's `name:` is exactly `harness-migration`. ## The one rule that makes this work **Every command runs against an isolated daemon user root.** Export it once and keep it exported for the whole session: ```bash export HARNESS_MIGRATION_WORK="$(cd "$(mktemp -d "${TMPDIR:-/tmp}/ha-migration.XXXXXX")" && pwd -P)" export HARNESS_DAEMON_USER_ROOT="$HARNESS_MIGRATION_WORK/daemon-user-root" mkdir -p "$HARNESS_DAEMON_USER_ROOT" ``` Without it, the CLI connects to whatever Harness daemon is already running on the machine and that daemon serves a different generation of the code. The failure does not announce itself as a conflict — it arrives as: ``` error code=missing_vertical ``` with `"origin":"daemon"` in the JSON receipt. **Match on those two fields, not on the hint text**: the hint describes an unavailable vertical, which is true but misleading, and it reads differently depending on which generation the running daemon serves. `code=missing_vertical` together with `"origin":"daemon"` means `HARNESS_DAEMON_USER_ROOT` is not set. Do **not** stop or uninstall the user's existing Harness to work around it: isolation is sufficient, and stopping their daemon interrupts work you are not responsible for restoring. The `cd … && pwd -P` wrapper around `mktemp` is what keeps the path readable. On macOS `$TMPDIR` already ends in a slash, so the template produces a doubled slash, and `/tmp` is itself a symlink into `/private/tmp`. Either way the raw path turns up in every receipt from here on and does not match what you see on the filesystem. `pwd -P` resolves both, once, at the start. Quoting is not involved: `"${TMPDIR:-/tmp}"/ha-migration.XXXXXX` and `"${TMPDIR:-/tmp}/ha-migration.XXXXXX"` are the same string to the shell, and neither one avoids the doubled slash. ### If your shell does not persist between commands The steps below accumulate exported variables and, in step 1, a shell function. An agent that gets a **fresh shell per tool call keeps none of them** — the second command runs with `HARNESS_DAEMON_USER_ROOT` unset and connects to the machine's own daemon, which is exactly the failure this section exists to prevent. This is the root cause of the downstream traps the later steps describe individually (zsh word-splitting in step 1, `nohup` not seeing the function in step 8, `env` resolving `ha` from `PATH` in step 9). Write the session state to a file and source it at the start of every later command: ```bash export HARNESS_MIGRATION_ENV="$HARNESS_MIGRATION_WORK/env.sh" cat > "$HARNESS_MIGRATION_ENV" <<EOF export HARNESS_MIGRATION_WORK='$HARNESS_MIGRATION_WORK' export HARNESS_MIGRATION_ENV='$HARNESS_MIGRATION_ENV' export HARNESS_DAEMON_USER_ROOT='$HARNESS_DAEMON_USER_ROOT' EOF # every later command: . "$HARNESS_MIGRATION_ENV" && <command> ``` Append each new `export` to that file as the steps below introduce it — `HA_ENTRY`, `ARCHIVE_SOURCE`, `WORK_SOURCE`, `TARGET_REPO`, `DRY_RUN`, `LEDGER_ARCHIVE`, `SOURCE_SHA_BEFORE`. Record `$HARNESS_MIGRATION_ENV` somewhere you will still have it later; it is the one path you must not lose. **Keep this file inside `$HARNESS_MIGRATION_WORK`, and if you write anything anywhere else, put the repository name in its filename.** `mktemp` already makes the work directory unique, but agents habitually drop scratch files into a shared session scratchpad instead — and `env.sh`, `dry-run.txt` and `apply.log` are the same name in every migration. When two migrations run in parallel, a sibling overwriting your env file is not hypothetical: it has happened, and it presents as your own variables quietly turning into someone else's, several steps after the damage. Either keep everything under `$HARNESS_MIGRATION_WORK`, or name it `env-<repo>.sh`, `dry-run-<repo>.txt`, `apply-<repo>.log`. ## 1. Fetch the current source **Node 24 or newer is required.** Check before cloning: ```bash node --version ``` The CLI is run from its TypeScript entry point, which relies on Node's native type stripping. On an older Node every command fails with ``` TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" ``` which reads like a missing build step and is not one — no amount of `npm run build` fixes it. Stop and tell the user to upgrade Node; nothing below works until they do. ```bash cd "$HARNESS_MIGRATION_WORK" git clone --depth 1 https://github.com/FairladyZ625/harness-anything.git ha-src cd ha-src npm install --no-audit --no-fund export HA_ENTRY="$HARNESS_MIGRATION_WORK/ha-src/packages/cli/src/index.ts" ha() { node "$HA_ENTRY" "$@"; } ha --version ``` Expect a version line. The repository is about 10 MB and install takes seconds. **`ha` here is a shell function, not an exported variable.** The obvious `export HA="node …/index.ts"` followed by `$HA --version` works in bash and silently fails in zsh, which does not word-split unquoted parameters: zsh passes `node …/index.ts` as a **single** argument and the receipt comes back `unsupported_command`. A function behaves identically in both shells. If your shell does not persist between commands, append both the variable and the function to the env file from the top of this skill, and source it every time: ```bash { echo "export HA_ENTRY='$HA_ENTRY'" echo 'ha() { node "$HA_ENTRY" "$@"; }'; } >> "$HARNESS_MIGRATION_ENV" ``` Two consequences worth knowing now rather than at step 8: - The function lives in the shell that defined it. Anything that runs in a **detached** process — see step 8 — must spell out `node "$HA_ENTRY" …` instead, and sourcing the env file does not change that. - `env -u HARNESS_DAEMON_USER_ROOT ha …`, which step 9 uses on purpose, does **not** see the function. `env` execs a program, so it resolves `ha` from `PATH` — the machine's own installation. That is exactly what step 9 wants, and step 9 says so again where it matters. **Do not look for `node_modules/.bin/ha`.** The published `bin` points at `dist/`, which a source checkout does not contain, so the linked binary is absent. Running the TypeScript entry directly is the supported path here and needs no build step. ## 1a. Confirm the symptom is a ledger-generation mismatch before importing Do this **after step 1 fetches the current source, before creating a destination or running `migrate import`**. The symptoms below are an entry point, not a decision: both kinds of ledger can produce them. | What you see | What it means | Next action | | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | `doc event envelope or payload is invalid` | Could be either an older ledger or a current ledger with pre-S4 doc cuts. | Run the read-only event scan below. | | daemon receipt `repo_attach_failed` or `repo_unavailable` while attaching the repository | The daemon could not build its projection; it does not identify the ledger generation. | Run the read-only event scan below; do not retry attach as a probe. | Set the source path once. The scan only reads its event files; it does not need the daemon and does not alter the source. ```bash export ARCHIVE_SOURCE="$(cd /absolute/path/to/repository-with-harness && pwd -P)" export CANONICAL_EVENT_CONTRACT="file://$HARNESS_MIGRATION_WORK/ha-src/packages/kernel/src/domain/doc-sync.contract.ts" node --input-type=module - "$ARCHIVE_SOURCE/harness/events" "$CANONICAL_EVENT_CONTRACT" <<'NODE' import { readdir, readFile } from 'node:fs/promises'; import { join } from 'node:path'; const [eventsRoot, contractUrl] = process.argv.slice(2); const { parseCanonicalEvent } = await import(contractUrl); const counts = { files: 0, parsed: 0, unknown_schema: 0, legacy_cut_shape: 0, other: 0 }; async function* eventFiles(dir) { for (const entry of await readdir(dir, { withFileTypes: true })) { const file = join(dir, entry.name); if (entry.isDirectory()) yield* eventFiles(file); else if (entry.isFile() && entry.name.endsWith('.json') && entry.name !== 'head.json') yield file; } } function hasLegacyCut(value) { if (!value || typeof value !== 'object') return false; if (Array.isArray(value)) return value.some(hasLegacyCut); const cut = value.baseLedgerSha; if (cut && typeof cut === 'object' && !Array.isArray(cut)) { const keys = Object.keys(cut).sort(); if (keys.length === 2 && keys[0] === 'repoId' && keys[1] === 'sha') return true; } return Object.values(value).some(hasLegacyCut); } for await (const file of eventFiles(eventsRoot)) { counts.files++; let text, event; try { text = await readFile(file, 'utf8'); event = JSON.parse(text); parseCanonicalEvent(text); counts.parsed++; } catch (error) { const message = error instanceof Error ? error.message : String(error); if (message === 'canonical event schema is unknown') counts.unknown_schema++; else if (message === 'doc event envelope or payload is invalid' && event?.schema === 'doc-event/v1' && hasLegacyCut(event)) counts.legacy_cut_shape++; else counts.other++; } } console.log(JSON.stringify(counts, null, 2)); NODE ``` Interpret the counts conservatively: - If `unknown_schema > 0` **or** `other > 0`, this is a previous-generation ledger rather than the narrow S4-only cut mismatch. Continue with the replay steps in this skill. The replay importer constructs current migration events rather than copying source event bytes. - If `legacy_cut_shape > 0`, `unknown_schema = 0`, and `other = 0`, the ledger is a current-generation ledger whose doc cuts predate S4. Replay handles it, and replay is the only supported path: **there is no in-place restamp migration, and none is planned.** An in-place restamp was built and evaluated; it was deliberately not shipped, because a tool that rewrites cut identity in place has to be trusted on a ledger nobody can re-derive, while replay reconstructs the destination from source events and leaves the source untouched. Know what replay costs you here: it remaps entity IDs and writes a new ledger, which is more than this ledger strictly needs — only its cut identity is stale. Budget for the ID remapping (see the ID mapping steps below) rather than looking for a narrower tool. If remapped IDs are genuinely unacceptable for your ledger, stop and report that; do not improvise a hand-edit of event bytes. - If all three failure counts are zero, the stream already parses under the current code. This is not a generation mismatch, so migration will not fix it; stop and investigate the reported symptom separately. If the scan itself cannot read the events directory or run the current parser, stop and report that failure. Do not infer the generation from `harness.yaml`: both generations can carry `schema: harness-anything/v1`. ## 2. Freeze and back up the source ledger Back up and digest **`harness/` only** — never the repository root. ```bash export WORK_SOURCE="$HARNESS_MIGRATION_WORK/legacy-copy" mkdir -p "$HARNESS_MIGRATION_WORK/backups" "$WORK_SOURCE" export LEDGER_ARCHIVE="$HARNESS_MIGRATION_WORK/backups/legacy-harness.tar" COPYFILE_DISABLE=1 tar -cf "$LEDGER_ARCHIVE" -C "$ARCHIVE_SOURCE" harness export SOURCE_SHA_BEFORE="$(COPYFILE_DISABLE=1 tar -cf - --exclude='harness/.git' -C "$ARCHIVE_SOURCE" harness | shasum -a 256 | awk '{print $1}')" printf 'source-before %s\n' "$SOURCE_SHA_BEFORE" tar -xf "$LEDGER_ARCHIVE" -C "$WORK_SOURCE" ``` `$WORK_SOURCE` now contains `harness/` and nothing else, which is all the importer reads — `--source` takes the repository root and descends into `harness/` itself. Three things this deliberately does **not** do, each for a reason worth knowing: - **No `git bundle`.** `harness/` is its own git repository and the outer repo ignores it (`/harness/` in `.gitignore`, `git ls-files harness/` returns nothing). A bundle of the outer repo therefore contains **zero** ledger content — it looks like a backup and protects nothing. - **No repository-root digest.** The root contains `.harness/`, which is ignored runtime state — locks, `write-journal`, `cache`, `script-runs`, `task-holders` — that any running daemon rewrites continuously. A root digest changes on its own between the before and after reads, so the "source untouched" check would report a false failure every time. A check that must be ignored to proceed is worse than no check. - **No `cp -a` of the root.** It copies `node_modules` and the whole `.git` directory, which the importer never reads. The digest excludes `harness/.git` for the same reason the root digest is excluded entirely: it is live metadata, not content. A `git fetch` from any mirror or a background maintenance run rewrites `packed-refs`, `FETCH_HEAD` and the reflog without touching a single ledger file, and the closing comparison then fails for a reason unrelated to the importer. This bit a real migration — an unrelated mirror fetch landed mid-run and the source looked modified when it was not. **The archive is not filtered**: a backup must carry the ledger's own git history, and only the digest needs the exclusion. The digest and the `tar` print nothing while they run. On a **small ledger they are effectively instant** — about a second each for a 963-event, 257 MB `harness/` — so if you are staring at a blank prompt for more than a few seconds, look at the size of what you are digesting rather than waiting. Only a genuinely large ledger takes tens of seconds. Either way, if it runs for many minutes you are digesting more than `harness/`; check the `-C` argument. Nothing may write to `$ARCHIVE_SOURCE/harness/` while the migration runs, or the closing digest will differ for a reason that has nothing to do with the importer. Show the user `$LEDGER_ARCHIVE` and **stop until they confirm one independent