antithesis-moog

# MOOG Operations ## Setup — CRITICAL **Always use the release binary, never `moog-head`.** ```bash cd /code/moog # Download the release binary (same as the GitHub Actions workflow uses). # Latest release is v0.5.1.5; asset naming changed at v0.5.x — the portable, # statically-linked CLI is now `moog-<ver>-x86_64-linux-musl.tar.gz` (it # extracts a single `moog` binary; moog-agent / moog-oracle are separate assets). curl -sL "https://github.com/cardano-foundation/moog/releases/download/v0.5.1.5/moog-0.5.1.5-x86_64-linux-musl.tar.gz" | tar xz -C ./tmp/ export PATH=/code/moog/tmp:$PATH source tmp/prod-setup.sh ``` > **Asset naming history:** ≤ v0.4.1.2 used `moog-vX.Y.Z.W-linux64.tar.gz`; > v0.5.x switched to per-arch names — `moog-X.Y.Z.W-x86_64-linux-musl.tar.gz` > (musl = static/portable), plus `.AppImage`, `.deb`, `.rpm`, and aarch64 > variants. Always match what `.github/workflows/cardano-node.yaml` downloads. **WARNING**: `moog-head` in `/code/moog/tmp/` is a dev build. It reports the same `--version` as the release but encodes Duration differently (`{"minutes":0}` instead of `0`), producing requests the oracle can't parse (`UnknownUpdateValidationFailure`). Never symlink `moog-head` as `moog`. Always download the release binary. The workflow at `.github/workflows/cardano-node.yaml` downloads from `cardano-foundation/moog` releases — match that exactly. ## Active Antithesis tenant: `amaru-cardano` (2026-05-26 →) Antithesis migrated Cardano testing from the `cardano` tenant (`cardano.antithesis.com` / user `cardano`) to `amaru-cardano` (`amaru-cardano.antithesis.com` / user `pragma`). The moog-agent binary at `851aac3-dirty` (= `v0.4.1.2`) **hardcodes** the old URL at `src/User/Agent/PushTest.hs:246`. The multi-tenant fix lives in https://github.com/cardano-foundation/moog/pull/93 (branch `feat/multi-tenant-antithesis`, head `e6157f6`); it reads `MOOG_ANTITHESIS_LAUNCH_URL` and `MOOG_REGISTRY` from env. Anything older than that PR cannot reach the new tenant; treat the deployed `image:` tag as part of the diagnosis. **Working credentials matrix (verify with curl before rotating anything — see `feedback_split_config_test_outside.md`):** | Field | Value | Source | |---|---|---| | Antithesis URL | `https://amaru-cardano.antithesis.com/api/v1/launch/amaru-cardano` | env `MOOG_ANTITHESIS_LAUNCH_URL` | | Antithesis user | `pragma` | env `MOOG_ANTITHESIS_USER` | | Antithesis password | per `secrets.yaml.antithesisPassword` | secret file | | GAR registry | `us-central1-docker.pkg.dev/molten-verve-216720/cardano-repository` (unchanged at the migration) | env `MOOG_REGISTRY` | **Direct API check (outside moog) — succeeds with the right user/pw, 403 otherwise:** ```bash ssh agent 'PW=$(sudo awk "/^antithesisPassword:/{sub(/^antithesisPassword:[[:space:]]*\"?/,\"\");sub(/\"?[[:space:]]*$/,\"\");print}" /secrets/moog-agent/new/secrets.yaml) && \ curl -sS -o /dev/null -w "%{http_code}\n" -u "pragma:$PW" \ -X POST https://amaru-cardano.antithesis.com/api/v1/launch/amaru-cardano \ -H "Content-Type: application/json" -d "{}"' # 200 → creds OK; 403 → creds rejected (and same 403 with no -u, so 403 alone doesn't distinguish) ``` ## Identities | Role | Wallet | Set with | |------|--------|----------| | Requester | `tmp/requester.json` | `being_requester` | | Agent | `tmp/agent.json` | `being_agent` | | Oracle | `tmp/oracle.json` | `being_oracle` | Check current identity: `being` ## Key commands | Command | Purpose | |---------|---------| | `moog token --no-pretty` | Full token state with pending requests | | `moog facts test-runs --whose cfhal` | List test runs (run as `being_requester` to auto-decrypt the Antithesis report URL in the `url` field) | | `moog retract -w <wallet> -o <outref>` | Retract a request you own | | `moog agent report-test -i <id> -w <wallet> --outcome unknown --duration 0 --url "reason"` | Force-finish a stale test (manual override) | | `moog antithesis runs --no-pretty` | List the Antithesis runs the agent reconciles against (see "How the agent reconciles results") | | `moog wallet info` | Show wallet owner hash | ## Reading Antithesis runs directly (`moog antithesis`, v0.5.1.0+) `moog antithesis` (added in v0.5.1.0) reaches the Antithesis report data through the Moog proxy — no browser, SSO cookie, or Playwright. All subcommands emit newline-delimited JSON; pair with `--no-pretty | jq`. Exit codes on `runs`: `0` success, `2` auth/SSO failure, `3` proxy/network failure, `4` invalid JSON from proxy. | Command | Purpose | |---------|---------| | `moog antithesis runs [--limit N] [--cursor C]` | List runs (paginated; `--limit` max 100). Each entry carries `run_id` (e.g. `46d4…-54-7`), `status` (`in_progress` / `completed` / `incomplete`), and a JSON `description` with the testRun `directory` (e.g. `testnets/cardano_node_governance`). | | `moog antithesis run --run-id ID` | Full details of one run. For `incomplete` runs it includes `failure_moment {input_hash, vtime}`; for `completed` runs it carries `links.triage_report` (the report URL). | | `moog antithesis properties --run-id ID` | Property pass/fail for a **completed** run (404 while still in progress). | | `moog antithesis events --run-id ID [--q TEXT]` | Stream matching events as NDJSON; free-text substring via `--q`; upstream caps at 50 results. | | `moog antithesis logs --run-id ID --input-hash H --vtime V` | Logs at a specific moment. Needs a real moment — `in_progress` runs 400 (`Invalid query parameters`); a degenerate `0/0` failure moment returns empty. | | `moog antithesis build-logs --run-id ID` | Stream the config-image **build** logs as NDJSON (`{timestamp, text, stream}`). Works regardless of run status — the fallback when there's no usable failure moment. | **Find a run, then read it** — e.g. the latest governance run's build logs: ```bash # Locate the run id by directory (the banner from prod-setup.sh precedes the JSON, # so select the JSON line explicitly): moog antithesis runs --limit 100 --no-pretty | grep '^{"data"' \ | jq -r '.data[] | select(.description|contains("governance")) | [.created_at, .status, .run_id] | @tsv' | head # Moment logs need input_hash + vtime from `run`; build-logs need only the id: moog antithesis build-logs --run-id 46d4033b0cbaec3715d6a7a77b4411cb-54-7 --no-pretty ``` ## Retract stuck requests (oracle crash-loop fix) ```bash # 1. List pending requests on token moog token --no-pretty | jq -c '.requests[] | {outref: .request.outputRefId, owner: .request.owner}' # 2. Match owner to wallet (check with: moog wallet info) # 3. Retract each owned request being_requester # or being_agent moog retract -w $MOOG_WALLET_FILE -o "<outref>" # 4. Oracle picks up changes automatically (no restart needed in theory, # but restart if it's stuck in a crash-loop reading stale state) ``` ## How the agent reconciles results (Antithesis API, v0.5.1.2+) **The agent derives outcomes from the Antithesis REST API, not email.** The service loop (commit `ca82717e`, shipped in v0.5.1.2+) lists Antithesis runs and matches each to an on-chain test-run by its rendered `description`, then advances the on-chain phase. Reconciliation rules live in `src/User/Agent/Antithesis/State.hs`: - **`pending` on-chain** → no matching Antithesis run yet ⇒ launch it; exactly one match ⇒ accept; many ⇒ flagged duplicate. - **`accepted` (running) on-chain** → when the matched run reaches a *terminal* API status, write a `finished` fact with this mapping (`terminalOutcome`): | Antithesis status | on-chain outcome | |---|---| | `completed` | `success` — **only if** the run has a `triage_report` link; without it the agent keeps waiting (success is never attested without the report) | | `incomplete` | **`failure`** | | `cancelled` | `failure` | | `starting` / `in_progress` / `unknown` | keep waiting | - **Terminal but no triage-report link** (`incomplete` / `cancelled`): finished as `failure` anyway, with a deterministic synthetic URL `antithesis://runs/<run_id>/no-triage-report` (`terminalNoReportOutcome`, fix `04a74a40` / issue #138) so these no longer stay `accepted` forever. So **`incomplete` has no on-chain representation of its own — it is recorded as `outcome: failure`** (the report URL if one exists, else the `no-triage-report` synthetic URL). The on-chain `duration` field is the *requested* hours, not actual runtime, so a truncated run is invisible once finished. ## Clean up stale "running" tests With the v0.5.1.2+ loop, terminal Antithesis runs auto-finish, so a genuinely stuck `accepted` fact now means either a **correlation miss** (the agent's `description` match failed — e.g. the run was launched by a different binary) or an agent running a **pre-v0.5.1.2** binary. The manual escape hatch is unchanged: ```bash # 1. Find accepted (running) tests on-chain moog facts test-runs --whose cfhal --no-pretty | jq -c '.[] | select(.value.phase == "accepted") | {id: .id[:16], commit: .key.commitId[:8], try: .key.try}' # 2. Confirm against the Antithesis API what status the run actually reached moog antithesis runs --limit 100 --no-pretty | grep '^{"data"' | jq -r '.data[] | [.status, .run_id] | @tsv' # 3. Report as unknown to finish them (manual override) being_agent moog agent report-test -i <full-test-run-id> -w $MOOG_WALLET_FILE --outcome unknown --duration 0 --url "stale-uncorrelated-run" ``` ## Infrastructure SSH | Host | Service | Container | |------|---------|-----------| | `oracle` | moog-oracle v0.4.1.2 | `oracle-moog-oracle-1` | | `agent` | moog-agent (release binary; the v0.5.1.2+ loop reconciles via the Antithesis API) | `agent-moog-agent-1` | Compose lives at `/opt/hal/infrastructure/moog/agent/docker-compose.yaml` on the `agent` host. Edits go in there, not in `/home/paolino/hal/...` (which is a checkout, not the deployed copy). ## Secrets layout ### Agent host (`agent`) ``` /secrets/moog-agent/ ├── old/ # previous rotation generation (kept for rollback) │ ├── secrets.yaml │ └── docker/config.json └── new/ # currently mounted by compose ├── secrets.yaml # githubPAT, antithesisPassword, trustedRequesters, │ # mnemonics or encryptedMnemonics, walletPassphrase, slackWebhook, │ # tokenId, mpfsHost, registry, antithesisUser, antithesisLaunchUrl, │ # pollIntervalSeconds, minutes, wait └── docker/ ├── config.json # docker auth (auths block with base64 _json_key:<sa>) └── sa-key.json # raw GCP service-account JSON (kept for regenerating config.json) agent-wallet (a separate compose secret) └── /secrets/moog-agent/new/agent.json (or wherever the agent-wallet.file points) ``` Compose: `/opt/hal/infrastructure/moog/agent/docker-compose.yaml`. ### Oracle host (`oracle`) ``` /home/paolino/secrets/moog/oracle/ ├── oracle.json # oracle wallet (mnemonics OR encryptedMnemonics) └── secrets.yaml # githubPAT (cfhal-owned, distinct from agent's PAT), # walletPassphrase (if oracle.json is encrypted) ``` Compose: `/opt/hal/infrastructure/moog/oracle/docker-compose.yaml`. Container env points `MOOG_WALLET_FILE=/run/secrets/oracle-wallet` and `MOOG_SECRETS_FILE=/run/secrets/secrets` at the in-container tmpfs paths the compose materialises from the two host files. **The oracle wallet CANNOT be rotated on a live token.** The on-chain token's `state.owner` is permanently bound to the wallet's pubkey-hash at mint time (`moog token state --no-pretty | jq .state.owner`). Rotating the oracle wallet means burning the token and reminting — which loses the entire test-run history. If the oracle mnemonics leak (as happened 2026-05-28 — see lessons), the only mitigations are (a) `moog wallet encrypt` so on-disk material isn't immediately usable without the passphrase, and (b) accepting that the preprod blast radius is limited to fraudulent inclusions on this token's history and drainable preprod ADA — no mainnet impact. Treat the oracle wallet file as if it were a mainnet hot key: never `cat`, never `head`, never `scp` it off the host. `old/` and `new/` may have *different* `githubPAT` / `antithesisPassword` / docker `config.json` values. When something fails on auth, diff the two — a half-applied rotation is a real failure mode. **`docker/config.json` is NOT the raw SA key.** It must be a real docker config: ```json { "auths": { "us-central1-docker.pkg.dev": {